[Opendnssec-develop] RE: Jenkins tests

[Sara Dickinson]

Hi All, 

I got thinking the other day about how we are using jenkins and wanted to propose an approach for discussion. We are still developing the framework, test coverage and learning how to write tests but I thought it was worth an email. There are two main things that I think need agreeing:

1) What is the general process for jenkins tests? 

2) What do we do with intermittent and platform dependant failures.

So my proposals are:

1) We aim for clean builds every night. 
- We can configure jenkins to do a full, clean build of everything in the early morning (e.g. 1 or 2 am) then easily tell by looking at the results the following morning if any code submitted the previous day has broken anything. 
- If everyone gets in the habit of running any relevant regression tests locally before submitting code to trunk then we should have lots of sunny skies :-) In theory the overnight builds should only ever break when platform dependancies pop up and hopefully these would be seen during the day anyway. 
- Obviously if new tests are being added or actively developed then there may be failures during the day, but if these tests are not passing by the time you head home then the recommendation would be that the test is turned off overnight. 

2) Assuming the above then we probably need to consider mechanisms to deal with 'awkward' failures. 
- I propose that if we have a test that fails intermittently then we turn if off or modify it until we can get it to run reliably (e.g.10-160-odscc10t160). 
- I think it is ok to turn off tests on specific platforms while testing/developing/bug fixing is in progress if having the test pass on say, 11 out of 12 platforms is helpful (rather than having to turn the whole test on an off every day). I think it is a better option than having a test failing and clouding the results for others. There may even be cases where a given platform doesn't support a tool needed for a given test in which case we would need a mechanism to suppress the test on a given platform. 

I'd like to know what people think about this. 

I also saw this the other day but don't think it has an big enough range for an international team :-)    
http://github.com/codedance/Retaliation#readme

Sara.