[Opendnssec-develop] RE: Regression testing update

[Sara Dickinson]

Hi All, 

So I have been having another look lately at the regression tests and how we could manage them better as we add more tests. (They are currently a mixture of imported scripts with a numeric naming convention and new tests added by us.) My plan is to do the following things:

- come up with a new naming convention for the test scripts
- come up with a way to track what is currently tested (including bug fixes)
- review which tests should be smoke/daily/weekly
- re-vamp the Test Coverage wiki page to more clearly see what tests still need adding

I've had a go at the first two so am looking for some feedback before moving any further.... More details on both the following suggestions are on the wiki page: https://wiki.opendnssec.org/display/OpenDNSSEC/Test+case+policy

Naming
-------------
My idea for naming is to split the names into 3 parts (based on the area tested - see the wiki). These parts would be separated by dashes, with word separated by underscores within the 3 parts. So for example we might have:

enforcer-keys-check_backup_required_works
signer-adaptors-many_dns_updates
general-repository-opendnssecXXX

I thought this was easier than using than a numbering system and since we currently run all the tests regardless of any failures then the ordering within the directory doesn't really matter (at the moment anyway). We could alternatively use a directory structure along these lines but I'm not convinced that having to always drill down further to the test scripts ultimately makes life easier...? Please let me know if you can think of improvements or alternatives to this. 

Tracking
-------------
On the tracking side of things I took the approach that the most reliable way to track things is to have a way to grab information directly from the tests rather than try to have a separate document that needs updating and so is likely to get out of date. I have written a small script that grabs info from the comments in the test scripts and generates a CSV file as output. It needs more work and really should be incorporated into the framework if we decide to adopt this approach - at the moment it is just to generate feedback. An example excel spreadsheet generated from the information extracted (including new suggested names for the existing tests) is attached to the wiki page for people to review. 

This does rely on the comments in the tests being accurate and up to date but that seems a good idea anyway!  Please try this out and let me know if you think this approach makes sense. 

Sara.