[Opendnssec-develop] RE: Regression testing update

[Matthijs Mekking]

On 11/26/2012 04:03 PM, Sara Dickinson wrote:
> Hi All,
>
> So I have been having another look lately at the regression tests and
> how we could manage them better as we add more tests. (They are
> currently a mixture of imported scripts with a numeric naming
> convention and new tests added by us.) My plan is to do the following
> things:
>
> - come up with a new naming convention for the test scripts - come up
> with a way to track what is currently tested (including bug fixes) -
> review which tests should be smoke/daily/weekly - re-vamp the Test
> Coverage wiki page to more clearly see what tests still need adding
>
> I've had a go at the first two so am looking for some feedback before
> moving any further.... More details on both the following suggestions
> are on the wiki page:
> https://wiki.opendnssec.org/display/OpenDNSSEC/Test+case+policy
>
> Naming ------------- My idea for naming is to split the names into 3
> parts (based on the area tested - see the wiki). These parts would be
> separated by dashes, with word separated by underscores within the 3
> parts. So for example we might have:
>
> enforcer-keys-check_backup_required_works
> signer-adaptors-many_dns_updates general-repository-opendnssecXXX

I like the idea of structuring tests, although I think those names may 
become quite long, especially later if we have to test weird, specific 
corner cases. 
(signer-adapters-many_dns_updates_all_ixfr_over_udp_and_some_packets_are_dropped)

Perhaps signer-adapters-XXXX where XXXX is just a number?

> I thought this was easier than using than a numbering system and
> since we currently run all the tests regardless of any failures then
> the ordering within the directory doesn't really matter (at the
> moment anyway). We could alternatively use a directory structure
> along these lines but I'm not convinced that having to always drill
> down further to the test scripts ultimately makes life easier...?
> Please let me know if you can think of improvements or alternatives
> to this.
>
> Tracking ------------- On the tracking side of things I took the
> approach that the most reliable way to track things is to have a way
> to grab information directly from the tests rather than try to have a
> separate document that needs updating and so is likely to get out of
> date. I have written a small script that grabs info from the comments
> in the test scripts and generates a CSV file as output. It needs more
> work and really should be incorporated into the framework if we
> decide to adopt this approach - at the moment it is just to generate
> feedback. An example excel spreadsheet generated from the information
> extracted (including new suggested names for the existing tests) is
> attached to the wiki page for people to review.
>
> This does rely on the comments in the tests being accurate and up to
> date but that seems a good idea anyway!  Please try this out and let
> me know if you think this approach makes sense.

I like this idea. I think the comments are mainly useful for tracking 
which JIRA issue is tested (which I know aren't many now).

Best regards,
   Matthijs

>
> Sara.
>
> _______________________________________________ Opendnssec-develop
> mailing list Opendnssec-develop at lists.opendnssec.org
> https://lists.opendnssec.org/mailman/listinfo/opendnssec-develop
>