[Opendnssec-develop] Fwd: Performance benchmarks

[Sara Dickinson]

Hi All, 

Firstly thanks to Jerry and SurfNet for getting the benchmarking hardware up and running! Now - what do we do with it?

I looked back at the minutes from the last OAB meeting and two of the specific things mentioned about benchmarking were that 

- the benchmarks should demonstrate that a "reasonable" sized registrar can use OpenDNSSEC comfortably
- benchmarking was higher priority than endurance testing because we need to asses 1.4 vs 2.0

My own notes from the meeting also mention that there was a discussion about the 2.0 performance target. We (the developers) asked the board to clarify the target as all that had been said up to that point was that 2.0 "should handle 50,000 zones". The feeling was that this was too vague to really measure, but there wasn't any real agreement on what the target was apart from wanting to demonstrate something like the first bullet point above. 

My notes also show that there was a discussion about whether we did 
- 'one-shot' type benchmarking (measure sign and re-sign) on a few zone size/number combinations but with several different config options (no. of threads, NSEC, shared keys, etc.) or
- more dynamic testing e.g. start with 1 small zone and make measurements as N lots of X zones or records are added, with probably less config options tested.  

With the dynamic testing we would get a plot of how signing time/memory/etc. changes as the system grows. The feeling was that the second approach would be more useful and we should pick some tests to start with, see how hard they are to do,  publish the data and get feedback from the users.  

In general I think the aim of the benchmarking tests we develop should include the following:
- do direct comparisons between the new and current enforcer...
- provide data and monitor performance (time taken, memory usage) for the development team as code is added during development phases
- give us useful information to feedback to the users
- be the basis of some standard set ups that we can document

Maybe we should start with 2 'dynamic' tests
- grow from 1 small zone to 1 large zone by adding records (add 10 lots of 100,000 records)
- grow from 1 small zone to many small zones by adding zones (add 10 lots of 5,000 zones)
and see what this give us. 

What do people think?

Sara.