[Opendnssec-develop] too many open files

[Matthijs Mekking]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As a follow up, mysql did not speed things up. The 5000 zones are stored
in the database, but signer configurations are still outputted equally slow.

Best regards,

Matthijs

On 10/05/2010 10:44 AM, Matthijs Mekking wrote:
> FYI,
> 
> r4047 fixes one more file descriptor leak, as well as a segfault due to
> a parse error in the backed up unsorted file.
> 
> Signing on my machine (2GB memory, 2x 1.6GHz processor, Ubuntu 10.04) is
> quite slow: Also created 5000 zones, using --no-xml (with sqlite3).
> 
> When starting, the enforcer finds zone every half minute. So signer
> configurations are outputted every half minute. That was yesterday,
> about 15:30 CET. At this moment, the enforcer found 761 zones of the
> 5000. Intervals between finding zones have increased to 5 minutes.
> Could it be an entropy issue? The only call to random is when updating
> the NSEC3 salt. I tried with NSEC and that looks slow too...
> 
> The signer does not pick up updates, since it does not reload the zone
> list yet when receiving 'ods-signer update thiszone' from the enforcer
> (It's a Pivotal story, working on that).
> 
> When reloading the signer, signing is very fast. However, when 8 audit
> processes are started at the same time, auditing gets slow. Per zone,
> auditing takes about 85 seconds of the total time of 86 seconds. This is
> with ruby 1.8.7.
> 
> To be continued...
> 
> Matthijs
> 
> 
> 
> On 09/30/2010 09:02 PM, Patrik Wallström wrote:
>> On Sep 30, 2010, at 7:08 PM, Patrik Wallström wrote:
>>>
>>> I am testing performance on a better machine now, with fast disks and a quad core xeon processor. Performance is much better. I seem to get 5000 signed domains without too much hassle.
>>>
>>> First, create 5000 domains, adding them using no-xml (with mysql), and exporting the xml zonelist. First problem here, the first three lines is not xml, it's the mysql config. Remove this by hand. HUP the enforcer, and the enforcer creates signer configs reasonable fast. The signer acts a little strange, so after restarting the signer, it signs all domains without too much problem.
>>>
>>> Add another 5000 domains, that one is a little bit slower with the enforcer. Export and HUP, and it creates signer configs. Here somewhere (had some dinner), the signer loops with this:
>>>
>>> Sep 30 18:53:21 dnslab ods-signerd: command handler accept error: Too many open files
>>>
>>> So this problem do still exist. It might be that the enforcer grabs all file descriptors, I don't know. But maybe the signer should be more resilient about it. So, we still have to find the file descriptor leak.
>>>
>>> But it looks better and better, performance wise.
> 
> 
>> And after putting the kids to sleep, I just tried to start ods again.
> 
>> Lots of these, of course, but look at the end.
> 
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1908more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1909many.org
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1909more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 190many.org
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 190more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1910many.org
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1910more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1911many.org
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1911more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1912many.org
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1912more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1913many.org
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1913more.org.state for reading: No such file or directory
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1914many.org.unsorted for reading: Too many open files
>> Sep 30 21:00:50 dnslab ods-signerd: error reading zone 1914many.org from file 1914many.org.unsorted
>> Sep 30 21:00:50 dnslab ods-signerd: unable to recover unsorted zone from file 1914many.org.unsorted: parse error
>> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1914many.org.task for reading: Too many open files
>> Sep 30 21:00:50 dnslab kernel: [23674.642930] ods-signerd[16728]: segfault at 0 ip 0000000000416f80 sp 00007fffbb5a05f0 error 6 in ods-signerd[400000+33000]
> 
_______________________________________________
Opendnssec-develop mailing list
Opendnssec-develop at lists.opendnssec.org
https://lists.opendnssec.org/mailman/listinfo/opendnssec-develop
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMsudeAAoJEA8yVCPsQCW5SXgH/iVav8GQ/VEU23iOboC96OTc
oF85+YpGM94cJ9YL9pdjpSB/PEHVJ5vb5DFRvBn5qziE1d2vrALwmen9F9ixGjkA
TP82ipSX+SX433XimV8h2cbrB7OT3KIbEqazxNU41fkjVZAzq2fmyw3bH60y3pIs
VG4txMBncnvwsMuSep7t9DYXbXmraY6+o3fo+2mnbBJO9SLRj++wSIlb4pRMnntj
8sMoGizv9E10ODlcKmsSGjYbLPIBZ6FZnTTe484VgxmnRlSw3WX71UO5A9MJ/7hq
Ll3gVIZovdshLQhVtm7KgDtzMOKzSpCQgaUleV4skwhTwTZlOYh1zx11C/2QpzI=
=J3LN
-----END PGP SIGNATURE-----