[Opendnssec-develop] too many open files

[Matthijs Mekking]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

FYI,

r4047 fixes one more file descriptor leak, as well as a segfault due to
a parse error in the backed up unsorted file.

Signing on my machine (2GB memory, 2x 1.6GHz processor, Ubuntu 10.04) is
quite slow: Also created 5000 zones, using --no-xml (with sqlite3).

When starting, the enforcer finds zone every half minute. So signer
configurations are outputted every half minute. That was yesterday,
about 15:30 CET. At this moment, the enforcer found 761 zones of the
5000. Intervals between finding zones have increased to 5 minutes.
Could it be an entropy issue? The only call to random is when updating
the NSEC3 salt. I tried with NSEC and that looks slow too...

The signer does not pick up updates, since it does not reload the zone
list yet when receiving 'ods-signer update thiszone' from the enforcer
(It's a Pivotal story, working on that).

When reloading the signer, signing is very fast. However, when 8 audit
processes are started at the same time, auditing gets slow. Per zone,
auditing takes about 85 seconds of the total time of 86 seconds. This is
with ruby 1.8.7.

To be continued...

Matthijs



On 09/30/2010 09:02 PM, Patrik Wallström wrote:
> On Sep 30, 2010, at 7:08 PM, Patrik Wallström wrote:
>>
>> I am testing performance on a better machine now, with fast disks and a quad core xeon processor. Performance is much better. I seem to get 5000 signed domains without too much hassle.
>>
>> First, create 5000 domains, adding them using no-xml (with mysql), and exporting the xml zonelist. First problem here, the first three lines is not xml, it's the mysql config. Remove this by hand. HUP the enforcer, and the enforcer creates signer configs reasonable fast. The signer acts a little strange, so after restarting the signer, it signs all domains without too much problem.
>>
>> Add another 5000 domains, that one is a little bit slower with the enforcer. Export and HUP, and it creates signer configs. Here somewhere (had some dinner), the signer loops with this:
>>
>> Sep 30 18:53:21 dnslab ods-signerd: command handler accept error: Too many open files
>>
>> So this problem do still exist. It might be that the enforcer grabs all file descriptors, I don't know. But maybe the signer should be more resilient about it. So, we still have to find the file descriptor leak.
>>
>> But it looks better and better, performance wise.
> 
> 
> And after putting the kids to sleep, I just tried to start ods again.
> 
> Lots of these, of course, but look at the end.
> 
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1908more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1909many.org
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1909more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 190many.org
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 190more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1910many.org
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1910more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1911many.org
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1911more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1912many.org
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1912more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: On Thu Sep 30 19:04:42 2010 I will sign zone 1913many.org
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1913more.org.state for reading: No such file or directory
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1914many.org.unsorted for reading: Too many open files
> Sep 30 21:00:50 dnslab ods-signerd: error reading zone 1914many.org from file 1914many.org.unsorted
> Sep 30 21:00:50 dnslab ods-signerd: unable to recover unsorted zone from file 1914many.org.unsorted: parse error
> Sep 30 21:00:50 dnslab ods-signerd: unable to open file 1914many.org.task for reading: Too many open files
> Sep 30 21:00:50 dnslab kernel: [23674.642930] ods-signerd[16728]: segfault at 0 ip 0000000000416f80 sp 00007fffbb5a05f0 error 6 in ods-signerd[400000+33000]
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMquV6AAoJEA8yVCPsQCW5XUQH/isM+NeKN/yA0GMZJ8c502/h
pWzg9hDD/nawvLOJJglSYJ5No400TSY7CbB5s+ZDp4aViEhVmTYSscEYpzSqM2UG
7JKInLid1ZB0xwn79ATU3i1zH7L52VwzoC4Zy750OrwByDH7r6y1nJQ7k27AB5zK
21O7cGJoSFcOdpjyzjGQnDJDIR6FiMmuInD90GNHQ8R3+ZMtar3uLGCrcFlOy+Eq
Z90LDd0Q2aPFr3JN1kk3w/KM65QZKzY6STtikXl75maMAz0ZsK6NH18VMj03x7Je
p6SHBnL9X1BL86wM6zQZfXvOxAZItZPe5LiBeKMI3r1ZMO49d220kaJee3ZqG5s=
=7fK+
-----END PGP SIGNATURE-----