[Opendnssec-user] Slow key generation with KSK rollover

[Erik P. Ostlyngen]

Hi again,

I've now made some additional observations. In order to narrow down
the problem, I tried to modify the hsm key factory in my enforcer
daemon to create 792 keys instead of just one. With debug logging
enabled, I can then see that the keys are generated quickly (3-4 keys
per sec). So it seems that the entropy is fine, and it is not the key
generation itself that is slowing down the process.

Also, if I run the signer in parallell with the enforcer while doing
key rollovers, I get some ugly looking errors from the signer:

Nov  8 10:43:00 server017 ods-signerd: CRITICAL: failed to sign zone
domain1.no: General error
Nov  8 10:43:00 server017 ods-signerd: back-off task [read] for zone
domain1.no with 240 seconds
Nov  8 10:47:01 server017 ods-signerd: ObjectFile.cpp(122): The
attribute does not exist: 0x00000002
Nov  8 10:47:01 server017 ods-signerd: [hsm] unable to get key: key
b4a13ec5cdcfb1d0bb38456dc0e44388 not found
Nov  8 10:47:01 server017 ods-signerd: [hsm] hsm_get_dnskey(): Got
NULL key
Nov  8 10:47:01 server017 ods-signerd: [hsm] unable to get key: hsm
failed to create dnskey
Nov  8 10:47:01 server017 ods-signerd: [zone] unable to publish
dnskeys for zone domain1.no: error creating dnskey
Nov  8 10:47:01 server017 ods-signerd: [tools] unable to read zone
domain1.no: failed to publish dnskeys (General error)
Nov  8 10:47:01 server017 ods-signerd: CRITICAL: failed to sign zone
domain2.no: General error
Nov  8 10:47:01 server017 ods-signerd: back-off task [read] for zone
domain2.no with 480 seconds

It seems to me that this happens if the signer tries to sign the zone
after a rollover is initiated, but before the new key is generated and
added to the key database (but this is a bit speculative since I do
not know the system very well). After the enforcer is done adding the
new KSK to the key database, I can do a signconf update and restart
the signer. The signer will then start signing the zone again without
errors (presumably).

Is this a known problem when rolling a big number of zones at the same
time, or am I doing it in the wrong way, or is my opendnssec
installation broken?

Regards,
Erik Østlyngen
Norid AS
www.norid.no


On 15/11/2019 11.14, Erik P. Ostlyngen wrote:
> Hi,
> 
> I'm using OpenDNSSEC 2.1.4, and I'm seeing some strange behaviour
> when I try to do a KSK rollover on a set of zones. I'm doing a
> rollover of all my zones within a given policy. The command I use
> is:
> 
> % sudo ods-enforcer key rollover --keytype KSK --policy mypolicy
> 
> Enforcer then starts to generate new keys for my 792 zones but this
> is done rather slowly, approximately 10 secs per key. Each time a
> key is generated, I see the following message in the log:
> 
> Nov 12 07:44:33 server01 ods-enforcerd: [hsm_key_factory_generate]
> 1 keys needed for 792 zones covering 31536000 seconds, generating
> 1 keys for policy mypolicy
> 
> I would expect it to say something like '792 keys needed for 792 
> zones' since I'm not using shared keys. Between every key
> generated, Enforcer seems to be looping through all the zones,
> logging messages like this:
> 
> Nov 12 08:00:17 server01 ods-enforcerd: [enforcer] update zone: 
> myzone.no Nov 12 08:00:17 server01 ods-enforcerd:
> [hsm_key_factory_get_key] no keys available Nov 12 08:00:17
> server01 ods-enforcerd: [enforcer] updatePolicy: No keys available
> in HSM for policy mypolicy, retry in 60 seconds
> 
> Is this the correct/expected behaviour or am I doing something
> wrong?
> 
> Regards, Erik Østlyngen Norid AS www.norid.no 
> _______________________________________________ Opendnssec-user
> mailing list Opendnssec-user at lists.opendnssec.org 
> https://lists.opendnssec.org/mailman/listinfo/opendnssec-user
>