[Opendnssec-user] ods-signer trying to sign long-removed zones

[Philip Paeps]

On 2021-06-15 13:22:08 (+0800), Philip Paeps via Opendnssec-user wrote:
> I upgraded OpenDNSSEC for freebsd.org this morning.  There were no 
> huge explosions.  Yet.  As far as I can tell.
>
> However, we do get a lot of these in the logs:
>
> Jun 15 05:18:57 ns-master ods-signerd[14648]: [hsm] unable to get key: 
> key d6c2bb972ef3cd75c57e234dfc8173b8 not found
> Jun 15 05:18:57 ns-master ods-signerd[14648]: [hsm] hsm_get_dnskey(): 
> Got NULL key
> Jun 15 05:18:57 ns-master ods-signerd[14648]: [hsm] unable to get key: 
> hsm failed to create dnskey
> Jun 15 05:18:57 ns-master ods-signerd[14648]: [zone] unable to prepare 
> signing keys for zone 1.0.1.0.0.0.0.2.0.5.c.f.7.0.6.2.ip6.arpa: error 
> getting dnskey
> Jun 15 05:18:57 ns-master ods-signerd[14648]: [worker[1]] CRITICAL: 
> failed to sign zone 1.0.1.0.0.0.0.2.0.5.c.f.7.0.6.2.ip6.arpa: General 
> error
> Jun 15 05:18:57 ns-master ods-signerd[14648]: back-off task [sign] for 
> zone 1.0.1.0.0.0.0.2.0.5.c.f.7.0.6.2.ip6.arpa with 480 seconds
>
> This is a zone we used to have a long time ago.  It was deleted from 
> zonelist.xml a long time ago (years).  'ods-enforcer zone list' does 
> not know about this zone.  So the database must have been updated.  
> However .. 'ods-signer zones' does know about this zone.  And it's 
> trying to sign it apparently.
>
> There are a couple of other zones in this state.
>
> I have tried 'ods-signer update all' and 'ods-signer clear 
> 1.0.1.0.0.0.0.2.0.5.c.f.7.0.6.2.ip6.arpa'.  Apparently to no avail.
>
> Is there a way to help ods-signer forget about these stale zones so 
> our log files stop growing in vain?

After a lot of grepping (and gnashing of teeth), I managed to make this 
go away.

It turns out there were stale <Zone> stanzas in 
/usr/local/var/opendnssec/enforcer/zones.xml referencing the deleted 
zones.  As mentioned earlier, there were also stale files named after 
these zones in /usr/local/var/opendnssec/signer.  Updating that 
zones.xml file to match reality made the problem go away.

I also found a /usr/local/var/opendnssec/enforcer/ods-signerd.core file 
with a timestamp around the time the zones were deleted.  That might 
explain why things were in an intermediate state.

Philip

-- 
Philip Paeps
Senior Reality Engineer
Alternative Enterprises