[Opendnssec-user] SOA version number decremented!

[Havard Eidnes]

Hi,

on the 26th of February I upgraded our OpenDNSSEC installation
from 1.4.7 + the right-before-Christmas patches to 1.4.9, since
the patches have now been integrated.  Thus, I needed to restart
OpenDNSSEC.

However, we're now experiencing a problem with one of our zones,
norid.no, which we can trace back to this upgrade and the
corresponding required restart of OpenDNSSEC.

The problem the zone is experiencing is that the RRSIG for the DNSKEY
records for the zone are in danger of expiring.  The reason for this
appears to be that our public master name server (downstream of
OpenDNSSEC) is now complaining:

Mar  2 22:42:42 ns named[182]: zone norid.no/IN: serial number (2016022600) received from master 158.38.0.175#53 < ours (2016022602)

and therefore (as it should) refusing to update the zone.

However, checking the zone contents from the OpenDNSSEC installation
(doing a zone transfer with "dig") reveals that it *does* have a newer
RRSIG record for the DNSKEY.  What's being published by ns.uninett.no
is:

norid.no. 3600 IN RRSIG DNSKEY 8 2 3600 20160309115055 20160217201022 57680 norid.no. [...]

whereas what we get as part of the zone transfer from our
OpenDNSSEC host is:

norid.no. 3600 IN RRSIG DNSKEY 8 2 3600 20160318225317 20160226120511 57680 norid.no. [...]

I.e. this is a newer signature.

I did notice that after OpenDNSSEC was restarted, it logged a
number of errors about what it refers to "backup files", which
seems to be the <zonename>.backup2 files:

Feb 26 14:03:10 hugin ods-signerd: [zone] corrupted backup file zone norid.no: read key error
Feb 26 14:03:10 hugin ods-signerd: [engine] unable to recover zone norid.no from backup, performing full sign

It then logged this message for the zone:

Feb 26 14:03:12 hugin ods-signerd: [xfrd] zone norid.no transfer done [notify acquired 0, serial on disk 2016022200, notify serial 0]

(not sure where it got the "serial on disk" from), and then it signed
the full zone:

Feb 26 14:05:11 hugin ods-signerd: [signconf] zone norid.no signconf: RESIGN[PT7200S] REFRESH[PT777600S] VALIDITY[PT1814400S] DENIAL[PT1814400S] JITTER[PT43200S] OFFSET[PT3600S] NSEC[50] DNSKEYTTL[PT3600S] SOATTL[PT3600S] MINIMUM[PT900S] SERIAL[datecounter]
Feb 26 14:05:19 hugin ods-signerd: [STATS] norid.no 2016022600 RR[count=330 time=0(sec)] NSEC3[count=202 time=0(sec)] RRSIG[new=513 reused=0 time=4(sec) avg=128(sig/sec)] TOTAL[time=4(sec)] 

However, this zone is published with a SOA version number which is now
lower than what it had emitted earlier, i.e. it's reset the "intra-day
SOA increment counter" back to zero.  Also, it's evident that absent
an update from the hidden master upstream of our OpenDNSSEC host, the
zone won't update its SOA version number on its own either, and
turning up logging reveals:

Mar  2 22:05:11 hugin ods-signerd: [worker[2]] sign zone norid.no ok: 517 of 517 RRsets succeeded
Mar  2 22:05:11 hugin ods-signerd: [worker[2]] write zone norid.no
Mar  2 22:05:11 hugin ods-signerd: [tools] skip write zone norid.no serial 2016030200 (zone not changed)

so this is currently stuck.  ("ods-signer queue | grep norid.no" has
numerous times said "I will sign norid.no" at a time ~2 hours in the
future, but no new zone file is generated, probably due to the above
optimization.


I'm sort of getting tired with OpenDNSSEC rejecting what it's written
itself.  It refused the ".backup2" files from 297 of our 368 zones
(not sure there were .backup2 files for the others).  ...and...  The
reason appears to be that there's no backward compatibility for the
";;Key" statement in the .backup2 files -- between 1.4.7 and 1.4.9,
the "rfc5011" field has been added, and the code in key_recover2()
doesn't accept such statements written by 1.4.7, so all the .backup2
files are invalidated.  And I would not be surprised if this is what's
causing OpenDNSSEC to lose count of its outgoing SOA version numbers,
causing this exact problem.

Sometime tomorrow I plan on doing a "null update" on the hidden master
for this zone, to unwedge this problem, but this really is a bug in
OpenDNSSEC -- one shall Never EVER decrement the SOA version number.

Best regards,

- Håvard