<div dir="ltr">Problem solved. And many thanks Sebastian for pointing to the right direction.<div>In fact I was well aware that Keyper uses the keymap.db for key mapping. The default location which can't be changed, at least I failed to find a way to change it is /root/Keyper/PKCS11Provider/keymap.db. I'm running both signer and enforcer as user opendnssec with a different home directory (/usr/local/ods), so as a fix I moved /root/Keyper to /usr/local/ods/Keyper and created a link in /root with name Keyper pointing to /usr/local/ods/Keyper and then all commands worked both as user opendnssec and user root. Month or two later I decided to separate the opendnssec binaries and data and moved the Keyper data to /ods-data/Keyper. The new setup continued using the same keys and it still worked well. The the problems started when I decided wipe the data and keys and the the signer failed to sign the zone because it was looking for the mapping of the keys at the old location /usr/local/ods/Keyper.</div><div>The fix was to change the home directory for user opendnssec.</div><div>Thank you again.</div><div><br></div><div>Emil</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 17, 2014 at 4:15 AM, Sebastian Castro <span dir="ltr"><<a href="mailto:sebastian@nzrs.net.nz" target="_blank">sebastian@nzrs.net.nz</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
On 17/12/14 12:56 am, Emil Natan wrote:<br>
> Hi Matthijs and thank you for your reply.<br>
><br>
<br>
</span>Hi Emil:<br>
<br>
Your problem seems really odd, but for some reason not strange. We've<br>
done some testing with the AEP Keyper, and it seems there is a mapping<br>
between key id and HSM used that lives in a BerkeleyDB file somewhere in<br>
the file system.<br>
<br>
I don't recall the location of the file at the moment, and don't have<br>
notes, but came across with something similar before.<br>
<br>
You can find where the file is while stracing the command<br>
<br>
ods-hsmutil generate Keyper rsa 1024<br>
<br>
Also you can try with ods-hsmutil to generate a DNSKEY from an existing<br>
key, perhaps the problem is your program doesn't have access to read the<br>
mapping file.<br>
<br>
If you run<br>
<br>
ods-hsmutil dnskey 39a954b0fccb0f5ed73614d5fc1a8144 test.<br>
<br>
as the root used should work, but if you run<br>
<br>
sudo -u opendnssec ods-hsmutil dnskey 39a954b0fccb0f5ed73614d5fc1a8144 test.<br>
<br>
it should fail.<br>
<br>
Let us know how it works, I'll ask internally to find out if someone<br>
remembers the name of the bloody file!<br>
<div><div class="h5"><br>
> Here is how it goes for me.<br>
><br>
> I start with:<br>
> Zone: Keytype: State: Date of next<br>
> transition:<br>
> XXX KSK active 2016-01-16<br>
> 09:49:45<br>
> XXX ZSK active 2015-04-18 22:40:55<br>
><br>
> root@debugsigner002:~# ods-hsmutil purge Keyper<br>
> Purging all keys from repository: Keyper<br>
> 12 keys found.<br>
><br>
> Are you sure you want to remove ALL keys from repository Keyper ?<br>
> (YES/NO) yes<br>
><br>
> Starting purge...<br>
> Key remove successful: fdd17d120d3e548a104dda856d84c770<br>
> ...<br>
> Key remove successful: db97ded0cc231c3908f8f20f5ce21229<br>
> Key remove successful: f81e4b2cb33eec780320b6ceeb6f6bb8<br>
> Purge done.<br>
><br>
> root@debugsigner002:~# /opt/Keyper/PKCS11Provider/inittoken<br>
> ...<br>
> PKCS11 Slot : 0<br>
> PKCS11 Label : aepkeyper<br>
> Keyper Model : Keyper Ent 1126<br>
> Keyper Serial :<br>
> Keyper version : 2.0<br>
> App : 020<br>
> ABL : 029<br>
> AL : 02<br>
> --------------------------------------------<br>
> Token initialised OK<br>
> ********************************************<br>
><br>
> To remove the zone I actually comment it out from zonelist.xml, then:<br>
><br>
> root@debugsigner002:~# ods-ksmutil update zonelist<br>
> zonelist filename set to /ods-data/etc/opendnssec/zonelist.xml.<br>
> kasp filename set to /ods-data/etc/opendnssec/kasp.xml.<br>
> Removing zone XXX from database<br>
> Notifying enforcer of new database...<br>
><br>
> I stopped both ODS daemons.<br>
><br>
> root@debugsigner002:~# ps auxww | grep ods<br>
> root 14452 0.0 0.0 11744 896 pts/2 S+ 13:31 0:00 grep<br>
> --color=auto ods<br>
><br>
> Initialize ODS, all the warnings are skipped, but no errors.<br>
><br>
> root@debugsigner002:~# ods-ksmutil setup<br>
><br>
> *WARNING* This will erase all data in the database; are you sure? [y/N] y<br>
> zonelist filename set to /ods-data/etc/opendnssec/zonelist.xml.<br>
> kasp filename set to /ods-data/etc/opendnssec/kasp.xml.<br>
> Repository Keyper found<br>
> No Maximum Capacity set.<br>
> RequireBackup set.<br>
> INFO: The XML in /ods-data/etc/opendnssec/conf.xml is valid<br>
> INFO: The XML in /ods-data/etc/opendnssec/zonelist.xml is valid<br>
> INFO: The XML in /ods-data/etc/opendnssec/kasp.xml is valid<br>
> Policy XXXTLD found<br>
><br>
> Generate new keys.<br>
><br>
> root@debugsigner002:~# ods-ksmutil key generate --policy XXXTLD<br>
> --zonetotal 1 --interval P2Y<br>
> Key sharing is Off<br>
> Info: converting P2Y to seconds; M interpreted as 31 days, Y interpreted<br>
> as 365 days<br>
> HSM opened successfully.<br>
> Info: 0 zone(s) found on policy "XXXTLD"<br>
> Info: Keys will actually be generated for a total of 1 zone(s) as<br>
> specified by zone total parameter<br>
> 2 new KSK(s) (2048 bits) need to be created for policy XXXTLD:<br>
> keys_to_generate(2) = keys_needed(2) - keys_available(0).<br>
> 6 new ZSK(s) (1024 bits) need to be created for policy XXXTLD:<br>
> keys_to_generate(6) = keys_needed(6) - keys_available(0).<br>
> *WARNING* This will create 2 KSKs (2048 bits) and 6 ZSKs (1024 bits)<br>
> Are you sure? [y/N]<br>
> y<br>
> Created KSK size: 2048, alg: 8 with id: 39a954b0fccb0f5ed73614d5fc1a8144<br>
> in repository: Keyper and database.<br>
> Created KSK size: 2048, alg: 8 with id: 47dc08d7c5be2104b18a9f7a1702e6b0<br>
> in repository: Keyper and database.<br>
> Created ZSK size: 1024, alg: 8 with id: 64504804f1dc34cd44fa83cbede95275<br>
> in repository: Keyper and database.<br>
> Created ZSK size: 1024, alg: 8 with id: ec77d359ccdde3e38b222423a5d2075f<br>
> in repository: Keyper and database.<br>
> Created ZSK size: 1024, alg: 8 with id: 669a0a563fa03c62fc58d20e85628b35<br>
> in repository: Keyper and database.<br>
> Created ZSK size: 1024, alg: 8 with id: e799a40efda79c8e98a76adc72470f6d<br>
> in repository: Keyper and database.<br>
> Created ZSK size: 1024, alg: 8 with id: 0618eb27e1061e37df6bf6a055c85160<br>
> in repository: Keyper and database.<br>
> Created ZSK size: 1024, alg: 8 with id: 424fbb66fbaf3605b4d935b473d1be01<br>
> in repository: Keyper and database.<br>
> NOTE: keys generated in repository Keyper will not become active until<br>
> they have been backed up<br>
> all done! hsm_close result: 0<br>
><br>
> I also mark the keys as backed up.<br>
><br>
> root@debugsigner002:~# ods-ksmutil backup prepare<br>
> Marked all repositories as pre-backed up at 2014-12-16 13:40:15<br>
> root@debugsigner002:~# ods-ksmutil backup commit<br>
> Marked all repositories as backed up at 2014-12-16 13:40:21<br>
><br>
> This time I stopped the signer and enforcer before setup, so I start them.<br>
><br>
> root@debugsigner002:~# ps auxww | grep ods<br>
> opendns+ 14492 0.0 0.5 128840 5548 ? Ss 13:42 0:00<br>
> /ods-bin/sbin/ods-enforcerd<br>
> opendns+ 14501 0.0 0.6 533744 7068 ? Ssl 13:42 0:00<br>
> /ods-bin/sbin/ods-signerd<br>
> root 14514 0.0 0.0 11744 896 pts/1 S+ 13:45 0:00 grep<br>
> --color=auto ods<br>
><br>
> I added the zone, again by editing zonelist.xml and ...<br>
><br>
> root@debugsigner002:~# ods-ksmutil update zonelist<br>
> zonelist filename set to /ods-data/etc/opendnssec/zonelist.xml.<br>
> kasp filename set to /ods-data/etc/opendnssec/kasp.xml.<br>
> Zone XXX found; policy set to XXXTLD<br>
> Notifying enforcer of new database...<br>
><br>
> And I end up with the same problem.<br>
><br>
> Dec 16 13:46:08 debugsigner002 ods-signerd: [hsm] libhsm connection ok<br>
> Dec 16 13:46:08 debugsigner002 ods-signerd: [hsm] unable to get key: key<br>
> 39a954b0fccb0f5ed73614d5fc1a8144 not found<br>
> Dec 16 13:46:08 debugsigner002 ods-signerd: [zone] unable to publish<br>
> dnskeys for zone XXX: error creating dnskey<br>
> Dec 16 13:46:08 debugsigner002 ods-signerd: [tools] unable to read zone<br>
> XXX: failed to publish dnskeys (General error)<br>
> Dec 16 13:46:08 debugsigner002 ods-signerd: [worker[1]] CRITICAL: failed<br>
> to sign zone XXX: General error<br>
><br>
> And ods-ksmutil can still list the keys:<br>
><br>
> root@debugsigner002:~# ods-ksmutil key list -v<br>
> Zone: Keytype: State: Date of next<br>
> transition (to): Size: Algorithm: CKA_ID:<br>
> Repository: Keytag:<br>
> XXX ZSK active 2015-04-19<br>
> 13:46:07 (retire) 1024 8 64504804f1dc34cd44fa83cbede95275<br>
> Keyper 5680<br>
> XXX KSK publish 2014-12-16<br>
> 17:51:07 (ready) 2048 8 39a954b0fccb0f5ed73614d5fc1a8144<br>
> Keyper 6962<br>
><br>
> I'll send you the full log off-list.<br>
> Thanks again.<br>
><br>
> Emil<br>
><br>
> On Tue, Dec 16, 2014 at 12:18 PM, Matthijs Mekking<br>
</div></div><span class="">> <<a href="mailto:matthijs@pletterpet.nl">matthijs@pletterpet.nl</a> <mailto:<a href="mailto:matthijs@pletterpet.nl">matthijs@pletterpet.nl</a>>> wrote:<br>
><br>
> Hi Emil,<br>
><br>
> Short: I tried to simulate your use case (with SoftHSM, on<br>
> ubuntu-trusty-64 VM), but it seems to work for me. Perhaps I used<br>
> slightly different commands? Can you share your used commands?<br>
><br>
> Best regards,<br>
> Matthijs<br>
><br>
><br>
> Audit trail:<br>
><br>
> I started with Keys:<br>
> Zone: Keytype: State: Date of next transition:<br>
</span>> <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>> KSK publish<br>
> 2014-12-16 23:55:02<br>
> <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>> ZSK active<br>
<span class="">> 2015-03-16 09:55:02<br>
><br>
> On 16-12-14 08:54, Emil Natan wrote:<br>
> > Good morning,<br>
> ><br>
> > I have a test environment with ODS 1.4.6 and Keyper HSM where signing<br>
> > zones was working until I decided to remove all keys and start from scratch.<br>
> > I removed all keys with "ods-hsmutil purge"\<br>
><br>
> $ sudo ods-hsmutil purge SoftHSM<br>
> Purging all keys from repository: SoftHSM<br>
> 2 keys found.<br>
><br>
> Are you sure you want to remove ALL keys from repository SoftHSM ?<br>
> (YES/NO) YES<br>
><br>
> Starting purge...<br>
> Key remove successful: 816416e1255a1724021895b531c0e313<br>
> Key remove successful: 615ef6c218cc6bc6d714a0742a07617b<br>
> Purge done.<br>
><br>
><br>
> > reinitialized the HSM\<br>
><br>
> Don't think this is necessary, but okay:<br>
><br>
> $ sudo softhsm --init-token --slot 0 --label "OpenDNSSEC"<br>
> The SO PIN must have a length between 4 and 255 characters.<br>
> Enter SO PIN:<br>
> The user PIN must have a length between 4 and 255 characters.<br>
> Enter user PIN:<br>
> The token has been initialized.<br>
><br>
><br>
> > removed the single zone I used to sign\<br>
><br>
</span>> $ sudo ods-ksmutil zone delete --zone <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>><br>
<div><div class="h5">> zonelist filename set to /etc/opendnssec/zonelist.xml.<br>
> Zone list updated: 1 removed, 0 added, 0 updated.<br>
><br>
><br>
> > reinitialized the database "ods-ksmutil setup"\<br>
><br>
> I think you should first stop the opendnssec service, but I will not do<br>
> that now:<br>
><br>
> $ sudo ods-ksmutil setup<br>
> *WARNING* This will erase all data in the database; are you sure?<br>
> [y/N] y<br>
> fixing permissions on file /var/opendnssec/kasp.db<br>
> zonelist filename set to /etc/opendnssec/zonelist.xml.<br>
> kasp filename set to /etc/opendnssec/kasp.xml.<br>
> Repository SoftHSM found<br>
> No Maximum Capacity set.<br>
> RequireBackup NOT set; please make sure that you know the potential<br>
> problems of using keys which are not recoverable<br>
> INFO: The XML in /etc/opendnssec/conf.xml is valid<br>
> INFO: The XML in /etc/opendnssec/zonelist.xml is valid<br>
> INFO: The XML in /etc/opendnssec/kasp.xml is valid<br>
> WARNING: In policy default, Y used in duration field for Keys/KSK<br>
> Lifetime (P1Y) in /etc/opendnssec/kasp.xml - this will be interpreted as<br>
> 365 days<br>
> WARNING: In policy lab, Y used in duration field for Keys/KSK Lifetime<br>
> (P1Y) in /etc/opendnssec/kasp.xml - this will be interpreted as 365 days<br>
> Policy default found<br>
> Info: converting P1Y to seconds; M interpreted as 31 days, Y interpreted<br>
> as 365 days<br>
> Policy lab found<br>
> Info: converting P1Y to seconds; M interpreted as 31 days, Y interpreted<br>
> as 365 days<br>
><br>
><br>
> > pregenerated new keys\<br>
><br>
> But you have no zones currently (you removed the single zone)?<br>
><br>
> $ sudo ods-ksmutil key generate --policy default --interval P1Y<br>
> Key sharing is Off<br>
> Info: converting P1Y to seconds; M interpreted as 31 days, Y interpreted<br>
> as 365 days<br>
> HSM opened successfully.<br>
> Info: 0 zone(s) found on policy "default"<br>
> No zones on policy default, skipping...<br>
><br>
><br>
> > added a zone\<br>
><br>
</div></div>> $ sudo ods-ksmutil zone add --zone <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>><br>
<span class="">> zonelist filename set to /etc/opendnssec/zonelist.xml.<br>
</span>> Imported zone: <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>><br>
<div><div class="h5">><br>
><br>
> > updated, restarted all services.<br>
><br>
> $ sudo ods-control stop<br>
> Stopping enforcer...<br>
> Stopping signer engine...<br>
> Engine shut down.<br>
><br>
> $ sudo ods-control start<br>
> Starting enforcer...<br>
> OpenDNSSEC ods-enforcerd started (version 1.4.6), pid 28343<br>
> Starting signer engine...<br>
> OpenDNSSEC signer engine version 1.4.6<br>
> Engine running.<br>
><br>
> > Everything seems to worked well, but the signer does not find one of the<br>
> > keys to sign the zone, more specifically the KSK. I went the above<br>
> > process few times, always ending with:<br>
> ><br>
> > Dec 16 09:40:27 debugsigner002 ods-signerd: [hsm] unable to get key: key<br>
> > f81e4b2cb33eec780320b6ceeb6f6bb8 not found<br>
> > Dec 16 09:40:27 debugsigner002 ods-signerd: [zone] unable to publish<br>
> > dnskeys for zone XXX: error creating dnskey<br>
> > Dec 16 09:40:27 debugsigner002 ods-signerd: [tools] unable to read zone<br>
> > XXX: failed to publish dnskeys (General error)<br>
> > Dec 16 09:40:27 debugsigner002 ods-signerd: [worker[4]] CRITICAL: failed<br>
> > to sign zone XXX: General error<br>
><br>
> For me, it finds the old key in the<br>
> `/var/opendnssec/tmp/example.com.backup2` file and decides it is<br>
> corrupted:<br>
><br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [hsm] libhsm<br>
> connection opened succesfully<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [engine] signer<br>
> started (version 1.4.6), pid 28355<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [hsm] unable to<br>
> get key: key 615ef6c218cc6bc6d714a0742a07617b not found<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [zone] unable to<br>
</div></div>> publish dnskeys for zone <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>>: error<br>
<span class="">> creating dnskey<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [zone] corrupted<br>
</span>> backup file zone <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>>: unable to publish<br>
<span class="">> dnskeys (General error)<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [engine] unable to<br>
</span>> recover zone <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>> from backup,<br>
<span class="">> performing full sign<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [signconf] zone<br>
</span>> <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>> signconf: RESIGN[PT7200S]<br>
<span class="">> REFRESH[PT259200S]<br>
> VALIDITY[PT1209600S] DENIAL[PT1209600S] JITTER[PT43200S] OFFSET[PT3600S]<br>
> NSEC[50] DNSKEYTTL[PT3600S] SOATTL[PT3600S] MINIMUM[PT3600S]<br>
> SERIAL[unixtime]<br>
> Dec 16 10:07:59 vagrant-ubuntu-trusty-64 ods-signerd: [STATS]<br>
</span>> <a href="http://example.com" target="_blank">example.com</a> <<a href="http://example.com" target="_blank">http://example.com</a>> 1418724479 RR[count=61 time=0(sec)]<br>
<div><div class="h5">> NSEC3[count=60<br>
> time=0(sec)] RRSIG[new=112 reused=0 time=0(sec) avg=0(sig/sec)]<br>
> TOTAL[time=0(sec)]<br>
><br>
><br>
> > The key exist in both HSM and database. ods-hsmutil lists it:<br>
> ><br>
> > root@debugsigner002:~# ods-hsmutil list | grep<br>
> > f81e4b2cb33eec780320b6ceeb6f6bb8<br>
> > Keyper f81e4b2cb33eec780320b6ceeb6f6bb8 RSA/2048<br>
> ><br>
> > ods-ksmutil shows it:<br>
> ><br>
> > root@debugsigner002:~# ods-ksmutil key list -v<br>
> > Keys:<br>
> > Zone: Keytype: State: Date of next<br>
> > transition (to): Size: Algorithm: CKA_ID:<br>
> > Repository: Keytag:<br>
> > XXX KSK active 2016-01-16<br>
> > 09:49:45 (retire) 2048 8 f81e4b2cb33eec780320b6ceeb6f6bb8<br>
> > Keyper 6061<br>
> > XXX ZSK active 2015-04-18<br>
> > 22:40:55 (retire) 1024 8 d2aa0ba9af0f41429d23ea387abb836a<br>
> > Keyper<br>
> ><br>
> > external tools - dnssec-keyfromlabel can use it.<br>
> > No other errors in the log.<br>
> ><br>
> > Any ideas what's wrong? Suggestions what else to try?<br>
> > Thanks.<br>
> ><br>
> > Emil<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Opendnssec-user mailing list<br>
> > <a href="mailto:Opendnssec-user@lists.opendnssec.org">Opendnssec-user@lists.opendnssec.org</a><br>
</div></div>> <mailto:<a href="mailto:Opendnssec-user@lists.opendnssec.org">Opendnssec-user@lists.opendnssec.org</a>><br>
<span class="">> > <a href="https://lists.opendnssec.org/mailman/listinfo/opendnssec-user" target="_blank">https://lists.opendnssec.org/mailman/listinfo/opendnssec-user</a><br>
> ><br>
><br>
> _______________________________________________<br>
> Opendnssec-user mailing list<br>
> <a href="mailto:Opendnssec-user@lists.opendnssec.org">Opendnssec-user@lists.opendnssec.org</a><br>
</span>> <mailto:<a href="mailto:Opendnssec-user@lists.opendnssec.org">Opendnssec-user@lists.opendnssec.org</a>><br>
<div class="HOEnZb"><div class="h5">> <a href="https://lists.opendnssec.org/mailman/listinfo/opendnssec-user" target="_blank">https://lists.opendnssec.org/mailman/listinfo/opendnssec-user</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Opendnssec-user mailing list<br>
> <a href="mailto:Opendnssec-user@lists.opendnssec.org">Opendnssec-user@lists.opendnssec.org</a><br>
> <a href="https://lists.opendnssec.org/mailman/listinfo/opendnssec-user" target="_blank">https://lists.opendnssec.org/mailman/listinfo/opendnssec-user</a><br>
><br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Sebastian Castro<br>
Technical Research Manager<br>
.nz Registry Services (New Zealand Domain Name Registry Limited)<br>
desk: <a href="tel:%2B64%204%20495%202337" value="+6444952337">+64 4 495 2337</a><br>
mobile: <a href="tel:%2B64%2021%20400535" value="+6421400535">+64 21 400535</a><br>
</font></span></blockquote></div></div>