[Opendnssec-user] Missing keys and various other problems on 2.0

[Casper Gielen]

Op 02-07-18 om 10:33 schreef Berry A.W. van Halderen:
> On 06/28/2018 02:06 PM, Casper Gielen wrote:
>> 25-06-18 om 17:05 schreef Casper Gielen:
>>>> Are you using SoftHSM as HSM?  If so, which version?
>>>> There is a known, resolved issue with certain versions.
>>>
>>> I just switched to SoftHSM 2.4.0, from Debian Unstable.
>>> I'll run it for a bit and see if anything improves.
>>
>>
>> After two days nothing has happened. That is, all keys seem to be in
>> exactly the same state as two days ago.
>>
>> Calling 'ods-enforcer enforce' manually does trigger something, but the
>> enforcer is not able to talk to our SQL server.
>>
>> Jun 28 11:52:16 ramachandra ods-enforcerd: DB prepare SQL SELECT
>> policy.id, policy.rev, policy.name, policy.description,
>> policy.signaturesResign, policy.signaturesRefresh,
>> policy.signaturesJitter, policy.signaturesInceptionOffset,
>> policy.signaturesValidityDefault, policy.signaturesValidityDenial, po
>> licy.signaturesValidityKeyset, policy.signaturesMaxZoneTtl,
>> policy.denialType, policy.denialOptout, policy.denialTtl,
>> policy.denialResalt, policy.denialAlgorithm, policy.denialIterations,
>> policy.denialSaltLength, policy.denialSalt, policy.denialSaltLastChange,
>> policy.keysTtl, policy.keysRetireSafety
>> , policy.keysPublishSafety, policy.keysShared, policy.keysPurgeAfter,
>> policy.zonePropagationDelay, policy.zoneSoaTtl, policy.zoneSoaMinimum,
>> policy.zoneSoaSerial, policy.parentRegistrationDelay,
>> policy.parentPropagationDelay, policy.parentDsTtl, policy.parentSoaTtl,
>> policy.parentSoaMinimum, policy.p
>> assthrough FROM policy WHERE policy.id = ?
>> Jun 28 11:52:16 ramachandra ods-enforcerd: DB prepare Err 2006: MySQL
>> server has gone away
>> Jun 28 11:52:16 ramachandra ods-enforcerd:
>> [hsm_key_factory_generate_task] generate for policy key [duration: 0]
>> Jun 28 11:52:16 ramachandra ods-enforcerd: [hsm_key_factory_generate]
>> repository LocalHSM role KSK
>> Jun 28 11:52:16 ramachandra ods-enforcerd: SELECT COUNT(*) FROM hsmKey
>> WHERE hsmKey.policyId = ? AND hsmKey.state = ? AND hsmKey.bits = ? AND
>> hsmKey.algorithm = ? AND hsmKey.role = ? AND hsmKey.isRevoked = ? AND
>> hsmKey.keyType = ? AND hsmKey.repository = ?
>> Jun 28 11:52:16 ramachandra ods-enforcerd: DB prepare SQL SELECT
>> COUNT(*) FROM hsmKey WHERE hsmKey.policyId = ? AND hsmKey.state = ? AND
>> hsmKey.bits = ? AND hsmKey.algorithm = ? AND hsmKey.role = ? AND
>> hsmKey.isRevoked = ? AND hsmKey.keyType = ? AND hsmKey.repository = ?
>> Jun 28 11:52:16 ramachandra ods-enforcerd: DB prepare Err 2006: MySQL
>> server has gone away
>>
>> After restarting the enforcer it connects correctly to MySQL and the
>> keys start advancing through the various states.
> 
> That would be known issue:
> https://issues.opendnssec.org/browse/OPENDNSSEC-913
> 
> There is some code which "keeps" the connection alive, but in case
> the connection goes anyway, it won't reconnect.  There are two
> reasons for a connection to get lost:

> - A deliberate restart of the database.  Althrough we should address
>   this, it's not a very frequent case.

So far I've not seen a relation between restarts MariaDB and my problem,
even though I've tried to break it by repeatedly restarting MariaDB.
I'll keep it in mind in case there is a pattern I've missed so far.

> - A too short timeout on the mysql/mariadb compared to how often
>   the enforcer wakes up to check zones.  Can also be adressed as above
>   but often also resolved by larger settings to interactive_timeout
>   (and possible wait_timeout, though that shouldn't).

Both are at 8 hours, that should be plenty, but just to be sure I've
increased both to 80 hours.

> 
>> I've added a cron-job that restarts the enforcer every 6 hours.
>> That's not ideal but should make clear if the problem is just that the
>> enforcer gets stuck and thus misses its deadlines, or if the problems go
>> deeper.

Due to a small mistake this cron-job never got installed on the system
and this morning the enforcer was stuck again, so I don't have an new
results.
I've fixed the problem and the enforcer got back to it. I hope to have
more information tomorrow.


-- 
Casper Gielen <cgielen at uvt.nl> | LIS UNIX
PGP fingerprint = 16BD 2C9F 8156 C242 F981  63B8 2214 083C F80E 4AF7

Universiteit van Tilburg | Postbus 90153, 5000 LE
Warandelaan 2 | Telefoon 013 466 4100 | G 236 | http://www.uvt.nl