[Opendnssec-develop] SoftHSMv2 and IPC

[Roland van Rijswijk - Deij]

Hi Jerry,

On 31 okt. 2012, at 08:42, Jerry Lundström <jerry at opendnssec.org> wrote:

> On Tue, Oct 30, 2012 at 10:19 PM, Rickard Bellgrim
> <rickard at opendnssec.org> wrote:
>>> Why do we have a semaphore per object?
>> 
>> For synchronizing changes between processes. There is a memory layer
>> and file layer. To minimize the number of semaphore, it would be
>> better to re-design this part.
> 
> I've been looking at the code and from what I can see it uses
> semaphores as a type of revision for the object tokens and files to
> know when to reread them from disk, please correct me if I'm wrong.

That is correct. This is my fault/doing, I opted for this as it seemed to be the most platform independent way of achieving simple and carefree IPC triggers. To improve the performance of SoftHSM v2 all data is kept in memory and only re-read from disk when necessary. This means there needed to be a way to trigger all running processes when an object is created or changed.

> Since we have a file per key (I assume) then there is another
> limitation that you must be aware of, the number of files in a
> directory is limited depending on the file system used. This is why
> most file based backends like this split the first part of the files
> names into 2-3 levels of directories to increase the number of files
> it can have.

There is a file per PKCS #11 object and a directory per token.

> The use of semaphores here is very strange in my view but it could work.

It is the simplest (i.e. most lightweight) way I could think of to do the inter-process signalling. But if it isn't available on some platforms (as you mentioned in an earlier e-mail) it might be better to replace this.

> There is one thing that came to mind, if instead of using semaphore
> you use the first bytes of the file and a revision marker then this
> type of object store could be used on a network based file system
> (NFS) running two active setups. Then put Lim (or similar) on top and
> you have a redundant network based (H)SM with REST/SOAP/* APIs.

I considered that when writing the code, but that meant IOPs which turned out to be much more expensive (in terms of cycles/delay) than the semaphore.

> I also noticed that in ObjectFile::commitTransaction it releases the
> mutex for the object, releases the transaction lock and puts it out of
> transaction before storing the object. And in store() it does not lock
> the mutex before checking if its in transaction. This can lead to
> another thread/process starting an transaction after store() checked
> that and alter information of the ObjectFile as it is storing it.
> commitTransaction should not release the mutex, it should be locked
> during the store().
> 
> And I see a lot of mutexes, transaction locks and locks on the files
> itself all in different orders which will eventually lead to dead
> locks.


Hmm… that may require some inspection.

Cheers,

Roland

-- Roland M. van Rijswijk - Deij
-- SURFnet bv
-- w: http://www.surfnet.nl/en/
-- t: +31-30-2305388
-- e: roland.vanrijswijk at surfnet.nl