[Opendnssec-develop] [OpenDNSSEC] #75: softHSM checks segfaults in current Debian unstable

[OpenDNSSEC]

#75: softHSM checks segfaults in current Debian unstable
------------------------------------------+---------------------------------
Reporter:  Ondřej Surý <ondrej@…>         |       Owner:  rb     
    Type:  defect                         |      Status:  new    
Priority:  major                          |   Component:  SoftHSM
 Version:  trunk                          |    Keywords:         
------------------------------------------+---------------------------------
 {{{
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -a
 Checking C_Initialize and C_Finalize: OK
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -b
 Checking C_GetInfo, C_GetFunctionList, C_GetSlotList, C_GetSlotInfo,
 C_GetTokenInfo, C_GetMechanismList, C_GetMechanismInfo: OK
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -c
 Checking C_OpenSession, C_CloseSession, C_CloseAllSessions, and
 C_GetSessionInfo: OK
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -d
 Checking C_Login and C_Logout: OK
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -e
 Checking C_SeedRandom and C_GenerateRandom: OK
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -f
 Segmentation fault
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -g
 Segmentation fault
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -i
 Checking C_DigestInit, C_Digest, C_DigestUpdate, and C_DigestFinal: OK
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -j
 Segmentation fault
 howl:/tmp/buildd/softhsm-1.1.2/checks# ./checks -k
 Segmentation fault
 howl:/tmp/buildd/softhsm-1.1.2/checks#
 }}}

 Backtrace:
 {{{
 (gdb) run -f
 Starting program: /tmp/buildd/softhsm-1.1.2/checks/checks -f
 [Thread debugging using libthread_db enabled]
 [New Thread 0xf77e0b70 (LWP 6175)]
 [Thread 0xf77e0b70 (LWP 6175) exited]

 Program received signal SIGSEGV, Segmentation fault.
 0xf7e0a925 in Botan::(anonymous namespace)::gmp_malloc(unsigned int) ()
 from /usr/lib/libbotan-1.8.2.so
 Current language:  auto
 The current source language is "auto; currently asm".
 (gdb) bt
 #0  0xf7e0a925 in Botan::(anonymous namespace)::gmp_malloc(unsigned int)
 () from /usr/lib/libbotan-1.8.2.so
 #1  0xf783658f in __gmpz_init () from /usr/lib/libgmp.so.3
 #2  0xf7e0c8a0 in Botan::GMP_MPZ::GMP_MPZ(Botan::BigInt const&) () from
 /usr/lib/libbotan-1.8.2.so
 #3  0xf7e0c338 in Botan::GMP_Engine::mod_exp(Botan::BigInt const&,
 Botan::Power_Mod::Usage_Hints) const () from /usr/lib/libbotan-1.8.2.so
 #4  0xf7e6a9c8 in Botan::Engine_Core::mod_exp(Botan::BigInt const&,
 Botan::Power_Mod::Usage_Hints) () from /usr/lib/libbotan-1.8.2.so
 #5  0xf7eae9ab in Botan::Power_Mod::set_modulus(Botan::BigInt const&,
 Botan::Power_Mod::Usage_Hints) const () from /usr/lib/libbotan-1.8.2.so
 #6  0xf7eaea72 in Botan::Power_Mod::Power_Mod(Botan::BigInt const&,
 Botan::Power_Mod::Usage_Hints) () from /usr/lib/libbotan-1.8.2.so
 #7  0xf7eaf251 in
 Botan::Fixed_Exponent_Power_Mod::Fixed_Exponent_Power_Mod(Botan::BigInt
 const&, Botan::BigInt const&, Botan::Power_Mod::Usage_Hints) () from
 /usr/lib/libbotan-1.8.2.so
 #8  0xf7eac8d8 in Botan::MillerRabin_Test::MillerRabin_Test(Botan::BigInt
 const&) () from /usr/lib/libbotan-1.8.2.so
 #9  0xf7ead140 in Botan::passes_mr_tests(Botan::RandomNumberGenerator&,
 Botan::BigInt const&, unsigned int) () from /usr/lib/libbotan-1.8.2.so
 #10 0xf7ea8b77 in Botan::random_prime(Botan::RandomNumberGenerator&,
 unsigned int, Botan::BigInt const&, unsigned int, unsigned int) () from
 /usr/lib/libbotan-1.8.2.so
 #11 0xf7f31833 in
 Botan::RSA_PrivateKey::RSA_PrivateKey(Botan::RandomNumberGenerator&,
 unsigned int, unsigned int) () from /usr/lib/libbotan-1.8.2.so
 #12 0x08052df8 in rsaKeyGen (session=0x806d658,
 pPublicKeyTemplate=0xffffd708, ulPublicKeyAttributeCount=6,
 pPrivateKeyTemplate=0xffffd6b4, ulPrivateKeyAttributeCount=7,
 phPublicKey=0xffffd78c,
     phPrivateKey=0xffffd788) at main.cpp:2247
 #13 0x080531f8 in C_GenerateKeyPair (hSession=2, pMechanism=0xffffd778,
 pPublicKeyTemplate=0xffffd708, ulPublicKeyAttributeCount=6,
 pPrivateKeyTemplate=0xffffd6b4, ulPrivateKeyAttributeCount=7,
     phPublicKey=0xffffd78c, phPrivateKey=0xffffd788) at main.cpp:2072
 #14 0x0804e1cc in runGenerateCheck (counter=5) at checks.c:719
 #15 0x08050ae4 in main (argc=2, argv=0xffffd864) at checks.c:94
 }}}

 All other failed checks segfaults in same function C_GenerateKeyPair.

 Installed libraries:
 {{{
 | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-
 pend
 |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 ||/ Name                                        Version
 Description
 +++-===========================================-===========================================-======================================================================================================
 ii  libbotan-1.8.2                              1.8.6-2
 multiplatform crypto library
 ii  libgmp3c2                                   2:4.3.2+dfsg-1
 Multiprecision arithmetic library
 }}}

-- 
Ticket URL: <http://trac.opendnssec.org/ticket/75>
OpenDNSSEC <http://www.opendnssec.org/>
OpenDNSSEC