[Opendnssec-develop] Re: [OpenDNSSEC] #75: softHSM checks segfaults in current Debian unstable

[OpenDNSSEC]

#75: softHSM checks segfaults in current Debian unstable
------------------------------------------+---------------------------------
Reporter:  Ondřej Surý <ondrej@…>         |       Owner:  rb     
    Type:  defect                         |      Status:  new    
Priority:  major                          |   Component:  SoftHSM
 Version:  trunk                          |    Keywords:         
------------------------------------------+---------------------------------

Comment(by Ondřej Surý <ondrej@…>):

 I have retried with botan 1.8.8 and it still segfaults, but with 1.8.8 the
 backtrace is slightly different (after recompiling with -O0):

 {{{
 (gdb) run -f
 Starting program: /tmp/buildd/softhsm-1.1.2/checks/checks -f
 [Thread debugging using libthread_db enabled]
 [New Thread 0x7ffff5b45910 (LWP 13809)]
 [Thread 0x7ffff5b45910 (LWP 13809) exited]

 Program received signal SIGSEGV, Segmentation fault.
 memcpy () at ../sysdeps/x86_64/memcpy.S:103
 103     ../sysdeps/x86_64/memcpy.S: No such file or directory.
         in ../sysdeps/x86_64/memcpy.S
 Current language:  auto
 The current source language is "auto; currently asm".
 (gdb) bt
 #0  memcpy () at ../sysdeps/x86_64/memcpy.S:103
 #1  0x00007ffff7a4a687 in Botan::(anonymous namespace)::gmp_realloc(void*,
 unsigned long, unsigned long) () from /usr/lib/libbotan-1.8.2.so
 #2  0x00007ffff600035a in __gmpz_realloc () from /usr/lib/libgmp.so.3
 #3  0x00007ffff5ff9650 in __gmpz_import () from /usr/lib/libgmp.so.3
 #4  0x00007ffff7a4bf61 in Botan::GMP_MPZ::GMP_MPZ(Botan::BigInt const&) ()
 from /usr/lib/libbotan-1.8.2.so
 #5  0x00007ffff7a4bb47 in Botan::GMP_Engine::mod_exp(Botan::BigInt const&,
 Botan::Power_Mod::Usage_Hints) const () from /usr/lib/libbotan-1.8.2.so
 #6  0x00007ffff7a8cce9 in Botan::Engine_Core::mod_exp(Botan::BigInt
 const&, Botan::Power_Mod::Usage_Hints) () from /usr/lib/libbotan-1.8.2.so
 #7  0x00007ffff7ac865a in Botan::Power_Mod::set_modulus(Botan::BigInt
 const&, Botan::Power_Mod::Usage_Hints) const () from
 /usr/lib/libbotan-1.8.2.so
 #8  0x00007ffff7ac8839 in
 Botan::Fixed_Exponent_Power_Mod::Fixed_Exponent_Power_Mod(Botan::BigInt
 const&, Botan::BigInt const&, Botan::Power_Mod::Usage_Hints) () from
 /usr/lib/libbotan-1.8.2.so
 #9  0x00007ffff7ac6466 in
 Botan::MillerRabin_Test::MillerRabin_Test(Botan::BigInt const&) () from
 /usr/lib/libbotan-1.8.2.so
 #10 0x00007ffff7ac6900 in
 Botan::passes_mr_tests(Botan::RandomNumberGenerator&, Botan::BigInt
 const&, unsigned int) () from /usr/lib/libbotan-1.8.2.so
 #11 0x00007ffff7ac35a5 in
 Botan::random_prime(Botan::RandomNumberGenerator&, unsigned int,
 Botan::BigInt const&, unsigned int, unsigned int) () from
 /usr/lib/libbotan-1.8.2.so
 #12 0x00007ffff7b2c73f in
 Botan::RSA_PrivateKey::RSA_PrivateKey(Botan::RandomNumberGenerator&,
 unsigned int, unsigned int) () from /usr/lib/libbotan-1.8.2.so
 #13 0x000000000040d672 in rsaKeyGen (session=0x447af0,
 pPublicKeyTemplate=0x7fffffffe170, ulPublicKeyAttributeCount=6,
 pPrivateKeyTemplate=0x7fffffffe0c0, ulPrivateKeyAttributeCount=7,
     phPublicKey=0x7fffffffdfa8, phPrivateKey=0x7fffffffdfa0) at
 main.cpp:2247
 #14 0x000000000040d113 in C_GenerateKeyPair (hSession=2,
 pMechanism=0x7fffffffdf80, pPublicKeyTemplate=0x7fffffffe170,
 ulPublicKeyAttributeCount=6, pPrivateKeyTemplate=0x7fffffffe0c0,
     ulPrivateKeyAttributeCount=7, phPublicKey=0x7fffffffdfa8,
 phPrivateKey=0x7fffffffdfa0) at main.cpp:2072
 #15 0x0000000000406729 in runGenerateCheck (counter=5) at checks.c:719
 #16 0x0000000000403f18 in main (argc=2, argv=0x7fffffffe338) at
 checks.c:94
 }}}

-- 
Ticket URL: <http://trac.opendnssec.org/ticket/75#comment:3>
OpenDNSSEC <http://www.opendnssec.org/>
OpenDNSSEC