[Opendnssec-develop] Re: [OpenDNSSEC] #75: softHSM checks segfaults in current Debian unstable

Fri Jan 22 13:29:38 UTC 2010

#75: softHSM checks segfaults in current Debian unstable
------------------------------------------+---------------------------------
Reporter:  Ondřej Surý <ondrej@…>         |       Owner:  rb     
    Type:  defect                         |      Status:  new    
Priority:  major                          |   Component:  SoftHSM
 Version:  trunk                          |    Keywords:         
------------------------------------------+---------------------------------

Comment(by Ondřej Surý <ondrej@…>):

 After some more debugging (linking with .a version of botan and gmp):

 There is something rotten in the kingdom of Denmark:

 softhsm compiled with -O2 backtrace:
 {{{
 Program received signal SIGSEGV, Segmentation fault.
 memcpy () at ../sysdeps/x86_64/memcpy.S:102
 102     ../sysdeps/x86_64/memcpy.S: No such file or directory.
         in ../sysdeps/x86_64/memcpy.S
 Current language:  auto
 The current source language is "auto; currently asm".
 (gdb) bt
 #0  memcpy () at ../sysdeps/x86_64/memcpy.S:102
 #1  0x0000000000482252 in gmp_realloc (ptr=0x0, old_n=8, new_n=48) at
 src/engine/gnump/gmp_mem.cpp:35
 #2  0x000000000053ff5b in __gmpz_realloc (m=0x64a208, new_alloc=6) at
 ../../mpz/realloc.c:51
 #3  0x000000000053e0a0 in __gmpz_import (z=0x64a208, count=6, order=-1,
 size=8, endian=0, nail=0, data=0x643b90) at ../../mpz/import.c:50
 #4  0x00000000004ff75b in GMP_MPZ (this=0x64a208, in=...) at
 src/engine/gnump/gmp_wrap.cpp:29
 #5  0x00000000004ff299 in GMP_Modular_Exponentiator (this=0x64a1e0, n=...)
 at src/engine/gnump/gmp_powm.cpp:27
 #6  0x00000000004ff409 in Botan::GMP_Engine::mod_exp (this=0x61e5b0,
 n=...) at src/engine/gnump/gmp_powm.cpp:50
 #7  0x00000000005076a5 in Botan::Engine_Core::mod_exp (n=..., hints=260)
 at src/libstate/pk_engine.cpp:164
 #8  0x00000000004ba251 in Botan::Power_Mod::set_modulus
 (this=0x7fffffffd740, n=..., hints=260) at
 src/math/numbertheory/pow_mod.cpp:58
 #9  0x00000000004ba014 in Power_Mod (this=0x7fffffffd740, n=...,
 hints=260) at src/math/numbertheory/pow_mod.cpp:19
 #10 0x00000000004baa7a in Fixed_Exponent_Power_Mod (this=0x7fffffffd740,
 e=..., n=..., hints=Botan::Power_Mod::NO_HINTS) at
 src/math/numbertheory/pow_mod.cpp:142
 #11 0x00000000004b9bcf in MillerRabin_Test (this=0x7fffffffd7b0, num=...)
 at src/math/numbertheory/numthry.cpp:342
 #12 0x00000000004b8a8c in Botan::passes_mr_tests (rng=..., n=..., level=1)
 at src/math/numbertheory/numthry.cpp:274
 #13 0x00000000004b7307 in Botan::random_prime (rng=..., bits=384,
 coprime=..., equiv=1, modulo=2) at src/math/numbertheory/make_prm.cpp:75
 #14 0x000000000044bedd in RSA_PrivateKey (this=0x6448f0, rng=...,
 bits=768, exp=65537, __in_chrg=<value optimized out>, __vtt_parm=<value
 optimized out>) at src/pubkey/rsa/rsa.cpp:68
 #15 0x000000000040d5f9 in rsaKeyGen (session=<value optimized out>,
 pPublicKeyTemplate=<value optimized out>, ulPublicKeyAttributeCount=6,
 pPrivateKeyTemplate=<value optimized out>,
     ulPrivateKeyAttributeCount=<value optimized out>, phPublicKey=<value
 optimized out>, phPrivateKey=0x7fffffffdf90) at main.cpp:2247
 #16 0x000000000040d9f1 in C_GenerateKeyPair (hSession=<value optimized
 out>, pMechanism=0x7fffffffdf70, pPublicKeyTemplate=0x7fffffffe160,
 ulPublicKeyAttributeCount=6,
     pPrivateKeyTemplate=0x7fffffffe0b0, ulPrivateKeyAttributeCount=7,
 phPublicKey=0x7fffffffdf98, phPrivateKey=0x7fffffffdf90) at main.cpp:2072
 #17 0x00000000004083b9 in runGenerateCheck (counter=5) at checks.c:719
 #18 0x0000000000405ba8 in main (argc=2, argv=0x7fffffffe328) at
 checks.c:94
 }}}

 softhsm compiled with -O0:
 {{{
 (gdb) run -f
 Starting program: /tmp/buildd/softhsm-1.1.2/checks/checks -f
 [Thread debugging using libthread_db enabled]
 [New Thread 0x7ffff5c3f910 (LWP 27915)]
 [Thread 0x7ffff5c3f910 (LWP 27915) exited]

 Program received signal SIGSEGV, Segmentation fault.
 0x00007ffff7dbce70 in vtable for Botan::Malloc_Allocator () from
 /usr/lib/libbotan-1.8.2.so
 (gdb) bt
 #0  0x00007ffff7dbce70 in vtable for Botan::Malloc_Allocator () from
 /usr/lib/libbotan-1.8.2.so
 #1  0x00007ffff7befc4d in gmp_malloc (n=8) at
 src/engine/gnump/gmp_mem.cpp:26
 #2  0x00007ffff5e71688 in __gmpz_init () from /usr/lib/libgmp.so.3
 #3  0x00007ffff7bf1805 in GMP_MPZ (this=0x44e2d8, in=...) at
 src/engine/gnump/gmp_wrap.cpp:27
 #4  0x00007ffff7bf126d in GMP_Modular_Exponentiator (this=0x44e2d0, n=...)
 at src/engine/gnump/gmp_powm.cpp:27
 #5  0x00007ffff7bf146d in Botan::GMP_Engine::mod_exp (this=0x4226a0,
 n=...) at src/engine/gnump/gmp_powm.cpp:50
 #6  0x00007ffff7c3e32d in Botan::Engine_Core::mod_exp (n=..., hints=260)
 at src/libstate/pk_engine.cpp:164
 #7  0x00007ffff7c798d5 in Botan::Power_Mod::set_modulus
 (this=0x7fffffffd700, n=..., hints=260) at
 src/math/numbertheory/pow_mod.cpp:58
 #8  0x00007ffff7c79698 in Power_Mod (this=0x7fffffffd700, n=...,
 hints=260) at src/math/numbertheory/pow_mod.cpp:19
 #9  0x00007ffff7c7a0fe in Fixed_Exponent_Power_Mod (this=0x7fffffffd700,
 e=..., n=..., hints=Botan::Power_Mod::NO_HINTS) at
 src/math/numbertheory/pow_mod.cpp:142
 #10 0x00007ffff7c792ff in MillerRabin_Test (this=0x7fffffffd770, num=...)
 at src/math/numbertheory/numthry.cpp:342
 #11 0x00007ffff7c781bc in Botan::passes_mr_tests (rng=..., n=..., level=1)
 at src/math/numbertheory/numthry.cpp:274
 #12 0x00007ffff7c763ef in Botan::random_prime (rng=..., bits=384,
 coprime=..., equiv=1, modulo=2) at src/math/numbertheory/make_prm.cpp:75
 #13 0x00007ffff7cd2999 in RSA_PrivateKey (this=0x4489e0, rng=...,
 bits=768, exp=65537, __in_chrg=<value optimized out>, __vtt_parm=<value
 optimized out>) at src/pubkey/rsa/rsa.cpp:68
 #14 0x00007ffff7fe55ad in rsaKeyGen (session=0x42faf0,
 pPublicKeyTemplate=0x7fffffffe160, ulPublicKeyAttributeCount=6,
 pPrivateKeyTemplate=0x7fffffffe0b0, ulPrivateKeyAttributeCount=7,
     phPublicKey=0x7fffffffdf98, phPrivateKey=0x7fffffffdf90) at
 main.cpp:2247
 #15 0x00007ffff7fe5022 in C_GenerateKeyPair (hSession=2,
 pMechanism=0x7fffffffdf70, pPublicKeyTemplate=0x7fffffffe160,
 ulPublicKeyAttributeCount=6, pPrivateKeyTemplate=0x7fffffffe0b0,
     ulPrivateKeyAttributeCount=7, phPublicKey=0x7fffffffdf98,
 phPrivateKey=0x7fffffffdf90) at main.cpp:2072
 #16 0x0000000000403e69 in runGenerateCheck (counter=5) at checks.c:719
 #17 0x0000000000401658 in main (argc=2, argv=0x7fffffffe328) at
 checks.c:94
 }}}

-- 
Ticket URL: <http://trac.opendnssec.org/ticket/75#comment:6>
OpenDNSSEC <http://www.opendnssec.org/>
OpenDNSSEC