dropbear fails on uClinux ARM NOMMU target with gcc-3.4.5 but not gcc-2.95.3

[George McCollister]

Matt Johnston wrote:

>On Fri, Jan 06, 2006 at 09:24:59AM -0600, George McCollister wrote:
>  
>
>>I'm building/running the version of dropbear 0.43 included with 
>>uClinux-dist-20051014. Target is little endian ARM CPU with no MMU. If I 
>>use the arm-elf-tools-20040427 toolchain everything builds and works 
>>fine. I cannot to the target using both ssh-rsa and ssh-dss. If I use my 
>>binutils-2.15.90.0.1.1 + gcc-3.4.5 toolchain everything builds and runs 
>>but...
>>
>>when the host connects with ssh-rsa it errors out with:
>>
>>RSA_public_decrypt failed: error:0407006A:rsa 
>>routines:RSA_padding_check_PKCS1_type_1:block type is not 01
>>debug1: ssh_rsa_verify: signature incorrect
>>key_verify failed for server_host_key
>>
>>when the host connects with ssh-dss the target errors out with:
>>
>>/bin/staticdropbear: dss.c: 366: buf_put_dss_sign: Assertion `writelen 
>><= 20' failed.
>>    
>>
>
>Both the DSS and RSA failures look like some particular
>big-number operation isn't giving the right result. Whether
>it's a compiler error or a code error is kind of hard to
>tell. The fact that one compiler works correctly makes me a
>bit suspicious of the gcc 3.4.5 compiler - crypto code seems
>to bring out many obscure compiler bugs. It's also plausible
>that libtommath (or libtomcrypt) is doing some broken struct
>aliasing or something, that a newer compiler won't cope
>with. 
>
>Does CFLAGS=-O0 make a difference?
>
>  
>
>>I haven't had any other trouble with the binutils-2.15.90.0.1.1 + 
>>gcc-3.4.5 toolchain, but I haven't done anything very math intensive 
>>like crypto. Are there any testsuites that can be compiled to 
>>exhaustively test the toolchain / target CPU?
>>    
>>
>
>LibTomCrypt and LibTomMath (the libraries used by Dropbear)
>both have testsuites. I'd suggest running testsuites for
>LibTomMath 0.27 and LibTomCrypt 0.96, as those are the
>versions bundled with Dropbear 0.43.
>(http://libtomcrypt.org/download.html
>and http://math.libtomcrypt.org/download.html)
>
>For LibTomMath, just "make test" in the top-level directory.
>It seems that LibTomMath's testsuite is run with
>"./mtest/mtest | ./test" - the test will run indefinitely,
>exiting on an error. mtest just generates a test set to work
>from.
>
>LibTomCrypt requires you to run "make" in the top-level
>first to build the library. Then copy the attached makefile
>to demos/test/makefile (there are a couple of errors), then
>run "make" in demos/test. You should then be able to ./test
>the program there.
>
>I assume some modifications will be required for
>cross-compiling etc.
>
>I can't really think of any other obvious things to try - I
>don't recall fixing any related issues in newer releases,
>though newer libtommath/libtomcrypt releases might improve
>things.
>
>Let me know how it goes.
>
>Cheers,
>Matt
>
>  
>
Going from -O2 to -O0 didn't help. I verified dropbear, libtommath, and 
libtomcrypt are now getting -O0. I'll try scouring through the ARM 
specific gcc flags and gcc configuration and see if I can find something 
that will fix it.

Cheers,
George McCollister



This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information.  If you have
received it in error, please notify the sender immediately and delete
the original.  Any other use of the email by you is prohibited.


This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information.  If you have
received it in error, please notify the sender immediately and delete
the original.  Any other use of the email by you is prohibited.