dropbear fails on uClinux ARM NOMMU target with gcc-3.4.5 but not gcc-2.95.3

[Matt Johnston]

On Fri, Jan 06, 2006 at 09:24:59AM -0600, George McCollister wrote:
> I'm building/running the version of dropbear 0.43 included with 
> uClinux-dist-20051014. Target is little endian ARM CPU with no MMU. If I 
> use the arm-elf-tools-20040427 toolchain everything builds and works 
> fine. I cannot to the target using both ssh-rsa and ssh-dss. If I use my 
> binutils-2.15.90.0.1.1 + gcc-3.4.5 toolchain everything builds and runs 
> but...
> 
> when the host connects with ssh-rsa it errors out with:
> 
> RSA_public_decrypt failed: error:0407006A:rsa 
> routines:RSA_padding_check_PKCS1_type_1:block type is not 01
> debug1: ssh_rsa_verify: signature incorrect
> key_verify failed for server_host_key
> 
> when the host connects with ssh-dss the target errors out with:
> 
> /bin/staticdropbear: dss.c: 366: buf_put_dss_sign: Assertion `writelen 
> <= 20' failed.

Both the DSS and RSA failures look like some particular
big-number operation isn't giving the right result. Whether
it's a compiler error or a code error is kind of hard to
tell. The fact that one compiler works correctly makes me a
bit suspicious of the gcc 3.4.5 compiler - crypto code seems
to bring out many obscure compiler bugs. It's also plausible
that libtommath (or libtomcrypt) is doing some broken struct
aliasing or something, that a newer compiler won't cope
with. 

Does CFLAGS=-O0 make a difference?

> I haven't had any other trouble with the binutils-2.15.90.0.1.1 + 
> gcc-3.4.5 toolchain, but I haven't done anything very math intensive 
> like crypto. Are there any testsuites that can be compiled to 
> exhaustively test the toolchain / target CPU?

LibTomCrypt and LibTomMath (the libraries used by Dropbear)
both have testsuites. I'd suggest running testsuites for
LibTomMath 0.27 and LibTomCrypt 0.96, as those are the
versions bundled with Dropbear 0.43.
(http://libtomcrypt.org/download.html
and http://math.libtomcrypt.org/download.html)

For LibTomMath, just "make test" in the top-level directory.
It seems that LibTomMath's testsuite is run with
"./mtest/mtest | ./test" - the test will run indefinitely,
exiting on an error. mtest just generates a test set to work
from.

LibTomCrypt requires you to run "make" in the top-level
first to build the library. Then copy the attached makefile
to demos/test/makefile (there are a couple of errors), then
run "make" in demos/test. You should then be able to ./test
the program there.

I assume some modifications will be required for
cross-compiling etc.

I can't really think of any other obvious things to try - I
don't recall fixing any related issues in newer releases,
though newer libtommath/libtomcrypt releases might improve
things.

Let me know how it goes.

Cheers,
Matt