dropbear fails on uClinux ARM NOMMU target with gcc-3.4.5 but not gcc-2.95.3

[Jamie Lokier]

George McCollister wrote:
> I'm building/running the version of dropbear 0.43 included with 
> uClinux-dist-20051014. Target is little endian ARM CPU with no MMU. If I 
> use the arm-elf-tools-20040427 toolchain everything builds and works 
> fine. I cannot to the target using both ssh-rsa and ssh-dss. If I use my 
> binutils-2.15.90.0.1.1 + gcc-3.4.5 toolchain everything builds and runs 
> but...
> 
> when the host connects with ssh-rsa it errors out with:
> 
> RSA_public_decrypt failed: error:0407006A:rsa 
> routines:RSA_padding_check_PKCS1_type_1:block type is not 01
> debug1: ssh_rsa_verify: signature incorrect
> key_verify failed for server_host_key
> 
> when the host connects with ssh-dss the target errors out with:
> 
> /bin/staticdropbear: dss.c: 366: buf_put_dss_sign: Assertion `writelen 
> <= 20' failed.
> 
> I haven't had any other trouble with the binutils-2.15.90.0.1.1 + 
> gcc-3.4.5 toolchain, but I haven't done anything very math intensive 
> like crypto. Are there any testsuites that can be compiled to 
> exhaustively test the toolchain / target CPU?

There are toolchain testsuites - look at GCC's testsuite.  Though, you
might find it hard to run in this environment.

I've also seen problems with dropbear-0.48 and gcc-3.4.3, but the
problems I see are different: spurious memory accesses to addresses
that shouldn't be accessed.

I switched from gcc-2.95 at the same time as moving from
dropbear-0.47, so haven't narrowed down whether it's a compiler
problem, or a dropbear problem.

Have you tried adding the -fno-strict-aliasing option, or even -O1 or
-O0, to compiler flags?  Type-based aliasing is an optimisation,
turned on at -O2 or higher (unless -fno-strict-aliasing is used),
which does tend to break programs which got away with certain things
before.  Expecially code which accesses memory in a variety of ways to
speed up bit-twiddling - typical of crypto code.  Maybe that's it?

-- Jamie