Hash mismatch when SSHing to dropbear server

[Matt Johnston]

On Thu, Aug 03, 2006 at 04:18:44PM +0200, Hans-Christian Egtvedt wrote:
> I've compiled dropbear for the avr32 platform, the compilation goes
> about without any problems, although there is alot of "differ in
> signedness" warnings.
> 
> The problem is that it works fine ssh'ing from the avr32 => avr32, but I
> can't ssh from my workstation (Intel P4 running Ubuntu Dapper).
> 
> Could the hash mismatch openssh reports on my workstation be related to
> a big-endian vs. little-endian bug/issue in the dropbear software?
> The avr32 is big-endian.

> hash mismatch
> debug1: ssh_rsa_verify: signature incorrect
> key_verify failed for server_host_key

> The same applies if I use a dss key.

> Any hints or tips to where I should start attacking would be great (-:

The error suggests that something is going wrong in the
big-number code (libtommath) - I don't _think_ it's just a
wrong-endianness issue. A first thing to try might be
compiling with -O0 or -O2 (rather than -Os as the default?),
and also trying -fno-strict-aliasing.

You could also try setting MP_16BIT at the top of
libtommath/tommath.h - that shouldn't make a difference, but
it's worth checking.

Let me know how it goes, if those don't help I'll have a
look further.

Matt