Hash mismatch when SSHing to dropbear server

[Matt Johnston]

On Fri, Aug 04, 2006 at 10:09:08AM +0200, Hans-Christian Egtvedt wrote:

> So the error corrects it self when I ssh localhost, but external targets
> is a no-go.
> 
> Could this be an alligment error?

Possibly? To try and figure what's going on, I've put a
version with some copius debugging of the RSA bits at
http://matt.ucc.asn.au/dropbear/testing/db-unrandom1.tar.gz
The random number generator has been made totally
deterministic, so subsequent runs on different hosts can be
compared.

Compile the server binary for your workstation and compile
the client for both the avr32 and the workstation. Run the
server on your workstation using the included "constant_rsa"
key - then I can compare what I'm seeing. You might want to
run it with "2> /dev/null" as it'll print a lot of junk.

Then run the debugging client both on the avr32 and your
workstation, as "./dbclient -v workstation 2> logfile".
If it's working correctly, you should see mostly the same
output on each host (I've put a copy of what I see in that
website dir). If it seems to hang for ages before showing a
password prompt, you might need to press "y" for the hidden
key confirmation prompt.

I'm guessing that you'll probably see different big-number
output on the avr32, which might help narrow things down.
I'm not entirely sure what to be looking for, but with some
extra dropbear_trace() and print_mp_int() calls in
libtommath/bn_mp_exptmod_fast.c it should be possible to
narrow down the differences to a particular bit of code.

Hopefully that might be of use.

Cheers,
Matt