[tech] Problems with Mylah's network connection

[David Adam]

For as long as I can remember, people have been complaining that the 
network is slow. "Whatever", I usually think, and go back to IRC.

Then I wrote a backups script that uses SSH and discovered that it would 
die occasionally with "Corrupted MAC on input. Disconnecting: Packet 
corrupt". Oops.

Further investigation reveals that an
  `ssh root at murasoi cat /dev/zero | pv > /dev/null`
pipe will eventually die with the same error message (i.e. the stream gets 
corrupted at some point), but it usually takes hours and up to twenty 
gigabytes of traffic. It happens between Mylah and Motsugo, too, but it 
doesn't seem to happen between Murasoi and anything else (including 
anything on the same switch as Mylah) - over 3TB transferred without a 
problem.

Interestingly even though both Mylah and Murasoi are on gigabit 
connections, the maximum throughput is more like 2-300 megabit as measured 
by pv(1); on other gigabit-enabled hosts on the machine room it is more 
like 800 megabit. The throughput also drops every few minutes to basically 
zero. iperf(1) shows similar information.

A little bit of analysis with tcpdump(8) shows that captures on Mylah show 
significant packet loss interrupting the TCP stream - lots of missed ACKs 
and retransmissions. I suspect this causing the throughput limitations and 
occasional pauses, but I'm not sure it is responsible for the corrupted 
packets.

I'm not really sure where to go from here. iperf is supposed to give some 
in-depth indication of TCP performance or dropped datagrams in UDP mode 
but does neither. The tcpdump traces are not particularly enlightening; 
not much is changing quickly in `netstat -s`.

I wonder about the performance of a 32-bit 33mHz gigabit network card but 
have no idea how to measure the PCI utilisation or interrupt frequency on 
Linux. (pcitop looked promising but only works on HP IA-64 machines.)

Anyone have any thoughts?

David