<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Disassembly of fast_s_mp_sqr() and other libtommath functions reveals gcc is utilizing the arm NEON SIMD instructions and registers for calculations involved with libtommath's mp_word scalar. Based on the 64-bit word corruption I see I'm guessing the SIMD registers
aren't being preserved/restored properly somewhere, probably during a context switch, specifically s16–s31 (d8–d15, q4–q7), which AAPCS says must be preserved and which I see being used in the disassembly of fast_s_mp_sqr(). I'lll write some test code later
today to see if this is the case, and if so, try to track down where and why the registers aren't being preserved.<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Horshack <horshack@live.com><br>
<b>Sent:</b> Saturday, March 21, 2020 1:11 AM<br>
<b>To:</b> Matt Johnston <matt@ucc.asn.au><br>
<b>Cc:</b> dropbear@ucc.asn.au <dropbear@ucc.asn.au><br>
<b>Subject:</b> Re: SSH key exchange fails 30-70% of the time on Netgear X4S R7800</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I have one of the failure paths isolated down to a single corrupt 64-bit word in memory, which required a significant amount of code instrumentation to achieve. I implemented a code execution
history buffer that gets filled at various checkpoints within s_mp_exptmod() and some of the modules called by it. To facilitate this history mechanism I packaged all of s_mp_exptmod()'s local variables inside a structure , which consists of saving the local
scalar vars in addition to crc32's of all the mp_int data structures with a separate crc32 of the mp_int.dp payload (data). When a failure occurs, ie one or more of the three back-to-back debug invocations of s_mp_exptmod yields a mismatching signed key result,
I dump out the history elements for each of the invocations to determine the first code checkpoint where failing invocation departed from the known correct invocation.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Here's a sample capture demonstrating.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Format is event #:, source code line #, crc32 of local scalars, crc32 of mp_int structures (minus dp field), and crc32 of all the mp_int dp data payloads. In this sample, the crc32 of the
dp data payload is different, which causes all subsequent crc32's for the remainder of the invocation to be difference since the data propagates through all the subsequent calculations performed in the routine.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><span style="font-size:10pt; font-family:"Courier New",monospace">1554: line=0492, crcLocalVars=6a08573e, crcMpIntNoDp=ab967993, crcMpIntDp=ded4078e crcRes=2554be5b, 0021 0005 0016 0002 0061
0003 0001</span><span><br>
</span>
<div><span style="font-size:10pt; font-family:"Courier New",monospace">1555: line=0488, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ad3e197a, crcMpIntDp=e71d5c11 crcRes=5ef59250, 0021 0005 0016 0002 0061 0004 0001</span><br>
</div>
<div><span style="font-size:10pt; font-family:"Courier New",monospace">1556: line=2049, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ad3e197a, crcMpIntDp=e71d5c11 crcRes=5ef59250, 0021 0005 0016 0002 0061 0004 0001</span><br>
</div>
<div><span style="font-size:10pt; font-family:"Courier New",monospace">1557: line=2062, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ab967993, crcMpIntDp=<b>21b13223</b> crcRes=a43fde70, 0021 0005 0016 0002 0061 0004 0001</span><b><br>
</b></div>
<div><span style="font-size:10pt; font-family:"Courier New",monospace">1558: line=0492, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ab967993, crcMpIntDp=<b>21b13223</b> crcRes=a43fde70, 0021 0005 0016 0002 0061 0004 0001</span><br>
</div>
<div><span style="font-size:10pt; font-family:"Courier New",monospace">1559: line=0501, crcLocalVars=7a3e1d2a, crcMpIntNoDp=ad3e197a, crcMpIntDp=<b>7691624d</b> crcRes=6d1388bc, 0021 0005 0016 0002 0061 0005 0001</span></div>
<div><span style="font-size:8pt; font-family:"Courier New",monospace"><br>
</span></div>
<div><span style="font-size:8pt; font-family:"Courier New",monospace"><span style="font-size:10pt">1554: line=0492, crcLocalVars=6a08573e, crcMpIntNoDp=ab967993, crcMpIntDp=ded4078e crcRes=2554be5b, 0021 0005 0016 0002 0061 0003 0001</span><span><br>
</span>
<div><span style="font-size:10pt">1555: line=0488, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ad3e197a, crcMpIntDp=e71d5c11 crcRes=5ef59250, 0021 0005 0016 0002 0061 0004 0001</span><br>
</div>
<div><span style="font-size:10pt">1556: line=2049, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ad3e197a, crcMpIntDp=e71d5c11 crcRes=5ef59250, 0021 0005 0016 0002 0061 0004 0001</span><br>
</div>
<div><span style="font-size:10pt">1557: line=2062, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ab967993, crcMpIntDp=<b>a2639ce4</b> crcRes=74f7dec6, 0021 0005 0016 0002 0061 0004 0001</span><b><br>
</b></div>
<div><span style="font-size:10pt">1558: line=0492, crcLocalVars=7dc8fe2c, crcMpIntNoDp=ab967993, crcMpIntDp=<b>a2639ce4</b> crcRes=74f7dec6, 0021 0005 0016 0002 0061 0004 0001</span><br>
</div>
<div><span style="font-size:10pt">1559: line=0501, crcLocalVars=7a3e1d2a, crcMpIntNoDp=ad3e197a, crcMpIntDp=<b>5e3343d2</b> crcRes=517ed1b0, 0021 0005 0016 0002 0061 0005 0001</span><br>
</div>
<span></span></span><br>
</div>
<span></span></div>
<div>I initially found the failure occurs at seemingly random places, affected mostly by the variances of code/data placement between builds, which also affects the frequency of failure. Through a lot of trial and error I was able to tease the failure down
to one of the simplest code paths (fast_s_mp_sqr), which required balancing debug code placement to keep the movement of the failure in control. fast_s_mp_sqr() does only basic arithmetic and is easy to follow. I haven't yet determined if the corrupt data
is pre-calculation or post-calculation due to the limits of how much data I can snapshot in the history buffer. Nevertheless I expanded the history mechanism to snapshot the specific mp_int that usually is corrupted via this path (s_mp_exptmod's local res
structure).</div>
<div><br>
</div>
<div>Here is correct vs corrupt mp_init at the specific execution point where it departs from the previous correction invocation. The data fields prefixed by : are the actual content of the mp_int - I've highlighted the mismatching crc32's and the mismatching
64-bit word:<br>
</div>
<div><br>
</div>
<div>Correct invocation:<br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">8057: line=2062, crcLocalVars=1d8f10b6, crcMpIntNoDp=80a0f0a7, crcMpIntDp=<b>e92a3e1f</b> crcRes=<b>02003870</b>, 0018 0005 0020 0002 0016 0002 0000</span><span><br>
</span>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 05297100 04f1e4e6 0fb47d28 0ab5d584 00b2778c 08656465 02cc79bb 05e280c3 - 073117bc 037170a2 0603ef41 0a73c7af 0388c6cd 08b543fa 055d90c9 006afe46</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0c4d0d2b 0b8753bf 0ba6b917 0dbc26af 0d5d541f 03cbd888 0a8b07bb 06ce141b - 0f2e2cdc 0d83829c 00b9e992 007a007e 0b35c3fa 0f97fa98 078b16e2 05681c5a</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 09e81cad 0fcb1b35 0f017b34 0828f9c8 08253004 02f4139f 07b97efe 03a2c2c6 - 0baf31f0 038dc84d 0ec2028d 0a4d2163 0b3d8f14 03a5b8a1 07656722 0636f515</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 047c6a4e 0249e773 074fdaae 0c7affcb 025e144e 0e6e524b 0369a7e6 005e5b18 - 07359ab7 094aa102 06e091dc 048578b3 0f2023d6 09e16318 0fb25f70 091e7d0c</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 00e038fe 01fe0be1 0c879fba 055feb36 05135c48 063ef5c4 062acf74 0e2ee213 - 0b32d4b4 01ac1beb 0df27135 0645d3a2 02f54fab 04524d06 0e21e0a0 01a58051</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0d0dd311 0b10815a 08044871 0bec8042 0473b083 0d99e620 0db94b72 07398f84 - 06930d29 021f81cd 0e96625a 0ffa3c78 0c9908d6 0fd6f904 0f5dcfd9 0bd6e140</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0357bd4b 0488f3a9 00ed811d 0c8a129f 0bde5ab5 0c61d340 042eea72 01fe06f5 - 018c9e3d 025ede93 0ce5786c 00c174de 0479c67d 06c711f5 052ebca1 093bf956</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 042b9b5e 06a62fce 0eef5130 0065890a 0ed4ef4d 0adc823d 0b7ab96f 04639d68 - 0484c7b5 0135f153 0818067f 00cffc19 0097dcba 016e355b 002e3d3e 051065cb</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0b41750c 049fb50f 0be87386 0d76e872 0de83a61
<b>04f8c371 07daa886</b> 03a70e50 - 0c79ea89 016660c2 0963ebd6 09d9b469 0abd18ff 02c370ac 0ad5b8ba 04846255
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0e7c9e10 03662210 00000011 00000000 00000000 00000000 00000000 00000000 - 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000</span><br>
</div>
<span></span><br>
</div>
Corrupt invocation:</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace; font-size:10pt">8057: line=2062, crcLocalVars=1d8f10b6, crcMpIntNoDp=80a0f0a7, crcMpIntDp=<b>5a521526</b> crcRes=<b>86bd8450</b>, 0018 0005 0020 0002 0016 0002 0000</span><span><br>
</span>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 05297100 04f1e4e6 0fb47d28 0ab5d584 00b2778c 08656465 02cc79bb 05e280c3 - 073117bc 037170a2 0603ef41 0a73c7af 0388c6cd 08b543fa 055d90c9 006afe46</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0c4d0d2b 0b8753bf 0ba6b917 0dbc26af 0d5d541f 03cbd888 0a8b07bb 06ce141b - 0f2e2cdc 0d83829c 00b9e992 007a007e 0b35c3fa 0f97fa98 078b16e2 05681c5a</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 09e81cad 0fcb1b35 0f017b34 0828f9c8 08253004 02f4139f 07b97efe 03a2c2c6 - 0baf31f0 038dc84d 0ec2028d 0a4d2163 0b3d8f14 03a5b8a1 07656722 0636f515</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 047c6a4e 0249e773 074fdaae 0c7affcb 025e144e 0e6e524b 0369a7e6 005e5b18 - 07359ab7 094aa102 06e091dc 048578b3 0f2023d6 09e16318 0fb25f70 091e7d0c</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 00e038fe 01fe0be1 0c879fba 055feb36 05135c48 063ef5c4 062acf74 0e2ee213 - 0b32d4b4 01ac1beb 0df27135 0645d3a2 02f54fab 04524d06 0e21e0a0 01a58051</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0d0dd311 0b10815a 08044871 0bec8042 0473b083 0d99e620 0db94b72 07398f84 - 06930d29 021f81cd 0e96625a 0ffa3c78 0c9908d6 0fd6f904 0f5dcfd9 0bd6e140</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0357bd4b 0488f3a9 00ed811d 0c8a129f 0bde5ab5 0c61d340 042eea72 01fe06f5 - 018c9e3d 025ede93 0ce5786c 00c174de 0479c67d 06c711f5 052ebca1 093bf956</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 042b9b5e 06a62fce 0eef5130 0065890a 0ed4ef4d 0adc823d 0b7ab96f 04639d68 - 0484c7b5 0135f153 0818067f 00cffc19 0097dcba 016e355b 002e3d3e 051065cb</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0b41750c 049fb50f 0be87386 0d76e872 0de83a61
<b>07156229 072adcf7</b> 03a70e50 - 0c79ea89 016660c2 0963ebd6 09d9b469 0abd18ff 02c370ac 0ad5b8ba 04846255
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">: 0e7c9e10 03662210 00000011 00000000 00000000 00000000 00000000 00000000 - 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000</span><br>
</div>
<span></span><br>
</div>
<div>
<div id="x_appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<span></span>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>I don't see any immediate relationship between the corrupt vs expected data or any unique attributes of the corrupt data over multiple captures I've done. The above mp_int is post-execution of fast_s_mp_sqr(), so any corruption occurring within its execution
will get folded in and propagated into a form that wont be immediately recognizable since it's undergone arithmetic operations within the routine.<br>
</span>
<div><br>
</div>
<div>The fact the corruption is always a single 64-bit word is a good clue. fast_s_mp_sqr() uses 64-bit scalars (mp_word) in its carry arithmetic logic - I'll be looking into the disassembly of the routine to dig deeper.</div>
<div><br>
</div>
<div>For reference here is the history structures used for the above dumps:</div>
<div><br>
</div>
<span></span><span style="font-family:"Courier New",monospace; font-size:10pt">typedef struct _LOCAL_VARS {</span><span style="font-family:"Courier New",monospace; font-size:10pt"> // local vars of s_mp_exptmod() packaged into a struct</span><span><br>
</span>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int *G;
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int *X;
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int *P;
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int *Y;
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int M[TAB_SIZE];</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int res;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_int mu;
</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> mp_digit buf;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> int err, bitbuf, bitcpy, bitcnt, mode, digidx, x, y, winsize;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">} LOCAL_VARS;</span><br>
</div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">typedef struct _HISTORY_ELEMENT {</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> ushort lineNumber;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> ushort pad;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> uint crcLocalVars;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> uint crcMpInt_WithoutDp; // mp_int structure excluding .dp</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> uint crcMpIntDp; // all mp_int's in LOCAL_VARS</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> uint crcRes;</span><span style="font-family:"Courier New",monospace; font-size:10pt"> // just LOCAL_VARS.res</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> uint resDp[160];</span><span style="font-family:"Courier New",monospace; font-size:10pt"> // content of LOCAL_VARS.res</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt"> ushort bitbuf, bitcpy, bitcnt, mode, digidx, x, y;</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">} HISTORY_ELEMENT;</span><br>
</div>
<div><br>
</div>
<div>Here is the CPU info:</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">root@OpenWrt:/tmp# cat /proc/cpuinfo</span><span><br>
</span>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">processor : 0</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">model name : ARMv7 Processor rev 0 (v7l)</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">BogoMIPS : 6.00</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU implementer : 0x51</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU architecture: 7</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU variant : 0x2</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU part : 0x04d</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU revision : 0</span><br>
</div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">processor : 1</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">model name : ARMv7 Processor rev 0 (v7l)</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">BogoMIPS : 12.50</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU implementer : 0x51</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU architecture: 7</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU variant : 0x2</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU part : 0x04d</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">CPU revision : 0</span><br>
</div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">Hardware : Generic DT based system</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">Revision : 0000</span><br>
</div>
<span style="font-family:"Courier New",monospace; font-size:10pt">Serial : 0000000000000000</span><br>
</div>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
And the first few messages of the kernel log showing version and detected CPU details:</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] Booting Linux on physical CPU 0x0</span><span><br>
</span>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] Linux version 4.14.171 (builder@buildhost) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r10947-65030d81f3)) #0 SMP Thu Feb 27 21:05:12 2020</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] CPU: ARMv7 Processor [512f04d0] revision 0 (ARMv7), cr=10c5787d</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] CPU: div instructions available: patching division code</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] OF: fdt: Machine model: Netgear Nighthawk X4S R7800</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] Memory policy: Data cache writealloc</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] On node 0 totalpages: 122880</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] free_area_init_node: node 0, pgdat c0a27880, node_mem_map dda39000</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] Normal zone: 960 pages used for memmap</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] Normal zone: 0 pages reserved</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] Normal zone: 122880 pages, LIFO batch:31</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] random: get_random_bytes called from 0xc09008dc with crng_init=0</span><br>
</div>
<span style="font-family:"Courier New",monospace; font-size:10pt">[ 0.000000] percpu: Embedded 15 pages/cpu s29388 r8192 d23860 u61440</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matt Johnston <matt@ucc.asn.au><br>
<b>Sent:</b> Friday, March 20, 2020 3:50 AM<br>
<b>To:</b> Horshack <horshack@live.com><br>
<b>Subject:</b> Re: SSH key exchange fails 30-70% of the time on Netgear X4S R7800</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hi,
<div class=""><br class="">
</div>
<div class="">That's an interesting failure. You should be able to disable SMP if you set <span class="" style="background-color:rgb(255,255,255)">maxcpus=1 as a kernel boot argument - not sure where you would set that for your device though.</span></div>
<div class=""><span class="" style="background-color:rgb(255,255,255)">I guess the other option is that a kernel syscall somewhere is clobbering registers, disabling SMP wouldn't avoid that...</span></div>
<div class=""><span class="" style="background-color:rgb(255,255,255)">Which kernel is it running, and what's the CPU (/proc/cpuinfo)?</span></div>
<div class=""><span class="" style="background-color:rgb(255,255,255)"><br class="">
</span></div>
<div class=""><span class="" style="background-color:rgb(255,255,255)">Cheers,</span></div>
<div class=""><span class="" style="background-color:rgb(255,255,255)">Matt</span></div>
<div class=""><span class="" style="background-color:rgb(255,255,255)"><br class="">
</span></div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Fri 20/3/2020, at 3:28 pm, Horshack <<a href="mailto:horshack@live.com" class="">horshack@live.com</a>> wrote:</div>
<br class="x_x_Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Update - I have isolated the intermittent issue down to the interchangeable functions s_mp_exptmod_fast() and s_mp_exptmod() - by default s_mp_exptmod_fast() is compiled instead of s_mp_exptmod() [BN_MP_EXPTMOD_FAST_C] but both functions intermittently fail
and I decided to use s_mp_exptmod() as my focus because it's slightly simpler.</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
s_mp_exptmod() is called indirectly by rsa.c::buf_put_rsa_sign()'s call to mp_exptmod(). For the intermittent failing case if I call mp_exptmod() / s_mp_exptmod() immediately again with the same source mp_int structures it yields the correct data. Example -
debug code bolded:<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<blockquote class="" style="border-color:rgb(200,200,200); border-left-width:3px; border-left-style:solid; padding-left:1ex; margin-left:0.8ex; color:rgb(102,102,102)">
<div class=""><span class=""><span class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> DEF_MP_INT(rsa_s_backup);</span><span class=""><br class="">
</span></b>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> DEF_MP_INT(rsa_s_backup_2);</b></span><br class="">
</div>
<span class=""></span><br class="">
</span></span></div>
<div class=""><b class=""><span class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> mp_copy (&rsa_s, &rsa_s_backup);</span><span class=""><br class="">
</span></span></b></div>
<div class=""><span class="">
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> mp_copy (&rsa_s, &rsa_s_backup_2);</b></span><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> if (mp_exptmod(&rsa_tmp1, key->d, key->n, &rsa_s) != MP_OKAY) {</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> dropbear_exit("RSA error");</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> } <span class="x_x_Apple-converted-space"> </span></span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> if (mp_exptmod(&rsa_tmp1, key->d, key->n, &rsa_s_backup) != MP_OKAY) {</b></span><b class=""><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> dropbear_exit("RSA error");</b></span><b class=""><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> } <span class="x_x_Apple-converted-space"> </span></b></span><b class=""><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> if (mp_exptmod(&rsa_tmp1, key->d, key->n, &rsa_s_backup_2) != MP_OKAY) {</b></span><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> <span class="x_x_Apple-converted-space"> </span></b></span><b class=""><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> dropbear_exit("RSA error");</b></span><b class=""><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><b class=""> } <span class="x_x_Apple-converted-space"> </span></b></span><b class=""><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> printf("after mp_exptmod\n");</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> dump_mp_int("rsa_s", &rsa_s);</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> dump_mp_int("rsa_s_backup", &rsa_s_backup);</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> dump_mp_int("rsa_s_backup_2", &rsa_s_backup_2);</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> comp_mp_int("rsa_s", "rsa_s_backup", &rsa_s, &rsa_s_backup);</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> comp_mp_int("rsa_s_backup", "rsa_s_backup_2", &rsa_s_backup, &rsa_s_backup_2);</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> mp_clear(&rsa_s_backup);</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> mp_clear(&rsa_s_backup_2);</span><br class="">
</b></div>
<b class=""><span class=""></span></b></span></div>
<br class="">
</blockquote>
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Sample output from a failure, which contains the first portion of each mp_int->dp. Bolded text has wrong data:<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<blockquote class="" style="border-color:rgb(200,200,200); border-left-width:3px; border-left-style:solid; padding-left:1ex; margin-left:0.8ex; color:rgb(102,102,102)">
<span class="" style="font-family:"Courier New",monospace; font-size:10pt">after mp_exptmod</span><span class=""><br class="">
</span>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s [0xbef6c358]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 4a 00 00 00 c0 00 00 00 00 00 00 00 30 e1 8f 00 J...........0...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s</span>->dp [0x008fe130]:</span><br class="">
</div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 05 fb c0 0f 68 91 ff 0a 9f 05 57 0b 35 a2 bd 05 ....h.....W.5...</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0010 57 ec a0 0b 34 3c b1 0f fa 8b b5 08 ed aa 9c 04 W...4<..........</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0020 7e 88 bb 04 12 42 51 05 9a 6d 7d 0a 98 ef 12 0c ~....BQ..m}.....</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0030 76 e0 f4 0f ea 89 d7 0c 87 b0 76 03 12 a1 2d 0e v.........v...-.</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0040 d7 3c df 06 0f 54 92 04 23 90 .<...T..#.</span><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup [0xbef6c398]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 4a 00 00 00 c0 00 00 00 00 00 00 00 00 d8 8f 00 J...............</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup</span>->dp [0x008fd800]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 ec 9f a0 01 d4 8e e8 07 c3 ae df 0b 45 61 e6 06 ............Ea..</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0010 a1 99 59 03 d7 49 24 02 50 a6 ac 0a de a2 5c 0d ..Y..I$.P.....\.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0020 cb b7 3c 05 33 cb da 08 28 10 f2 04 14 69 d6 07 ..<.3...(....i..</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0030 8c 8e a5 04 f5 fc 92 0c ba 88 d9 04 71 b4 b2 08 ............q...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0040 bc 4f c7 0d de 73 f9 06 0d bf .O...s....</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup_2 [0xbef6c3a8]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 4a 00 00 00 c0 00 00 00 00 00 00 00 e0 d1 8f 00 J...............</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup_2</span>->dp [0x008fd1e0]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 ec 9f a0 01 d4 8e e8 07 c3 ae df 0b 45 61 e6 06 ............Ea..</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0010 a1 99 59 03 d7 49 24 02 50 a6 ac 0a de a2 5c 0d ..Y..I$.P.....\.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0020 cb b7 3c 05 33 cb da 08 28 10 f2 04 14 69 d6 07 ..<.3...(....i..</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0030 8c 8e a5 04 f5 fc 92 0c ba 88 d9 04 71 b4 b2 08 ............q...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0040 bc 4f c7 0d de 73 f9 06 0d bf .O...s....</span><br class="">
</div>
<span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s and rsa_s_backup differ</span><br class="">
</blockquote>
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Sometimes it's the second or third call that yields the incorrect data. In this instance it was the second call:<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<blockquote class="" style="border-color:rgb(200,200,200); border-left-width:3px; border-left-style:solid; padding-left:1ex; margin-left:0.8ex; color:rgb(102,102,102)">
<span class="" style="font-family:"Courier New",monospace; font-size:10pt"></span><span class="" style="font-family:"Courier New",monospace; font-size:10pt">after mp_exptmod</span><span class=""><br class="">
</span>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s [0xbe9a6358]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 4a 00 00 00 c0 00 00 00 00 00 00 00 30 c1 40 02 J...........0.@.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s</span>->dp [0x0240c130]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 25 b9 db 00 ec 62 00 0d 80 2d b0 0d 00 13 d3 06 %....b...-......</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0010 3f ec 8b 0a af 5d e9 03 2d f4 4b 0c 6c 3c 72 08 ?....]..-.K.l<r.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0020 5d 52 6a 08 21 4c dd 01 a2 59 1a 03 33 16 97 0f ]Rj.!L...Y..3...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0030 c7 69 c2 08 0b 61 d6 03 b9 86 fc 01 27 15 c8 0c .i...a......'...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0040 dd 03 b1 04 78 c7 9f 0f d8 9c ....x.....</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup [0xbe9a6398]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 4a 00 00 00 c0 00 00 00 00 00 00 00 00 b8 40 02 J.............@.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup</span>->dp [0x0240b800]:</span><br class="">
</div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 df 86 0c 0a 6c 2f 68 09 f9 a1 37 01 26 02 e7 0b ....l/h...7.&...</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0010 69 5c b8 0e 0b 95 3a 0d 26 24 00 0e 97 6f dc 0b i\....:.&$...o..</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0020 64 95 ed 0a c0 75 53 03 66 3d ff 0b 26 4b ce 09 d....uS.f=..&K..</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0030 89 12 d2 03 9b 9b 0b 09 19 2c 5a 00 2c 99 fc 0b .........,Z.,...</span><br class="">
</b></div>
<div class=""><b class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0040 ea ad 61 09 38 e1 6a 0a 49 a5 ..a.8.j.I.</span><br class="">
</b></div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup_2 [0xbe9a63a8]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 4a 00 00 00 c0 00 00 00 00 00 00 00 e0 b1 40 02 J.............@.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s_backup_2</span>->dp [0x0240b1e0]:</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0000 25 b9 db 00 ec 62 00 0d 80 2d b0 0d 00 13 d3 06 %....b...-......</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0010 3f ec 8b 0a af 5d e9 03 2d f4 4b 0c 6c 3c 72 08 ?....]..-.K.l<r.</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0020 5d 52 6a 08 21 4c dd 01 a2 59 1a 03 33 16 97 0f ]Rj.!L...Y..3...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0030 c7 69 c2 08 0b 61 d6 03 b9 86 fc 01 27 15 c8 0c .i...a......'...</span><br class="">
</div>
<div class=""><span class="" style="font-family:"Courier New",monospace; font-size:10pt"> 0040 dd 03 b1 04 78 c7 9f 0f d8 9c</span><span class="" style="font-family:"Courier New",monospace; font-size:10pt"><span class="x_x_Apple-converted-space"> </span>
....x.....</span><br class="">
</div>
<span class="" style="font-family:"Courier New",monospace; font-size:10pt">rsa_s and rsa_s_backup differ</span><br class="">
</blockquote>
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
I have heavily instrumented s_mp_exptmod() but due to the complexity of the calcualtions performed it's proving very difficult to root down to the issue. What I can tell so far is the failure point within s_mp_exptmod() varies from instance to instance, which
is odd because the only potential variant between my three, back-to-back invocations are the memory allocations (buffer locations) triggered by mp_exptmod(), although the invocations usually get provided the same buffer addresses. I tried various scaffolding
code on the core memory allocation routines to isolate any buffer overruns/overwrites the logic might be performing, including padding each allocation by a large block of bytes, but the intermittent failure case still occurs. The behavior I'm observing almost
appears as if the execution context is being corrupted (ie, processor registers) because the failure point moves around the various elements of the logic within the routine from one failure to the next - sometimes I see an early-stage mp_int structure with
the wrong data, sometimes one that has undergone many transformations - all within s_mp_exptmod().<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Do you know if OpenWRT has any way to disable SMP at runtime, or a method or technique to provide a critical section around a block of code to prevent any preemptive task switches?<br class="">
</div>
<div class="" style="font-family:Helvetica; font-size:13px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div id="x_x_appendonsend" class=""></div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<hr tabindex="-1" class="" style="display:inline-block; width:1166.1875px">
<div id="x_x_divRplyFwdMsg" dir="ltr" class=""><font class="" style="font-size:11pt" face="Calibri, sans-serif"><b class="">From:</b><span class="x_x_Apple-converted-space"> </span>Horshack <<a href="mailto:horshack@live.com" class="">horshack@live.com</a>><br class="">
<b class="">Sent:</b><span class="x_x_Apple-converted-space"> </span>Thursday, March 19, 2020 7:11 AM<br class="">
<b class="">To:</b><span class="x_x_Apple-converted-space"> </span>Matt Johnston <<a href="mailto:matt@ucc.asn.au" class="">matt@ucc.asn.au</a>><br class="">
<b class="">Cc:</b><span class="x_x_Apple-converted-space"> </span><a href="mailto:dropbear@ucc.asn.au" class="">dropbear@ucc.asn.au</a> <<a href="mailto:dropbear@ucc.asn.au" class="">dropbear@ucc.asn.au</a>><br class="">
<b class="">Subject:</b><span class="x_x_Apple-converted-space"> </span>Re: SSH key exchange fails 30-70% of the time on Netgear X4S R7800</font>
<div class=""> </div>
</div>
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Thanks Matt, I'll give that a shot when I get a build environment set up for the server-side/openwrt.<span class="x_x_Apple-converted-space"> </span><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I also plan to look at the RSA blinding logic in buf_put_rsa_sign(). Considering the intermittency of the issue I'm thinking the issue has some correlation or dependency to the
random data generated or transformed by that logic. Crypto is well outside my core competency so it'll be slow-going.<br class="">
</div>
<div class="">
<div id="x_x_x_appendonsend" class=""></div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<hr tabindex="-1" class="" style="display:inline-block; width:1166.1875px">
<div id="x_x_x_divRplyFwdMsg" dir="ltr" class=""><font class="" style="font-size:11pt" face="Calibri, sans-serif"><b class="">From:</b><span class="x_x_Apple-converted-space"> </span>Matt Johnston <<a href="mailto:matt@ucc.asn.au" class="">matt@ucc.asn.au</a>><br class="">
<b class="">Sent:</b><span class="x_x_Apple-converted-space"> </span>Thursday, March 19, 2020 7:04 AM<br class="">
<b class="">To:</b><span class="x_x_Apple-converted-space"> </span>Horshack <<a href="mailto:horshack@live.com" class="">horshack@live.com</a>><br class="">
<b class="">Cc:</b><span class="x_x_Apple-converted-space"> </span><a href="mailto:dropbear@ucc.asn.au" class="">dropbear@ucc.asn.au</a> <<a href="mailto:dropbear@ucc.asn.au" class="">dropbear@ucc.asn.au</a>><br class="">
<b class="">Subject:</b><span class="x_x_Apple-converted-space"> </span>Re: SSH key exchange fails 30-70% of the time on Netgear X4S R7800</font>
<div class=""> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hi,
<div class=""><br class="">
</div>
<div class="">The first thing I'd try would be to build with -O0 compilation flags to rule out compiler optimisations doing something strange. </div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Matt</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Thu 19/3/2020, at 3:42 pm, Horshack <<a href="mailto:horshack@live.com" class="">horshack@live.com</a>> wrote:</div>
<br class="x_x_x_x_Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Update - I cloned and built the dbclient source so I could enable the debug tracing facility to get more information about the 'Bad hostkey signature'. The intermittent failure is detected in recv_msg_kexdh_reply() -> buf_rsa_verify() -> mp_cmd(). If I bypass
the buf_rsa_verify() call then the session proceeds normally without issue, which indicates everything else in the key exchange is working 100% of the time. I'll dig deeper to see why the signed host key sent by the server is wrong.<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="">
</div>
<div class="" style="font-family:Helvetica; font-size:13px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<hr tabindex="-1" class="" style="display:inline-block; width:605.625px">
<div id="x_x_x_x_divRplyFwdMsg" dir="ltr" class=""><font class="" style="font-size:11pt" face="Calibri, sans-serif"><b class="">From:</b><span class="x_x_x_x_Apple-converted-space"> </span>Horshack <br class="">
<b class="">Sent:</b><span class="x_x_x_x_Apple-converted-space"> </span>Wednesday, March 18, 2020 9:36 AM<br class="">
<b class="">To:</b><span class="x_x_x_x_Apple-converted-space"> </span><a href="mailto:dropbear@ucc.asn.au" class="">dropbear@ucc.asn.au</a><span class="x_x_Apple-converted-space"> </span><<a href="mailto:dropbear@ucc.asn.au" class="">dropbear@ucc.asn.au</a>><br class="">
<b class="">Subject:</b><span class="x_x_x_x_Apple-converted-space"> </span>SSH key exchange fails 30-70% of the time on Netgear X4S R7800</font>
<div class=""> </div>
</div>
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Hi,</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I have a strange issue on my Netgear X4S R7800. Running either DD-WRT or OpenWrt, approximately 30-70% of my SSH login attempts fail. For OpenSSH clients the error reported is "error
in libcrypto". For the PuTTY client the error is more descriptive - "Signature from server's host key is invalid". The failure occurs even when using the OpenSSH client built in to OpenWrt itself (ie, SSH'ing into the router from the router via an existing
remote SSH session).</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The failure appears to be at the tail end of the key exchange, before authentication. I've tried varying the cipher (aes128-ctr / aes256-ctr), the MAC (hmac-sha1 / hmac-sha2-256),
and the key exchange algo (curve25519-sha256 /<span class="x_x_Apple-converted-space"> </span><a href="mailto:curve25519-sha256@libssh.org" class="">curve25519-sha256@libssh.org</a><span class="x_x_Apple-converted-space"> </span>/ diffie-hellman-group14-sha256
/ diffie-hellman-group14-sha1) but the intermittent failure still occurs. The frequency of failure is about the same for all these configuration options except for diffie-hellman-group14-sha256, which fails much more frequently - it sometimes takes hundreds
of attempts to succeed. Perhaps that will provide a clue to the underlying cause.<br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Once an SSH login succeeds the connection is stable. However if I initiate a manual rekey operation via ~R then the key re-exchange fails. The router is otherwise very stable with
no noticeable issues.</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I'm an embedded firmware engineer but have never worked on DD-WRT/OpenWrt firmware or dropbear. I have a conceptual understanding of the key exchange algo but haven't looked at the
actual code of any implementation including Dropbear's. I'm seek ideas on how to troubleshoot this issue. Considering the problem is intermittent I'm thinking it's some variant in the key generation/exchange algorithm that's failing due to some issue with
the router, or a more remote possibility, an issue with the Dropbear implementation.<br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Here are pastebin links to the PuTTY full debug logs (w/raw data dumps) for both the failure and success cases:</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Failure Case:<span class="x_x_x_x_Apple-converted-space"> </span><a href="https://pastebin.com/MS2BtFmW" target="_blank" rel="noopener noreferrer" class="">https://pastebin.com/MS2BtFmW</a><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Success Case:<span class="x_x_x_x_Apple-converted-space"> </span><a href="https://pastebin.com/c4j66Ga9" target="_blank" rel="noopener noreferrer" class="">https://pastebin.com/c4j66Ga9</a><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The only message I see from dropbear for a failed connection attempt is:</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><span class=""><a href="http://authpriv.info/" class="">authpriv.info</a><span class="x_x_Apple-converted-space"> </span>dropbear[15948]: Child connection from 192.168.1.249:54819<br class="">
</span><span class=""><a href="http://authpriv.info/" class="">authpriv.info</a><span class="x_x_Apple-converted-space"> </span>dropbear[15948]: Exit before auth: Disconnect received</span><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Thanks!</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</div>
</body>
</html>