<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Binny,<div class=""><br class=""></div><div class="">I think regardless of what Dropbear's doing with pipes (closed sessions etc), there is probably something wrong with the Linux kernel.</div><div class="">As far as I know userspace can't trigger D state even intentionally (I'd be interested if anyone knows a way though).</div><div class="">-K is unrelated, that just sends some SSH traffic at a certain interval.</div><div class=""><br class=""></div><div class="">If you can reproduce it, "echo t > /proc/sysrq-trigger" might be helpful - you can look at "dmesg" for stack traces of all threads and see what the stuck processes are doing in the kernel.</div><div class="">Was there anything unusual in dmesg on the problematic machine?</div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Matt</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mon 14/10/2019, at 12:44 pm, Jeshan, Binny <<a href="mailto:Binny.Jeshan@netscout.com" class="">Binny.Jeshan@netscout.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Dear Matt,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Thank you for your response. Here is our situation said below...<o:p class=""></o:p></span></div><ol start="1" type="1" style="margin-bottom: 0in; margin-top: 0in;" class=""><li class="MsoListParagraph" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Gisha, sans-serif;" class="">This has happened to one or two of those users out of a thousand such devices that are deployed. We have never seen this reported since many years now. When the issue was reported, we could only take out the below logs from the user unit. User is in an end customer deployment.<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Gisha, sans-serif;" class="">We do not use NFS, therefore when we checked the disk stats of all processes, nothing was holding on to it, and all other disk r/w operations were working normal.<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: Gisha, sans-serif;" class="">We do not have strace in our target device to debug, nor the problem is reproducible to us easily. It happened only once, and to really debug the problem we have to simulate such condition in our lab.<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></li></ol><div style="margin: 0in 0in 0.0001pt 0.25in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">As of now after reading some user guides, I wonder if the below precaution should be added. But I am not sure whether it will help when such situations where Open pipes exist in the process. I believe they are from common_session_init() and session_loop().<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Will my approach be right if I use this below flag -K in my situation of drop bear process? I have a feeling that the Open IPC pipes is also doing sort of an I/O operation that leads to this state of D. And the said user has a habit of not closing the SSH session properly with an exit or logout from the terminal, leaving it to idle-close or abrupt close the terminal.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><b class=""><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class="">-K <i class="">timeout_seconds</i></span></b><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Verdana, sans-serif; color: rgb(68, 68, 68);" class="">Ensure that traffic is transmitted at a certain interval in seconds. This is useful for working around firewalls or routers that drop connections after a certain period of inactivity. The trade-off is that a session may be closed if there is a temporary lapse of network connectivity. A setting if 0 disables keepalives.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Please advise your thoughts with the above. I am still working on recreating the problem by just forcing these pipes that are kept open to be there forever in that state. Any other suggestions may help.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Thanks for your help again,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Binny<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Matt Johnston <<a href="mailto:matt@ucc.asn.au" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">matt@ucc.asn.au</a>><span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, October 9, 2019 6:56 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Jeshan, Binny <<a href="mailto:Binny.Jeshan@netscout.com" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">Binny.Jeshan@netscout.com</a>>;<span class="Apple-converted-space"> </span><a href="mailto:dropbear@ucc.asn.au" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">dropbear@ucc.asn.au</a>;<span class="Apple-converted-space"> </span><a href="mailto:rwoodsmall@gmail.com" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">rwoodsmall@gmail.com</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: Dropbear processes getting into uninterruptible I/O process "D" state<o:p class=""></o:p></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width: 651px; background-color: rgb(252, 155, 134); background-position: initial initial; background-repeat: initial initial;"><tbody class=""><tr class=""><td style="padding: 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;" class=""><span style="" class="">This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.</span><o:p class=""></o:p></div></td></tr></tbody></table></div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">Hi Binny,<br class=""><br class="">I don't think it's related to 2019.78<br class=""><br class="">Usually D state means something else is wrong on the system, bad NFS mounts or IO devices. Can you strace the stuck processes?<br class=""><br class="">Cheers,<br class="">Matt<o:p class=""></o:p></p><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On 9 October 2019 10:53:56 am GMT+07:00, "Jeshan, Binny" <<a href="mailto:Binny.Jeshan@netscout.com" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">Binny.Jeshan@netscout.com</a>> wrote:<o:p class=""></o:p></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Dear all in the mailing list,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">We are seeing a problem with 2018.76 version of the dropbear SSH server which exhibits the following:<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">The processes go into uninterruptible “D” state and lies there, couldn’t be killed nor shutdown.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><img border="0" width="559" height="220" id="Picture_x0020_8" src="cid:image001.jpg@01D57E82.261498F0" style="width: 5.8263in; height: 2.2916in;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">The stuck processes in the bad state show the below behavior of two open pipes that are not closed.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><img border="0" width="601" height="657" id="Picture_x0020_3" src="cid:image002.jpg@01D57E82.261498F0" style="width: 6.2638in; height: 6.8402in;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">The last process with PID 394 seems to work fine, and has no open IPCs.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-family: Gisha, sans-serif;" class="">When I look at the release notes of 2019.78, I see this: “</span><span style="" class="">2019.78 - 27 March 2019<o:p class=""></o:p></span></pre><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 10pt; font-family: "Courier New";" class="">- Fix dbclient regression in 2019.77. After exiting the terminal would be left<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 10pt; font-family: "Courier New";" class=""> in a bad state. Reported by Ryan Woodsmall<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">“<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Is the problem that we see is same as you fixed? Please suggest. Your feedback and ideas will help.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Thanks<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Gisha, sans-serif;" class="">Binny</span></div></blockquote></div></div></div></blockquote></div><br class=""></div></body></html>