[solved]ssh freezes during intermittent output (or read)

whoops · 2016-12-30 10:41:27

Hi!

I've had this problem for a long time. Client was always arch and I hoped the problem would go away when I migrated the server to arch. It didn't. I don't know how to debug this - partially because it seems to be the opposite of the problem "everyone else" is having with ssh (mine does work great both when it's idle AND when something dumps a lot of text very fast).

ssh session with those freezes:
- journalctl -f usually freezes after less than a minute.
- pacman often freezes when it should be asking for confirmation
- had the same problem, just MUCH worse, with apt-get before I migrated (only way to make it run was basically "yes | apt-get & disown", then pray)
- piping to "sort" or anything that collects / delays output will often freeze
- rsync will run for a loooong time sometimes before it freezes (probably when it doesn't find anything to talk about for too long)

ssh session running those usually works fine:
- journalctl pager or no-pager (as long as it isn't -f) will usually work without problems, even if I dump a whole weeks worth of log spam at once.
- cat / compiling / find / anything that generated a LOT of output at once + without pauses usually runs without problems
- nano / vim - can leave them open for the whole day, no problem.
- just the shell: same thing. No input / output required, doesn't ever time out or freeze.
- sudo - irritatingly - has always worked fine, even if I don't respond to it immediately

Very rarely, the ssh session recovers a few minutes after the command is finished or starts generating output but usually not and I can see from a second ssh that sometimes only seconds after I ran the freezing command, its process dies for mysterious reasons (shell that owns it closed?). If I hit ctrl+c at least a 10 times soon enough after the freeze starts, chances that it recovers after 30 seconds or so seem greatly enhanced.

And that's what I did so far:
- I tried messing with the "*Alive*" settings on both server and client but only managed to make it much worse (I get timeouts in addition to the freezes if I mess with the servers settings... client settings only seems to change how long the freezes last before the ssh client closes).
- Checked + tested my MTU for some reason. I'm not sure why - Google made me do it but didn't tell me what to do with the result oft those tests. *shrug* my mtu... looks... probably fine I guess?
- Tried mosh, which works "great". Except for almost all the things that aren't working with ssh anyway, I really need client-side scroll-back and I didn't even manage to get server-side scroll-back working with mosh - it might even be impossible with the current versions of mosh/tmux/etc.
- Also tried randomly turning various optional ssh & sshd settings on and off with no (or at least no positive) effect.

Any ideas how to get a more reliable ssh?

Last edited by whoops (2017-01-25 23:29:23)

paulkerry · 2016-12-31 23:10:22

ssh is normally totally reliable with out of the box settings.

I'm thinking on the lines of you having networking issues (dns, network driver, wired/wireless settings, bad nic) or a firewall setting could be causing this?

whoops · 2017-01-07 13:44:44

I haven't been able to find any obvious "networking issues"... since I don't know what exactly to look for, that's not saying much though (as I wrote above, I messed around with ping / MTU etc a bit... which looked ok'ish, as far as I can see... but yes, that's not particularity "far").

So... my next step would be "wait until I get a new ISP in a few weeks, hope that the problem magically fixes itself"... unless there's something else I should try in the meantime?
Or is there something I should test / log / record (just in case the problem really disappears, so there's something documented for "posteriority")?

ewaller · 2017-01-07 17:06:24

I don't have the same issues as do you. My issues are the timing out and having the connection shut down. As a workaround, what happens if you run a tmux session inside of your ssh session?

whoops · 2017-01-08 09:05:18

With tmux, "journalctl -f" appear to work for a while longer, but then ssh times out (and breaks the terminal / mouse button creates random binary strings).
Had an ssh session without tmux idle at the same time and that one's still working fine.
And another one with just "journalctl -f", which froze before even one new line came through.

edit: I guess it really was my ISP somehow. Wasn't able to figure out how / what exactly the problem was. After switching ISP (and changing nothing else) it's working flawlessly now.

Last edited by whoops (2017-01-25 23:29:02)

Arch Linux

#1 2016-12-30 10:41:27

[solved]ssh freezes during intermittent output (or read)

#2 2016-12-31 23:10:22

Re: [solved]ssh freezes during intermittent output (or read)

#3 2017-01-07 13:44:44

Re: [solved]ssh freezes during intermittent output (or read)

#4 2017-01-07 17:06:24

Re: [solved]ssh freezes during intermittent output (or read)

#5 2017-01-08 09:05:18

Re: [solved]ssh freezes during intermittent output (or read)

Board footer