[Solved] Keeping remote commands running after terminating ssh session

keiichiiownsu12 · 2019-04-21 22:25:19

I am currently trying to run a remote command on a lab server at my university (we will call the lab server Machine C) from a laptop or desktop from home (we will call this Machine A). Because my university uses the eduroam network, I cannot ssh into Machine C directly unless I am on the local network (i.e. no way of configuring port forwarding on the router. Also, I don't want to request VPN access from my school as I hear it is a pain to get permission from IT anyway). To get around this, I am currently SSH'ing via tor into a secondary computer at my lab, an old laptop that I set up (we will call this Machine B), and am using to act as a bridge between Machine A and C.
I.e.
Machine A --tor/ssh --> Machine B --LAN/ssh--> Machine C.

This works fine for the most part, however I have a program that I need to run on Machine C that will take more than 24 hours to complete. I am worried that the command will potentially exit if the ssh connection between Machine A and B is broken, and thus might potentially corrupt data on Machine C.

So, I am wondering if it is possible to run a command remotely on Machine C from Machine B, terminate the SSH connection between Machine A and Machine B, but have the program continue running in the background on Machine C. I have read here that I can potentially run the command in a tmux session, detach the tmux instance (i.e. using Ctrl + b + d), and then safely terminate the SSH session between Machine A and B.

The only caveat I think of is I do not have root access on Machine C, so I cannot install tmux on it. Therefore, I would have to 1) SSH into Machine B, 2) start a tmux session, 3) SSH from Machine B to Machine C, 4) run the needed program on Machine C, 5) detach the tmux session from Machine B, then 6) terminate the ssh session between Machine A and B.

My only question is: does this sound like it can work? I set up Machine B to always be on, however I am worried of something occurring that might accidentally terminate the ssh session from Machine B and C prematurely, thus causing the program to exit before it finishes. Would this just be something I have to contend with? Alternatively, could I just "copy" the tmux binary from either Machine A or B via scp to Machine C, then run a tmux session on Machine C that way?

I created a test program to run on Machine B via a tmux session., I ssh'ed via Machine A to B, started tmux, ran the program, detached tmux, exited the ssh session, SSH'ed again from A to B, re-attached the tmux session, and found the test program still running. However, despite said success, I am still reluctant to try out the above stated plan.

Last edited by keiichiiownsu12 (2019-04-22 05:59:53)

rowdog · 2019-04-21 23:44:15

man nohup

https://wiki.archlinux.org/index.php/Al … ter_logoff

Trilby · 2019-04-22 02:22:49

Are you sure that machine C doesn't have screen or tmux? If it uses one of the Grid Engine systems, you may need to load the 'dot-kit' for it, but at least one of screen or tmux would likely be available on machine C.

But even if that's not the case, you do not need root access to install tmux locally for your user. Then you can set up your shell profile to automatically start/reattach tmux on log in - that way there is no need to specifically detach: just end your ssh session and tmux will keep running, next time you log in, you'll pick up where you left off.

Last edited by Trilby (2019-04-22 02:24:02)

keiichiiownsu12 · 2019-04-22 03:13:44

It appears that machine C does have screen enabled (not too surprised, even if it is MAC/OSx). I just SSH'ed into machine C (i.e. A to B to C), invoked the screen command to verify it was there and exited. However I also killed the process labeled "screen" as well before exiting. Just to make sure, this didn't kill any other processes did it? (I.e. other lab members have shell clients open running different data processing pipelines)

Also, to clarify: when logging in to Machine C, I log in as a shared user (i.e. everyone in the lab logs in with the same credentials. Only admins have access to root account). Even so, would I still be able to set up said shell profile in the way you described, without affecting other individuals logging in to the shared user for Machine C? If I wanted to install tmux, could I install the binary to a personal directory then run it (I presume via make install)? Or could I precompile the binary on another machine, transfer it via scp to a personal directory on Machine C, then run it? (If it's easier, I'll just try and use screen instead)

Trilby · 2019-04-22 03:53:04

keiichiiownsu12 wrote:

Just to make sure, this didn't kill any other processes did it?

Normally I'd say no, this wouldn't be possible on any sane system - you'd not have access to kill other user's processes. But ...

kelichilownsu12 wrote:

Also, to clarify: when logging in to Machine C, I log in as a shared user (i.e. everyone in the lab logs in with the same credentials.

That's absolutely horrific. So even if you get screen running with your process, someone else could stop it at any time. Are you really sure this is how it works?

keiichiiownsu12 wrote:

Even so, would I still be able to set up said shell profile in the way you described, without affecting other individuals logging in to the shared user for Machine C?

Not if what you say is true. If there is a single shared user, there is a single shell config ... which is really really horrible. Can no one add anything to the shellrc, or can anyone add anything to everyone's shell rc? I'm not sure which would be worse.

keiichiiownsu12 wrote:

If I wanted to install tmux, could I install the binary to a personal directory then run it (I presume via make install)?

Yes to all of that, except the `make install`. You would not run `make install`, you'd just build it with `make`, then run it from there or move it to your own folder. Normally I'd say you should then add that folder to your PATH variable, but if the above-mentioned-lunacy is correct, you probably can't/shouldn't modify the PATH variable.

keiichiiownsu12 wrote:

Or could I precompile the binary on another machine, transfer it via scp to a personal directory on Machine C, then run it? (If it's easier, I'll just try and use screen instead)

No, don't do this, it will be built against the wrong libs and will not run. But you said this is a Mac ... you may be S.O.L. on building tmux regardless. Mac's often don't even have a compiler available by default. Use screen. Personally I prefer tmux, but as you will not be able to set up a .tmux.conf anyways, you can't do any customizing anyways - so tmux or screen will get the job done equally well in their "vanilla" state.

I suppose under these circumstances (with a shared user, are you really really sure that's what it is), just using `nohup` would be simpler and a bit safer.

Last edited by Trilby (2019-04-22 03:56:31)

keiichiiownsu12 · 2019-04-22 04:43:31

To answer your question, yes, none of us have a unique login to the server -- technically the "server" is just a normal Mac Desktop Pro setup as a server. So yeah, it is pretty insecure (then again, me using tor to log in might also not be that secure, even if I am using the. . .whatever option in tor that requires

 HidAuthServ <address> .onion <cookie>

for an individual to access said hidden service)

I confirmed Machine C has both make and a gcc compiler present, so I may be able to build tmux from source. However, in order to not risk anything too much, I am going to attempt to do this locally tomorrow instead of via ssh, just to be sure no random thing happens without me physically being there.

Also, one more alternative: we do have a machine with an Ubuntu install running (i.e. lab server with Ubuntu installed). Good thing too, since this machine DOES NOT have shared accounts. I will be having an account setup later in the month, so I would have to test this when that happens. Worth the wait, or would it be better to proceed with the above plan -- i.e. precompile tmux on the Mac and see if it works?

(Also,just as a full disclosure. Reason why I am doing it this way and not through the official VPN service at my school is because, when I brought up the possibility to one of my PIs, they shied the possibility away due to both security and legal concerns. Other people at the lab said requesting VPN service through the school IT service is a pain anyway. There also isn't any serious need for me to be doing this atm -- i.e. not currently traveling or anything like that -- however the given flexibility is desired. I also would like a comment if tor is indeed secure enough for my purposes, or should I explore other, more secure alternatives?)

Last edited by keiichiiownsu12 (2019-04-22 04:52:34)

ngoonee · 2019-04-22 05:50:33

Your current ssh-into-B-using tor is likely already against the terms of use for your university. It's likely the IT administration is not competent enough to notice this though (I've run similar setups before at my current and previous institute), however this does not 'legalise' anything.

As an academic staff at a university, my advise would be to get things done properly. It doesn't even need to be a VPN, just requesting for a port forward for a high-numbered port so you can run SSH would be sufficient (then again, most universities would not entertain that request without justification). If you're not able to get things done properly, then avoid hacking your own solutions in the above manner. You risk your candidature and/or employment for a bit of convenience.

keiichiiownsu12 · 2019-04-22 05:59:29

ngoonee wrote:

As an academic staff at a university, my advise would be to get things done properly. It doesn't even need to be a VPN, just requesting for a port forward for a high-numbered port so you can run SSH would be sufficient (then again, most universities would not entertain that request without justification). If you're not able to get things done properly, then avoid hacking your own solutions in the above manner. You risk your candidature and/or employment for a bit of convenience.

That is kinda what I was thinking when I started reflecting on this earlier. I could potentially jeopardize other parts of the network, even if I am using "stealth" client authentication over tor (did test this earlier, and could not log in without a specific modification to my torrc file).

So yeah, I guess I will just shutdown the tor service anyway. If I need to run some sort of command, can always just head to the university, ask someone already there, or just wait. Like you are advising, it definitely does not seem like the end justifies the means here. And I certainly don't want my graduate candidacy revoked if something stupid occurs. . . (And yes, this is against uni policy. . .)

Anyway, for now I will mark the topic as "solved". If an admin or moderator reads this, please mark this thread for deletion or something like that if appropriate.

Last edited by keiichiiownsu12 (2019-04-22 06:00:43)

quinoa42 · 2019-04-22 06:23:24

I'd generally recommend just use tmux for this purpose.

Arch Linux

#1 2019-04-21 22:25:19

[Solved] Keeping remote commands running after terminating ssh session

#2 2019-04-21 23:44:15

Re: [Solved] Keeping remote commands running after terminating ssh session

#3 2019-04-22 02:22:49

Re: [Solved] Keeping remote commands running after terminating ssh session

#4 2019-04-22 03:13:44

Re: [Solved] Keeping remote commands running after terminating ssh session

#5 2019-04-22 03:53:04

Re: [Solved] Keeping remote commands running after terminating ssh session

#6 2019-04-22 04:43:31

Re: [Solved] Keeping remote commands running after terminating ssh session

#7 2019-04-22 05:50:33

Re: [Solved] Keeping remote commands running after terminating ssh session

#8 2019-04-22 05:59:29

Re: [Solved] Keeping remote commands running after terminating ssh session

#9 2019-04-22 06:23:24

Re: [Solved] Keeping remote commands running after terminating ssh session

Board footer