[Solved] Testing of sshd service on remote server

stri · 2020-04-28 15:50:56

I want to rent a bare metal server and avoid downtime in case an upgrade goes wrong. I'm looking for a robust way to test if the sshd service is running properly (so that I will be able to access the server remotely via ssh), and if not, automatically roll back to a previous kernel/initramfs/root filesystem.

I already figured out the latter (i.e. rollback) using three partitions on the server, and local testing in a VM before deployment, but the question is about the former:

Since I use systemd-boot, I want to use "boot counting" and

boot-complete.target

for testing sshd (and in case the test fails, reboot the previous kernel/initramfs/root filesystem).

https://www.freedesktop.org/software/sy … ete.target

So which services would I require for this boot-complete.target, to get a robust test for remote access via ssh? Testing that sshd is up and running is a no-brainer I suppose, but is that enough? Obviously, proper testing of ssh access can only be done trying to connect from outside, but I want to check as many necessary preconditions from the server itself. Are there any best practices? Thanks a lot for your help!

Last edited by stri (2020-04-30 16:07:43)

stri · 2020-04-28 16:00:51

Is there a way to test if the firewall is open for ssh from *inside* the firewall, except e.g. grepping the /etc/iptables/iptables.rules?

sabroad · 2020-04-28 16:19:53

stri wrote:

I want to rent a bare metal server and avoid downtime in case an upgrade goes wrong. I'm looking for a robust way to test if I can access the server via ssh, and if not, automatically roll back to a previous kernel/initramfs/root filesystem.

If you want "robust", I'd strongly suggest setting up the baremetal as a minimal hypervisor and hosting systems on top.

Check out LXD for container-based OS "virt", or libvirt for traditional QEMU virt. Both can snapshot, revert and show the boot process.

There's lots of unforeseeable that can go wrong in a boot, and unfixable remotely without out-of-band management (such as iDRAC or iLO).

stri · 2020-04-28 17:14:05

Thanks for the reply, but consider this:

I was planning on qemu/kvm, but I need to update my hypervisor once in a while, hence my question.

I only mentioned "VM" in the post above because I do all the configuration in a VM, but then deploy it on bare metal (I'm not sure how to test nested qemu/kvm, but that's ok because I can later do it on the server and reboot to the old system if the guest VM's don't spin up as they should.)

I can currently afford one single server without out-of-band management, as I found those pretty expensive, and I want it as robust and highly available as possible, given the means I have. And yes, I'm aware that if my disks fail on that single server or other such things happen, all my services fail.

The only thing I could think of is to launch a shell script that sleeps for 1 minute after rebooting into the new initram/kernel/root fs, and if there is no intervention from outside, then triggers a reboot into the old system. If I manage to login via ssh and kill the script before the 1-minute timeout, it doesn't reboot.

Trilby · 2020-04-28 17:16:44

Sorry if it sounds OT, but given the description of your needs and capacity, why are you getting a bare-metal server? That really sounds like the wrong tool for the job. A quality VPS will give you all the same control, but much more stability and assurance of uptime or repair if something goes horribly wrong.

stri · 2020-04-28 17:43:30

Probably true, but I find the computing very expensive on the VPS!

Trilby · 2020-04-28 18:02:36

Really? What kind of specs are you looking for? Generally a VPS could be cheaper.

stri · 2020-04-28 20:59:35

I'm open for suggestions...I only ever checked the largest VPS provider, but it is so terribly expensive that I didn't even bother looking at the others, because I thought they simply can't be that much cheaper if they want to make money.

What I need is as much computing power as possible, for a given price. Memory and persistent storage don't matter much, networking is only moderately important.

Example large cloud provider (on-demand instance):
8 vCPU, 16 GiB Memory: 0.34 per hour. That's 252.96 per month.

Example rootserver provider:
AMD Ryzen 7 3700X Octa-Core 63GB Memory: less than 80 Euro per month, however: One month setup fee!

I have not tested the speed, but the Ryzen has a base frequency of 3.6 Ghz. That's not bad.

I'm not sure what a vCPU is nowadays, but it used to be a hyperthread, and I doubt it's much better than a core of the Ryzen.

I never understood the business model with the VPS. I think I do understand their newer business model with the serverless stuff. I host my home page on S3 with their CDN in front. It's very slow if it's not in the CDN cache (nobody ever looks at my home page, so it usually isn't cached:), but it's a reasonable price and extremely robust. I also use a "serverless" MySQL-like database there (which almost never gets used), and I'm also happy with it. However, when it comes to servers, I think it's very expensive if you use it for more than a couple of weeks.

Last edited by stri (2020-04-28 21:00:13)

loqs · 2020-04-28 21:26:00

VPS services I am familiar with normally bill monthly and you have a reduced price compared to an on demand hourly billing rate I associate with cloud services.
Cloud services do offer discounts for sustained use and prepayment.

Trilby · 2020-04-28 21:59:11

A VPS through Linode for 8 cores is $160/mo (0.24/hr) and with that you get 32GB ram. I think their prices are good but certainly not the cheapest (I might subjectively rate them as best value though, their customer service is awesome). OVH was a bit cheaper last time I checked, but not drastically.

EDIT: OVH's website is showing 8 cores for $31.28/mo in their Elite plan. I'm not sure what the catch is as that does sound too cheap to be true.

Just OT curiosity, what kind of work is the machine going to be doing such that you need CPU power but have little to no concern with RAM? I've worked a bit in informatics, and while more cores are better, RAM is often a bigger concern.

Last edited by Trilby (2020-04-28 22:03:26)

stri · 2020-04-28 22:48:00

Ok thanks for the suggestions, maybe I was too fixated on the one cloud company. Meanwhile I saw that most of the bare metal providers offer VPS too.

I use it for computational fluid dynamics simulations, I need (far) less memory than e.g. a typical 1 core to 2 gb ratio. I will need to send data in between the simulations. Not too much, but maybe more than a typical residential home connection can handle.

I also need to be able to update the server during simulations which altogether will take on the order of a couple of months. That's a limit I can set according to my financial constraints, by configuring the application. The faster the server, the more it can do during that time.

Ideally I would run the applications in VMs. It would be much more convenient and portable. But I have to check the overhead. I would accept some overhead.

Edit: By the way, I don't want to go off topic, but in case I do end up going with the bare metal server, is there a way to test if the firewall is open for ssh from *inside* the firewall, except e.g. grepping the /etc/iptables/iptables.rules?

Last edited by stri (2020-04-28 22:51:06)

sabroad · 2020-04-29 10:14:37

stri wrote:

So which services would I require for this boot-complete.target, to get a robust test for remote access via ssh?

I'd try some sort of dead man's switch: if it can be presumed the machine is restarting attended, then require the boot to check for the existence of a file which you'd have to place by ssh (within 15m). That way, pretty much guaranteed ssh access (including issues such as key management).

deadmanswitch.service
edit: this is ass-backwards and wrong but hopefully you get the gist.

[Unit]
ConditionPathExists=!/tmp/deadmanswitch

[Service]
Type=oneshot
ExecStart=/usr/lib/systemd/system-bless-boot bad

deadmanswitch.timer

[Timer]
OnBootSec=15m

If you're confident on this setup, you can optionally get the service to reboot after marking bad (or reboot out-of-band).

That said though- if you're going minimal hypervisor I'd just treat it as immutable: if it needs an update then migrate VMs (eg. at end of 30-day contracts).

Last edited by sabroad (2020-04-29 10:16:55)

stri · 2020-04-30 15:58:34

Yes thanks, I do get gist!

Wow, "dead man's switch", never heard that expression before, but it makes a lot of sense!

Arch Linux

#1 2020-04-28 15:50:56

[Solved] Testing of sshd service on remote server

#2 2020-04-28 16:00:51

Re: [Solved] Testing of sshd service on remote server

#3 2020-04-28 16:19:53

Re: [Solved] Testing of sshd service on remote server

#4 2020-04-28 17:14:05

Re: [Solved] Testing of sshd service on remote server

#5 2020-04-28 17:16:44

Re: [Solved] Testing of sshd service on remote server

#6 2020-04-28 17:43:30

Re: [Solved] Testing of sshd service on remote server

#7 2020-04-28 18:02:36

Re: [Solved] Testing of sshd service on remote server

#8 2020-04-28 20:59:35

Re: [Solved] Testing of sshd service on remote server

#9 2020-04-28 21:26:00

Re: [Solved] Testing of sshd service on remote server

#10 2020-04-28 21:59:11

Re: [Solved] Testing of sshd service on remote server

#11 2020-04-28 22:48:00

Re: [Solved] Testing of sshd service on remote server

#12 2020-04-29 10:14:37

Re: [Solved] Testing of sshd service on remote server

#13 2020-04-30 15:58:34

Re: [Solved] Testing of sshd service on remote server

Board footer