Emergency log gathering without ssh access

roobre · 2014-03-11 18:40:20

Hi there,

First of all, sorry about the undescriptive title, but I did not know how to really put this situation in short.

Let's describe the situation first. Please, read this carrefully as it's crucial for understanding the problem:
I have a remote machine, lets call it A, only accesible via SSH. It can also be rebooted, even if SSH access has been lost.
This machine can also be booted into "rescue mode". This rescue mode is basically another machine, let's call it B, which can acces A's HDD. This way, if something breaks on A, you can shut it down, power on B, revert the configuration changes, power off B and boot A (A and B can't be powered on at the same time).

So, today I updated my arch box (A, B is never touched), and a systemd update alert showed up, announcing that file /etc/udev/rules.d/80-net-name-slot.rules, which controls static interface naming, will now be located at /etc/udev/rules.d/80-net-setup-link.rules, and that a reboot was recommended for systemd maintenance and blah, blah. So I rebooted. And A stopped responding to ping.

I suspect the problem is related to the network configuration: Interface names changed, so the machine does not connect to the net anymore.

In order to confirm this, I'd like to issue some debugging commands, like ip addr, ip link and dmesg.

Any idea about how I could approach this?

Best regards, and thanks in advance ^^.

WonderWoofy · 2014-03-11 19:25:49

It sounds as though you have absolutely no connection to the machine from either A or B. So unless you can connect somehow, I don't see how this can be accomplished.

I assume that you have physical access to the machine if you are able to reboot it w/o connectivity. If that is the case, then you might try just grabbing a USB keyboard, hooking it up and blindly trying to either run dhcpcd or start/enable the dhcpcd.service. Either of those things will simply run dhcpcd on all available network interfaces, hopefully getting you an IP address that you can hook up to. Of course, this assumes you have a dhcp service running within your LAN.

roobre · 2014-03-11 19:30:44

I have a remote machine, lets call it A, only accesible via SSH.

Nope, I can only power it on and off, and access to the machine's storage when its off.

Last edited by roobre (2014-03-11 19:31:15)

jasonwryan · 2014-03-11 19:46:26

Write a simple script that grabs the output of the commands you want and writes them to a file. Write a service file that runs the script after the network is up. Write both to A from B then power A up and wait for the scripts to work. Then read the file from B when they have completed.

Ugly, but all I can think of.

roobre · 2014-03-11 19:50:54

I'll give it a try.
I'll need to enable that systemd unit symlinking by hand tho, so I'm not sure if it will work. Stay tuned

Last edited by roobre (2014-03-11 19:51:21)

roobre · 2014-03-12 11:05:43

Ok, so here is the script I wrote:

[root@B (chrooted A) ~]# cat /root/debug.sh 
#!/bin/sh

/usr/bin/ip link > /root/ip_link
/usr/bin/ip addr > /root/ip_addr

/usr/bin/dmesg > /root/dmesg

And the unit file I used:

[root@B (chrooted A) ~]# cat /etc/systemd/system/debug.service 
[Unit]
Description=Debug log

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/root/debug.sh 2> /root/debug.sh.log
ExecStop=/root/debug.sh 2> /root/debug.sh.log
KillMode=none

[Install]
WantedBy=multi-user.target

Note: I have also tried by adding a dependency on Network.target

I symlinked the file to the multi-user.target.wants folder:

[root@B (chrooted A) ~]# ls -l /etc/systemd/system/multi-user.target.wants/
lrwxrwxrwx 1 root root 33 Mar 12 11:28 debug.service -> /etc/systemd/system/debug.service

But my /root folder is untouched, and the files the script should write the output to (/root/{ip_link,ip_addr,dmesg,debug.sh.log}) are never created.
Given this fact, I suspect it's not a network failure but something bigger... However, if you think those files are not being created by any reason other than the machine is not actually booting, please tell me.

EDIT: I discovered I could reach the journal from the chrooted environment (why the hell did'nt I tried it first?), so I'm currently investigating it. Thanks for help guys

Best regards, and thanks for helping ^^

Last edited by roobre (2014-03-12 11:28:29)

WonderWoofy · 2014-03-12 15:15:19

The ExecStart line is not a shell. If you want to use redirects, either put them in the script itself or wrap the command in a shell.

In the script seems like it would be my preference:

 /usr/bin/ip link 2> /root/debug.sh.log 1>/root/ip_link

But in the service file, you could also do:

 ExecStart=/bin/sh -c "/root/debug.sh 2> /root/debug.sh.log"

Last edited by WonderWoofy (2014-03-12 15:17:47)

Arch Linux

#1 2014-03-11 18:40:20

Emergency log gathering without ssh access

#2 2014-03-11 19:25:49

Re: Emergency log gathering without ssh access

#3 2014-03-11 19:30:44

Re: Emergency log gathering without ssh access

#4 2014-03-11 19:46:26

Re: Emergency log gathering without ssh access

#5 2014-03-11 19:50:54

Re: Emergency log gathering without ssh access

#6 2014-03-12 11:05:43

Re: Emergency log gathering without ssh access

#7 2014-03-12 15:15:19

Re: Emergency log gathering without ssh access

Board footer