Arch Linux

minimouse

Member

Registered: 2013-03-14

Posts: 9

[ABANDONED] Arch install from Debian rescue boot

I've been using an arch linux server for some time without any problems. But recently it rebooted for some reason, and seems to hang during startup. The server is hosted on OVH (Kimsufi) and there are good tools for remote reboot, and net boot for problem solving. So, I can login to the server in "rescue" net boot mode, mount the file systems, and look around to see what's wrong. In /var/log/boot I see entries like this:

Thu Mar 14 03:12:23 2013: :: Adjusting system time and setting kernel time zone    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Starting udev daemon    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Triggering udev uevents    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Loading user-specified modules    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Waiting for udev uevents to be processed    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Configuring virtual consoles    [BUSY] %@   [DONE]
Thu Mar 14 03:12:23 2013: :: Bringing up loopback interface    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Checking filesystems    [BUSY] /: clean, 40317/655360 files, 421706/2621440 blocks
Thu Mar 14 03:12:23 2013: /home: clean, 7673/60366848 files, 178937740/241436672 blocks
Thu Mar 14 03:12:23 2013:    [DONE]
Thu Mar 14 03:12:23 2013: :: Remounting root and API filesystems    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Mounting local filesystems    [BUSY] mount: sysfs is already mounted or /sys busy
Thu Mar 14 03:12:23 2013:    [FAIL]
Thu Mar 14 03:12:23 2013: :: Activating swap    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Initializing random seed    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Removing leftover files    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: :: Saving dmesg log    [BUSY]    [DONE]
Thu Mar 14 03:12:23 2013: INIT: Entering runlevel: 3

It seems to look fine except that there should be more following the last line. The previous successful boot was about 11 months ago and looked like this:

Fri Apr 27 17:14:11 2012: :: Adjusting system time and setting kernel timezone    [BUSY] libkmod: index_mm_open: open(/lib/modules/3.2.13-grsec-xxxx-grs-ipv6-64/modules.de
Fri Apr 27 17:14:11 2012: libkmod: kmod_search_moddep: could not open moddep file '/lib/modules/3.2.13-grsec-xxxx-grs-ipv6-64/modules.dep.bin'
Fri Apr 27 17:14:11 2012: libkmod: kmod_search_moddep: could not open moddep file '/lib/modules/3.2.13-grsec-xxxx-grs-ipv6-64/modules.dep.bin'
Fri Apr 27 17:14:11 2012: libkmod: kmod_search_moddep: could not open moddep file '/lib/modules/3.2.13-grsec-xxxx-grs-ipv6-64/modules.dep.bin'
Fri Apr 27 17:14:11 2012:    [DONE]
Fri Apr 27 17:14:11 2012: :: Starting UDev Daemon    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Triggering UDev uevents udevd[1625]: open(/lib/modules/3.2.13-grsec-xxxx-grs-ipv6-64/modules.dep.bin, O_RDONLY|O_CLOEXEC): No such file or directory
Fri Apr 27 17:14:11 2012:    [BUSY]
Fri Apr 27 17:14:11 2012:    [DONE]
Fri Apr 27 17:14:11 2012: :: Waiting for UDev uevents to be processed    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Bringing up loopback interface    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Checking Filesystems    [BUSY] /: clean, 25767/655360 files, 164076/2621440 blocks
Fri Apr 27 17:14:11 2012: /home: clean, 11/60366848 files, 1953451/241436672 blocks
Fri Apr 27 17:14:11 2012:    [DONE]
Fri Apr 27 17:14:11 2012: :: Remounting Root    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Mounting Local Filesystems    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Activating Swap    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Configuring Time Zone    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Initializing Random Seed    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Removing Leftover Files    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Setting Hostname: ks3097942.kimsufi.com    [BUSY]    [DONE]
Fri Apr 27 17:14:11 2012: :: Setting Consoles to UTF-8 mode    [BUSY]    [DONE]
Fri Apr 27 17:14:12 2012: :: Loading Keyboard Map: en    [BUSY]    [FAIL]
Fri Apr 27 17:14:12 2012: :: Saving dmesg Log    [BUSY]    [DONE]
Fri Apr 27 17:14:12 2012: INIT: Entering runlevel: 3
Fri Apr 27 17:14:12 2012: error: "kernel.sysrq" is an unknown key
Fri Apr 27 17:14:12 2012: :: Starting Syslog-NG    [BUSY]    [DONE]
Fri Apr 27 17:14:12 2012: :: Starting Network    [BUSY]    [DONE]
Fri Apr 27 17:14:12 2012: :: Starting crond daemon    [BUSY]    [DONE]
Fri Apr 27 17:14:12 2012: :: Starting BIND    [BUSY]    [DONE]
Fri Apr 27 17:14:13 2012: :: Starting Secure Shell Daemon    [BUSY]    [DONE]

I'm a bit clueless on how to proceed, any suggestions?

Last edited by minimouse (2013-03-22 03:05:02)

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 29,520

Website

Re: [ABANDONED] Arch install from Debian rescue boot

It looks like it hasn't been updated in what, 6 months, a year, more?

Upgrade, and transition to systemd.  Initscripts are no longer supported.

---

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

I'm not sure how to do that since it won't boot. I tried the following from the rescue boot:

mount /dev/sda1 /mnt
chroot /mnt
pacman -Syu

Some of it seems to work, but not entirely:

:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
:: The following packages should be upgraded first :
    pacman
:: Do you want to cancel the current operation
:: and upgrade these packages now? [Y/n]

resolving dependencies...
looking for inter-conflicts...

Targets (2): libarchive-3.1.2-1  pacman-4.0.3-7

Total Installed Size:   6.73 MiB
Net Upgrade Size:       0.27 MiB

Proceed with installation? [Y/n]
(2/2) checking package integrity                             [################################] 100%
error: GPGME error: Bad file descriptor
error: GPGME error: Bad file descriptor
error: libarchive: missing required signature
error: pacman: missing required signature
error: failed to commit transaction (invalid or corrupted package (PGP signature))
Errors occurred, no packages were upgraded.

If I say No to upgrade pacman first, it still fails with a lot of errors like this:

...
error: xfsprogs: missing required signature
error: xmlrpc-c: missing required signature
error: failed to commit transaction (invalid or corrupted package (PGP signature))
Errors occurred, no packages were upgraded.

65kid

Member

From: Germany

Registered: 2011-01-26

Posts: 663

Re: [ABANDONED] Arch install from Debian rescue boot

The previous successful boot was about 11 months ago

wow, that must be some kind of record. To be honest, you have missed so many big changes in this time (package signatures, systemd move, /usr/lib move etc... etc ...) that a complete reinstall is probably easier and faster.

If you use Arch, you are expected to keep up with the news and update your system accordingly. If you want a system that runs for month or even years without requiring a reboot, Arch is probably not the best choice.

teateawhy

Member

From: GER

Registered: 2012-03-05

Posts: 1,138

Website

Re: [ABANDONED] Arch install from Debian rescue boot

What you did is explicitly stated as wrong in the archwiki:

Bad Practices:
...
- Updating once a year.

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

Well actually I did update more often than once a year, though clearly not often enough... and didn't reboot after last updates. Lesson learned.

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 29,520

Website

Re: [ABANDONED] Arch install from Debian rescue boot

How often had you been updating?  It is possible that you have stayed current with most updates, but still had never yet booted with systemd to work out any kinks in the transition.

There is (was?) a wiki page specific to going through the systemd transition step-by-step; that'd be worth a look.

---

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

After reviewing systemd transition etc. I agree that complete reinstall is probably faster and easier. However I'm not sure how to go about it. OVH offers automated reinstallation but it reformats the entire disk, and it would be much preferable to preserve the /home partition. The arch download page basically says "use pacman or boot from ISO". Is there no way to just copy a fresh install onto the root partition?

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

I found this article Install From Existing Linux which seems to be just the ticket.

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

Ok, all reinstalled and it seems to boot ok. But there is no response to ping or ssh so I've obviously done something wrong. This is the output of journalctl:

Mar 14 22:23:43 ks3097942.kimsufi.com systemd-journal[1745]: Allowing runtime journal files to grow
Mar 14 22:23:43 ks3097942.kimsufi.com systemd-journal[1745]: Journal started
Mar 14 22:23:43 ks3097942.kimsufi.com systemd-fsck[1586]: /: clean, 32374/655360 files, 259621/26214
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Started File System Check on Root Device.
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Starting Remount Root and Kernel File Systems...
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Started Remount Root and Kernel File Systems.
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Starting Local File Systems (Pre).
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Reached target Local File Systems (Pre).
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Starting Load Random Seed...
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Started Load Random Seed.
Mar 14 22:23:43 ks3097942.kimsufi.com systemd[1]: Started udev Coldplug all Devices.
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Found device ST31000524AS.
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Starting File System Check on /dev/sda2...
Mar 14 22:23:44 ks3097942.kimsufi.com systemd-fsck[2656]: /home: clean, 47980/60366848 files, 179300
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Started File System Check on /dev/sda2.
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Mounting /home...
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Mounted /home.
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Starting Local File Systems.
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Reached target Local File Systems.
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Starting Trigger Flushing of Journal to Persistent
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Starting Recreate Volatile Files and Directories..
Mar 14 22:23:44 ks3097942.kimsufi.com systemd-journal[1745]: Allowing system journal files to grow t
Mar 14 22:23:44 ks3097942.kimsufi.com systemd[1]: Started Trigger Flushing of Journal to Persistent
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started Recreate Volatile Files and Directories.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting System Initialization.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Reached target System Initialization.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Daily Cleanup of Temporary Directories.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started Daily Cleanup of Temporary Directories.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting D-Bus System Message Bus Socket.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Listening on D-Bus System Message Bus Socket.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Sockets.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Reached target Sockets.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Basic System.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Reached target Basic System.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Permit User Sessions...
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting D-Bus System Message Bus...
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started D-Bus System Message Bus.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Login Service...
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started Permit User Sessions.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Getty on tty1...
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started Getty on tty1.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Login Prompts.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Reached target Login Prompts.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started Login Service.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Multi-User.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Reached target Multi-User.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Graphical Interface.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Reached target Graphical Interface.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Starting Update UTMP about System Runlevel Changes
Mar 14 22:23:45 ks3097942.kimsufi.com systemd-logind[2671]: New seat seat0.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Started Update UTMP about System Runlevel Changes.
Mar 14 22:23:45 ks3097942.kimsufi.com systemd[1]: Startup finished in 6s 71ms 941us (kernel) + 2s 43

I still have all files from the previous installation stashed in a backup folder for reference. There was no /etc/hostname - just an entry in /etc/hosts. But I read that it's not necessary to edit /etc/hosts anymore now that systemd is used. I tried using hostnamectl from rescue mode (with mounts and chroot as for pacman) but it gives this error:

Failed to get D-Bus connection: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

So I created /etc/hostname. dhcpcd seems to do the right thing, obtaining lease for the correct IP address. What am I missing?

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

I've read a ton of wiki pages, and I'm still clueless. Suggestions anyone? It seems so close to working properly...

Is dmesg output written to disk anywhere? Running dmesg from rescue/chroot just shows information from the rescue boot. Is journalctl output (quoted above) the Arch equivalent, or is there more information elsewhere? Is there a way to make the output more verbose in hopes of spotting a problem?

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

I managed to get grub bootstrap from hard disk working. Of course that didn't change the lack of response to ping/ssh after boot. I still have to reboot to rescue mode in order to look at the log. After consulting this wiki page I tried adding verbose, and then debug to the menu.lst as suggested. But it didn't seem to affect the messages displayed by journalctl - is that page out of date? There don't seem to be any messages about networking in the journalctl output (quoted above).  I'm at a disadvantage because I don't have any working arch system to compare with. Are there missing network messages, and if so what would they look like? Why would it fail to start network, yet not display any message about it? Is there some way to improve the boot diagnostic output?

DSpider

Member

From: Romania

Registered: 2009-08-23

Posts: 2,273

Re: [ABANDONED] Arch install from Debian rescue boot

I managed to get grub bootstrap from hard disk working. Of course that didn't change the lack of response to ping/ssh after boot. I still have to reboot to rescue mode in order to look at the log. After consulting this wiki page I tried adding verbose, and then debug to the menu.lst as suggested. But it didn't seem to affect the messages displayed by journalctl - is that page out of date? There don't seem to be any messages about networking in the journalctl output (quoted above).  I'm at a disadvantage because I don't have any working arch system to compare with. Are there missing network messages, and if so what would they look like? Why would it fail to start network, yet not display any message about it? Is there some way to improve the boot diagnostic output?

What menu.lst? I have marked that wiki page as out of date.

Instead of adding "verbose", remove the "quiet" parameter from /etc/default/grub and re-generate a new grub.cfg (recommended).
https://wiki.archlinux.org/index.php/Gr … mmended.29

This is how you chroot:
https://wiki.archlinux.org/index.php/Chroot

And this is how you check the network connection (read the entire thing, from the top, twice):
https://wiki.archlinux.org/index.php/Ne … figuration

---

"How to Succeed with Linux"

I have made a personal commitment not to reply in topics that start with a lowercase letter. Proper grammar and punctuation is a sign of respect, and if you do not show any, you will NOT receive any help (at least not from me).

minimouse

Member

Registered: 2013-03-14

Posts: 9

Re: [ABANDONED] Arch install from Debian rescue boot

What menu.lst? I have marked that wiki page as out of date.

Instead of adding "verbose", remove the "quiet" parameter from /etc/default/grub and re-generate a new grub.cfg (recommended).
https://wiki.archlinux.org/index.php/Gr … mmended.29

This is how you chroot:
https://wiki.archlinux.org/index.php/Chroot

And this is how you check the network connection (read the entire thing, from the top, twice):
https://wiki.archlinux.org/index.php/Ne … figuration

Thanks for your reply. menu.lst was present in boot folder from previous version. I tried removing the quiet parameter but it did not seem to change the journalctl output.

chroot is working well, it's how I installed arch to the root partition, how I run journalctl, etc.

I have read the network configuration page several times but most of it is not helpful for a headless server with only ssh available to reach it. Of course when network is not functioning I cannot ssh so commands mentioned in the page such as hostnamectl, dmesg, systemctl are not useful.

Anyway I eventually gave up and wiped the server, using OVH tools to reinstall from scratch.