Arch Linux

hede0n

Member

From: UK

Registered: 2015-01-19

Posts: 10

[SOLVED] -user error- Cannot login to system restored via rsync

Hi

I need a little help, obviously I am doing something wrong here.

I have made system backup of Arch partiotion as described in Arch wiki here to different ext4 partition. Then I have booted to Manjaro on same PC, have wiped Arch partition, and then restored backed up files to wiped partition using rsync.
I have also modified fstab on restored system and updated grub. So far, so good.

But after reboot to restored system, it is not booting to gnome at all, with warning: [FAILED] Failed to mount Configuration File System. See "systemctl status sys-kernel-config.mount" for details.
Switching to tty2 I am not able to login either as user, nor as root (login incorrect)

Does anybody can point out to me what I am doing wrong here? What am I missing?
I would appreciate any help. Thanks

[SOLVED]

Turns out that if you run rsync as standard user it will change file ownership of all data to that user. Make sure that you run it as ROOT, even if bash accepts this syntax without root privileges.

Last edited by hede0n (2015-01-28 10:08:01)

EscapedNull

Member

Registered: 2013-12-04

Posts: 129

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Can you post the output of "systemctl status sys-kernel-config.mount"? Also ensure that /sys exists and is empty on the new filesystem. You might want to try "journalctl -p 3 -b -0" to find more errors, too. You'll probably need to use rescue media to run these commands.

hede0n

Member

From: UK

Registered: 2015-01-19

Posts: 10

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Thanks for reply.

Shortly...

Also ensure that /sys exists and is empty on the new filesystem.

I didn't know that, but just checked and /sys folder do exist and is empty.

Can you post the output of "systemctl status sys-kernel-config.mount"?

That's the problem, when I chroot to this system and run it I get:

sh-4.3# systemctl status sys-kernel-config.mount
Running in chroot, ignoring request.

You might want to try "journalctl -p 3 -b -0" to find more errors, too.

sh-4.3# journalctl -p 3 -b -0
-- Logs begin at Tue 2015-01-20 18:38:09 UTC, end at Thu 2015-01-22 23:42:12 UTC. --
sh-4.3#

EscapedNull

Member

Registered: 2013-12-04

Posts: 129

Re: [SOLVED] -user error- Cannot login to system restored via rsync

I forgot that systemctl won't work inside a chroot. Since journalctl seems to work okay in the chroot, try something like

journalctl --unit=sys-kernel-config.mount

inside the chroot.

Also try

journalctl -p 3

inside the chroot. The -b option might confuse journalctl when running inside a chroot (I'm unsure about the internals of how systemd identifies a particular "boot"). The -p option stands for "--priority" where a lower number is more urgent. Adjust it as you see fit. Scroll to the bottom to see the newest messages, and look for anything out of the ordinary that might give us some clues.

You did use 'rsync -aAx ...' when you backed up and restored the files, right?

hede0n

Member

From: UK

Registered: 2015-01-19

Posts: 10

Re: [SOLVED] -user error- Cannot login to system restored via rsync

I forgot that systemctl won't work inside a chroot. Since journalctl seems to work okay in the chroot, try something like

journalctl --unit=sys-kernel-config.mount

inside the chroot.

Also try

journalctl -p 3

inside the chroot. The -b option might confuse journalctl when running inside a chroot (I'm unsure about the internals of how systemd identifies a particular "boot"). The -p option stands for "--priority" where a lower number is more urgent. Adjust it as you see fit. Scroll to the bottom to see the newest messages, and look for anything out of the ordinary that might give us some clues.

You did use 'rsync -aAx ...' when you backed up and restored the files, right?

I have used "rsync -aAXv" to be precise, and yes, on both operations.

Now, those outputs:

sh-4.3# journalctl --unit=sys-kernel-config.mount
-- Logs begin at Tue 2015-01-20 18:38:09 UTC, end at Thu 2015-01-22 23:42:12 UTC. --
sh-4.3#

looks like it is empty. Output of journalctl -p 3 is similiar, what I mean is last entry is from 22nd January, the day when I have created this backup. I have restored it, and tried to boot on 25th January, but there is no entries on this date:

-- Reboot --
Jan 22 22:55:59 Arch-Napierdalator kernel: radeon 0000:01:00.0: Invalid ROM contents
Jan 22 22:56:09 Arch-Napierdalator pulseaudio[540]: GetManagedObjects() failed: org.freedesktop.systemd1.LoadFailed: Unit dbus-org.blue
Jan 22 22:58:08 Arch-Napierdalator systemd[1]: Got invalid poll event on pipe.
Jan 22 23:15:05 Arch-Napierdalator systemd-udevd[2725]: inotify_add_watch(7, /dev/sdc2, 10) failed: No such file or directory
Jan 22 23:15:05 Arch-Napierdalator systemd-udevd[2725]: inotify_add_watch(7, /dev/sdc2, 10) failed: No such file or directory
Jan 22 23:15:05 Arch-Napierdalator systemd-udevd[2725]: inotify_add_watch(7, /dev/sdc2, 10) failed: No such file or directory
Jan 22 23:15:05 Arch-Napierdalator systemd-udevd[2725]: inotify_add_watch(7, /dev/sdc2, 10) failed: No such file or directory
Jan 22 23:15:05 Arch-Napierdalator systemd-udevd[2725]: inotify_add_watch(7, /dev/sdc2, 10) failed: No such file or directory
Jan 22 23:15:05 Arch-Napierdalator systemd-udevd[2725]: inotify_add_watch(7, /dev/sdc2, 10) failed: No such file or directory
Jan 22 23:15:06 Arch-Napierdalator systemd-udevd[2724]: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jan 22 23:15:06 Arch-Napierdalator systemd-udevd[2724]: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jan 22 23:15:06 Arch-Napierdalator systemd-udevd[2724]: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jan 22 23:15:06 Arch-Napierdalator systemd-udevd[2724]: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jan 22 23:15:06 Arch-Napierdalator systemd-udevd[2724]: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jan 22 23:15:06 Arch-Napierdalator systemd-udevd[2724]: inotify_add_watch(7, /dev/sdc1, 10) failed: No such file or directory
Jan 22 23:24:32 Arch-Napierdalator kernel: EXT4-fs (sdc1): Unrecognized mount option "umask=022" or missing value
Jan 22 23:27:43 Arch-Napierdalator kernel: EXT4-fs (sdc1): Unrecognized mount option "umask=022" or missing value
Jan 22 23:30:24 Arch-Napierdalator kernel: EXT4-fs (sdc1): Unrecognized mount option "umask=0022" or missing value
Jan 22 23:31:20 Arch-Napierdalator kernel: EXT4-fs (sdc1): Unrecognized mount option "uid=1000" or missing value
Jan 22 23:39:47 Arch-Napierdalator sudo[4122]: hedeon : a password is required ; TTY=pts/0 ; PWD=/home/hedeon ; 

Could it be that I have simply mounted / partition with wrong options? I have blindly followed Arch wiki article, and edited my fstab, not using RW option.
I cannot test it right now but could it be the case?

my fstab:

#/dev/sda7          /        ext4         rw,relatime,data=ordered         0      1
/dev/sda7           /        ext4                 defaults                 0      1

commented out entry is the one I used to use before backup/restore.

Thanks

Last edited by hede0n (2015-01-26 14:50:03)

EscapedNull

Member

Registered: 2013-12-04

Posts: 129

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Output of journalctl -p 3 is similiar, what I mean is last entry is from 22nd January, the day when I have created this backup. I have restored it, and tried to boot on 25th January, but there is no entries on this date:

Hmm, systemd should log something at every boot, if it is able to write to the journal. Try editing /etc/systemd/journald.conf and adding "ForwardToConsole=yes" (and optionally "TTYPath=..."), so you can see the systemd log while the system is booting. Use "MaxLevelConsole=err" (or your desired log level) to prevent the scrollback buffer from filling up with useless information.

Could it be that I have simply mounted / partition with wrong options? I have blindly followed Arch wiki article, and edited my fstab, not using RW option.
I cannot test it right now but could it be the case?

my fstab:

#/dev/sda7          /        ext4         rw,relatime,data=ordered         0      1
/dev/sda7           /        ext4                 defaults                 0      1

commented out entry is the one I used to use before backup/restore.

Thanks

I guess it would be worth a try using the same options as you previously did. Also make sure that your bootloader is passing the "rw" flag along with "root=". For Grub, you can type 'e' when you see the grub menu, and manually add the 'rw' flag once for testing purposes. You can use dumpe2fs(8) to see the default mount options for the filesystem in question, and tune2fs(8) to change them. It's unlikely, but possible, that the new filesystem is setup to be mounted 'ro' by default (although it will be remounted with the fstab options before init is run anyway, I believe).

hede0n

Member

From: UK

Registered: 2015-01-19

Posts: 10

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Hmm, systemd should log something at every boot, if it is able to write to the journal. Try editing /etc/systemd/journald.conf and adding "ForwardToConsole=yes" (and optionally "TTYPath=..."), so you can see the systemd log while the system is booting. Use "MaxLevelConsole=err" (or your desired log level) to prevent the scrollback buffer from filling up with useless information.

I did as you have said, and although I could not scroll through output of console (simply don't know how) I have seen enough. Errors displayed were referring to Read Only file system...
I have checked... All backed up folders had hedeon user (me) as owner. So obviously I have f...d up something. Either before making a backup, or when creating backup, most probably second option, otherwise I would have problems booting before backup operation. I swear, I have used -aAXv for rsync, could mounting options of the drive I have backed to somehow overwrite this?
Will it be simply a case of changing all system folders ownership to root, or more complicated than that?

EscapedNull

Member

Registered: 2013-12-04

Posts: 129

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Hmm, systemd should log something at every boot, if it is able to write to the journal. Try editing /etc/systemd/journald.conf and adding "ForwardToConsole=yes" (and optionally "TTYPath=..."), so you can see the systemd log while the system is booting. Use "MaxLevelConsole=err" (or your desired log level) to prevent the scrollback buffer from filling up with useless information.

I did as you have said, and although I could not scroll through output of console (simply don't know how) I have seen enough. Errors displayed were referring to Read Only file system...
I have checked... All backed up folders had hedeon user (me) as owner. So obviously I have f...d up something. Either before making a backup, or when creating backup, most probably second option, otherwise I would have problems booting before backup operation. I swear, I have used -aAXv for rsync, could mounting options of the drive I have backed to somehow overwrite this?
Will it be simply a case of changing all system folders ownership to root, or more complicated than that?

I was hoping it would be a read-only filesystem, or incorrect ownership/permissions, but I wasn't expecting both. But at least we have something to go on now. For future reference, you can use Shift+PgUp and Shift+PgDn to scroll on the console (and many terminal emulators), even though the scrollback buffer is relatively short.

First I would try to get your root partition mounting read-write. This involves your boot loader, your fstab, and in some cases the filesystem's embedded metadata.
1. Boot loader (GRUB; I'm afraid I can't give advice on syslinux or any other boot loader)
  1a. Press 'e' when you see the GRUB menu at bootup. You have five seconds, unless you changed that from the default. This allows you to make a one-time change to your boot loader config, which is good for testing. Put the 'rw' flag after your root flag (add a root flag if you don't have one). Example: "... root=/dev/sda1 rw"
  1b. Modify the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub. It should match whatever line worked in 1a. Skip this if 'rw' was already present in 1a, since GRUB tries to detect the appropriate root filesystem automatically. Don't forget to run grub-mkconfig after making changes to this file!
2. /etc/fstab
  2a. add 'rw' to your root partition's mount options. This usually only affects the filesystem after init is started, however it is still relevant in this case.
  2b. If you use systemd in early userspace, you may need to run mkinitcpio after this change. If you don't know, then you probably are not using systemd in early userspace.
3. Embedded metadata: use dumpe2fs(8) and tune2fs(8) to view and change your filesystem's metadata, particularly the default mount options. Make sure one of them is 'rw'.

Reboot and cross your fingers. I recommend leaving 'ForwardToConsole=yes' until you are able to login successfully.

Now onto the ownership problem. If you ran rsync as a non-root user, that would cause the wrong ownership symptoms you're seeing (in theory, rsync should have warned you if that was the case). If this is causing you to be unable to login, then it's a simple fix. Boot from the Arch live CD, mount and chroot, then run `pacman -S base`. This will reinstall all packages in the 'base' group, including 'filesystem'. These files will be reinstalled with the correct ownership. There may be some lingering ownership problems on files that are not tracked by pacman, but reinstalling 'base' should hit all of the important ones. Use the mount(8) command to ensure the filesystem is mounted *without* nosetuid,noexec,nodev and the like.

[Edit: actually it's probably best to fix the ownership problem first, as that will prevent you from logging in regardless of a read-only root]

Last edited by EscapedNull (2015-01-27 02:58:03)

hede0n

Member

From: UK

Registered: 2015-01-19

Posts: 10

Re: [SOLVED] -user error- Cannot login to system restored via rsync

I did as you said and started and started with ownership problem, chroot to that system and installed and installed base and base-devel packages again. I had a warning on nearly every package parent directory permissions mismatch. Then I have edited grub on boot as you suggested and added rw to root flag, boot, and somewhat success. I didn't see any red [failed] entries during boot. Also there was no 'err' output from journalctl in console, at least in visible part of it (scrolling didn't work for me, maybe funny keyboard is to blame). But I still was not able to log in, still it would't accept users, nor root password.

I will be honest, I gave up. I have formatted all partitions with arch, and manjaro, and installed arch from scratch again... The entire exercise is to learn how to have two bootable cloned arch systems, and use second one as kind of snapshot of first one, made every week or so. Little point in recovering it if I made backup wrong way. Lesson learned, now I know that if bash accepted the command without su or sudo, it doesn't mean everything is OK (rsync gave no warning, I have double checked).

I would like to thank you for your help and time. You have helped me with understanding few concepts, and showed me new tricks (despite my age). Thanks again.

Last edited by hede0n (2015-01-28 10:01:58)

EscapedNull

Member

Registered: 2013-12-04

Posts: 129

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Sorry we couldn't come up with a fix. I hope the knowledge you gained through the experience becomes useful in the future. There's a learning curve to every Linux, especially Arch, and I recommend you do anything you can not to get discouraged. If that means reformatting and moving on, then that's a smart thing to do.

The entire exercise is to learn how to have two bootable cloned arch systems, and use second one as kind of snapshot of first one, made every week or so.

You might want to look into Btrfs and/or LVM snapshots in this case. Neither will give you two different filesystems, but they do have advantages over the rsync method. LVM uses block-level snapshots, and Btrfs uses block+metadata snapshots, meaning that snapshots are much more robust against permissions problems, ownership, hardlinks, link-loops, etc. Probably the most important advantage of Btrfs/LVM snapshots is they they only store the differences between the time the snapshot was taken, and "now", resulting in far less disk usage. You can also selectively take and apply snapshots on various parts of the filesystem(s) (Btrfs subvolumes, LVM logical volumes), i.e. to restore your root directory to last week, but keep your home directory current. Keep in mind, they are not alternatives to cold backups for irreplaceable files. There is no best solution for every use case, but you might find "real" snapshots more reliable and easier to work with than file-by-file backups with rsync.

See you around the forum

hede0n

Member

From: UK

Registered: 2015-01-19

Posts: 10

Re: [SOLVED] -user error- Cannot login to system restored via rsync

Sorry we couldn't come up with a fix.

Oh... you probably could, but I just figured out that repercussions of my mistake will hit me in a face one day or another, even I we would fix problem with logging in now. As system was 3-4 days old, and not critical, at some point it made more sense just to rebuild it.

There's a learning curve to every Linux, especially Arch, and I recommend you do anything you can not to get discouraged.

You are kidding? I absolutely love it about Arch. In just over a week of using it I have learned more stuff than for few years of using Ubuntu/Mint. Once I will get more confident with it I will install it on rest of my desktops/laptops, 2 virtual servers, and Raspberry Pi

You might want to look into Btrfs and/or LVM snapshots in this case

Well, this is definitely intriguing... Especially when you say it could be easier to work with than rsync copying, because yesterday when I tried to boot clone system, original one was loaded. I have edited fstab correctly (I believe), so I think this is due to the fact I have copied /boot folder as well. Will need to investigate.

Anyway thanks again for help, and see you around