You are not logged in.
I've searched the forums and found Moving /usr To Folder On Different Partition, but that was inconclusive.
General websearching led me to something that seems nearly the answer, but it's for Debian, so the instructions fall at my last hurdle: namely that I don't know enough about this to translate it into Arch.
I've had no problem following the instructions on /usr as a separate partition in the past, but now I need to do something more complex and mount my /usr (and other directories) from lower down the filetree on another partition.
The reasons aren't because I'm running out of space or anything that can be resolved a different way, it's just (hopefully) a solution to a problem I will otherwise have when attempting to demonstrate a proof of concept for something else altogether — for which purpose I need to simulate an LTSP style fat client login from a 'thin' client. But, before I go to the (potentially uncompensated) expense/trouble of obtaining all the necessary hardware for a PXE boot/LDAP server and three NAS boxes to stand in for three SANs on three physically isolated networks, I thought I'd try simulating them on a separate partition — for what I want to demonstrate, the login and authentication aren't important and can safely be assumed to have been successful before any of what I am doing would happen, so all I need do (I hope) is create three folders on a partition to represent the three SANs, a folder in each to represent a RAID box on each and a bunch of subdirectories to hold appropriate data structures for my proof on concept to operate upon; never mind ACL/SELinux/AppArmor/whatever, standard permissions will do fine for my purposes at this stage, which, again, won't even be my problem in the long term anyway (I'm 'selling' an architecture, not a mechanism).
The reason I'm attempting it with Arch is because it's what I know and, once again, the specific platform isn't particularly relevant, so it's better I feel not to introduce unnecessary complication by trying ti use a different solution (like LTSP) that I don't know and could get bogged down troubleshooting instead of getting on with things.
So, as said, I found something for Debian that seems very close to what I need.
You create unit files for the directories you want to bind to their counterpart mount points on the system partition and let systemd handle the out of sequence/late mounting.
That much is simple enough and even I, with my utterly pitiful knowledge of systemd, have been able to adapt the given examples (I think).
A simple example would be having systemd mount a separate /boot — not strictly necessary, since that's easily handled in fstab but, what the hey, why not be consistent, eh? It's not like a /boot will be necessary for a PXE booted system anyway, so it's really just an exercise in 'stating the obvious' as it were, but it indicates the shape of things to come in the simplest way.
[Unit]
Description=Main Boot Mount
DefaultDependenices=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target
[Mount]
What=/dev/disk/by-uuid/83b5cefa-9819-4d1e-9e53-66da51f7f458
Where=/boot
Type=ext2
Options=defaults,errors=remount-ro
[Install]
WantedBy=multi-user.targetBind mounting anything from the other filesystem requires it to be mounted first — /srv/seems as good a place as any to mount it to for now (it kinda makes sense when you consider the purpose of /srv as Data for services provided by this system, there doesn't seem to be anywhere else obvious in the FSH standard, and it saves cluttering up the root system with spurious /data subtrees, so it'll do for my purposes here).
[Unit]
Description=Main Domain_000 Mount
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target
[Mount]
What=/dev/disk/by-uuid/cafda08e-85b5-426f-915a-a82844d3bf88
Where=/srv/org_000/domain_000
Type=ext4
Options=defaults,errors=remount-ro
[Install]
WantedBy=multi-user.targetOkay, so far so good.
I can now bind mount the profile for a specific user, or employee role, on a specific device (make and model of remote office desktop machine, for instance).
Let's try it with their client's /home structure:
[Unit]
Description=Org_000.Domain_000.Host_00000 Home Mount
DefaultDependenices=no
Conflicts=umount.target
Before=local-fs.target umount.target
RequiresMountsFor=/srv/org_000/domain_000
[Mount]
What=/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home
Where=/home
Type=none
Options=bind,x-systemd.requires-mounts-for=/srv/org_000/domain_000
[Install]
WantedBy=multi-user.targetHell, let's do it with /etc as well, why not?
[Unit]
Description=Org_000.Domain_000.Host_00000 Etc Mount
DefaultDependenices=no
Conflicts=umount.target
Before=local-fs.target umount.target
RequiresMountsFor=/srv/org_000/domain_000
[Mount]
What=/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/etc
Where=/etc
Type=none
Options=bind,x-systemd.requires-mounts-for=/srv/org_000/domain_000
[Install]
WantedBy=multi-user.targetAnd so on and so forth until every meaningful top level filetree I want to move has been accounted for; probably at least /opt, /root and /var, if not the likes of /proc, /sys and all the others — a mountpoint's just a mountpoint, after all, so, as long as the system knows where to find it when necessary, it should, theoretically, make no difference (it's not strictly necessary, as the client will be passed a small ramfs image to mount the 'virtual' filetrees to, I imagine, but the 'fat client' needs somewhere to start from when it reads its system partition on whichever RAID box it resides on).
Now, however, comes the hurdle where my lack of knowledge lets me down.
The instructions for Debian discuss user-definable scripts in /etc/initramfs-tools. Apparently, most of the predefined ones are in /usr/share/initramfs-tools and in the scripts directory there are a number of subdirectories with suffixes of -top, -bottom, and -premount — the top scripts run first, followed by the premount, and then the bottom (so init-top runs first, the others run after that and init-bottom is the last thing executed).
The author then mentions that the template script shows init sourcing /usr/share/initramfs-tools/hook-functions to load common functions but that only works if /usr has already been found — so, as some other scripts use a copy that is in the boot environment, located at /usr/share/initramfs-tools/scripts/functions, that's what they used as their template instead.
If I were using Debian, mine would look like this, therefore (NB, further investigation into how Arch might do this led me to add the remark # [OR SHOULD IT BE /new_root ?]:
#!/bin/sh
PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
. scripts/functions
# Note: our root is on /root right now, not /
# [OR SHOULD IT BE /new_root ?]
mount /dev/disk/by-uuid/cafda08e-85b5-426f-915a-a82844d3bf88 /root/srv/org_000/domain_000
mount /dev/disk/by-uuid/83b5cefa-9819-4d1e-9e53-66da51f7f458 /root/boot
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr /root/usr
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/root /root/root
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home /root/home
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt /root/opt
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var /root/varIt also seems that, the more unit files used to mount other locations, the less that needs to go in this script — meaning that (possibly) the only necessary entries here might be:
mount /dev/disk/by-uuid/cafda08e-85b5-426f-915a-a82844d3bf88 /root/srv/org_000/domain_000
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr /root/usrOr even simply ...?
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr /root/usrI'm pretty sure the unit files (ha!) want to go in /etc/systemd/system. And, further investigation leads me to suspect that the script might need to go in /etc/systemd, /usr/share/mkinitcpio, or possibly even /usr/lib/initcpio, but I'm really uncertain about it and not convinced my best guess as to which (/usr/share/mkinitcpio ?) would lead to anything other than failure.
Likewise, whilst I'm pretty confident I'm on the right track with the remark about /new_root, I'm not sure I am.
As said, this is beyond my knowledge now and I really don't want to get it wrong. I've got a spare drive set up to install Arch to, but it going wrong and having to re-image it afresh, whilst not overly onerous, would be tedious (I'd much rather get it right first time).
So, if some kind soul would like to set me straight about this, I'd be ever so grateful. Yes, it's counter to how things 'ought' to be done, but humour me — systemd is, seemingly, perfectly capable of handling it and it would save me a lot of time learning how to do it with an LTSP solution (that is really beyond the scope of what I need to do for a proof of concept concerning a logical/physical data management architecture and the tools to manage it) ... and also (potentially wasted) expenditure on hardware I have no use for and I can ill afford right now.
Thanks in advance (and sorry about the looooooong post).
BD
Offline
If you change WantedBy to initrd-root-device.target install and enable the unit files does the systemd hook include them in the initrd?
I do not understand why you would need anything from initramfs-tools.
Edit:
systemd uses /sysroot/ as the mount point for the rootfs so all subtree mounts need to be relative to that.
Why are you ordering the main mount after swap?
Last edited by loqs (2021-11-09 22:46:22)
Offline
If you change WantedBy to initrd-root-device.target install and enable the unit files does the systemd hook include them in the initrd?
Hi, thanks for replying!
Sorry, I'm in over my head at this stage, so just to clarify, are you suggesting I modify the unit files?
FROM
[Install]
WantedBy=multi-user.target
TO
[Install]
WantedBy=initrd-root-device.target
I can't test this until this evening (it's about 14:00 here now), but I'll give it a go and see what happens.
I do not understand why you would need anything from initramfs-tools.
My (probably misguided) understanding after reading about stuff on mkinitcpio — in particular about automated generation and runtime hooks (which latter, like the original material I used as a starting point, mentions early/late-run processes).
And this here
initrd.target
This is the default target in the initramfs, similar to
default.target in the main system. It is used to mount the
real root and transition to it. See bootup(7) for more
discussion.initrd-fs.target
systemd-fstab-generator(3) automatically adds dependencies of
type Before= to sysroot-usr.mount and all mount points found
in /etc/fstab that have the x-initrd.mount mount option set
and do not have the noauto mount option set. It is also
indirectly ordered after sysroot.mount. Thus, once this
target is reached the /sysroot/ hierarchy is fully set up, in
preparation for the transition to the host OS.initrd-root-device.target
A special initrd target unit that is reached when the root
filesystem device is available, but before it has been
mounted. systemd-fstab-generator(3) and
systemd-gpt-auto-generator(3) automatically setup the
appropriate dependencies to make this happen.initrd-root-fs.target
systemd-fstab-generator(3) automatically adds dependencies of
type Before= to the sysroot.mount unit, which is generated
from the kernel command line's root= setting (or equivalent).initrd-usr-fs.target
systemd-fstab-generator(3) automatically adds dependencies of
type Before= to the sysusr-usr.mount unit, which is generated
from the kernel command line's usr= switch. Services may
order themselves after this target unit in order to run once
the /sysusr/ hierarchy becomes available, on systems that
come up initially without a root file system, but with an
initialized /usr/ and need to access that before setting up
the root file system to ultimately switch to. On systems
where usr= is not used this target is ordered after
sysroot.mount and thus mostly equivalent to
initrd-root-fs.target. In effect on any system once this
target is reached the file system backing /usr/ is mounted,
though possibly at two different locations, either below the
/sysusr/ or the /sysroot/ hierarchies.
Plus...
...discovering the presence of /usr/lib/initcpio, /usr/lib/initcpio/hooks and /usr/lib/initcpio/install directories on my system and, after looking inside the usr files in the latter two, learning that
This provides a support for mounting /usr via a late running hook. No
configuration is needed, as the mount options will be pulled directly from the
fstab on the real root device.
... and figuring that, if mkinitcpio were going to generate the initramfs come what may, I'd better make sure I accommodate that.
I'm likely just confused about things though.
Edit:
systemd uses /sysroot/ as the mount point for the rootfs so all subtree mounts need to be relative to that.
Okay, I (probably erroneously) got the impression that /sysroot/ was a substitution for "wherever it actually is" (device UUID, top of the chroot environment, or wherever) — kinda like Windows' %SystemDrive% and %SystemRoot%.
Are you saying that I should be using it as a string literal instead of /new_root ?
Why are you ordering the main mount after swap?
Again, probably because I'm in over my head and don't know enough about how systemd works.
But, this
initrd-usr-fs.target
systemd-fstab-generator(3) automatically adds dependencies of
type Before= to the sysusr-usr.mount unit, which is generated
from the kernel command line's usr= switch. Services may
order themselves after this target unit in order to run once
the /sysusr/ hierarchy becomes available, on systems that
come up initially without a root file system, but with an
initialized /usr/ and need to access that before setting up
the root file system to ultimately switch to. On systems
where usr= is not used this target is ordered after
sysroot.mount and thus mostly equivalent to
initrd-root-fs.target. In effect on any system once this
target is reached the file system backing /usr/ is mounted,
though possibly at two different locations, either below the
/sysusr/ or the /sysroot/ hierarchies.
... gave me the impression that I'd better have the partition on which the target usr directory resides already mounted before it (systemd) reaches initrd-root-device.target
Last edited by BeastlyDoglick (2021-11-10 14:17:18)
Offline
loqs wrote:If you change WantedBy to initrd-root-device.target install and enable the unit files does the systemd hook include them in the initrd?
Hi, thanks for replying!
Sorry, I'm in over my head at this stage, so just to clarify, are you suggesting I modify the unit files?
FROM
[Install]
WantedBy=multi-user.targetTO
[Install]
WantedBy=initrd-root-device.target
Yes
loqs wrote:Edit:
systemd uses /sysroot/ as the mount point for the rootfs so all subtree mounts need to be relative to that.Okay, I (probably erroneously) got the impression that /sysroot/ was a substitution for "wherever it actually is" (device UUID, top of the chroot environment, or wherever) — kinda like Windows' %SystemDrive% and %SystemRoot%.
Are you saying that I should be using it as a string literal instead of /new_root ?
Yes.
loqs wrote:Why are you ordering the main mount after swap?
Again, probably because I'm in over my head and don't know enough about how systemd works.
But, this
initrd-usr-fs.target
systemd-fstab-generator(3) automatically adds dependencies of
type Before= to the sysusr-usr.mount unit, which is generated
from the kernel command line's usr= switch. Services may
order themselves after this target unit in order to run once
the /sysusr/ hierarchy becomes available, on systems that
come up initially without a root file system, but with an
initialized /usr/ and need to access that before setting up
the root file system to ultimately switch to. On systems
where usr= is not used this target is ordered after
sysroot.mount and thus mostly equivalent to
initrd-root-fs.target. In effect on any system once this
target is reached the file system backing /usr/ is mounted,
though possibly at two different locations, either below the
/sysusr/ or the /sysroot/ hierarchies.... gave me the impression that I'd better have the partition on which the target usr directory resides already mounted before it (systemd) reaches initrd-root-device.target
Still not seeing the connection to swap.target. Are you planning on using usr command line switch? Not sure it would accommodate your use case.
Offline
BeastlyDoglick wrote:FROM [...] TO
Yes
Done.
Placed them in /etc/systemd/system and enabled them.
But, it's a bit academic right now, because I'm getting a mkinitcpio error:
ERROR: file not found: 'fsck.none'Which is followed by a warning:
WARNING: errors were encountered during the build. The image may not be complete.I suspect it's to do with the bind mounts meaning there's no file system given for fsck to fsck, even though there are explicit pass values of '2' in the fstab
My genfstab generated fstab is as folllows (tthere's an awful lot of 'none' in there):
# Static information about the filesystems.
# See fstab(5) for details.
# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sda3 LABEL=Org_000
UUID=778f442d-909a-4c3f-bfec-9b6e90206315 / ext4 rw,relatime 0 1
# /dev/sda4 LABEL=Boot
UUID=83b5cefa-9819-4d1e-9e53-66da51f7f458 /boot ext2 rw,relatiime 0 2
# /dev/sda2 LABEL=Domain_000
UUID=cafda08e-85b5-426f-915a-a82844d3bf88 /srv/org_000/domain_000 ext4 rw,relatime 0 2
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/etc
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/etc /etc none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr /usr none rw,bind 0 2
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home /home none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt /opt none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/root
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/root /root none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var /var none rw,bind 0 0
# /dev/sda1
UUID=3b2d1080-fa07-4b53-be2b-572190b4cecf none swap defaults 0 0Whatever the problem is, it doesn't boot — just locks up o the extent that even the power button doesn't work (I had to yank both the power and the battery).
BeastlyDoglick wrote:Are you saying that I should be using it as a string literal instead of /new_root ?
Yes
After further reading about systemd, I had a (possibly/probably counterproductive) epiphany and ignored the script issue — put in a usr unit file instead.
Is this likely to be why my initramfs won't build? Do, I need to be putting something somewhere other than the usr, fsck and shutdown hooks hooks in /etc/mkinitcpio.conf? (fsck was in there by default anyway).
Or is it that I'm trying to fsck a filesystem that is reported as not being a filesystem, so (as far as fsck is concerned) there's nothing there to fsck?
It's late now and I'm too tired to do any more tonight — I'll just make (even more) mistakes.
I'll give it a go with a pass value of '0' tomorrow — see if it builds at least and, if it does, worry about what to do about fsck-ing the mounts afterwards.
Still not seeing the connection to swap.target. Are you planning on using usr command line switch? Not sure it would accommodate your use case.
I probably just misunderstood the original guide. It seemed simple enough: mounting an SD card, on which the /home, /opt and /usr were direcories that were bind-mounted late. The script was part of a discussion of what needed to be done to the initramfs build script(s), so I simply made a note of it, adapting it (where I could) to match my own config, with the intention of seeing how it would translate to Arch afterwards.
It may be unnecessary, or it may be the root of my build problem, but I'm too tired to think straight right now — I'll start with the simple process of elimination by trying a pass of '0' on /usr and /etc tomorrow and see where (if anywhere) it gets me. If that doesn't point me in the right direction, I'll have to turn my attention to the pivot/swap and start delving deeper into how systemd does it. I'm not convinced that's the problem though and you're probably right that it's not relevant/necessary.
Last edited by BeastlyDoglick (2021-11-11 02:21:01)
Offline
If the root is a network file system is there a need for any bind mounting on the client apart from as part of the POC? Can the directory structure not be generated on the server?
Last edited by loqs (2021-11-11 20:40:39)
Offline
It can (and ultimately will be), that way, yes — so there won't be any need for my shenanigans, no.
But I don't really need to trouble myself about the ins and outs of how they manage the client/server side of things — standard thin client approach, LTSP style fat client, local or remote login, RHEL, SUSE, Debian ... it's all the same to me. I'm just attempting to simulate that setup on a single HDD right now, for demonstration purposes. I'll be selling people an approach, architecture and the tools to deal with it, not the hardware/system itself, so I just need to be able to demonstrate it working (albeit emulated and in a limited manner), without having to traipse into a meeting with a laptop, a router, a PXE boot/LDAP sever, three NAS boxes and an unfortunate amount of cabling — because there's no way I'll even want to (let alone will) persuade anyone to go to the necessary lengths of building it on their premises for the purposes of a speculative pitch (that bit comes when I've sold them on the idea and they want to see it work 'for real' before committing to a lot of disruption).
If I can't simulate it perfectly with complete separation from the root filesystem (everything on the simulation) it won't be the end of the world (the principle and pretty much all of the tooling will still hold true), but the fewer holes that can be picked in it right from the outset the better — if I can say "Certainly, let me show you," rather than "Well, I can't demonstrate that feature right now but, trust me, the theory is sound, honest"), it's a lot more impressive and people are less sceptical.
In the meantime, I've done some further investigation/testing.
Seemingly it has nothing to do with whether the bind mount is fsck-ed in /etc/fstab or not — a pass no. of '0' made no difference (same error and warning)
Doing a diff between my own initramfs and the 'incomplete' one produced mostly time/date-stamp differences, but a few filesize diffs and a couple of additions as well (nothing to raise eyebrows about though, as far as I could see).
In the buildconfig, HOOKS=(base udev autodetect modconf block filesystems usr fsck keyboard shutdown), which is what's supposed to be there, if you follow the wiki on mkinitcpio — I'm not sure how significant the order is though.
In initramfs/org_000-domain_000-host_00000/hooks, there were (as you'd expect) additional shutdown and usr scripts.
The usr script is:
#!/usr/bin/ash
run_latehook() {
local usr_source mountopts passno realtab=/new_root/etc/fstab
if [ -f "$realtab" ]; then
if usr_source=$(findmnt -snero source --tab-file="$realtab" -T /usr); then
mountopts=$(findmnt -snero options --tab-file="$realtab" -T /usr)
# returning the passno column requires util-linux >= 2.23. for
# older versions which do not support this column, always fsck.
passno=$(findmnt -snero passno --tab-file="$realtab" -T /usr 2>/dev/null)
if [ -z "$passno" ] || [ "$passno" -gt 0 ]; then
fsck_device "$usr_source"
fi
msg ":: mounting '$usr_source' on /usr"
mount "$usr_source" /new_root/usr -o "$mountopts"
fi
fi
}
# vim: set ft=sh ts=4 sw=4 et:Given that realtab=/new_root/etc/fstab and your remarks re /sysroot/ being a string literal, are you suggesting it should be realtab=/sysroot/etc/fstab instead? (Whilst there's a initramfs/org_000-domain_000-host_00000/new_root mountpoint, there's no equivalent /sysroot, so I'm guessing I'd have to create it and then edit the init script to change all instances of /new_root to /sysroot?)
There is, of course, the fsck_device() entry in the init_functions script that could be meeting the if [ ! -b "$1" ] test, but it's the wrong error message.
The single thing that really stands out is the absence of a fsck.none in the initramfs usr/bin, but I'm stumped as to why mkinitcpio would think it needed one — unless it's a builtin in one of the binaries, it can only be as a result of some paramter being passed that's based on the fstab entry for the filesystem type.
But I thought I'd pursue it anyway.
First of all, I tried being a smartarse: extracted the initramfs-linux.img with lsinitcpio -x, copied fsck.ext4 to fsck.none, recompressed the image with the build_image function from the /usr/bin/mkinitcpio script, as per Recompressing a modified extracted image, copied the new initramfs-linux.img, reboot and ... no dice — it just hung on the ramdisk initialisation.
So, then I rebooted from the Arch iso, mounted everything, created a copy of fsck.ext4 in /usr/bin as fsck.none, ran mkinitcpio -P and ... success! The image built with no errors beyond the usual warnings about possibly missing firmware for wd719x, aic94xx and xhci_pci.
Rebooted and ... nope, not having it.
Tried changing the .mount unit WantedBy entries back to multi-user.target
Still nogo.
Tried disabling the units altogether.
No change.
Short of there being some reason to add the aforementioned entries from the original script to mount the data partition and /usr off /new_root (I'm guessing in one of the scripts in /usr/lib/initcpio or /usr/lib/initcpio/install), I'm stumped.
Well, okay, there's the add_fsck function in /usr/lib/initcpio/install/fsck and/or the add_binary command in the build function in /usr/lib/initcpio/usr, but I'm loathe to play around with those unnecessarily — although the former is clearly why the default fsck.ext2 and fsck.ext3 in the original image are symlinks rather than a physical binary as it was after my attempts.
Any ideas?
Last edited by BeastlyDoglick (2021-11-12 13:51:54)
Offline
HOOKS=(base udev autodetect modconf block filesystems usr fsck keyboard shutdown)No systemd hook. Without it the initrd will use ash scripts such as the usr script, the root fs will be mounted to /newroot and systemd will not be used until after switching to /newroot
With the systemd hook run time ash scripts will not be used and the root fs will be mounted to /sysroot and systemd units will be used in the initrd. https://www.freedesktop.org/software/sy … ootup.html
For demonstration purposes why not use virtual machines rather than bare metal?
Last edited by loqs (2021-11-12 00:51:17)
Offline
HOOKS=(base udev autodetect modconf block filesystems usr fsck keyboard shutdown)No systemd hook. Without it the initrd will use ash scripts such as the usr script, the root fs will be mounted to /newroot and systemd will not be used until after switching to /newroot
With the systemd hook run time ash scripts will not be used and the root fs will be mounted to /sysroot and systemd units will be used in the initrd.
AhA!
See ... said I was in over my head when it came to systemd.
Do I need to remove the usr hook (or any others), or just add the systemd hook?
Does the order matter?
For demonstration purposes why not use virtual machines rather than bare metal?
Because that would be cleverer than I am. I was working on a tricky bit of code recently and got bogged down debugging it for eight hours, trying to work out why the end result was correct when two of the intermediate stages were producing what appeared to be wildly incorrect values. By the time I wrapped my head around what was going on (and realised the intermediate results weren't incorrect, I was just misinterpreting the syntax of a couple of command structures), it was to realise that, completely unaware of what I was doing, I'd been working on two solutions to the same problem simultaneously ... and one of them (the one I had ignored) had not only already solved it five hours previously but was, in fact, a better one than the one I'd wasted another five hours on *sigh*
I honestly just didn't think of using VMs.
But, in my defence, I also don't want to risk unfortunate lapses/hiccups — like my laptop deciding to offload a VM to swap behind my back, for instance (it's had a disconcerting tendency to swap a lot recently). People at that level are often impatient and judgmental and (like not traipsing in with all the boxes and cables) I'd rather not give them any reason to dismiss it on spurious grounds because my machine misbehaves just at the wrong moment. Bare metal is as fast as it gets and I want the demo to be as slick as can be for a boardroom 'elevator pitch.'
It's not a bad idea though; I'l keep it in mind for if/when I have to concede defeat with the current approach — thanks!
Last edited by BeastlyDoglick (2021-11-12 01:04:19)
Offline
I was looking at that only last night, but too tired by that stage to do anything sensible (like understand enough of it).
And today I had my brainwaves and went off down the track of messing around with the initramfs image instead.
*sigh* gotta work smarter, not harder!
[ I'm gonna have to pick your brains about systemd-nspawn later: I have an idea for being able to make use of unlimited encrypted datapools on an encrypted LVM schema — but I need to learn a lot more about systemd and systemd-nspawn first]
Last edited by BeastlyDoglick (2021-11-12 01:21:28)
Offline
The systemd hook should come first. See Mkinitcpio#HOOKS for the hooks it replaces.
Edit:
Do note from the table you will need to write your own systemd equivalent of the net hook for the real system.
Last edited by loqs (2021-11-12 01:25:00)
Offline
The systemd hook should come first. See Mkinitcpio#HOOKS for the hooks it replaces.
I've read and re-read that page so many times, I all but went cross-eyed the last time I saw the word 'HOOKS'.
But I just didn't glean that from it.
Sure there's this:
Note: Runtime hooks are only used by busybox init. systemd hook triggers a systemd based init, which does not run any runtime hooks but uses systemd units instead.But it's a little too cryptic for me: the what is clear but not the how — it's probably just my ignorance of systemd (I'm sure it'll be clear as day to me once I know more).
Offline
Edit:
Do note from the table you will need to write your own systemd equivalent of the net hook for the real system.
Thanks!
I foresee some more reading in my immediate future.
Not now though — now I have to go to bed again (before I stay up all night again and find myself staring blankly at the word 'HOOKS' for thirty minutes straight again).
Last edited by BeastlyDoglick (2021-11-12 01:33:17)
Offline
The systemd hook should come first.
Removed the fsck.none ffrom /usr/bin.
Added the systemd hook as the first in (the effective) /etc/mkinitcpio.conf
HOOKS=(systemd base udev autodetect modconf block filesystems usr fsck shutdown keyboard).
Edited the .mount unit files back to WantedBy=initrd-root-device.target
Confirmed the existence (and content) of (the effective) /etc/systemd/system/initrd-root-device.target.wants.
Ran 'mkinitcpio -P'.
Got same complaint about lack of 'fsck.none'.
Copied /usr/bin/fsck.ext4 to /usr/bin/fsck.none.
Edited the add_fsck function in /usr/lib/initcpio/install/fsck to add add_symlink /usr/bin/fsck.none fsck.ext4.
Re-ran 'mkinitcpio - P'.
No complaint and the image built.
Rebooted,
Go as far as initframfs image initialisation.
Got no further.
If this other person successfully booted after binding their /usr from another filesytem, the only difference I can see between what they did and what I'm attempting is that they
1. relocated /home, /opt and /usr, but not /etc;
2. added those extra lines to a build script from their /usr/share/initramfs-tools/scripts/functions.
What I really don't understand is the complaint about the missing /usr/bin/fsck.none. I've never heard of such a thing — you don't fsck nothing, you just don't invoke fsck on the 'nothing' in the first place (i.e. dump pass = 0 0). Where is that coming from?
Offline
Do you get the message about fsck.none for the creation of the fallback image as well?
If you remove all the units apart from the unit to mount /srv does that work?
What boot options are you using?
Offline
Do you get the message about fsck.none for the creation of the fallback image as well?
No, just the main initramfs.
If you remove all the units apart from the unit to mount /srv does that work?
Cheers. Will give it a try and get back to you.
What boot options are you using?
Defaults: installed Arch and left it at that — no switches, no extra modules (vanilla config).
Offline
If you remove all the units apart from the unit to mount /srv does that work?
Disabled all but the /srv mount.
Removed add_symlink /usr/bin/fsck.none fsck.ext4 from /usr/lib/initcpio/install/fsck
Ran 'mkinitcpio -P'.
Same complaint as before.
Unless it can find a /usr/bin/fsck.none, it just isn't happy.
But, even after it does, it can't complete the initramfs mount.
Perhaps there's something to this after all?
#!/bin/sh
PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
. scripts/functions
# Note: our root is on /root right now, not /
# [OR SHOULD IT BE /new_root ?]
mount /dev/disk/by-uuid/cafda08e-85b5-426f-915a-a82844d3bf88 /root/srv/org_000/domain_000
mount /dev/disk/by-uuid/83b5cefa-9819-4d1e-9e53-66da51f7f458 /root/boot
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr /root/usr
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/root /root/root
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home /root/home
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt /root/opt
mount -o bind /root/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var /root/varLast edited by BeastlyDoglick (2021-11-13 17:33:59)
Offline
You have a fstab entry for /usr which is a bind mount and are using both the usr and autodetect hooks?
By boot options I meant what options is the boot loader passing to kernel.
What are the differences between the good and bad initrd now?
Offline
You have a fstab entry for /usr which is a bind mount and are using both the usr and autodetect hooks?
Rebuilt it.
Read the wiki article on mkinitcpio a little more closely (it could do with being a trifle more obvious) and didn't add both the systemd and usr hooks, just the systemd hook:
HOOKS=(systemd base udev autodetect modconf block filesystems keyboard fsck)FSTAB
# /dev/sda3 LABEL=Org_000
UUID=52c5acf5-5101-45cc-9097-8043584493f3 / ext4 rw,relatime 0 1
# /dev/sda4 LABEL=Boot
UUID=e2dd56d1-fc2c-4e2e-a293-f62a2cd3364d /boot ext2 rw,relatime 0 2
# /dev/sda2 LABEL=Domain_000
UUID=782c7e72-b0f5-4e30-aa8e-280dfa1c117e /srv/org_000/domain_000 ext4 rw,relatime 0 2
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/etc
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/etc /etc none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home /home none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt /opt none rw,bind 0 0
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr /usr none rw,bind 0 2
# /mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var
/mnt/srv/org_000/domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var /var none rw,bind 0 0
# /dev/sda1
UUID=3b2d1080-fa07-4b53-be2b-572190b4cecf none swap defaults 0 0Whether I enabled only the <systemroot>.mount unit file or all of them, however, mkinitcpio was still not having it without there being a /usr/bin/fsck.none binary.
After creating that binary it built without the need to modify the the add_fsck function in /usr/lib/initcpio/install/fsck — it did it without that before though, so that's just for the sake of clarity, and, furthermore, last time I edited the file to tell it it to add a symlink, I noticed that, unlike /usr/bin/fsck.ext2 and /usr/bin/fsck.ext3, the fsck.none binary got included in the initramfs anyway, so I'm not sure how significant it is to make that change.
It made no difference whether only the <systemroot>.mount unit file was enabled or all of them — it built successfully, so long as the binary existed.
After reboot, it got further this time.
Sadly though:
Starting version 249.6-3-arch
Org_000: clean, [...] files [...] blocks
ERROR: Root device mounted successfully, but /sbin/init does not exist
Bailing out [...]So, seemingly, having both the systemd and the usr hooks in mkinitcpio.conf causes a(nother) problem.
By boot options I meant what options is the boot loader passing to kernel.
GRUB.CFG
### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
[...]
insmod part_gpt
insmod ext2
set root='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2 782c7e72-b0f5-4e30-aa8e-280dfa1c117e
else
search --no-floppy --fs-uuid --set=root 782c7e72-b0f5-4e30-aa8e-280dfa1c117e
fi
font="/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr/share/grub/unicode.pf2"
fi
[...]
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-52c5acf5-5101-45cc-9097-8043584493f3' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
set root='hd0,gpt4'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt4 --hint-efi=hd0,gpt4 --hint-baremetal=ahci0,gpt4 e2dd56d1-fc2c-4e2e-a293-f62a2cd3364d
else
search --no-floppy --fs-uuid --set=root e2dd56d1-fc2c-4e2e-a293-f62a2cd3364d
fi
echo 'Loading Linux linux ...'
linux /vmlinuz-linux root=UUID=52c5acf5-5101-45cc-9097-8043584493f3 rw loglevel=3 quiet
echo 'Loading initial ramdisk ...'
initrd /intel-ucode.img /amd-ucode.img /initramfs-linux.img
}
[...]That's what you meant, yes?
What are the differences between the good and bad initrd now?
I exported the initramfs image with
1 no /usr/bin/fsck.binary
.1.1 all mount unit files enabled and
.1.2 just the systemroot unit file enabled
2 a /usr/bin/fsck.binary
.2.1 all mount unit files enabled and
.2.2 just the systemroot unit file enabled
I could diff them and give you the results, but I suspect it'd go quicker, if I just gave you an archive and you looked at them yourself, rather than waiting for me to not supply you with what you're really looking for — up to you, let me know which you prefer.
Offline
Remove all the .mount unit files for the bind mounts.
Comment out all the bind mount lines in the fstab. That should hopefully stop the fsck errors.
What is the contents systemroot unit?
Edit:
With respect to the full implementation you are not on using PXE over wireless? As my recollection is that was never implemented.
Last edited by loqs (2021-11-14 19:35:57)
Offline
Remove all the .mount unit files for the bind mounts.
Comment out all the bind mount lines in the fstab. That should hopefully stop the fsck errors..
Okay.
I'm uncertain how the system will know where to instatll/update files after that though — wont it just try to install/update to /etc, /usr and so on, instead of the target destination of the bind mounts?
That's not insignificant, as (along with the solutions themselves), I need to be able to demo that updates to the solutions are are seamless too by actually applying an update process and part of it is that the tools reside in the same location(s) as other files on the system(s) — if you install your binaries to <network>/device_0000/it/profiles/systems/local/hosts/host_00000/usr then my tools will need to be installed to <network>/device_0000/it/profiles/systems/local/hosts/host_00000/opt/<my tools> (or wherever you choose to install them) ... it can't rely upon your having to install and maintain them on an external host (it needs to be self-referential, if you see what I mean).
What is the contents systemroot unit?
srv-org_000-domain_000.mount
[Unit]
Description=Main Domain_000 Mount
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target
[Mount]
What=/dev/disk/by-uuid/cafda08e-85b5-426f-915a-a82844d3bf88
Where=/srv/org_000/domain_000
Type=ext4
Options=defaults,errors=remount-ro
[Install]
WantedBy=initrd-root-device.targetEdit:
With respect to the full implementation you are not on using PXE over wireless? As my recollection is that was never implemented.
Ultimately, that won't be my problem: how people manage their logins is up to them, I just need thing to work, once users have logged in; which is why being able to demo things on a single drive would make life a lot simpler for me, rather than run the gauntlet of glitches and hitches thanks to needing to configure my system to log into theirs first (or transfer it all to a system they have, if they even have one available for that purpose, which is unlikely, I imagine) — I can just brush that aspect aside and get on with demoing how my stuff works, once users are logged in.
Offline
Removing that one file allows the system to boot?
I want to isolate what single change is triggering the boot failure.
Offline
Removing that one file allows the system to boot?
I don't see how it can.
There's nothing on the systemroot to boot — all the files are in the bind mounted directories, the systemroot just contains empty directories as mount points for them.
So, it'll mount the boot and systemroot partitions, bind the Domain_000 partition to the /srv/org_000/domain_000 directory and then ...
What?
There's nothing in /usr to include in the initramfs-linux.img and no /etc/mkinitcpio.conf to tell it what to put in it anyway.
I want to isolate what single change is triggering the boot failure.
I mean, I'll give it a go anyway, but I don't think anything's going to happen other than error messages about missing files (and not just fsck.none, but everything it wants for the initramfs).
Offline
I do not understand what changes you are making that causes the system to stop booting.
Post #19 made it no clearer to me.
Why would there be error messages about missing files other than fsck.none coming from the bind mount entries in the fstab?
Edit:
As there are .mount units for all the bind mounts why are there fstab entries for them? That would create more .mount units with different paths as systemd has now switched root and seem to invite more issues and the entries can not be used by the time the fstab is processed.
Edit2:
Create a second install of Arch on a usb thumb drive. Add a boot entry for that install (it could optionally uses the main installs kernel and fallback image). That fulfills the POC of switching installations.
Last edited by loqs (2021-11-14 21:29:23)
Offline
Maybe this will make it clearer.
There is a harddrive with 4 main partitions
[SWAP]
Domain_000
Org_000
Boot
On Boot there are the usual boot files — I have to boot my laptop.
On Org_000 there are the standard directories (from /boot to /var). They are empty — not one single file in any of them.
On Domain_000 there are three folders: data_local, data_share_local and data_share_public, representing three SANs.
data_local is org internal stuff (HR, Finance, internal IT stuff, Legal, Sales data, you get the idea)
data_share_local is stuff that is shared within the organisation and to which more people need access (Marketing materials and assets, employee information like the company handbook, technical data for support team members, contact data for sales partners/suppliers, stuff that isn't confidential, but not for outsiders)
data_share_public is for material that can either be made public or non-confidential information acquired from third-party sources (it's also how things like webservers and intranet servers are securely accessed from within the domain, without employees needing to go out via the Internet to do so).
However many separate networks/SANs you have, no matter how physically isolated from each other, I can accommodate that (Hell, I can even accommodate you, if you're a server farm and you don't just host business clients, you host other server farms!). But to demonstrate that, all I need is three (one for each type of organisational structure you might have: confidential, internally shared and public).
Because experience has long since taught me to be prepared, I know in advance, that being able to accommodate awkward questions like "What if our employees use thin clients?" or "What if we want to put it all in the 'cloud'?" is important, so I want to be prepared for that eventuality; if their employees use thin clients or log in remotely and pick up their roaming profile, or just connect to a cloud server somewhere, I can't turn to them and say "Well, sorry, I can't accommodate that, independent machines with full installation of the OS and apps only," or when they ask "Well, okay, but what if we want to install and run your solution that way as well?" reply with "Oh,you can't do that, you have to have a dedicated machines to run it separately" — I'll get (at best) a polite "Thanks, we'll let you know" (meaning "Goodbye, see you never.").
So, as I said, rather than traipse in with three NAS boxes, a router, a PXE boot/LDAP server and a lot of ugly cabling and try to look good in the boardroom (which that never will), or try to convince a potential client to set that up for me in advance (and then waste time installing stuff to them, configure them, copy over example data, etc.), I want to emulate it with my single Domain_000 partition and three folders.
I don't actually need to log in because, as I said, that's not my problem, all I need to do is demonstrate how things work when someone does log in.
For the purposes of what we are discussing, the only thing that matters is the -data_local filletree.
The directories below that represent physical devices on which data is stored (RAID boxes, JBoD 'arrays', NAS boxes, whatever).
Below those, are actual data organised however you like: business, finance, hr, it, marketing, sales, whatever you like.
And, as an example, I have created a structure that shows how a thin client, or remote login setup might be organised:
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000 (with /host_00000 representing an entire device image).
And Domain_000/data_local/device_0000/it/profiles/remote/users/user_00000 as an example of organising things by user/role instead of hardware.
If, for example, you want to have admin users log into a device profile and then administer the system from it, using my solution, my solution has to be accessible from exactly the same place in exactly the same way.
So, I need to install my own operating system, on my own laptop, to that structure.
My laptop has to boot, therefore, not from the system partition but from the host_00000 profile.
Just for the purpose of not shafting myself entirely, I'm keeping my /boot partition as the real thing — first of all, it represents the login process (the user logs in somehow and I boot my laptop somehow) and, secondly, it means I have a fighting chance of getting my laptop booted if the worst happens and I need to drop to a GRUB command line).
Everything else, however, needs to be as though I were logging in from a thin client, or picking up a roaming profile remotely/from a cloud server.
So, I have to install my Arch system and my own solution/tools to
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/etc
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/home
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/opt
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/root
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/usr
Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/var
... which is enough to demonstrate that it can accommodate that kind of setup with my solution, not just fat hosts connected to a network or VPN.
Therefore, there is nothing on the system partition for mkinitcpio to find and include in the initramfs — it's all in Domain_000/data_local/device_0000/it/profiles/systems/local/hosts/host_00000/<subdirectoy>
is that clearer now?
Last edited by BeastlyDoglick (2021-11-14 23:15:56)
Offline