Arch Linux

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

[SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

EDIT: changed the procedure (edited bellow), tried again and failed at

 grub-mkconfig -o /boot/grub/grub.cfg 

with error

 /usr/bin/grub-probe: error: failed to get canonical path of '/dev/ata-VBOX_HDD_SOME_UUID-part1' 

now I guess this is my main problem and what really doesn't make sense is the weird /dev/UUID notation in the error as I don't have any idea where it came from, maybe from

zpool create zroot /dev/disk/by-id/id-to-partition

?

Hello,
so I wanted to tinker with ZFS and I decided to try it within VirtualBox as to get tho whole installation procedure right before trying it on live machine, but after a reboot it newer loads, it just opens the VirtualBox UEFI shell.

I successfully created Archiso with zfs-git as noted in wiki.

Setup Partitions (all of them GPT, I doublechecked)
/dev/sda - 8GB virtualbox disk (place to create all those ZFS pools and stuff)
/dev/sda1 - set with gdisk as bf00 (not formated in any way as the wiki said)
/dev/sdb - 2GB external usb stick (place to instal GRUB)
/dev/sdb1 set with gdisk as ef00
then mkfs.fat -F32 /dev/sdb1

Setup VirtualBox
In setting I checked on "Enable EFI" as to emulate UEFI system and also "PAE/NX".
I mounted the Archiso with VirtualBox, it worked as I was able to boot and begin the install process.

The actual installation process is bellow, so please, any suggestions and improvements are welcomed, so far I suspect that the VirtualBox is not able to boot grub from external usb stick.

wipefs -a /dev/sda
wipefs -a /dev/sdb
gdisk /dev/sda		##new  gpt + set bf00 
gdisk /dev/sdb		##new  gpt + set ef00 
mkfs.fat -F32 /dev/sdb1

zpool create zroot /dev/disk/by-id/id-to-partition

zfs create zroot/home -o mountpoint=/home
zfs create zroot/root -o mountpoint=/root ## cannot mount, dir not empty
## zfs create zroot/var -o mountpoint=/var

zfs set mountpoint=/ zroot

zfs set mountpoint=/home zroot/home
zfs set mountpoint=/root zroot/root
## zfs set mountpoint=/var zroot/var

## zfs set xattr=sa zroot/var
## zfs set acltype=posixacl zroot/var

zpool set bootfs=zroot zroot
zpool export zroot
zpool import -d /dev/disk/by-id -R /mnt zroot

## cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

mkdir /mnt/boot
mount /dev/sdb1 /mnt/boot

zpool set cachefile=/etc/zfs/zpool.cache zroot

pacstrap -i /mnt base base-devel
## genfstab -U /mnt >> /mnt/etc/fstab
genfstab -U -p /mnt
genfstab -U -p /mnt >> /mnt/etc/fstab
nano /mnt/etc/fstab ## comment out zroot

arch-chroot /mnt /bin/bash

echo '[demz-repo-archiso]' >> /etc/pacman.conf
echo 'SigLevel = Required DatabaseOptional TrustedOnly' >> /etc/pacman.conf
echo 'Server = http://demizerone.com/$repo/$arch' >> /etc/pacman.conf
pacman-key -r 0EE7A126
pacman-key --lsign-key 0EE7A126
pacman -Syy
pacman -S zfs-git
modprobe zfs

rm /etc/locale.gen
touch /etc/locale.gen
echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen
locale-gen

touch /etc/locale.conf
echo 'LANG=en_US.UTF-8' >> /etc/locale.conf

nano /etc/mkinitcpio.conf >> HOOKS="base udev autodetect modconf block keyboard zfs filesystems"
mkinitcpio -p linux

pacman -S grub efibootmgr
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=grub --recheck
grub-mkconfig -o /boot/grub/grub.cfg

xxxxxxxxxxxxxxx

touch /etc/hostname
echo 'a123' >> /etc/hostname

passwd
exit

umount /mnt/boot
zfs umount -a
zpool export zroot
##umount -R /mnt
reboot

Last edited by avojevlavo (2016-01-28 00:51:41)

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

In the wiki article the “id-to-partition” is in italics, which means you have to replace that with the ID of the partition you want to use.
However, since you are dedicating the whole disk to zfs, you'll get better performance by specifying the disk's ID instead of the partition's ID (and if you do that, you don't need to partition the disk as zfs will do that).

If you do dedicate the whole disk to zfs I personally, since your boot disk is a small thumb drive, would use BIOS booting with syslinux

Head_on_a_Stick

Member

From: London

Registered: 2014-02-20

Posts: 7,732

Website

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

/dev/sda1 - set with gdisk as bf00 (not formated in any way as the wiki said)

Please provide a link to this ArchWiki statement.

Partition type BF00 is defined as "Solaris root"

See http://www.rodsbooks.com/gdisk/walkthrough.html

EDIT: Ooops, sorry for the noise

*slinks back into shadows*

Last edited by Head_on_a_Stick (2016-01-19 14:38:55)

---

Are you infected with Wetiko?

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@Head_on_a_Stick The OP has already linked to the relevant wiki article. However, the partition type is not the issue here - since we're dealing with ZFS, Solaris root is the correct partition type for the data disk.

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@Head_on_a_Stick -- wiki link, section partition scheme BIOS with GPT, however, I adjusted that since I am doing UEFI install, so no 2M BIOS boot partition (ef02) or 512M Ext boot partition (8300).

@ukhippo, yes I noticed the italics, and what I actually typed as a command was something like

zpool create zroot /dev/disk/by-id/ata-VBOX_HDD_SOME_UUID-part1

and see your point, will try the whole disk method.
I was thinking about syslinux since GRUB is giving me the finger in this instance, how would one boot syslinux on a usb stick?

Honestly this bootloader thing makes me cringe, as I'm not certain how after mkinitcpio with ZFS hook does the kernel (that was saved to /boot) know which disk to address as to look for ZFS filesystem.

mouseman

Member

From: Outta nowhere

Registered: 2014-04-04

Posts: 291

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

you'll get better performance by specifying the disk's ID instead of the partition's ID

Don't mean to derail the thread, but do you have a reliable source that explains that in more detail?

As far as I'm aware as long as you align your partitions and run a recent ZFS version that detects and configures ashift properly, I see no reason why it would matter whether you'd use an entire disk or just a partition.

Thanks!

tom.ty89

Member

Registered: 2012-11-15

Posts: 897

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

VBOX_HDD_SOME_UUID

That's not really UUID, but model and serial number of the drive:

[tom@localhost ~]$ ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Jan 20 10:18 ata-INTEL_SSDSA2MH080G1HP_CVEM843100XN080DGN -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 20 10:18 ata-INTEL_SSDSA2MH080G1HP_CVEM843100XN080DGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 20 10:18 ata-INTEL_SSDSA2MH080G1HP_CVEM843100XN080DGN-part2 -> ../../sda2
lrwxrwxrwx 1 root root  9 Jan 20 17:15 ata-INTEL_SSDSC2BW240A4_CVDA339603M02403GN -> ../../sdb
lrwxrwxrwx 1 root root  9 Jan 20 10:18 ata-PIONEER_BD-RW_BDR-S08 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jan 20 10:18 wwn-0x5001517387d62d33 -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 20 10:18 wwn-0x5001517387d62d33-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 20 10:18 wwn-0x5001517387d62d33-part2 -> ../../sda2
lrwxrwxrwx 1 root root  9 Jan 20 17:15 wwn-0x55cd2e404b836150 -> ../../sdb

I doubt that it has anything to do with the zpool command. Not sure if grub(-mkconfig) supports ZFS nicely though.

What's the exact grub-install command you ran?

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

I was thinking about syslinux since GRUB is giving me the finger in this instance, how would one boot syslinux on a usb stick?

Honestly this bootloader thing makes me cringe, as I'm not certain how after mkinitcpio with ZFS hook does the kernel (that was saved to /boot) know which disk to address as to look for ZFS filesystem.

The syslinux wiki page https://wiki.archlinux.org/index.php/Syslinux gives full install details; do them at the point in the install where you installed grub. For BIOS booting, all I do is:

pacman -S syslinux mtools gptfdisk
syslinux-install_update -i -a -m

Then you just need to edit/replace /boot/syslinux/syslinux.cfg and add entries. Again, the wiki page has examples.
For zfs, instead of specifying “root=/dev/sdXN” just put “zfs=POOLNAME” (e.g. zfs=zroot)

The zfs hook adds a runtime script to the initramfs, and that script does the pool import etc.

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

you'll get better performance by specifying the disk's ID instead of the partition's ID

Don't mean to derail the thread, but do you have a reliable source that explains that in more detail?

As far as I'm aware as long as you align your partitions and run a recent ZFS version that detects and configures ashift properly, I see no reason why it would matter whether you'd use an entire disk or just a partition.

Thanks!

It's been quite a few years since I last used Solaris, but the recommendation was always to use whole disks so that zfs could enable the hdd's write caching. Otherwise it did not use write caching in case the other partitions contained filesystems susceptible to corruption. I just assumed that was the case for zfsonlinux (my bad, should never assume!)

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@tom.ty89

That's not really UUID, but model and serial number of the drive:
What's the exact grub-install command you ran?

Yes, sorry its by-id as recomended in arch wiki on zfs.

The command was:

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=grub --recheck
grub-mkconfig -o /boot/grub/grub.cfg

@ukhippo
Thank you, that was informative, but when I have more disks in zroot as created with for example this:

zpool create zroot /dev/disk/by-id/id-to-disk1
zpool create zroot /dev/disk/by-id/id-to-disk2
zpool create zroot /dev/disk/by-id/id-to-disk3

Do I need to run mkinitcpio again, or the zfs hook just tells the kernel to look for solaris partitions and import zroot from any disk it finds with zroot(as defined by bootloader)?

pacman -S syslinux mtools gptfdisk
syslinux-install_update -i -a -m

But this is only for bios? It will be different for UEFI? (or do I have some bad assumptions?)

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

Thank you, that was informative, but when I have more disks in zroot as created with for example this:

zpool create zroot /dev/disk/by-id/id-to-disk1
zpool create zroot /dev/disk/by-id/id-to-disk2
zpool create zroot /dev/disk/by-id/id-to-disk3

Do I need to run mkinitcpio again, or the zfs hook just tells the kernel to look for solaris partitions and import zroot from any disk it finds with zroot(as defined by bootloader)?

You don't create multi-disk pools that way. You specify all the disks in a single command:

zpool create zroot /dev/disk/by-id/id-to-disk1 /dev/disk/by-id/id-to-disk2 \
     /dev/disk/by-id/id-to-disk3

If at a later time you want to add another disk to the pool, use zpool add:

zpool add zroot /dev/disk/by-id/id-to-new-disk

You shouldn't need to run mkinitcpio again, though doing so would update the zpool.cache file in the initramfs.

pacman -S syslinux mtools gptfdisk
syslinux-install_update -i -a -m

But this is only for bios? It will be different for UEFI? (or do I have some bad assumptions?)

The arch wiki page for syslinux I linked to in my post (https://wiki.archlinux.org/index.php/Syslinux) tells you how to do it for UEFI. Alternatively, use systemd-boot if you're sticking with UEFI.

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@ukhippo
I found the recommendation for using whole disks: http://zfsonlinux.org/faq.html#PerformanceConsideration
Syslinux wiki states that its not possible to use UEFI in VirtualBox, so I am out of luck, but I liked the install and config more than GRUBs, so I tried it with this old disk of mine 120gb (5400rpm) and it worked nicely Thanks you (and can you maybe share your syslinux.conf?)

So now I have other problem to solve, disk arrangement, migration and SLOG + L2ARC.

Question for a moderator, should I close this thread and mark it solved, even if it was solved with a workaround, and start a different thread to ask questions about ZFS disk arrangement, migration and SLOG+L2ARC? Or just continue it here?

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

Since you asked, this is my syslinux.cfg:

prompt 0
default arch
F1 syslinux.cfg
label arch
        linux ../vmlinuz-linux
        initrd ../intel-ucode.img,../initramfs-linux.img
        append zfs=ocean rw zfs_import_dir=/dev/disk/by-id
label fallback
        linux ../vmlinuz-linux
        initrd ../intel-ucode.img,../initramfs-linux-fallback.img
        append zfs=ocean rw zfs_import_dir=/dev/disk/by-id
label hdt
        com32 hdt.c32
label reboot
        com32 reboot.c32
label poweroff
        com32 poweroff.c32

This just boots straight into arch unless I force the 'boot:' prompt. At the prompt I can press F1 to view the contents of my syslinux.cfg (easiest way to see what's defined in it - seemed pointless having a separate "help" file to maintain). The 'zfs_import_dir' setting is a temporary fix because my system keeps setting the pool disks to be 'sdX' rather than the IDs. Keep meaning to find out why that is, but it's low priority.

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

Since you asked, this is my syslinux.cfg:

thanks anyway, but now I have different problem, as the first thing I do after arch-chroot I add do

echo '[demz-repo-core]' >> /etc/pacman.conf
echo 'SigLevel = Required DatabaseOptional TrustedOnly' >> /etc/pacman.conf
echo 'Server = http://demizerone.com/$repo/$arch' >> /etc/pacman.conf
pacman-key -r 0EE7A126
pacman-key --lsign-key 0EE7A126
pacman -Syy
pacman -S zfs-git
modprobe zfs

it installs, but modprobe zfs fails with:

modprobe: FATAL: Module zfs not found in directory /lib/modules/4.3.3-2-ARCH

is this a problem (guess so)? The second strange thing is after I create zpool it somehow adds a new partition sda9 (8MB, bf07 - solaris reserved 1), even though I partitioned the whole disk in gdisk as sda1 (solaris root and used zpool create with disk by-id), what is that about?

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

You need to follow the ordering in the wiki pages. Essentially:
You need to add zfs support to the running archiso kernel (install zfs from demz-repo-archiso)
Create zfs pool etc. and setup Zfs before you arch-chroot.
Inside the arch-chroot the only ZFS things you do are:
• setup pacman.conf to use demz-repo-core
• install the zfs-git package.
• add the zfs hook in mkinitcpio.conf before running mkinitcpio.

The "odd" partitioning is expected if you give zfs the whole disk - it repartitions the disk to zfs’s needs, which is why you don't bother partitioning if you are giving zfs the whole disk.

Last edited by ukhippo (2016-01-24 13:14:21)

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@ukhippo
Well still no luck, the same installation modprobe zfs problem.

I followed the steps from your last post except that I also ran the modprobe zfs command, sorry for stupid question but are you saying that it's not needed and that it can screw up the installation?

I finished the install anyway just to see what will happen, I boot to Uefi shell find the appropriate fs, used the ls command to see if the vmlinuz-linux is there and then just start it, result is: kernel panic - not syncing: VFS: unable to mount root fs on unknown block

So I guess I will try to recreate the archiso, right now I am using week old image, and I am almost 99% sure I used the demz-repo-archiso because the modprobe zfs didn't occur the first time I tried the installation in virtualbox.

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

When Arch-chrooted I run

systemctl
Failed to connect to bus: no such file or directory

Why is this happening?
/tears hair off

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@ukhippo
Well still no luck, the same installation modprobe zfs problem.

I followed the steps from your last post except that I also ran the modprobe zfs command, sorry for stupid question but are you saying that it's not needed and that it can screw up the installation?

Why are you trying to modprobe in the chroot? What do think it will achieve?
It won't screw up the installation in this case, because the command fails.

I finished the install anyway just to see what will happen, I boot to Uefi shell find the appropriate fs, used the ls command to see if the vmlinuz-linux is there and then just start it, result is: kernel panic - not syncing: VFS: unable to mount root fs on unknown block

you just can't execute vmlinuz-linux; you have to pass the kernel parameters and initrd as well.
See the EFISTUB wiki page and in particular the Using UEFI Shell section

When Arch-chrooted I run

systemctl
Failed to connect to bus: no such file or directory

Why is this happening?
/tears hair off

The chroot isn't a fully running system; its purpose is to setup enough so you can boot into it. You complete the rest of the installation in the booted system.

Last edited by ukhippo (2016-01-25 14:40:51)

avojevlavo

Member

Registered: 2010-05-27

Posts: 40

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

@ukhippo

Why are you trying to modprobe in the chroot? What do think it will achieve?
It won't screw up the installation in this case, because the command fails.

The chroot isn't a fully running system; its purpose is to setup enough so you can boot into it. You complete the rest of the installation in the booted system.

Yes and yes, this is what I get for installing when I should be sleeping, not thinking.

you just can't execute vmlinuz-linux; you have to pass the kernel parameters and initrd as well.
See the EFISTUB wiki page and in particular the Using UEFI Shell section

Yes, It works nicely now (I first booted manually from the UEFI shell to verify that ZFS is ok and than tinkered with syslinux), the one odd thing is that in syslinux.cfg I had to use

linux /vmlinuz-linux
initrd /initramfs-linux.img
append zfs=zroot rw

instead of yours config (note the lack of .. )

linux ../vmlinuz-linux
initrd ../intel-ucode.img,../initramfs-linux.img
append zfs=ocean rw

Marking as SOLVED (even though I didn't manage the virtualbox and GRUB setup), thank you ukhippo you have been very helpfull, have a nice day

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: [SOLVED] Install procedure wrong, Arch in VBox+GRUB+ZFS+UEFI

Could be a difference between UEFI and legacy (BIOS) booting - though my system has a UEFI firmware, I use legacy booting.