You are not logged in.

#1 2019-10-14 05:49:09

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

zfs-import-cache refuses to mount two out of my 3 zpools

This is pretty infuriating and it must be something related to the way I have the zpools configured because at first I thought it was Ubuntu being stupid, but it happens in Arch as well.

I have 3 zpools: nvme, hgst, and storage and for some reason it only ever imports nvme automatically. I have everything that I needed started but it still refuses to mount the other two

I took a look at the journal for zfs-import-cache and it says this, which is confusing to me because they definitely exist

Oct 14 01:36:01 server systemd[1]: Starting Import ZFS pools by cache file...
Oct 14 01:36:01 server zpool[1073]: cannot import 'hgst': no such pool or dataset
Oct 14 01:36:01 server zpool[1073]: cannot import 'storage': no such pool or dataset
Oct 14 01:36:01 server zpool[1073]:         Destroy and re-create the pool from
Oct 14 01:36:01 server zpool[1073]:         a backup source.
Oct 14 01:36:01 server zpool[1073]:         Destroy and re-create the pool from
Oct 14 01:36:01 server zpool[1073]:         a backup source.
Oct 14 01:36:01 server systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 01:36:01 server systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Oct 14 01:36:01 server systemd[1]: Failed to start Import ZFS pools by cache file.

Importing them manually works perfectly

 [bran@server ~]$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nvme   952G   328G   624G        -         -    18%    34%  1.00x    ONLINE  -

 [bran@server ~]$ sudo zpool import -d /dev/disk/by-id storage

 [bran@server ~]$ zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nvme      952G   328G   624G        -         -    18%    34%  1.00x    ONLINE  -
storage    65T  47.3T  17.7T        -         -     1%    72%  1.00x    ONLINE  -

 [bran@server ~]$ sudo zpool import -d /dev/disk/by-id hgst

 [bran@server ~]$ zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
hgst     10.9T   330M  10.9T        -         -     0%     0%  1.00x    ONLINE  -
nvme      952G   328G   624G        -         -    18%    34%  1.00x    ONLINE  -
storage    65T  47.3T  17.7T        -         -     1%    72%  1.00x    ONLINE  -

I also have docker running, that reads from and writes to the storage pool, so even when they're not imported, docker starts and creates the directories under /mnt/storage and then when I go to import the storage pool it complains that the directory isn't empty so I need to stop docker, do sudo rm -rf /mnt/storage/* then import the pools and restart docker.

So first off, why does zpool-import-cache complain that the pools don't exist, when they clearly do, and secondly, how can I make it so that docker doesn't start until the pools are successfully imported?

I've been using ZFS for years and have never had this problem.

Last edited by brando56894 (2019-10-14 05:53:35)

Offline

#2 2019-10-14 06:21:33

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: zfs-import-cache refuses to mount two out of my 3 zpools

I can't help you with the import (I actually have the same problem on an Ubuntu host at the moment, but haven't figured it out yet).
For the second part though:

brando56894 wrote:

how can I make it so that docker doesn't start until the pools are successfully imported?

Add this to your docker systemd service(s):

[unit]
After=zfs-import-cache.service
Requires=zfs-import-cache.service
RequiresMountsFor=/mnt/storage

You can put that in an override file (eg `systemctl edit docker`) so you're not blatting the distribution unit file (that will get overwritten during update)

Offline

#3 2019-10-14 09:05:52

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

Re: zfs-import-cache refuses to mount two out of my 3 zpools

I can't help you with the import (I actually have the same problem on an Ubuntu host at the moment, but haven't figured it out yet).

Interesting! Maybe ZoL has a bug in the new version?

Thanks for the systemd unit, I knew it had to be something like that!

Offline

#4 2019-10-14 23:50:22

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: zfs-import-cache refuses to mount two out of my 3 zpools

I actually spent some more time looking at this after your post yesterday. In my case, it appears my HP "Smart" Array HBA is delaying initialization of the drives until too late in the boot process - so probably not related to your issue.  Someone did suggest removing /etc/zfs/zpool.cache as theirs had gotten out-of-date; it didn't help me, but maybe worth trying for you?

Offline

#5 2019-10-15 04:39:15

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

Re: zfs-import-cache refuses to mount two out of my 3 zpools

That may actually be my issue as well then because I'm using a 4 port LSI HBA. For the past few years I've been using server and workstation Intel boards, and would always see the HBA initialization screen since it would take a good 30 seconds for the RAM checks and everything else to load, before even the initial splashscreen loaded, then the HBA would initialize, show me the splashscreen again, then proceed to boot Linux.

I bought an AMD Threadripper 2970x when it was on sale for $500 off like 2 months ago but didn't have any of the other necessary hardware for it. I wanted a server/workstation board that fit my needs, but none were available. I did find one but it had no release date, so I ended up getting a high end gaming board which had pretty much everything else, except IPMI. It boots up cold to the login prompt in about 30 seconds, all I see is the initial splashscreen, then immediately to GRUB, and then it loads Linux, no screens about the HBA initializing, which I thought was a little odd, but chalked it up this being a prosumer/gamer board and not a workstation board. Also I haven't had a system running AMD hardware in probably about 15 years haha

Of course about a month later they released that server board I was looking at!

I deleted my cache as well, but that didn't fix anything. Since the NVMe drives are M.2 and connected directly to the motherboard that seems to be the reason, since the drives in the other two pools are all connected to the HBA since they're 3.5" WD Reds.

I guess I'll have to turn off quick boot and/or figure out other ways to delay the boot process.

Would it be possible to have the zfs-mount service delayed until the end of the boot process instead of having it run shortly after the kernel is loaded?

Last edited by brando56894 (2019-10-15 04:50:10)

Offline

#6 2019-10-15 05:07:29

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: zfs-import-cache refuses to mount two out of my 3 zpools

Offline

#7 2019-10-15 05:21:12

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

Re: zfs-import-cache refuses to mount two out of my 3 zpools

That may actually work, not sure what the subsystem name would be though. All my HBA drives do have the "WorldWide Name" that start with wwn-0x5 so it could probably match to that or the standard ata-modelname. My OS drive is a 256 GB NVMe M.2 so it won't mess with that.

Edit: also that docker unit modification didn't work, docker is still running even though only one pool is available :-/

Last edited by brando56894 (2019-10-15 09:01:35)

Offline

#8 2019-11-03 19:50:28

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

Re: zfs-import-cache refuses to mount two out of my 3 zpools

@fukawi2 There's actually a really simple solution to this in the Arch ZFS wiki that we seemed to both have missed haha

All you have to do is add options scsi_mod scan=sync to /etc/modprobe.d/zfs.conf and then regenerate your initrd. This halts the boot process until your HBA is initialized and your disks are available. I just tested it out and it works perfectly!

Last edited by brando56894 (2019-11-03 19:58:33)

Offline

#9 2019-11-03 23:53:38

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: zfs-import-cache refuses to mount two out of my 3 zpools

Thanks! I'll give that a try now smile

Offline

#10 2019-11-04 02:26:22

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

Re: zfs-import-cache refuses to mount two out of my 3 zpools

Sure thing buddy! Now I don't have to buy a new motherboard haha

Offline

#11 2019-11-04 06:34:51

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: zfs-import-cache refuses to mount two out of my 3 zpools

Well FWIW, that did not work on my Ubuntu server, but this isn't the place for me to investigate that so I'll leave it at that for now.

Offline

#12 2019-11-04 07:01:05

brando56894
Member
From: NYC
Registered: 2008-08-03
Posts: 681

Re: zfs-import-cache refuses to mount two out of my 3 zpools

Really? That's pretty interesting. I wonder why it works for you and not me...

Offline

#13 2021-12-14 15:53:10

AveryFreeman
Member
Registered: 2017-11-25
Posts: 27

Re: zfs-import-cache refuses to mount two out of my 3 zpools

It's interesting, I keep hearing this thing about delaying the boot process because people assume that zpool.cache represents all your pools

In fact, if you run strings /etc/zfs/zpool.cache you can see it's only one pool at a time

you can create a cache file for each one of your pools manually by stopping the zfs-import-cache.service, deleting /etc/zfs/zpool.cache and making new ones with

zpool set cachefile=/etc/zfs/<yourCacheNameDecision>.cache <yourPoolNameYouWantToCache>

Problem is, if you cat /lib/systemd/system/zfs-import-cache.service you can immediately see the systemd unit is hard-coded to a .cache file named /etc/zfs/zpool.cache and has no apparent function for loading more than one.

It should be easy enough to create a script that loads whatever .cache file is in whatever directory you want to create them in using the current zfs-import-cache.service file as a template.

Or just edit the line:

ExecStart=/sbin/zpool import -c /etc/zfs/<cacheFileOne>.cache -aN && /sbin/zpool import -c /etc/zfs/<cacheFileTwo>.cache -aN && . . .  

I'm not sure if this is valid, as I have very little experience creating unit files, but in bash you can do a simple one-liner for loop:

ExecStart=for i in /etc/zfs/*.cache; do /sbin/zpool import -c /etc/zfs/"$i" -aN; done

Last edited by AveryFreeman (2021-12-14 16:05:37)

Offline

#14 2021-12-20 00:07:52

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: zfs-import-cache refuses to mount two out of my 3 zpools

I no longer work for the company with the Ubuntu server and the missing pool, but it only had a single pool so still don't think that would have solved it.

ExecStart=for i in /etc/zfs/*.cache; do /sbin/zpool import -c /etc/zfs/"$i" -aN; done

`ExecStart` isn't run with a shell, so you'd need to wrap the command in a script, or just bash -c

Offline

Board footer

Powered by FluxBB