Arch Linux

predmijat

Member

Registered: 2014-09-30

Posts: 39

[SOLVED] zfs-mount.service - cannot mount: directory is not empty

Hey,

First some history:

- I've updated the kernel to 4.5.2-1 without checking if appropriate ZFS version is available (dumb, I know, but I got spoiled by demz-repo)
- System boots, but obviously can't mount ZFS
- I downgrade linux, linux-headers, and linux-api-headers, rebuild nvidia package and everything looks good: system boots, I get graphical environment, ZFS datasets are mounted...except not really...

systemctl --failed --all
  UNIT              LOAD   ACTIVE SUB    DESCRIPTION
● zfs-mount.service loaded failed failed Mount ZFS filesystems

sudo systemctl start zfs-mount.service
Job for zfs-mount.service failed because the control process exited with error code. See "systemctl status zfs-mount.service" and "journalctl -xe" for details.

and

sudo journalctl -xe

***

-- Unit zfs-mount.service has begun starting up.
May 08 22:03:22 hqr-workstation zfs[24512]: cannot mount '/storage': directory is not empty
May 08 22:03:22 hqr-workstation systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
May 08 22:03:22 hqr-workstation systemd[1]: Failed to start Mount ZFS filesystems.

***

Why is this happening? I can access every single dataset under /storage, I have services which use some of those datasets running with no issues, but this error message is bugging me...can someone explain why this happened, and how to fix it?

Thanks

edit: I've reinstalled zfs-linux-git, zfs-utils-linux-git, spl-linux-git, and spl-utils-linux-git, rebooted, but zfs-mount.service still fails.

Last edited by predmijat (2016-05-09 12:50:19)

drozdu

Member

Registered: 2012-02-20

Posts: 13

Re: [SOLVED] zfs-mount.service - cannot mount: directory is not empty

I had exactly the same problem yesterday.
Just check your zfs mount point. This directory should not exist.
In my case pacman created package cache inside it.

predmijat

Member

Registered: 2014-09-30

Posts: 39

Re: [SOLVED] zfs-mount.service - cannot mount: directory is not empty

That helped a bit. I had a Docker container started with -v /storage/services/gitlab, and "zfs mount -a" was complaining about that particular path. I'm guessing that when ZFS failed to start (after my kernel upgrade), Docker created that path anyway, which in turn prevented ZFS to mount it after I downgraded kernel.

I removed traces of that container, but "zfs mount -a" would still complain about /storage. I didn't know how to locate which files were also created, so I upgraded the kernel again, rebooted (ZFS fails to start), did a "ls -R" on /storage and it listed all datasets just under /storage (depth 1, so nothing under that first level) - is that how it should be even though ZFS didn't start?

"tree *" on /storage gives 0 directories and 0 files, so I'm not sure what is causing the "cannot mount '/storage': directory is not empty" message when I do a "zfs mount -a".

I also removed all symlinks to /storage/* I had in my home directory since I remember that I had some troubles with managing datasets which had symlinks, not sure what it was.

edit: success!

Here's what happened - after kernel upgrade, ZFS couldn't start, but the Docker container I've mentioned (not just that, I also had some other stuff symlinked to some datasets) started and created stuff on /storage/...whatever

I've disabled zfs.target service, rebooted, noted that everything under /storage is empty, did a "mv /storage /storage.bak", and started zfs.target without issues! After I had ZFS working I could reenable all services.

I'm marking this as solved, but I'd like to hear how to handle this in the future - I don't want to mess up ZFS every time it fails to start...do I need to edit each service not to start unless ZFS started too? What is the best way to do this?

Last edited by predmijat (2016-05-09 12:53:09)

Koopa

Member

Registered: 2012-07-20

Posts: 19

Re: [SOLVED] zfs-mount.service - cannot mount: directory is not empty

- I've updated the kernel to 4.5.2-1 without checking if appropriate ZFS version is available (dumb, I know, but I got spoiled by demz-repo)

If you were using the demz-repo before you might like to know that it moved on (or around) April 25th to archzfs.

predmijat

Member

Registered: 2014-09-30

Posts: 39

Re: [SOLVED] zfs-mount.service - cannot mount: directory is not empty

I'm using archzfs repo, I've mentioned demz-repo because it had hard links and prevented updating the kernel before it was safe.