You are not logged in.

#1 2015-04-17 19:10:07

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

[solved] ZFS on Linux does not detect pools on LVM

I started using ZFS on Linux just recently so I do not know whether the problem was introduced by version 0.6.4, but after reboot my pools remain undetected.

I just created a new ZFS pool called "bigdata" and run

zpool set cachefile=/etc/zfs/zpool.cache bigdata

This step is advised against by ZOL devs, but I didn't have cache when I first rebooted and the pool was still not detected.

Anyway, I followed the ZFS arch wiki to the letter and have the ZFS service running, but whenever I reboot, the pool does not mount automatically and if I run "zpool status" I get "no pools available".

Only after I run either

zpool import 
zpool import bigdata

or

zpool import -a -f

"zpool status" starts showing the pool and it gets mounted.
I create this thread because even talking to a dev at ZOL irc channel produced no solution.

For now, my workaround is to add to crontab

@reboot         sleep 10 ; zpool import -a -f

but this is not a fix and should not be necessary.

Last edited by Lockheed (2015-04-19 15:48:24)


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#2 2015-04-17 20:23:06

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed, thanks for opening the thread. A few questions:

  • What's the version of the ZFS packages that you use (e.g. pacman -Q | egrep '(spl|zfs)')?

  • What's the configuration of your pool? In particular, if I understand correctly, it's not a root pool, right? The output from zpool status and the contents of /etc/fstab might be helpful as well.

  • What's the output from systemctl status zfs.target; systemctl status zfs-mount.service; and systemctl status zfs-import-cache.service?

Cheers,
Luchesar


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#3 2015-04-17 20:34:08

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

# pacman -Q | egrep '(spl|zfs)'
spl-git 0.6.4_r0_gcd69f02_3.19.3_3-1
spl-utils-git 0.6.4_r0_gcd69f02_3.19.3_3-1
zfs-git 0.6.4_r0_gd07a163_3.19.3_3-1
zfs-utils-git 0.6.4_r0_gd07a163_3.19.3_3-1

The configuration of my pool is a data partition on secondary disk (my root is on btrfs).

# zpool status
  pool: bigdata
 state: ONLINE
  scan: resilvered 336G in 4h11m with 0 errors on Fri Apr 10 14:29:59 2015
config:

	NAME         STATE     READ WRITE CKSUM
	bigdata      ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    2T-vol2  ONLINE       0     0     0
	    2T-vol1  ONLINE       0     0     0

errors: No known data errors

Yes, this is a RAID1 mirror setup, however it has no bearing on the problem because I only created the mirror after I was unable to solve the automount issue. Orignally, the pool was just one partition (2T-vol2).

There is nothing in fstab that relates to this pool or mountpoint.

● zfs.target - ZFS startup target
   Loaded: loaded (/usr/lib/systemd/system/zfs.target; enabled; vendor preset: enabled)
   Active: active since Fri 2015-04-17 19:10:12 CEST; 3h 19min ago

Apr 17 19:10:12 nanoBox systemd[1]: Starting ZFS startup target.
Apr 17 19:10:12 nanoBox systemd[1]: Reached target ZFS startup target.
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/usr/lib/systemd/system/zfs-mount.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2015-04-17 19:10:10 CEST; 3h 19min ago
  Process: 670 ExecStart=/usr/bin/zfs mount -a (code=exited, status=0/SUCCESS)
 Main PID: 670 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/zfs-mount.service

Apr 17 19:10:10 nanoBox systemd[1]: Started Mount ZFS filesystems.
● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2015-04-17 19:10:10 CEST; 3h 19min ago
  Process: 496 ExecStart=/usr/bin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)
 Main PID: 496 (code=exited, status=1/FAILURE)

Apr 17 19:10:10 nanoBox zpool[496]: cannot import 'bigdata': no such pool or dataset
Apr 17 19:10:10 nanoBox zpool[496]: Destroy and re-create the pool from
Apr 17 19:10:10 nanoBox zpool[496]: a backup source.
Apr 17 19:10:10 nanoBox systemd[1]: zfs-import-cache.service: main process exited, code=exited, status=1/FAILURE
Apr 17 19:10:10 nanoBox systemd[1]: Failed to start Import ZFS pools by cache file.
Apr 17 19:10:10 nanoBox systemd[1]: Unit zfs-import-cache.service entered failed state.
Apr 17 19:10:10 nanoBox systemd[1]: zfs-import-cache.service failed.

So there is a problem with the cache service, but please bear in mind ZOL devs claim that mounting using cache is deprecated and should not be used (details: http://pastebin.com/raw.php?i=gSfn8tHB ).


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#4 2015-04-17 20:39:11

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Could you please also paste the output of systemctl status zfs-import-scan.service?


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#5 2015-04-17 20:48:26

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

# systemctl status zfs-import-scan.service -l
● zfs-import-scan.service - Import ZFS pools by device scanning
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-scan.service; static; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Fri 2015-04-17 19:10:10 CEST; 3h 37min ago
           ConditionPathExists=!/etc/zfs/zpool.cache was not met

Apr 17 19:10:10 nanoBox systemd[1]: Started Import ZFS pools by device scanning.

Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#6 2015-04-17 20:56:06

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Thanks a lot, Lockheed! I think I see what the problem might be: importing the pool by scanning /dev/disk/by-id is performed by zfs-import-scan.service, but that service won't start if the ZFS cache file exists -- regardless of whether it's used or, indeed, if it's a valid cache file at all.

Could you please rename or move /etc/zfs/zpool.cache and see if that makes a difference? I'm almost sure it will, but let's see.

Last edited by kerberizer (2015-04-17 20:57:29)


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#7 2015-04-17 21:24:04

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

Did that. The result after reboot:

# zpool status
no pools available

After running manually the usual zpool import -a -f, a new zpool.cache file was created.

Last edited by Lockheed (2015-04-17 21:26:47)


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#8 2015-04-17 21:28:11

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Could you please paste again the status of the zfs services, especially of zfs-import-scan?


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#9 2015-04-17 21:29:41

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

● zfs.target - ZFS startup target
   Loaded: loaded (/usr/lib/systemd/system/zfs.target; enabled; vendor preset: enabled)
   Active: active since Fri 2015-04-17 23:15:03 CEST; 14min ago

Apr 17 23:15:03 nanoBox systemd[1]: Starting ZFS startup target.
Apr 17 23:15:03 nanoBox systemd[1]: Reached target ZFS startup target.
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/usr/lib/systemd/system/zfs-mount.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2015-04-17 23:15:00 CEST; 14min ago
  Process: 676 ExecStart=/usr/bin/zfs mount -a (code=exited, status=0/SUCCESS)
 Main PID: 676 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/zfs-mount.service

Apr 17 23:15:00 nanoBox systemd[1]: Started Mount ZFS filesystems.
● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2015-04-17 23:15:00 CEST; 14min ago
  Process: 506 ExecStart=/usr/bin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)
 Main PID: 506 (code=exited, status=1/FAILURE)

Apr 17 23:15:00 nanoBox zpool[506]: cannot import 'bigdata': no such pool or dataset
Apr 17 23:15:00 nanoBox zpool[506]: Destroy and re-create the pool from
Apr 17 23:15:00 nanoBox zpool[506]: a backup source.
Apr 17 23:15:00 nanoBox systemd[1]: zfs-import-cache.service: main process exited, code=exited, status=1/FAILURE
Apr 17 23:15:00 nanoBox systemd[1]: Failed to start Import ZFS pools by cache file.
Apr 17 23:15:00 nanoBox systemd[1]: Unit zfs-import-cache.service entered failed state.
Apr 17 23:15:00 nanoBox systemd[1]: zfs-import-cache.service failed.
● zfs-import-scan.service - Import ZFS pools by device scanning
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-scan.service; static; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Fri 2015-04-17 23:15:00 CEST; 14min ago
           ConditionPathExists=!/etc/zfs/zpool.cache was not met

Apr 17 23:15:00 nanoBox systemd[1]: Started Import ZFS pools by device scanning.

Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#10 2015-04-17 21:42:22

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Hmmm, what's the current setting of the 'cachefile' pool property? Also, could you please paste the output of zdb -CC.

And another thing that comes to mind: when you renamed/moved the cache file, was the pool imported?


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#11 2015-04-17 21:50:05

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

Not sure what do you mean by "cachfile pool property.

# zdb -CC
bigdata:
    version: 5000
    name: 'bigdata'
    state: 0
    txg: 133916
    pool_guid: 5054081668121566837
    errata: 0
    hostid: 1431326176
    hostname: 'nanoBox'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 5054081668121566837
        children[0]:
            type: 'mirror'
            id: 0
            guid: 17406290375263017915
            whole_disk: 0
            metaslab_array: 35
            metaslab_shift: 33
            ashift: 12
            asize: 1000114487296
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 4638536950754333635
                path: '/dev/mapper/2T-vol2'
                whole_disk: 0
                DTL: 91
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 18348444556819304730
                path: '/dev/mapper/2T-vol1'
                whole_disk: 0
                DTL: 89
                create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

Yes, I renamed cache file when the pool was imported (i.e. visible and mounted).


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#12 2015-04-17 21:55:34

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed wrote:

Not sure what do you mean by "cachfile pool property.

Sorry, that would be zpool get cachefile bigdata.

Yes, I renamed cache file when the pool was imported (i.e. visible and mounted).

I think the cache file gets updated on export. I'm sorry if that gets boring, but could you please export the pool, then rename or move the cache file, and then reboot the system. The problem is that the last time we tested the cache file was still there when zfs-import-scan was run, and I suspect it was the export operation that had "resurrected" it.


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#13 2015-04-17 21:57:05

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

kerberizer wrote:

could you please export the pool, then rename or move the cache file, and then reboot the system

But let's first see what the current cachefile property is.


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#14 2015-04-17 23:31:57

demizer
Member
From: Gilroy, CA
Registered: 2010-03-03
Posts: 111
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

I have not run into this issue. FWIW, when I inspect zfs-import-scan.service, I get the same posted as above:

demizer@lithium ❯❯❯ syss zfs-import-scan.service
● zfs-import-scan.service - Import ZFS pools by device scanning
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-scan.service; static; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Wed 2015-04-15 19:05:55 PDT; 1 day 21h ago
           ConditionPathExists=!/etc/zfs/zpool.cache was not met

I am working on setting up a testing environment so we can test for stuff like this.

Offline

#15 2015-04-18 00:37:05

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Here are some key points that I see. Apologies for getting them in a bit raw form, but it's really getting very late here.

  • Lockheed hasn't enabled the ZFS hook, because his pool is not a root one, and thus the pool must be imported by the respective systemd services. This is not the case for many of us, who do have ZFS root pools, so we don't really get to test these particular services.

  • If I'm not wrong, Lockheed also uses dm-crypt (likely LUKS), which adds an additional layer of complexity. For example, before this commit, the dm devices could be unavailable at the time the zfs-import-{cache,scan} services fire.

  • The cache file is recreated on several occasions, including, I believe, on pool export. Hence why our first test above failed, because even though Lockheed had removed the cache file, when the system was rebooted, the file got recreated during the shutdown phase.

  • The cache file is still pretty much necessary for some setups. The reason is that the zfs-import-scan service simply runs /usr/bin/zpool import -d /dev/disk/by-id -aN, and so any devices managed by the device mapper, which show in /dev/mapper, will not be scanned. Unless I'm missing something important here, the only way to import dm-crypt devices at present is through the cache file by the zfs-import-cache service instead.

  • It's therefore important to understand why zfs-import-cache fails in Lockheed's case. Despite that after the abovementioned commit this service waits for the cryptsetup.target to activate, I wouldn't entirely rule out the possibility that the device mapper devices are for some reason unavailable at the time zpool import is run by zfs-import-cache.


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#16 2015-04-18 05:03:18

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

# zpool get cachefile bigdata
NAME     PROPERTY   VALUE      SOURCE
bigdata  cachefile  -          default

@kerberizer, you are correct, I am using dm-crypt on that drive.
This is my setup: one dm-crypt (LUKS) container -> LVM (2 partitions on one dm-crypt container) -> ZFS mirror


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#17 2015-04-18 09:41:03

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed wrote:

This is my setup: one dm-crypt (LUKS) container -> LVM (2 partitions on one dm-crypt container) -> ZFS mirror

OK, I see now what you had in mind explaining about RAID1. Using LVM between ZFS and dm-crypt seems rather unusual to me (ZFS is a volume manager on its own, after all) and I wasn't expecting it. But it could, in the end, be the culprit of the problem, because it's very possible that zfs-import-cache.service gets started after the dm-crypt devices get available, but before the LVM ones do.

As I seldom use LVM outside of root device contexts, could you please explain how have you set it up: is it enabled in the initramfs and/or via the lvm2-monitor systemd service?


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#18 2015-04-18 10:07:09

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

I understand ZFS is also volume manager, but I am using LVM because:
1 - when I was setting this up, I was unfamiliar with ZFS volume management. In fact, I still am.
2 - I need to be able to have the option to convert this raid1 ZFS setup into, for example, single ZFS + single XFS or BTRFS without the necessity of copying the data to any other disk, if in the future I decide to abandon ZFS. This is a remote system so I need to be able to do this when I am thousands of kilometres away.

I did no extra configuration of this system with regards to lvm other than what comes with stock arch iso. Here is the info I hope will answer your questions:

# systemctl status lvm2-monitor
● lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
   Loaded: loaded (/usr/lib/systemd/system/lvm2-monitor.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
# cat /etc/mkinitcpio.conf 
MODULES="btrfs radeon mmc_core mmc_block sdhci sdhci-pci lz4 lz4_compress"
BINARIES=""
FILES=""
HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

Last edited by Lockheed (2015-04-18 10:08:51)


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#19 2015-04-18 13:38:48

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed wrote:

I need to be able to have the option to convert this raid1 ZFS setup into, for example, single ZFS + single XFS or BTRFS without the necessity of copying the data to any other disk, if in the future I decide to abandon ZFS. This is a remote system so I need to be able to do this when I am thousands of kilometres away.

Yes, I see your point here. I'm wondering if putting a GPT or even an MBR wouldn't have provided a simpler and thus more robust solution. LVM certainly adds some undesirable complexity, especially for filesystems like ZFS and Btrfs, and will likely affect performance too.

Anyway, let's see if we can find a solution for your current construct -- hopefully one that will also be universal, i.e. not affecting people who don't use LVM.

I did no extra configuration of this system with regards to lvm other than what comes with stock arch iso. Here is the info I hope will answer your questions

Yes, you've read my mind, thanks!

Would it be convenient for you to do another test? Could you create the file /etc/systemd/system/zfs-import-cache.service.d/start-after-lvm.conf (the directory zfs-import-cache.service.d most certainly will need to be created as well) with the following contents:

[Unit]
After=lvm2-lvmetad.socket

I'm not really sure if that will delay the start of zfs-import-cache.service enough, because, I think, the LVM devices will appear for sure only after the lvm2-pvscan@{device} service runs (which actually simply does a pvscan on a device). But that's the best that comes to my mind right now. I'll try to make a test setup similar to yours meanwhile, but unfortunately can't promise on any specific time frame.

P.S. I just did a quick check on one server where I have LVM (but not beneath ZFS), and if the above doesn't work, you might give it another try by substituting lvm2-lvmetad.service for lvm2-lvmetad.socket in the After= section.


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#20 2015-04-18 16:38:45

demizer
Member
From: Gilroy, CA
Registered: 2010-03-03
Posts: 111
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed wrote:

I understand ZFS is also volume manager, but I am using LVM because:
1 - when I was setting this up, I was unfamiliar with ZFS volume management. In fact, I still am.
2 - I need to be able to have the option to convert this raid1 ZFS setup into, for example, single ZFS + single XFS or BTRFS without the necessity of copying the data to any other disk, if in the future I decide to abandon ZFS. This is a remote system so I need to be able to do this when I am thousands of kilometres away.

This is a wild setup. So I am uncertain if this setup worked before, and stopped working when 0.6.4 was released? If that is the case, then there is likely a problem somewhere. However, if it never worked at all, it is probably because you are using ZFS in a way that is not widely used/tested.

From your first post it seems you can import it manually. Have you tried using "zfs_force=1" on the kernel command line?

See:

mkinitcpio -H zfs

Offline

#21 2015-04-18 22:08:59

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

I changed my setup by eliminating LUKS and LVM, and now I have simple MBR->ZFS. The pool gets mounted automatically so it appears the problem was LVM or LUKS+LVM.

However, taking the opportunity, I did some benchmarks to compare BTRFS and ZFS performance on the same (single, non-mirrored) partition, and the results are surprising.

The test commands are:

hdparm -Tt /dev/sda
cd /mnt/hdd
dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc
echo 3 > /proc/sys/vm/drop_caches
dd if=tempfile of=/dev/null bs=1M count=1024
dd if=tempfile of=/dev/null bs=1M count=1024
rm tempfile

The average, consistent results for BTRFS (relatime, no compression):

hdparm: 130.47 MB/sec
Write:			129 MB/s
Read:			131 MB/s 
Buffer cache: 	 1.5 GB/s

The results for ZFS were inconsistent and kept varying (noatime, no compression):

hdparm: 80.47 MB/sec
Write:			74.7 MB/s - 80 MB/s - 98 MB/s - 77.2 MB/s
Read:			74.7 MB/s - 166 MB/s - 122 MB/s - 156 MB/s
Buffer cache: 	 116 MB/s - 117 MB/s - 205 MB/s - 164 MB/s

So it appears as if ZFS was 2x slower in writes, and marginally faster at reads, but only every other time. Also surprising is the low hdparm score. I would think this remains unaffected by filesystem because it tests the raw device (sda) rather than partition (sda1).

For comparison, XFS results on the same partition:

hdparm:	130.15 MB/sec
Write:			126 MB/s
Read:			136 MB/s 
Buffer cache: 	 1.5 GB/s

Can someone comment on those results? If ZFS is that much slower on writes then I feel I should reconsider whether I should use it.

Last edited by Lockheed (2015-04-19 06:42:47)


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#22 2015-04-19 14:39:45

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed wrote:

I changed my setup by eliminating LUKS and LVM, and now I have simple MBR->ZFS. The pool gets mounted automatically so it appears the problem was LVM or LUKS+LVM.

Pretty sure the problem has been LVM. As @demizer also noted, that's an unusual setup and I'm not even sure if we would've been able to find an all-fitting solution, considering that the opposite scenario, LVM over ZFS (to be more precise, over zvol), might be more prevalent. And then we'll probably face the chicken & egg problem: should we first init the LVM volumes or the ZFS pools.

Anyway, glad to see that you've settled on a stable solution.

However, taking the opportunity, I did some benchmarks to compare BTRFS and ZFS performance on the same (single, non-mirrored) partition, and the results are surprising.

I wouldn't actually be surprised by the results. ZFS has not been designed that much for speed (which is not to say that it can't be reasonably fast) as for flexibility, feature set and, most importantly, reliability. One must also always keep in mind that ZFS has been designed initially for Solaris and porting to other operating systems may introduce some inefficiency. Most of the speed and efficiency problems with Linux I believe have already been ironed out, but if you are after speed, ZFS might indeed not be your best choice. And if you will use it on a single disk, I'd say that both ZFS and Btrfs would be suboptimal choices.

So it appears as if ZFS was 2x slower in writes, and marginally faster at reads, but only every other time. Also surprising is the low hdparm score. I would think this remains unaffected by filesystem because it tests the raw device (sda) rather than partition (sda1).

It should be unaffected, even if the partition were to be tested (it's still below the filesystem layer), and my guess is that there must've been some unaccounted for disk activity at the time, which has likely also affected the filesystem test. But again, even without such possible interference, I wouldn't be surprised to see ZFS being somewhat slower than the other filesystems. For me, even on the desktop, that's an insignificant issue, considering how ZFS never failed me despite some really unpleasant situations it has been put through. Btrfs, if we take it as comparison, is going to be a fine filesystem, I believe, but a filesystem usually matures in about a decade, so it's still not stable enough for me. I'll probably repeat myself here, but in the end it's really a question of identifying your priorities.


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#23 2015-04-19 14:49:06

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

Thank you for your help and explanations.
I understand ZFS is not abour performance, however 50% performance of BTRFS still surprised me.

You noted that for a single disk ZFS might not be the best choice. Why so? This is a server disk, mostly for storage of important data. Would ZFS non-mirrored not provide better data protection than XFS or ext4?


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

#24 2015-04-19 15:36:41

kerberizer
Member
From: Sofia, BG
Registered: 2014-02-01
Posts: 25
Website

Re: [solved] ZFS on Linux does not detect pools on LVM

Lockheed wrote:

I understand ZFS is not abour performance, however 50% performance of BTRFS still surprised me.

Filesystem testing is quite a complicated matter: many different use cases, many possible load patterns. As you can imagine, having a relational database puts a very different stress on the filesystem compared to a video streaming service. But even this is a very simplified picture: for example, there are significant enough differences between the various SQL implementations, streaming of live events differs from streaming pre-recorded video, etc, etc. Therefore, for myself and the requirements of my work, I've given up on speed benchmarking. The question that I ask myself is rather: is this filesystem fast enough for my needs, will I feel restricted by its speed or not?

I can't really say if those 50% that you've seen are surprising or even if they are meaningful at all. Even from a statistical point of view, the data are simply not enough to make a reasonable judgement. The only somewhat more comprehensive tests that I recall having seen, are those at Phoronix, and they seemed rather inconclusive. There have been large variations in the measurements, but one could probably guess that, on average, ZFS does tend to lag to a degree speedwise behind the other filesystems. But keep in mind as well, that the last such test is from almost two years ago, and ZoL has made much progress since then.

While I'm not a filesystems expert, if speed is really important for you, my advice would be to make tests with different filesystems as close to what the real-life use would be as possible.

You noted that for a single disk ZFS might not be the best choice. Why so? This is a server disk, mostly for storage of important data. Would ZFS non-mirrored not provide better data protection than XFS or ext4?

Perhaps you're mistaking the data integrity checksums with ECC. Both Btrfs and ZFS use checksums only for validation of data integrity, i.e. that the data blocks haven't changed on disk after they had been written. But when a damage is detected, those checksums cannot be used to reconstruct the original piece of information. The only way to do this is through data redundancy.

Technically, ZFS allows you to set a property on a dataset called 'copies': this would instruct ZFS to keep each data block of the dataset in up to three different locations on a vdev -- in your case, the vdev would be the physical disk (or partition). So, if sectors holding one of these blocks become damaged, ZFS could read the data from one of the other copies. But, of course, disks quite often fail completely, and the 'copies' property cannot save you from such failure (I think Btrfs has a similar mechanism, but it's just as restricted in terms of data recovery).

The only reasonable way of protection against total disk failures is, unsurprisingly, disk-level redundancy: mirror, raid{5,6}, etc. So, in comparison to XFS or ext4, when used on a single disk, ZFS and Btrfs offer only the benefit of knowing that your data has gone bad.


“Don't climb the mountain to conquer it. Climb it to conquer yourself.”

Offline

#25 2015-04-19 16:05:15

Lockheed
Member
Registered: 2010-03-16
Posts: 1,473

Re: [solved] ZFS on Linux does not detect pools on LVM

Thanks for this exhaustive clarification.

What is your opinion on Raid1 mirror made on one physical disk with two partitions? Does it make sense?
My limited understanding is that it keeps 100% read speed, lose 50% write speed and in exchange protects against bitrot. But the last time I discussed it I found the idea universally criticised but the above positive was not addressed nor refuted.

On the other hand, if bit rot is not a real-life problem, then the is no point to it on today's drives which already in contain protection against such issues.


Laptop: ThinkPad T420s, i7/i5, 16GB RAM, 1080p IPS mod, Arch | HTPC/Server/fw: Zotac AQ01, A4-5000 Kabini, 8GB, Arch/lxd NethServer

Offline

Board footer

Powered by FluxBB