You are not logged in.

#1 2021-08-24 14:45:44

stefano
Member
Registered: 2011-04-09
Posts: 287

ZFS pool won't grow [SOLVED]

I have a zfs pool of 4 disks in a raidz-2 configuration in a media server. (The Arch system is on a different drive, the zfs pool is used for data storage only).
The pool was originally created with 4x2Tb drives, but I have been replacing them with 4Tb as they failed. The pool how has 4xTb drive, but it refuses to grow to accommodate the new available storage.

I used the standard procedure:

1. Replace drives one by one with "zpool replace pool old_drive new_drive", then waiting for the resilvering to complete.

2. Once all four were replaced, I turn autoexpand off and on again with "zpool set autoexpand=off[on]  pool"

However, the pool size did not grow:

[stefano@polus ~]$ zpool status -c size
  pool: zfsdatapool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 1.69T in 20:53:28 with 0 errors on Mon Aug 23 20:20:16 2021
config:

        NAME                        STATE     READ WRITE CKSUM  size
        zfsdatapool                 ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            sda                     ONLINE       0     0     0  3.6T
            wwn-0x5000c500b2d7861e  ONLINE       0     0     0  3.6T
            wwn-0x5000c50079420af9  ONLINE       0     0     0  3.6T
            wwn-0x50014ee2bbdb66cb  ONLINE       0     0     0  3.6T

errors: No known data errors
[stefano@polus ~]$ zpool list
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfsdatapool  7.25T  6.97T   286G        -     7.25T      -    96%  1.00x    ONLINE  -

With 4x4Tb (nominal) disks in a radiz2 configuration I would expect to see approximately 11 to to 12 Tb of available space, not 7.25Tb. 

The four drives have no particular partitioning scheme---they were formatted by zfs from there original (new) condition. Here is the layout (sdb is the ssd drive holding the Arch system):

[stefano@polus ~]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   3.6T  0 disk 
├─sda1   8:1    0   3.6T  0 part 
└─sda9   8:9    0     8M  0 part 
sdb      8:16   0 111.8G  0 disk 
├─sdb1   8:17   0     1M  0 part 
└─sdb2   8:18   0 111.8G  0 part /
sdc      8:32   0   3.6T  0 disk 
├─sdc1   8:33   0   3.6T  0 part 
└─sdc9   8:41   0     8M  0 part 
sdd      8:48   0   3.6T  0 disk 
├─sdd1   8:49   0   3.6T  0 part 
└─sdd9   8:57   0     8M  0 part 
sde      8:64   0   3.6T  0 disk 
├─sde1   8:65   0   3.6T  0 part 

What am I doing wrong?

Last edited by stefano (2021-08-31 16:07:53)

Offline

#2 2021-08-24 21:29:46

TheSgtBilko
Member
Registered: 2013-08-13
Posts: 89

Re: ZFS pool won't grow [SOLVED]

If the vdev was configured as raidz (raidz1): One disk redundancy, you get three disk worth of data out of four disks total.
If the vdev was configured as raidz2: Two disk redundancy, you get two disk worth of data out of four disks total.

7.25TB data space from 4x4TB in raidz2 seems to be correct.

Offline

#3 2021-08-26 17:52:25

stefano
Member
Registered: 2011-04-09
Posts: 287

Re: ZFS pool won't grow [SOLVED]

My bad, I got confused after several attempts and reading countless zfs pages. But the problem is real, in spite of my clumsiness in describing it.  Here is a second attempt:

Facts:

1. The zfs pool was initially created with 4x2TB disks in a raidz-2 configuration, i.e. with two extra disks.
2. The total capacity was around 3.6T, as it should have been (about 50% of total)
3. Disks were progressively replaced to 4x4TB
4. Autoexpand feature was set to on

Problem:
4. Capacity should be about 50%, or between 7TB and 8TB, but it is still stuck at 3.6TB

zpool output:

[stefano@polus ~]$ zpool status -c size
  pool: zfsdatapool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 1.69T in 20:53:28 with 0 errors on Mon Aug 23 20:20:16 2021
config:

        NAME                        STATE     READ WRITE CKSUM  size
        zfsdatapool                 ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            wwn-0x5000c50079420edf  ONLINE       0     0     0  3.6T
            wwn-0x5000c500b2d7861e  ONLINE       0     0     0  3.6T
            wwn-0x5000c50079420af9  ONLINE       0     0     0  3.6T
            wwn-0x50014ee2bbdb66cb  ONLINE       0     0     0  3.6T

errors: No known data errors

So, 4x3.6 TB disks in raidz-2 should give me ~7.5 TB total, as per zpool list

[stefano@polus ~]$ zpool list
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfsdatapool  7.25T  6.95T   306G        -     7.25T      -    95%  1.00x    ONLINE  -

   
so far so good, but unfortunately df disagrees:

[stefano@polus ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
dev             3.9G     0  3.9G   0% /dev
run             3.9G  896K  3.9G   1% /run
/dev/sdb2       110G   22G   83G  21% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G     0  3.9G   0% /tmp
zfsdatapool     3.5T  3.4T   36G  99% /data
tmpfs           793M     0  793M   0% /run/user/1001

and zfs list concurs:

[stefano@polus ~]$ zfs list
NAME          USED  AVAIL     REFER  MOUNTPOINT
zfsdatapool  3.37T  36.0G     3.37T  /data

At this point I'm utterly confused. Why am I not seeing the whole capacity "zpool list" reports in both df's and zfs's outputs? More substantially, how much capacity do I have? I have tried unmounting/exporting / reimporting the pool to no avail. Neither did rebooting. I found an old github report (https://github.com/openzfs/zfs/issues/3953) on a similar issue that attributed the problem to a stale file handle that a regular reboot should solve. Not so in my case.

Offline

#4 2021-08-27 14:59:38

TheSgtBilko
Member
Registered: 2013-08-13
Posts: 89

Re: ZFS pool won't grow [SOLVED]

I don't know when you turned "on" autoexpand. But my guess is that it has to be done before you start replacing the disks.

You should probably read up on the following command (man zpool-online): zpool online [-e] pool device...

If the currently stored data is crucial, make sure you have a backup



Also unless you have reasons not to, you should attend to the zpool status message about upgrading feature sets.

Offline

#5 2021-08-31 16:07:29

stefano
Member
Registered: 2011-04-09
Posts: 287

Re: ZFS pool won't grow [SOLVED]

TheSgtBilko wrote:

I don't know when you turned "on" autoexpand. But my guess is that it has to be done before you start replacing the disks.

You should probably read up on the following command (man zpool-online): zpool online [-e] pool device...

If the currently stored data is crucial, make sure you have a backup



Also unless you have reasons not to, you should attend to the zpool status message about upgrading feature sets.


Ah, offlining/onlining all the devices in order did the trick, thanks a lot. I am not sure why autoexpand did not work---the online docs I consulted instructed to set it to on *after* the resilvering. I guess they were wrong. In general, it seems growing zraid pools is not a very common occurrence. Oracle has a very single line on their page on this topic:

"To expand a LUN, use the zpool online –e command. By default, a LUN that is added to a pool is not expanded to its full size unless the autoexpand pool property is enabled. If the property is disabled, use the command to expand the LUN automatically. You can run the command regardless of whether the LUN is offline or already online."

which seems to suggest the autoexpand option method should have worked and that offlining the devices was actually redundant (but it didn't hurt, thankfully).

Offline

Board footer

Powered by FluxBB