You are not logged in.

#1 2019-02-04 20:18:51

Jason P.
Member
Registered: 2015-02-26
Posts: 171

[solved] pvscan command not reading lvmetad cache ??

Hi,

I have a strange issue here with LVM. I normally do either suspend or reboot in my machine, but the other day and for some reason I don't remember I did a power off. The previous one was months ago.

And now the strange thing... Since that moment I can see on boot logs several lines like these ones:

lvm[498]:   /dev/sdd: open failed
lvm[498]:   /dev/sde: open failed
lvm[498]:   /dev/sdf: open failed
...

Which could make sense (I'm not completely sure) because each systemd lvm2-pvscan@.service unit (there are three, one per disk) runs with the flag --cache:

ExecStart=/usr/bin/lvm pvscan --cache --activate ay 254:1

I have the feeling that something is not working fine though. First, because those "errors" weren't on the logs before, or at least I didn't see them. Second and most important thing, when I run pvscan command from terminal I also get same errors along with proper results. That could be a pointer of something going wrong with lvmetad, which I have enabled in my lvm.config.

➜ # ~  pvscan --cache
  /dev/sdd: open failed
  /dev/sde: open failed
  /dev/sdf: open failed
  /dev/sdg: open failed
➜ # ~  pvscan
  /dev/sdd: open failed
  /dev/sde: open failed
  /dev/sdf: open failed
  /dev/sdg: open failed
  PV /dev/mapper/ssd         VG ssd             lvm2 [<118,74 GiB / <108,74 GiB free]
  PV /dev/sdb1               VG storage         lvm2 [<931,51 GiB / 0    free]
  PV /dev/mapper/crypt_hdd   VG hdd             lvm2 [<465,76 GiB / <73,76 GiB free]
  Total: 3 [1,48 TiB] / in use: 3 [1,48 TiB] / in no VG: 0 [0   ]

To me looks like pvscan is falling back to scanning for some reason and not reading cache. Or is this normal?

Edit 1 => Update title adding lvmetad keyword

Last edited by Jason P. (2019-02-11 14:22:47)

Offline

#2 2019-02-06 23:45:35

jamespharvey20
Member
Registered: 2015-06-09
Posts: 129

Re: [solved] pvscan command not reading lvmetad cache ??

What are "/dev/sd{d,e,f,g}"?  Are they devices used by LVM?

You could post a link to the journalctl output for the entire boot.

If it happens again, try seeing if you can run something like "# dd if=/dev/sdX of=/dev/null bs=1M count=100" for each of the drives, to see if that has problems.

Offline

#3 2019-02-07 00:04:47

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [solved] pvscan command not reading lvmetad cache ??

does disabling lvmetad help? lvmetad is history anyway.

Release 2.03.00
This is the first release from the 2.03 branch.

Major changes:

* Deduplication and compression - support for VDO volumes.
* Dropped clvmd with lvmlockd as a replacement.
* Dropped liblvm2app with D-Bus API or libblockdev as possible replacement.
* Dropped lvmetad without replacement.

Plus many changes inside.

(archlinux is on 2.02 branch right now)

You have a crypt device there, if your initramfs is setting up LUKS and LVM, you might get away with masking the pvscan unit altogether. (Do not run this experiment without a rescue system at hand to undo it.)

Offline

#4 2019-02-07 00:28:33

jamespharvey20
Member
Registered: 2015-06-09
Posts: 129

Re: [solved] pvscan command not reading lvmetad cache ??

frostschutz wrote:

does disabling lvmetad help? lvmetad is history anyway.
,,,
* Dropped lvmetad without replacement.

What the... Do you know anything more about this?  Love how their documentation is so lacking.  They can't be dropping everything back to repeated disk scans...

Offline

#5 2019-02-07 00:31:13

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [solved] pvscan command not reading lvmetad cache ??

Offline

#6 2019-02-07 00:34:32

jamespharvey20
Member
Registered: 2015-06-09
Posts: 129

Re: [solved] pvscan command not reading lvmetad cache ??

Thanks, I sit corrected.

Offline

#7 2019-02-09 17:27:01

Jason P.
Member
Registered: 2015-02-26
Posts: 171

Re: [solved] pvscan command not reading lvmetad cache ??

jamespharvey20 wrote:

What are "/dev/sd{d,e,f,g}"?  Are they devices used by LVM?

No clue what they are. How can I find out?

jamespharvey20 wrote:

You could post a link to the journalctl output for the entire boot.

I notice this on every boot so this log of my other issue with LVM should be fine.

jamespharvey20 wrote:

If it happens again, try seeing if you can run something like "# dd if=/dev/sdX of=/dev/null bs=1M count=100" for each of the drives, to see if that has problems.

➜ # ~  l /dev | grep sd
brw-rw----  1 root disk        8,   0 feb  9 17:47 sda
brw-rw----  1 root disk        8,   1 feb  9 17:47 sda1
brw-rw----  1 root disk        8,   2 feb  9 17:47 sda2
brw-rw----  1 root disk        8,  16 feb  9 17:47 sdb
brw-rw----  1 root disk        8,  17 feb  9 17:47 sdb1
brw-rw----  1 root disk        8,  32 feb  9 17:47 sdc
brw-rw----  1 root disk        8,  48 feb  9 17:47 sdd
brw-rw----  1 root disk        8,  64 feb  9 17:47 sde
brw-rw----  1 root disk        8,  80 feb  9 17:47 sdf
brw-rw----  1 root disk        8,  96 feb  9 17:47 sdg
drwxr-xr-x  2 root root            60 feb  9 17:47 ssd
➜ # ~  dd if=/dev/sdd of=/dev/null bs=1M count=100 
dd: No se puede abrir '/dev/sdd': No se ha encontrado el medio
➜ # ~  dd if=/dev/sdg of=/dev/null bs=1M count=100 
dd: No se puede abrir '/dev/sdg': No se ha encontrado el medio

I don't know what are these devices for and looks like they produce some kind of error when reading from them ("media not found"). I'm wondering if they could be leftovers of removable devices or LVM snapshots? Also, take a look at my partition scheme here: https://termbin.com/ipdq4

Offline

#8 2019-02-09 17:35:16

Jason P.
Member
Registered: 2015-02-26
Posts: 171

Re: [solved] pvscan command not reading lvmetad cache ??

frostschutz wrote:

does disabling lvmetad help? lvmetad is history anyway.

Release 2.03.00
This is the first release from the 2.03 branch.

Major changes:

* Deduplication and compression - support for VDO volumes.
* Dropped clvmd with lvmlockd as a replacement.
* Dropped liblvm2app with D-Bus API or libblockdev as possible replacement.
* Dropped lvmetad without replacement.

Plus many changes inside.

(archlinux is on 2.02 branch right now)

I didn't know that. So essentialy lvmetad is no more helpful since version 2.03 because discovering drives and volumens on the fly is more performant now than reading from a file??

frostschutz wrote:

You have a crypt device there, if your initramfs is setting up LUKS and LVM, you might get away with masking the pvscan unit altogether. (Do not run this experiment without a rescue system at hand to undo it.)

Just curiosity beyond the workaround itself. What are the advantages/disadvantages of having the whole LUKS/LVM setup done with initramfs? I'm a little scared of doing such changes. Maybe at some point I give myself time to do a transition from what I have right now to another partition scheme.

Offline

#9 2019-02-09 17:49:43

Jason P.
Member
Registered: 2015-02-26
Posts: 171

Re: [solved] pvscan command not reading lvmetad cache ??

I've just found out by chance running pvscan with verbose option:

#device/dev-cache.c:723           Found dev 8:64 /dev/sde - new.
#device/dev-cache.c:763           Found dev 8:64 /dev/disk/by-id/usb-Generic-_SM_xD-Picture_20070818000000000-0:1 - new alias.
#device/dev-cache.c:763           Found dev 8:64 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.3:1.1-scsi-0:0:0:1 - new alias.

I have a multicard reader plugged...

Last edited by Jason P. (2019-02-09 17:57:17)

Offline

#10 2019-02-10 13:41:48

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [solved] pvscan command not reading lvmetad cache ??

many SD card readers do indeed present themselves as 4 removable drives to the OS.

No idea how to verify sde to sdf are such devices, sorry.


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#11 2019-02-11 13:59:17

Jason P.
Member
Registered: 2015-02-26
Posts: 171

Re: [solved] pvscan command not reading lvmetad cache ??

#device/dev-cache.c:1212          Creating list of system devices.
#device/dev-cache.c:723           Found dev 8:48 /dev/sdd - new.
#device/dev-cache.c:763           Found dev 8:48 /dev/disk/by-id/usb-Generic-_Compact_Flash_20070818000000000-0:0 - new alias.
#device/dev-cache.c:763           Found dev 8:48 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.3:1.1-scsi-0:0:0:0 - new alias.
#device/dev-cache.c:723           Found dev 8:64 /dev/sde - new.
#device/dev-cache.c:763           Found dev 8:64 /dev/disk/by-id/usb-Generic-_SM_xD-Picture_20070818000000000-0:1 - new alias.
#device/dev-cache.c:763           Found dev 8:64 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.3:1.1-scsi-0:0:0:1 - new alias.
#device/dev-cache.c:723           Found dev 8:80 /dev/sdf - new.
#device/dev-cache.c:763           Found dev 8:80 /dev/disk/by-id/usb-Generic-_SD_MMC_20070818000000000-0:2 - new alias.
#device/dev-cache.c:763           Found dev 8:80 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.3:1.1-scsi-0:0:0:2 - new alias.
#device/dev-cache.c:723           Found dev 8:96 /dev/sdg - new.
#device/dev-cache.c:763           Found dev 8:96 /dev/disk/by-id/usb-Generic-_MS_MS-Pro_20070818000000000-0:3 - new alias.
#device/dev-cache.c:763           Found dev 8:96 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.3:1.1-scsi-0:0:0:3 - new alias.

sd[d..g] are all different slots of this multi card reader and they are presented to the system under /dev. I don't know if this normal or expected.

Another thing is that any time I try to access one of them (empty) I got an error:

➜ # ~  dd if=/dev/sdd of=/dev/null bs=1M count=100 
dd: No se puede abrir '/dev/sdd': No se ha encontrado el medio

* Media not found

I don't have any special drivers installed for this device.

Last edited by Jason P. (2019-02-11 14:00:39)

Offline

#12 2019-02-11 14:22:12

Jason P.
Member
Registered: 2015-02-26
Posts: 171

Re: [solved] pvscan command not reading lvmetad cache ??

There's still a device present on the system even after removing the media, so this is totally expected. What happens here is that LVM tries to filter out devices which don't have enough size for a physical volume. In this case those devices have not any media inserted hence the error when opening them to check the size.

Offline

Board footer

Powered by FluxBB