You are not logged in.

#1 2013-04-29 17:31:17

bodysativa
Member
Registered: 2012-11-04
Posts: 17

cryptsetup luksOpen hangs

Hi,

I have an LUKS-encrypted container file ("bin.001") on my unencrypted ext4 partition. This container holds an LVM physical volume. Normally I've wrote a small bash script which basically does this thing to mount it:

# create /dev/loop0
losetup -f bin.001

# mount /dev/loop0 to /dev/mapper/bin001
cryptsetup luksOpen /dev/loop0 bin001

# mount LVM image to my pool
pvscan
vgscan
vgchange -ay mypool

# mount LVM pool to a dir
mount /dev/mypool/lvol0 /home/bodysativa/data/private

It worked some time, but later I've stopped using the data stored inside this container. After a month or two (probably after lots of installed updates), I can't mount it anymore. It hangs on the "cryptsetup luksOpen" command.

First it asks me for a password, which I remember 100% correctly, because I'm using a password manager to store it, so i'm just copying and pasting it (same method as I used back then it worked). Then, it hangs. I can provide a wrong password, but there's no difference - it hangs and only ^C helps to exit to the command prompt. I've used 'strace' to try to see what is the reason for this hang, but it appears that it waits for some semaphore and looks like a plain deadlock:

write(6, "Has\305\202o dla /home/bodysativa/bin.001: ", 32Hasło dla /home/bodysativa/bin.001: ) = 32
ioctl(6, SNDCTL_TMR_CONTINUE or SNDRV_TIMER_IOCTL_GPARAMS or TCSETSF, {B38400 opost isig icanon -echo ...}) = 0
read(6, "\n", 512)                      = 1
ioctl(6, SNDCTL_TMR_CONTINUE or SNDRV_TIMER_IOCTL_GPARAMS or TCSETSF, {B38400 opost isig icanon echo ...}) = 0
write(6, "\n", 1
)                       = 1

<- Here it asks me for the password

close(6)                                = 0
brk(0x24c1000)                          = 0x24c1000
open("/dev/loop0", O_RDONLY)            = 6
fstat(6, {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0
ioctl(6, BLKSSZGET, 512)                = 0
close(6)                                = 0
stat("/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0
open("/dev/loop0", O_RDWR)              = 6
ioctl(6, BLKROGET, 0)                   = 0
ioctl(6, BLKGETSIZE64, 104857600000)    = 0
close(6)                                = 0
open("/dev/urandom", O_RDONLY)          = 6
read(6, "\212.", 2)                     = 2

<- Here it creates the blocking semaphore

semget(0xd4d2e8a, 1, IPC_CREAT|IPC_EXCL|0600) = 753671
semctl(753671, 0, SETVAL, 0x1)          = 0
semctl(753671, 0, GETVAL, 0xffffffffffffffff) = 1
close(6)                                = 0
semop(753671, {{0, 1, 0}}, 1)           = 0
semctl(753671, 0, GETVAL, 0xffffffffffffffff) = 2
open("/dev/loop0", O_RDONLY)            = 6
ioctl(6, BLKRAGET, 256)                 = 0
close(6)                                = 0
ioctl(5, DM_DEV_CREATE, 0x24a1210)      = 0
ioctl(5, DM_TABLE_LOAD, 0x24a1210)      = 0
ioctl(5, DM_DEV_SUSPEND, 0x24a1110)     = 0
semget(0xd4d2e8a, 1, 0)                 = 753671
semctl(753671, 0, GETVAL, 0xffffffffffffffff) = 2
semop(753671, {{0, -1, IPC_NOWAIT}}, 1) = 0

<- Next line it hangs, and I hit ^C.

semop(753671, {{0, 0, 0}}, 1^CProcess 4460 detached
 <detached ...>

(the kernel semaphore usage suggested me to use 'kernel & hardware' section for this post, if it's wrong then I'm sorry). Did anyone had a similar problem? Do I miss something obvious? Could anyone sched some light on this?

Thanks in advance for any help.

Offline

#2 2013-04-29 22:10:37

johni
Member
Registered: 2012-02-03
Posts: 102

Re: cryptsetup luksOpen hangs

Are you SURE that the "losetup -f bin.001" is connecting the image to loop0?   Using "-f" causes it to use the first free loop device available.  If something else already has loop0 allocated, it could be putting it on another loop device.

Try changing it to find an available loop device first, and then use the reported loop device.

Like this:

# create loop device
loop_dev=$(losetup -f)

losetup $loop_dev bin.001

# mount /dev/loop0 to /dev/mapper/bin001
cryptsetup luksOpen $loop_dev bin001

# mount LVM image to my pool
pvscan
vgscan
vgchange -ay mypool

# mount LVM pool to a dir
mount /dev/mypool/lvol0 /home/bodysativa/data/private

Offline

#3 2013-04-30 06:01:33

bodysativa
Member
Registered: 2012-11-04
Posts: 17

Re: cryptsetup luksOpen hangs

Thanks for the reply.

Yes I'm sure it uses the correct loop device, because it happens also when I'm doing it manually when giving proper args. The script I've provided is just "pseudocode", because the real script is a little bit more complicated (no need for anyone to dig through it), but it boils down to the excerpt I've pasted. The thing is that if I type in the commands manually, after checking proper loop allocation via "losetup -a", "cryptsetup luksOpen" hangs.

Last edited by bodysativa (2013-04-30 06:19:46)

Offline

#4 2013-04-30 10:18:05

aesiris
Member
Registered: 2012-02-25
Posts: 97

Re: cryptsetup luksOpen hangs

Is it the same if you let cryptsetup do the loop managing by itself?

cryptsetup luksOpen bin.001 bin001

And check if the output of "cryptsetup luksDump bin.001" looks ok.

Also look at the output when using the --debug or --verbose switch

Offline

#5 2013-04-30 15:43:24

bodysativa
Member
Registered: 2012-11-04
Posts: 17

Re: cryptsetup luksOpen hangs

Hi, here's the --debug --verbose log:

[root@hydra bodysativa]# cryptsetup --debug --verbose luksOpen bin.001 bin001 --verbose
# cryptsetup 1.5.1 processing "cryptsetup --debug --verbose luksOpen bin.001 bin001 --verbose"
# Running command luksOpen.
# Locking memory.
# Allocating crypt device bin.001 context.
# Trying to open and read device bin.001.
# Initialising device-mapper backend library.
# Trying to load LUKS1 crypt type from device bin.001.
# Crypto backend (gcrypt 1.5.2) initialized.
# Reading LUKS header of size 1024 from device bin.001
# Key length 32, device size 204800000 sectors, header size 2050 sectors.
# Timeout set to 0 miliseconds.
# Password retry count set to 3.
# Password verification disabled.
# Iteration time set to 1000 miliseconds.
# Activating volume bin001 [keyslot -1] using [none] passphrase.
# dm version   OF   [16384] (*1)
# dm versions   OF   [16384] (*1)
# Detected dm-crypt version 1.12.1, dm-ioctl version 4.23.1.
# Device-mapper backend running with UDEV support enabled.
# dm status bin001  OF   [16384] (*1)
# Interactive passphrase entry requested.
Hasło dla bin.001:
# Trying to open key slot 0 [ACTIVE_LAST].
# Reading key slot 0 area.
# Allocating a free loop device.
# Trying to open and read device /dev/loop1.
# Calculated device size is 256 sectors (RW), offset 8.
# DM-UUID is CRYPT-TEMP-temporary-cryptsetup-1616
# Udev cookie 0xd4dce15 (semid 851973) created
# Udev cookie 0xd4dce15 (semid 851973) incremented to 1
# Udev cookie 0xd4dce15 (semid 851973) incremented to 2
# Udev cookie 0xd4dce15 (semid 851973) assigned to CREATE task(0) with flags DISABLE_SUBSYSTEM_RULES DISABLE_DISK_RULES DISABLE_OTHER_RULES (0xe)
# dm create temporary-cryptsetup-1616 CRYPT-TEMP-temporary-cryptsetup-1616 OF   [16384] (*1)
# dm reload temporary-cryptsetup-1616  OFRW    [16384] (*1)
# dm resume temporary-cryptsetup-1616  OFRW    [16384] (*1)
# temporary-cryptsetup-1616: Stacking NODE_ADD (254,2) 0:0 0600 [verify_udev]
# temporary-cryptsetup-1616: Stacking NODE_READ_AHEAD 256 (flags=1)
# Udev cookie 0xd4dce15 (semid 851973) decremented to 1
# Udev cookie 0xd4dce15 (semid 851973) waiting for zero

So it seems the blocking semaphore is for udev interaction. I'm using systemd-udevd, is this a problem? Also I've checked that there are some temporary devices created in /dev/mapper, as stated in the debug log:

(2:513)$ ls -la /dev/mapper/
razem 0
drwxr-xr-x  2 root root     120 04-30 17:23 .
drwxr-xr-x 18 root root    3340 04-30 17:23 ..
crw-------  1 root root 10, 236 04-30 17:22 control
lrwxrwxrwx  1 root root       7 04-30 17:22 temporary-cryptsetup-1499 -> ../dm-0
lrwxrwxrwx  1 root root       7 04-30 17:23 temporary-cryptsetup-1575 -> ../dm-1
lrwxrwxrwx  1 root root       7 04-30 17:23 temporary-cryptsetup-1616 -> ../dm-2

I can read those dm's, but I can't tell what data it contains, it's not LVM and not the content of bin.001:

(2:527)$ dd if=/dev/dm-3 | xxd | head -n 10
0000000: 6b7b 5699 a96b 4a79 2af1 b5f1 59d7 5d5c  k{V..kJy*...Y.]\
0000010: b01e d81b e484 c65a b5ec f51f fb70 6394  .......Z.....pc.
0000020: 1f96 dac0 eecf bb96 b09b 69dd 92ee f9e9  ..........i.....
0000030: 4fe3 4757 0cac 761f f6d2 04db 08de 95bd  O.GW..v.........
0000040: 7b76 99a5 89c2 7f34 efc6 e645 0917 2bc1  {v.....4...E..+.
0000050: e00b 0870 e499 0fd1 0621 1d4a c1ec 4adf  ...p.....!.J..J.
0000060: 49b6 828f f5a9 af9a ecdc bee8 30ae 3a90  I...........0.:.
0000070: b251 e8f5 10c0 39e3 3039 554d 870f 999c  .Q....9.09UM....
0000080: 7883 964e 7ed7 78ad dd54 8b5d fa30 2160  x..N~.x..T.].0!`
0000090: b26c 1196 1eb7 02c3 b956 5044 b631 2d6d  .l.......VPD.1-m

This data changes depending on what password I provide, but it still isn't LVM nor anything meaningful even when I provide the correct password.

Here's the luksDump log (I've censored it a little bit, hope you don't mind!):

(2:507)$ cryptsetup luksDump bin.001
LUKS header information for bin.001

Version:        1
Cipher name:    aes
Cipher mode:    cbc-essiv:sha256
Hash spec:      sha1
Payload offset: 4096
MK bits:        256
MK digest:      [...]
MK salt:        dd 33 [...] 
                [...]
MK iterations:  xxxxx
UUID:           <valid uuid>

Key Slot 0: ENABLED
        Iterations:             xxxxxx
        Salt:                   [...]
                                [...] 
        Key material offset:    x
        AF stripes:             xxxx
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

In case you're wondering, I'm running an up-to-date Arch:

(2:510)$ uname -a
Linux hydra 3.8.10-1-ARCH #1 SMP PREEMPT Sat Apr 27 12:36:59 CEST 2013 x86_64 GNU/Linux

(2:511)$ systemd --version
systemd 202
+PAM -LIBWRAP -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ

My less than basic understanding of udev and the scope of systemd-udevd prevents me from knowing where to look for the problem... sad

Offline

#6 2013-04-30 17:01:35

aesiris
Member
Registered: 2012-02-25
Posts: 97

Re: cryptsetup luksOpen hangs

I do not know what may be causing your problem.
While you wait for an idea from someone else, see if your problem is related to the new LVM autoactivation by udev.
It has caused boot problems for many people with LVM on LUKS: https://bbs.archlinux.org/viewtopic.php?id=158012
Related FS: https://bugs.archlinux.org/task/30908
This is a thread about the new feature in the mailing list: https://mailman.archlinux.org/pipermail … 23956.html (I was not able to find the whole thread, maybe it is archived somewhere else)

Offline

#7 2013-05-01 10:23:34

bodysativa
Member
Registered: 2012-11-04
Posts: 17

Re: cryptsetup luksOpen hangs

Thanks.

I did some google search for the problem earlier, found some bug reports and forum posts, but mostly related to LVM + boot phase. Since in my case cryptsetup hangs even when I provide wrong password, I would think that there's no chance to know that the container holds LVM data, so it shouldn't trigger LVM autoactivation code, if I understand this correctly.

Moreover, I've just checked and I can't even perform luksFormat on a new container:

(2:503)$ dd if=/dev/zero of=testcontainer bs=1M count=100
100+0 przeczytanych recordów
100+0 zapisanych recordów
skopiowane 104857600 bajtów (105 MB), 3,61109 s, 29,0 MB/s

(2:505)$ losetup -f testcontainer

(2:506)$ losetup -a                                                                                                                                                                                 
[...]
/dev/loop3: [2049]:57805452 (/home/bodysativa/testcontainer)

(2:508)$ sudo cryptsetup --verbose --debug luksFormat /dev/loop3
# cryptsetup 1.5.1 processing "cryptsetup --verbose --debug luksFormat /dev/loop3"
# Running command luksFormat.
# Locking memory.

WARNING!
========
To nieodwołalnie nadpisze dane na /dev/loop3.

Are you sure? (Type uppercase yes): YES
# Allocating crypt device /dev/loop3 context.
[...]
# temporary-cryptsetup-2442: Stacking NODE_ADD (254,5) 0:0 0600 [verify_udev]
# temporary-cryptsetup-2442: Stacking NODE_READ_AHEAD 256 (flags=1)
# Udev cookie 0xd4d7fc0 (semid 917512) decremented to 1
# Udev cookie 0xd4d7fc0 (semid 917512) waiting for zero
<-- HANG
^C 

What I've also discovered is the use of the 'dmsetup udevcomplete_all' command. When I invoke it few times in another terminal session during the "waiting for cookie" deadlock of cryptsetup process, it unblocks and it finishes successfully (only when I provide a correct password of course). Then I can issue standard pvscan/lvscan/vgchange LVM commands to mount this LVM image. So it works! But I don't think I should have to issue this "dmsetup udevcomplete_all" command.

Last edited by bodysativa (2013-05-01 10:41:26)

Offline

#8 2013-05-03 07:38:59

Strike0
Member
From: Germany
Registered: 2011-09-05
Posts: 1,429

Re: cryptsetup luksOpen hangs

I just tried it and can't reproduce your hang. Cryptsetup luksformat It works just fine, also after the losetup "-f" flag.

Your cryptsetup is outdated though, version 1.5.1 was deprecated abt 3 months ago. I don't think this is your problem, but other outdated packages (systemd, util-linux, ..) may well be! Please confirm the system is uptodate to avoid ghosthunting.

edit: The news aesiris pointed to is important when you update.

Last edited by Strike0 (2013-05-03 08:20:31)

Offline

Board footer

Powered by FluxBB