System won't boot after upgrade of util-linux to 2.24.1-1

telpeloth · 2014-02-19 10:11:45

Dear colleagues, I have a riddle for you to solve.

Background: I have a 64GB SSD drive as root partition:

/dev/sdc: PLEXTOR PX-64M5S

I do use grub2, and grub2 has correct UUID for the partition:

root@telpeloth ~ # blkid /dev/sdc1
/dev/sdc1: UUID="54d265a6-cc48-4c82-ac76-392890bae696" TYPE="ext4" PARTUUID="775c81b9-01" 

root@telpeloth ~ # grep UUID /boot/grub/grub.cfg
    linux /boot/vmlinuz-linux root=UUID=54d265a6-cc48-4c82-ac76-392890bae696 rw  quiet
    linux /boot/vmlinuz-linux root=UUID=54d265a6-cc48-4c82-ac76-392890bae696 rw  quiet
        linux   /boot/vmlinuz-linux root=UUID=54d265a6-cc48-4c82-ac76-392890bae696 rw  quiet
                linux   /boot/vmlinuz-linux root=UUID=54d265a6-cc48-4c82-ac76-392890bae696 rw  quiet

Now: this device had been partitioned using cfdisk (which, as I know now does not care for alignment) and used to host nilfs2 filesystem.
After several issues with it, I backed up content, repartitioned disk using fdisk this time and created ext4 partition, restored backup, updated grub.cfg and reinstalled grub, everything worked like a charm, until yesterday, where I did pacman -Syyu and system refused to start, stating that disk with UUID=54d265a6-cc48-4c82-ac76-392890bae696 could not be found and dropping me in the initramfs rescue shell.
I did run a blkid in there are here's what I found: (I am running blkid from untarred util-linux-2.24.1-1, but the output is the same as in initramfs)

root@telpeloth ~/pacs # LD_LIBRARY_PATH=./usr/lib ./usr/bin/blkid /dev/sdc1
2 root@telpeloth ~/pacs # LD_LIBRARY_PATH=./usr/lib ./usr/bin/blkid /dev/sdc                                                                                                                                      :(
/dev/sdc: UUID="1f011eb1-a143-4799-80c8-7ebf7a220413" TYPE="nilfs2" PTUUID="775c81b9" PTTYPE="dos"

Funny thing -- this UUID is almost for sure the same as my old partition's was

This is the output from blkid from util-linux-2.24-2

root@telpeloth ~/pacs # blkid 
/dev/sda1: UUID="09616928-c357-4917-a27e-94f6b14ffdd2" TYPE="xfs" 
/dev/sdb1: UUID="f8fdd4ea-4ec1-4b35-b504-dec94201b768" TYPE="xfs" 
/dev/sdc1: UUID="54d265a6-cc48-4c82-ac76-392890bae696" TYPE="ext4" PARTUUID="775c81b9-01" 
/dev/sdd1: UUID="3fe24f4d-60e5-4ae5-bdb2-db97ed51d01d" TYPE="xfs" PARTUUID="a98a5f17-01"

It took me a while to figure out that udev relies on blkid to populate /dev/disk/* symlinks, but once I did, I began to wonder what would cause such behaviour. I though there had to be some residues in first few bytes that new blkid examined, but old one did not. Therefore, what I did, is I backed up content of /, did

dd if=/dev/zero of=/dev/sdc bs=1M count=6

Recreated partition using first cfdisk (without alignment) and then fdisk (with alignment), restored grub and everything, but the issue persisted.

Now, since I ran out of ideas, I downgraded util-linux, recreated initramfs and successfully booted.

Any ideas?

Rexilion · 2014-02-19 10:28:46

You recreated it. Is the stale UUID not detected anymore? Did you change it in your fstab?

telpeloth · 2014-02-19 10:37:42

What you see is the output of blkid after I was able to boot again, so yes, this can be recreated on demand, and yes, the stale UUID can be accessed even now, after dd'ing, repartitioning, recreating ext4 partition etc. The correct UUID is present in fstab right now, system is fully operable.
Yet, using blkid I got from untarred util-linux-2.24.1-1 I get this output -- I just ssh'd to my home machine to prove it:

telpeloth@jakub-barc ~ % ssh telpeloth.pl
Last login: Wed Feb 19 11:35:10 2014 from [job-location-deleted]
telpeloth@telpeloth ~ % sudo su -
root@telpeloth ~ # cd pacs 
root@telpeloth ~/pacs # LD_LIBRARY_PATH=./usr/lib ./usr/bin/blkid /dev/sdc
/dev/sdc: UUID="1f011eb1-a143-4799-80c8-7ebf7a220413" TYPE="nilfs2" PTUUID="775c81b9" PTTYPE="dos" 
root@telpeloth ~/pacs #

Rexilion · 2014-02-19 11:43:30

I think it's intentional that blkid of an entire disk shows the same UUID despite you dd-ing it? Taking that into account it's not stale anymore but intended.

Maybe the disk is not there in time for blkid to notice it (i.e. a race condition?). This could explain the fact that it 'seems' work, but does not.

Arch Linux

#1 2014-02-19 10:11:45

System won't boot after upgrade of util-linux to 2.24.1-1

#2 2014-02-19 10:28:46

Re: System won't boot after upgrade of util-linux to 2.24.1-1

#3 2014-02-19 10:37:42

Re: System won't boot after upgrade of util-linux to 2.24.1-1

#4 2014-02-19 11:43:30

Re: System won't boot after upgrade of util-linux to 2.24.1-1

Board footer