You are not logged in.

#1 2022-07-03 17:09:47

cmdrsweeper
Member
Registered: 2015-09-05
Posts: 12

[SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

I ran an update as I use the ZFS kernel module that seems to have a strange issue going on at the moment.
No matter, I use the precompiled one in the meantime.
But as I proceed to install it, I get to the step of mkinitcpio rebuilding the kernel with all the modules and it fails at the bsdtar step where it makes the initramfs files.
The error is basically "bsdtar write error" in the pacman output.

If I go looking in the dmesg, I can see something really strange:
[358794.345811] bsdtar[1965308]: segfault at 1 ip 00007feadb015e18 sp 00007ffc84523b50 error 4 in libc.so.6[7feadb010000+178000]
Everytime mkinitcpio runs it segfaults bsdtar.

So far I have tried:
Reinstalling zstd and libarchive to see if I could make the culprit disappear.

Any ideas?

Last edited by cmdrsweeper (2022-07-03 22:39:54)

Offline

#2 2022-07-03 19:22:36

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

The error is basically "bsdtar write error" in the pacman output.

Don't paraphrase, https://bbs.archlinux.org/viewtopic.php?id=57855

Are you out of space on the boot partition? Or is it mounted read-only?

Offline

#3 2022-07-03 20:41:30

cmdrsweeper
Member
Registered: 2015-09-05
Posts: 12

Re: [SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

seth wrote:

The error is basically "bsdtar write error" in the pacman output.

Don't paraphrase, https://bbs.archlinux.org/viewtopic.php?id=57855

Are you out of space on the boot partition? Or is it mounted read-only?

Sorry it is not a paraphrase...
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux.img
bsdtar: Write error
==> ERROR: Image generation FAILED: bsdtar (step 1) reported an error

That is the output of the error in Pacman, nothing more, nothing less.
I checked the drive for space defiencies, I have 14.6 Gb out of 240 Gb spent, so space is present, to make sure it wasn't lying, I also copied the Vmlinuz file which is fairly big into the same directory /boot and it copies.
Touching and making text files is also perfectly fine, so I can write.
Dmesg does give the segfault message however.

Last edited by cmdrsweeper (2022-07-03 20:42:04)

Offline

#4 2022-07-03 20:59:08

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: [SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

What is the output of the following commands?

dfc # or df -h if dfc is not installed
lsblk
findmnt
ls -lhd /boot /boot/initramfs-linux*

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#5 2022-07-03 21:21:54

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

Also

bsdtar --zstd -cf /tmp/foo.zst /etc/locale.gen

Is there a coredump next to the segfault message?

Offline

#6 2022-07-03 22:29:32

cmdrsweeper
Member
Registered: 2015-09-05
Posts: 12

Re: [SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

Alright, that making of the zst archive worked, and I didn't get a segfault.
However there is a Code section in the dmesg log trailing the segfaults with mkinitcpio that I notice is the same everytime.

[359975.637397] bsdtar[2034009]: segfault at 1 ip 00007f2e46d80e18 sp 00007fff84fe61b0 error 4 in libc.so.6[7f2e46d7b000+178000]
[359975.637414] Code: 44 89 a4 24 b0 00 00 00 48 8b 74 24 28 45 31 c0 48 8b 7c 24 30 48 8b 44 24 08 48 83 c4 78 5b 5d 41 5c 41 5d 41 5e 41 5f ff e0 <4c> 8b 29 e9 7a fc ff ff 4d 89 03 48 39 f2 0f 85 18 ff ff ff 4c 39

As for the other information requested, I will post them afterwards here.
This is my home server, so it is a CLI only setup so fairly stripped down if that information is useful.

Output of df -h

Filesystem              Size  Used Avail Use% Mounted on
dev                     3.9G     0  3.9G   0% /dev
run                     3.9G  940K  3.9G   1% /run
archzfs/ROOT/default    230G   12G  218G   5% /
tmpfs                   3.9G     0  3.9G   0% /dev/shm
tmpfs                   3.9G  4.0K  3.9G   1% /tmp
/dev/md127              5.4T  5.3T  143G  98% /ICHRaid
archzfs/data/home/root  218G  256K  218G   1% /root
archzfs/data/home       218G  256K  218G   1% /home
tmpfs                   795M     0  795M   0% /run/user/1000

Output of lsblk:

NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda         8:0    0 238.5G  0 disk
├─sda1      8:1    0     7M  0 part
└─sda2      8:2    0 238.5G  0 part
sdb         8:16   0 238.5G  0 disk
├─sdb1      8:17   0     7M  0 part
└─sdb2      8:18   0 238.5G  0 part
sdc         8:32   0   1.8T  0 disk
└─md127     9:127  0   5.5T  0 raid6 /ICHRaid
sdd         8:48   0   2.7T  0 disk
└─sdd1      8:49   0   1.4T  0 part
sde         8:64   0   3.6T  0 disk
└─sde1      8:65   0   1.4T  0 part
  └─md127   9:127  0   5.5T  0 raid6 /ICHRaid
sdf         8:80   0   2.7T  0 disk
└─sdf1      8:81   0   1.4T  0 part
  └─md127   9:127  0   5.5T  0 raid6 /ICHRaid
sdg         8:96   0   1.8T  0 disk
└─md127     9:127  0   5.5T  0 raid6 /ICHRaid
sdh         8:112  0   3.6T  0 disk
└─sdh1      8:113  0   1.4T  0 part
  └─md127   9:127  0   5.5T  0 raid6 /ICHRaid

The output from findmnt:

TARGET                                        SOURCE            FSTYPE      OPTIONS
/                                             archzfs/ROOT/default
                                                                zfs         rw,nodev,relatime,xattr,posixacl
├─/proc                                       proc              proc        rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                  systemd-1         autofs      rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12559
│   └─/proc/sys/fs/binfmt_misc                binfmt_misc       binfmt_misc rw,nosuid,nodev,noexec,relatime
├─/sys                                        sys               sysfs       rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                      securityfs        securityfs  rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                            cgroup2           cgroup2     rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore                            pstore            pstore      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                               bpf               bpf         rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                         debugfs           debugfs     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                       tracefs           tracefs     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections                  fusectl           fusectl     rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config                        configfs          configfs    rw,nosuid,nodev,noexec,relatime
├─/dev                                        dev               devtmpfs    rw,nosuid,relatime,size=4059500k,nr_inodes=1014875,mode=755,inode64
│ ├─/dev/shm                                  tmpfs             tmpfs       rw,nosuid,nodev,inode64
│ ├─/dev/pts                                  devpts            devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/mqueue                               mqueue            mqueue      rw,nosuid,nodev,noexec,relatime
│ └─/dev/hugepages                            hugetlbfs         hugetlbfs   rw,relatime,pagesize=2M
├─/run                                        run               tmpfs       rw,nosuid,nodev,relatime,mode=755,inode64
│ ├─/run/credentials/systemd-sysusers.service ramfs             ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ └─/run/user/1000                            tmpfs             tmpfs       rw,nosuid,nodev,relatime,size=813984k,nr_inodes=203496,mode=700,uid=1000,gid=1000,inode64
├─/tmp                                        tmpfs             tmpfs       rw,nosuid,nodev,nr_inodes=1048576,inode64
├─/ICHRaid                                    /dev/md127        ext4        rw,relatime,stripe=512
├─/root                                       archzfs/data/home/root
│                                                               zfs         rw,nodev,relatime,xattr,posixacl
└─/home                                       archzfs/data/home zfs         rw,nodev,relatime,xattr,posixacl

As for the last one, the .bak files is me trying to test the writeability without deleting the broken files to see if there was another issue that caused blockage.
Output of ls -lhd /boot /boot/initramfs-linux*

drwxr-xr-x 3 root root  8 Jul  3 18:53 /boot
-rw------- 1 root root 13 Jul  3 19:16 /boot/initramfs-linux-fallback.img
-rw------- 1 root root 13 Jul  3 18:52 /boot/initramfs-linux-fallback.img.bak
-rw------- 1 root root 13 Jul  3 19:15 /boot/initramfs-linux.img
-rw------- 1 root root 13 Jul  3 18:52 /boot/initramfs-linux.img.bak

And if it wasn't clear, the root of Arch is installed on ZFS, the same with /boot by making a bare minimum featureset on the pool to make it GRUB compatible.
It has 2x SSDs in mirroring with GRUB installed to both of them in case one dies.
Output of zfs list which also has docker making a lot of snapshots, but this issue occurred yesterday which according to zpool history is almost half a month after the docker setup.

NAME                                                                                         USED  AVAIL     REFER  MOUNTPOINT
archzfs                                                                                     12.6G   218G       96K  none
archzfs/ROOT                                                                                12.6G   218G      144K  none
archzfs/ROOT/default                                                                        12.6G   218G     11.2G  /
archzfs/ROOT/default/098a4709afbb6d7c321dd44c36ec6b53a7e37d305391f55abefe1706d41238b8       16.5M   218G     31.8M  legacy
archzfs/ROOT/default/0c442290951f4000954ad49d98e6e0959986197512184896e85bb0570b2b08da        296K   218G     31.9M  legacy
archzfs/ROOT/default/21693d62a23fe22360caaf10524ace51b2199728dc511789a0e196a69a9221ab       1.45M   218G      326M  legacy
archzfs/ROOT/default/21693d62a23fe22360caaf10524ace51b2199728dc511789a0e196a69a9221ab-init   240K   218G      326M  legacy
archzfs/ROOT/default/2bad5ee0beb3a8b16421e5ab584c00ebacf19405ce0e1f8ab59facbe2927871a       5.32M   218G     99.1M  legacy
archzfs/ROOT/default/3e12fb7064b64fe4bdef244327f43118227de7dc1d26bf32198f0ba65d13f27b        188K   218G      101M  legacy
archzfs/ROOT/default/449e6da233fd7abe72ad51c7a186285cb2a86c6b33b02eb7faff1761113dfe5c        573M   218G      735M  legacy
archzfs/ROOT/default/4de1c58d4c63ba6504cfa43af524fd53ed8c35d64daa2cd17db827007ef0cb5c        480K   218G      326M  legacy
archzfs/ROOT/default/56db52523e609876286d3b9a98d55d0521ac3a4ef8005e267eeaff1e23c3dea4        164K   218G      101M  legacy
archzfs/ROOT/default/65a19065634eb8a5310c6841b89df95ba12d8b170c6c736487b19e4179da699f       68.3M   218G      164M  legacy
archzfs/ROOT/default/6ed34ff2ead64ba183a10391a8a426cace455b23950e7573e5e0170c8c5012e6        300M   218G      735M  legacy
archzfs/ROOT/default/6ed34ff2ead64ba183a10391a8a426cace455b23950e7573e5e0170c8c5012e6-init   228K   218G      735M  legacy
archzfs/ROOT/default/75f833cdb15547d398b2f998a6db0c3a146248661e420ab45f047fe1dacb2a0c       5.30M   218G     14.0M  legacy
archzfs/ROOT/default/76c6f404f29b58be48ebedbbe0d9e3646dbc4abf256a6220a1004f468f32f4ba       8.85M   218G     8.84M  legacy
archzfs/ROOT/default/7ffb25b7e83c7a08e7648e296d732c159b250700a3efbdc63f3e81b9beec7e30        288K   218G      735M  legacy
archzfs/ROOT/default/92fceb085b23b8c2c0d22ae0ddb252a7c8b896f390d1051b45ea5fe8a47fb552        316K   218G      164M  legacy
archzfs/ROOT/default/a35e27375bf89ba9ab7957300d67967569ff14498ecddc57fe8dfac8e4b9ff48       7.64M   218G     16.2M  legacy
archzfs/ROOT/default/b3e6f85c9718d4ff93f665c1d448e67334bc38fda4700e3905f8c4904c967808        176K   218G     31.9M  legacy
archzfs/ROOT/default/b54918dd0443da5402f7e76aa438c6a94f801e4b9612cdcd4e79029eefec376f        296M   218G      326M  legacy
archzfs/ROOT/default/bed62a088d91db1bbe0804da0200996e3ce4b2d2ac80c3c733657f143548b805       94.0M   218G     93.9M  legacy
archzfs/ROOT/default/d3a987d4aa21fd9dd6466e4383b161ec8f680a5904aeb2b8a53e36caba5099b5       7.64M   218G      101M  legacy
archzfs/ROOT/default/f412f810617e72967fff5aa292cd1479a4064cc2f220c9750114426311734ec1        164K   218G     16.2M  legacy
archzfs/data                                                                                 588K   218G      144K  none
archzfs/data/home                                                                            444K   218G      240K  /home
archzfs/data/home/root                                                                       204K   218G      204K  /root

Last edited by cmdrsweeper (2022-07-03 22:30:27)

Offline

#7 2022-07-03 22:38:40

cmdrsweeper
Member
Registered: 2015-09-05
Posts: 12

Re: [SOLVED] Pacman induced weirdness, bsdtar segfaults and mkinitcpio doe

I will call this solved right now, sitting down thinking a bit, it does trace back to libc, which is owned by the arch package glibc.
What I did was a ran a "pacman -Syu glibc" which put in the updates and made it reinstalled libc and now it ran without segfaulting and I seem to have a proper filesize for the initramfs rather than 13 bytes.
I will mark this as solved for now so hopefully if someone else finds this years for now, they may be able to fix theirs.

Thank you both for your help.

Offline

Board footer

Powered by FluxBB