Arch Linux

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

[BTRFS Permissions] Almost Everything is Read-Only After Updating

I have no idea what exactly caused this but I'm in a painfully difficult situation where the vast majority of files / directories on my OS drive are read-only and "sudo chmod -R g+rwx" / "sudo chmod -R u+rwx" doesn't fix the problem.
It's severe enough that I can't even update pacman databases because of it:

neko-san@ARCH ~> doas pacman -Syu
doas (neko-san@ARCH) password: 
:: Synchronizing package databases...
error: failed to synchronize all databases (unable to lock database)
neko-san@ARCH ~ [1]> 

The only inkling of an idea I've been able to glean from this mess is that it might be BTRFS related but I have no idea:

neko-san@ARCH ~> btrfs check --readonly --progress /dev/nvme0n1p2
Opening filesystem to check...
ERROR: /dev/nvme0n1p2 is currently mounted, use --force if you really intend to check the filesystem
neko-san@ARCH ~ [1]> btrfs check --readonly --progress --force /dev/nvme0n1p2
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/nvme0n1p2
UUID: 5761f276-9f47-4195-a5b2-c21ff4f3fea5
[1/7] checking root items                      (0:00:01 elapsed, 1265039 items checked)
data backref 35500421120 root 5 owner 84289339 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 35500421120 root 5 owner 84289339 offset 0 found 1 wanted 0 back 0x55c612d23640
incorrect local backref count on 35500421120 root 5 owner 84289339 offset 4096 found 0 wanted 1 back 0x55c6085409e0
backref disk bytenr does not match extent record, bytenr=35500421120, ref bytenr=0
backpointer mismatch on [35500421120 1486848]
[2/7] checking extents                         (0:00:05 elapsed, 133603 items checked)
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache                (0:00:01 elapsed, 203 items checked)
[4/7] checking fs roots                        (0:00:03 elapsed, 109210 items checked)
[5/7] checking csums (without verifying data)  (0:00:01 elapsed, 569621 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 8 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 200912424960 bytes used, error(s) found
total csum bytes: 193258168
total tree bytes: 2188738560
total fs tree bytes: 1795637248
total extent tree bytes: 136314880
btree space waste bytes: 395675114
file data blocks allocated: 379134967808
 referenced 198146822144
neko-san@ARCH ~> uname -a
                                            btrfs --version
                                            btrfs fi show
                                            btrfs fi df /
Linux ARCH 5.17.9-258-tkg-pds-llvm #1 TKG SMP PREEMPT Fri, 20 May 2022 02:30:55 +0000 x86_64 GNU/Linux
btrfs-progs v5.17
Label: none  uuid: 5761f276-9f47-4195-a5b2-c21ff4f3fea5
        Total devices 1 FS bytes used 187.11GiB
        devid    1 size 198.11GiB used 198.11GiB path /dev/nvme0n1p2

Label: 'extraStorage'  uuid: 9d14e17d-66d4-41e7-95a4-8125f75b4d2b
        Total devices 1 FS bytes used 2.27TiB
        devid    1 size 2.32TiB used 2.31TiB path /dev/sda3

Data, single: total=194.01GiB, used=185.06GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=4.09GiB, used=2.05GiB
GlobalReserve, single: total=374.73MiB, used=0.00B
neko-san@ARCH ~> doas dmesg > /mnt/extraStorage/dmesg.log
doas (neko-san@ARCH) password:
neko-san@ARCH ~>

dmesg.log

I've sent for help on the BTRFS mailing list and this is the response I've received thus far on the matter:

On Fri, May 20, 2022 at 09:13:25PM +0000, Neko-san wrote:
> Greetings,
>
> I'm having a pretty severe issue right now where I don't have permission access to many of the files / directories on my system (namely my home folder and some others) and superuser privileges aren't helping to fix these issues. I'd really like to not have to reinstall Arch Linux just because of this... I'd appreciate assistance...
>
> I have no idea what's caused it and this is the only sensible conclusion I've been able to come to thus far...
>
> ```
> neko-san@ARCH ~> btrfs check --readonly --progress /dev/nvme0n1p2
> Opening filesystem to check...
> ERROR: /dev/nvme0n1p2 is currently mounted, use --force if you really intend to check the filesystem

   You can't expect an fsck on a mounted filesystem to produce any useful output.

> neko-san@ARCH ~ [1]> btrfs check --readonly --progress --force /dev/nvme0n1p2

   ... and so this is useless, I'm afraid.

[snip]

> [ 1646.159335] systemd-journald[387]: Failed to rotate /var/log/journal/63f413b0af0c43ae9345a082aaf00bd3/user-1000.journal: Read-only file system

   That would be a read-only filesystem, then. Most likely caused by
some serious error on the FS, which caused it to go read-only to
protect itself. You'll have to go back in dmesg to see what happened
to cause that. There's most likely a kernel oops in there at around
the same time that the systemd journal started spewing those errors.

   Hugo.

--
Hugo Mills             | You shouldn't anthropomorphise computers. They
hugo@... carfax.org.uk | really don't like that.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

If anyone else familiar with how to help with this issue, I'd greatly appreciate it...

Last edited by N3k0-san (2022-05-20 21:56:48)

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

New dmesg.log - provides the information about an error in btrfs

Zod

Member

From: Hoosiertucky

Registered: 2019-03-10

Posts: 629

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

I'm guessing it's not a really good idea to cause people to download that file.

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

What, the log? I hope that's a joke.
The log was output to a different file system; it couldn't possibly be problematic regardless because the file system with the problem is read-only.
At least make some sense before you think about saying things like that.

Last edited by N3k0-san (2022-05-20 23:27:16)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 21,617

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

SMART log on that nvme drive? https://wiki.archlinux.org/title/Solid_ … NVMe#SMART

Irets

Member

Registered: 2019-02-01

Posts: 143

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

What, the log? I hope that's a joke.

For future, it would *a lot* better if you'd use something like http://ix.io/ instead of sending Discord links of .tar.gz archives containing text files.

https://wiki.archlinux.org/title/List_o … n_services

Last edited by Irets (2022-05-23 15:12:54)

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

@Irets
I wasn't even aware the Arch wiki had information on pastebin services; also, it hadn't occurred to me that using direct links hosted by Discord's CDN would've been an issue but I'll keep that in mind

@V1del
I'll try that in a minute and come back with a log

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

@Videl smart.log

It's almost as if each subsequent reboot I do makes things worse; I rebooted and my browser extensions are inexplicably gone and pacman is essentially broken because it can't upgrade anything

Yesterday, I had downgraded btrfs-progs to one version down in a feeble attempt to see if it was just the package causing this to no avail; now, I can't even upgrade it back to 5.17 because it throws weird messages about files for the package already being on the file-system... :s

arkapravoghosh1

Member

Registered: 2021-12-20

Posts: 6

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

I had the same problem recently and I had to reinstall Linux. I'm using Kali Linux with BTRFS as of now cuz last week after some update my Arch Linux BTRFS converted the whole root partition to read-only file system I don't even know why. I thought I was the only one having this problem.

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

Still haven't received any response from anyone in the BTRFS mailing list in regards to helping with this issue...
I expected BTRFS to be more stable than this... Like, I understand it's still under development but what isn't in open-source?
I've been recommending it for quite some time now and now I have to "stick my foot in my mouth," so to speak, because I can't even get support for when something goes wrong with it...

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 21,617

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

Are you able to update the kernel? There have been some btrfs fixes in at least 5.17.7 maybe you just happened to get a build that has some fault? FWIW smart looks fine so unlikely to be a HW fault. Can you access the drive properly from a live disk?

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

My kernel is 5.17.9... I've also tried a different kernel 5.17 with the same problem... (Not that I can really install packages anymore with pacman; attempting to perform pacman transactions *looks* like they go through initially but they don't and ends up triggering the BTRFS issue afterwards...)
I can try accessing from a live disk but I wouldn't be sure where to go from there

N3k0-san

Member

From: Elsweyr

Registered: 2021-08-14

Posts: 67

Re: [BTRFS Permissions] Almost Everything is Read-Only After Updating

Still haven't got any word from the mailing list...
As for the drive being mounted, I booted into the live ISO and ran it unmounted and I got this as a result:

[1/7] checking root items
[2/7] checking extents
data backref 35500421120 root 5 owner 84289339 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 35500421120 root 5 owner 84289339 offset 0 found 1 wanted 0 back 0x5563c0ab36b0
incorrect local backref count on 35500421120 root 5 owner 84289339 offset 4096 found 0 wanted 1 back 0x5563bd7460c0
backref disk bytenr does not match extent record, bytenr=35500421120, ref bytenr=0
backpointer mismatch on [35500421120 1486848]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: 5761f276-9f47-4195-a5b2-c21ff4f3fea5
found 201850998784 bytes used, error(s) found
total csum bytes: 194338616
total tree bytes: 2213855232
total fs tree bytes: 1821523968
total extent tree bytes: 134987776
btree space waste bytes: 395442185
file data blocks allocated: 370105331712
 referenced 199064424448

However, the shell then threw some kind of error after this for some reason
Wish I could get help with this; I've had this issue for basically a month now and I have people breathing down my neck to do stuff that requires me to have this machine...