Crashes and read-only file systems

edeloa · 2012-12-26 07:28:12

Notice: I apolgize for the long post, I've tried to be as thorough as possible. I have searched everywhere for possible solutions, but things I've found end up being temporary workarounds or don't apply to my situation. Any help, even as simple as, "have you checked out XYZ log, it's hidden here", would be greatly appreciated. Thanks

I'm not sure what exactly caused the issues below, but they did start to happen within a day of running pacman -Syu. I hadn't run that since I first installed Arch on December 2nd of this year.

Setup:

Thinkpad 2436CTO
UEFI/GPT
SSD drive
Partitions: UEFISYS, Boot, LVM
The LVM is encrypted and is broken up as: /root, /var, /usr, /tmp, /home
All LVM file systems are EXT4 (used to have /var and /tmp as ReiserFS)

The first sign that something was wrong was gnome freezing. Gnome would then crash and I'd get booted back to the shell with all filesystems mounted as read-only. I started having the same issues as this OP:

https://bbs.archlinux.org/viewtopic.php?id=150704

At the time, I had /var and /tmp as ReiserFS, and would also get reiserfs_read_locked_inode errors.

When shutting down (even during non-crashed sessions) I would notice this during shutdown:

Failed unmounting /var
Failed unmounting /usr

Followed by a ton of these:

device-mapper: remove ioctl on <my LVM group> failed: Device or resource busy

Nother of these errors had ever appeared before.

After hours of looking for solutions (and not finding any that worked) I was convinced (without any proof) that my Reiser file systems were corrupt and so I reformatted my entire SSD and started anew - not the Arch way, I know I set all logical volumes as EXT4.

After started anew, I noticed

device-mapper: remove ioctl on LVM_SysGroup failed: Device or resource busy

was still showing up, even with just a stock Arch setup (maybe even when powering off via Arch install ISO, don't remember). After a lot of searching, I found that most people judged it a harmless error, so I ignored it and continued setting up Arch.

I set up Gnome and a basic LAMP server, and everything seemed to work for a couple of hours. Soon after, I got the same old issues back. The System-journald issue came back and per the workaround on https://bbs.archlinux.org/viewtopic.php?id=150704 and a couple other places, I rotated the journals and stopped journald from saving to storage. That seemed to stop THOSE errors from at least overwhelming the shell, but I would still get screen freezes, crashes, and read-only file systems.

I had to force the laptop to power off, since poweroff/reboot/halt commands weren't working (would get errors regarding the filesystems mounted as read-only).

I utilized all disk checking functions possible. From running the tests (SMART test included) that came as part of my laptop's BIOS to full blown fsck. All tests showed the drive was working fine, and Fsck would show everything was either clean, or

Clearing orphaned inode ## (uid=89, gid=89, mode=0100600, size=###
Free blocks count wrong (###, counted=###)

Which I would opt to fix. Nothing serious, though.

I could safely boot back into Arch and use the system fine until the system decides to freeze/crash and do the above all over again.

The sure way of recreating this for me is to run a cron job on a local site I'm developing. After a brief screen freeze (mouse still moveable but everything is otherwise unreponsive) I'll systemctl status mysqld.service and notice that mysqld went down.

It seems that it's at this point my file systems are mounted as read only, as trying to do virtually anything results in:

unable to open /var/db/sudo/...: Read-only file system

After some time, X/Gnome crashes and I get sent back to shell with

ERROR: file_stream_metrics.cc(37)
RecordFileError() err = 30 source = 1 record = 0
Server terminated successfully (0)

Closing log file.or_delegate.h(30)] sqlite erro1, errno 0: SQL logic error or missing database[1157:1179

rm: cannot remove '/tmp/serverauth.teuroEBhtl': Read-only file system

Before all this happened, I was using Arch just fine for a few weeks. I wiped the drives and started anew, and this still happens with just the minimal number of packages installed.

I've searched for solutions to each individual problem, but come across a hack that doesn't solve anything (like turning off storing logs for journal), or the solution doesn't apply to my case.

At this point, I'm so overwhelmed I'm not even sure where exactly to pick up figuring this issue out.

Thanks in advance for any help

cfr · 2012-12-26 22:16:39

Did this occur when you booted from the live/install media?

What is your current set up? That is, partitions, filesystems etc. I take it you have not yet reinstalled X but are in the default CLI following installation?

If turning off log storage didn't help, reenable it so that you may at least stand a chance of finding something useful.

What services, if any, are you running? What non-default daemons etc.?

Does it happen if you keep the machine off line?

Have you done pacman -Syu since installation and dealt with any *.pacnew files?

Last edited by cfr (2012-12-26 22:17:57)

Arch Linux

#1 2012-12-26 07:28:12

Crashes and read-only file systems

#2 2012-12-26 22:16:39

Re: Crashes and read-only file systems

Board footer