You are not logged in.

#1 2010-12-29 14:59:03

jocom
Member
Registered: 2009-04-29
Posts: 74

[SOLVED] Filesystem goes read-only

Hi,

I've got a Samba share on

/srv/samba_share

And /srv is mounted on /dev/sda2.

Now I was copy pasting something in this Samba share on the MS-Windows laptop of my wife. And all of a sudden this wasn't possible anymore because the filesystem was read-only. So I checked my logs

$ su -c 'tail /var/log/messages.log'
Password: 
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 04 c5 01 35 00 00 08 00
Dec 29 13:48:22 localhost kernel: lost page write due to I/O error on sda2
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda] Unhandled error code
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 04 c5 01 35 00 00 08 00
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda] Unhandled error code
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
Dec 29 13:48:22 localhost kernel: sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 04 c5 01 35 00 00 08 00
Dec 29 14:08:22 localhost -- MARK --
Dec 29 14:51:27 localhost -- MARK --

Especially the second line of the output, the lost page write bothers me.

[edit]
By the way, the output of mount says that /srv is mounted (rw) on /dev/sda2.

$ mount
...
/dev/sda2 on /srv type ext4 (rw)
...

[/edit]

  1. What does the lost page write mean?

  2. How do I fix this read-only back to read-write? Note: Several services use the data stored in /srv

  3. How did this happen, and how can I act to make sure it does not happen again?

Any hinst or tips would be great!

jocom

Last edited by jocom (2011-01-01 18:33:49)

Offline

#2 2010-12-30 16:44:14

Daniel_F
Member
From: Brazil
Registered: 2010-12-28
Posts: 69

Re: [SOLVED] Filesystem goes read-only

Hmm....  This looks like hardware failure !!

Is this an internal disk or a USB plugged one ??

Offline

#3 2010-12-30 18:48:52

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

It's an internal disk. How does that help?

jocom

Offline

#4 2010-12-30 21:18:22

Daniel_F
Member
From: Brazil
Registered: 2010-12-28
Posts: 69

Re: [SOLVED] Filesystem goes read-only

If it was a bus-powered USB disk it could be insuficient power from the USB port.

Since it's an internal disk I would say check the cables and start considering the possibility of a defective disk.

To answer your questions on the first post:

1.   What does the lost page write mean?

It means that it failed to write something to the disk. (at this point it is possible that some files got corrupted)


2.   How do I fix this read-only back to read-write? Note: Several services use the data stored in /srv

You could try remounting it with:

mount -o remount,rw /srv

3.   How did this happen, and how can I act to make sure it does not happen again?

Like I said, it may be time to consider replacing the disk.

Backing-up your files would also be a wise thing to do !


Sorry if this is not really helpfull...

Offline

#5 2010-12-30 21:30:25

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

Ok, thanks for your reply.

But the remounting actually still gives the error codes. Probably a reboot helps? But it will not fix the problem...

Offline

#6 2010-12-31 00:24:20

Daniel_F
Member
From: Brazil
Registered: 2010-12-28
Posts: 69

Re: [SOLVED] Filesystem goes read-only

Maybe the filesystem got corrupted...

try running

e2fsck -f -y -v /dev/sda2

Offline

#7 2010-12-31 08:36:41

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

Hmz, there are still problems:

# e2fsck -f -y -v /dev/sda2
e2fsck 1.41.12 (17-May-2010)
e2fsck: Attempt to read block from filesystem resulted in short read while trying to open /dev/sda2
Could this be a zero-length partition?

It clearly is not zero length. It is about 60Gb.

Mounting gives the following problems. I do not know if that helps?

# mount -a
mount: special device LABEL=srv does not exist
# mount /dev/sda2 /srv
mount: you must specify the filesystem type

I want to thank you for trying to help me...

Offline

#8 2010-12-31 15:13:34

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: [SOLVED] Filesystem goes read-only

How recent is the hardware? If it is a machine with state of the art components then it might be the case that there are still a few things to fine tune in the drivers, if the hardware is not very recent then it may be just a flaky connection, the hardware showing its age or some rare error state not handled by the drivers.

I guess you could use smartctl to keep an eye on the smart parameters and see if the hard disk is dying or if it is something else.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#9 2010-12-31 15:40:34

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

Thanks for your reply. The disk is some 4 years old, so it is not that new smile

Could you elaborate a bit more on smartctl? What is it meant for, and how should I use it?

Thank you!

Offline

#10 2010-12-31 16:18:47

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: [SOLVED] Filesystem goes read-only

From your reply I assume the rest of the hardware is also 4 years old, it may be the power supply may be starting to cause problems.

The utility smartctl is provided by smartmontools, it allows you to read the S.M.A.R.T. attributes from the hard disk (google it for more info). The smart parameters should indicate an impending failure of the hard disk, however they are not 100% accurate, if some "Pre-fail" attribute says it is failing then you should backup as soon as possible, otherwise backup anyway tongue and keep an eye on how things behave. It may be just a rare problem caused by an higher load than usual or some other problem not caused by the hard disk itself.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#11 2010-12-31 16:25:33

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

Ok, thanks for that info.

As I've had the problem a few times before, with this same hard disk, I think it indeed is wise to back-up and fix a new disk.
But, before that, I'll give the smartctl utility a try.

Offline

#12 2010-12-31 16:50:18

thisoldman
Member
From: Pittsburgh
Registered: 2009-04-25
Posts: 1,172

Re: [SOLVED] Filesystem goes read-only

jocom,

At this point, it might be wisest to assume hard drive failure is imminent.  If I were in your position I would backup the full drive immediately.

A howto for using smartmontools and smartctl is at the Linux Journal site: http://www.linuxjournal.com/magazine/mo … t?page=0,0.

The home site for Smartmontools wiki is http://sourceforge.net/apps/trac/smartmontools/wiki.

If your problems are partition failure that is software caused, not due to hardware failure Testdisk might help: http://www.cgsecurity.org/wiki/TestDisk.

To run these from a live CD, with your /dev/sda disk unmounted, I'd use SystemRescueCD.  It comes with both smartmontools and testdisk and also the firefox web browser to access online instructions.

Offline

#13 2010-12-31 17:04:19

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

Ok, but how do I back-up /srv, if I can no longer mount it?
Should I try a reboot, it might probably be able to mount, then. But it would cost me my uptime sad [Allthough that probably is not important at the moment.]

Offline

#14 2010-12-31 18:03:09

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: [SOLVED] Filesystem goes read-only

If /srv is on a different partition as it seems to be you can try using ddrescue to make an image of that partition and then try to recover things from the image.

If it is an error caused by software then you can take your time to try to recover the data (not counting the downtime nuisance), however if the problem is with the hardware I would try to make an image of the whole disk, replace it and recover/restore the data from the image. Not wanting to scare you (too much tongue) if it is an hardware problem you better backup everything you can before the disk fails for good.

After this is sorted out you need to take some time to consider the best course of action, either keep business as usual and expect things don't break, do regular backups, which we all talk about and recommend but probably not many of us do properly tongue, or consider using some redundancy in the form of raid (it's not a replacement for regular backups though).


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#15 2010-12-31 19:03:59

thisoldman
Member
From: Pittsburgh
Registered: 2009-04-25
Posts: 1,172

Re: [SOLVED] Filesystem goes read-only

I'll second ddrescue.  But here's an alternative using dd, tested by creating an image of a mountable /var partition, while it was unmounted.  I tested all steps just 5 minutes ago.

mkdir /media/image
dd if=/dev/sda2 of=/media/image/sda2.img    # Create an image
e2fsck -f /media/image/sda2.img             # Correct any errors if possible
mkdir /mnt/custom                           # Create a mount point
mount -o loop /media/image/sda2.img /mnt/custom    # Mount the image as a loop device

If it's successful, you can then make your backups.  If it's completely successful, you can dd the image back to the partition.

dd if=/media/image/sda2.img of=/dev/sda2
e2fsck -f /dev/sda2

Offline

#16 2011-01-01 13:43:25

jocom
Member
Registered: 2009-04-29
Posts: 74

Re: [SOLVED] Filesystem goes read-only

Thanks for all these great tips!

As R00KIE pointed out I don't have a good backup system. I think I will go to my computer shop next monday to fix some good backup drive, because this quite scared me actually.

As for now I acted as follows:

  1. Reboot, try to remount /srv and /opt [they are on the same disk].
    Result: Bulks of errors. Lots of "soft resetting link" and "link to slow".

  2. Another reboot, which kicked me in single user mode. I was had to remount / and was able to alter /etc/fstab, so that /srv and /opt were not mounted by default.

  3. Shutdown system, open the casing.

  4. Clean internals with vacuum cleaner. Unplug all harddisk cables and plug them back again.

  5. Fire up the system. Mount /srv.

  6. Try to make backup of important data on /srv.

This seemed to work, and there are no errors anymore. Probably some cable was loose, somehow. But I still do not really trust the situation, and will fix some backup routine.

Thanks again for all the tips and tricks. I will definitely run some extra checks and utilities from now on!

jocom

P. S.: Running "e2fsck -f -y -v" on both /srv and /opt says everything is fine. No errors, no bad blocks.

Last edited by jocom (2011-01-01 13:50:06)

Offline

#17 2011-01-01 16:03:58

Daniel_F
Member
From: Brazil
Registered: 2010-12-28
Posts: 69

Re: [SOLVED] Filesystem goes read-only

Glad to see you got it working !    Please add [SOLVED] to the topic's title.

Offline

#18 2011-01-01 16:13:42

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: [SOLVED] Filesystem goes read-only

Probably just some loose cable as you say or some oxidation in the contacts (will sata prove less reliable than pata in that respect?).

Either way you were lucky and the fallbacks worked as they should (remounting read only) and it seems there was no data loss (the hard disk will probably still be alive and kicking for a long while). All that is left to say is: nothing better than a good scare to make all of us setup some sort of backup strategy tongue


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

Board footer

Powered by FluxBB