Partition becomes corrupted?

nomorewindows · 2014-02-07 23:17:10

I have a ext3fs that is repeatedly getting corrupt. It happens to be my /var/cache/pacman/pkg partition. It happened about a month ago and I recreated the partition with mkfs.ext3 and started all over again. I'm about to fill the partition up again, and when I run pacman I get

:: Proceed with installation? [Y/n] 
:: Retrieving packages ...
error: failed retrieving file 'wxpython2.8-2.8.12.1-1-x86_64.pkg.tar.xz' from xxx.xxx.xxx.xxx:15678 : Failed writing body (20 != 1440)
error: failed retrieving file 'wxpython2.8-2.8.12.1-1-x86_64.pkg.tar.xz' from mirrors.kernel.org : Failed writing body (20 != 1440)
warning: failed to retrieve some files
error: failed to commit transaction (download library error)
Errors occurred, no packages were upgraded.

I thought a first it was my pacserve giving me fits because it did it last night on a package file containing ++ and that pacserve didn't parse ++ with its' http standard filenames.
Then I figured out it was my partition acting up again.
I unmount /var/cache/pacman/pkg and run e2fsck on it and I get endless errors on just about any inode on the partition.

R00KIE · 2014-02-07 23:25:28

Did you check the drive's smart values?

nomorewindows · 2014-02-07 23:39:52

R00KIE wrote:

Did you check the drive's smart values?

The Seek error rate has a high value, and the Hardware ECC Recovered is up, but this is still a good number if its' recovering them, and realloc sector count is still a single digit. I have notcied at times a funny message in the past about the bus resetting itself. I haven't seen them in awhile, but here's dmesg as of now

EXT4-fs (sda12): mounting ext3 file system using the ext4 subsystem
EXT4-fs (sda12): warning: mounting fs with errors, running e2fsck is recommended
EXT4-fs (sda12): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs error (device sda12): ext4_validate_block_bitmap:368: comm pacman: bg 18: block 589824: invalid block bitmap
EXT4-fs error (device sda12): ext4_validate_block_bitmap:368: comm pacman: bg 19: block 622592: invalid block bitmap
EXT4-fs error (device sda12): ext4_validate_block_bitmap:368: comm pacman: bg 20: block 655360: invalid block bitmap
EXT4-fs error (device sda12): ext4_validate_block_bitmap:368: comm pacman: bg 23: block 753666: invalid block bitmap
EXT4-fs (sda12): error count: 8
EXT4-fs (sda12): initial error at 1391347281: ext4_validate_block_bitmap:368
EXT4-fs (sda12): last error at 1391813322: ext4_validate_block_bitmap:368

I did a badblocks /dev/sda12 and it didn't come back with anything.

Last edited by nomorewindows (2014-02-07 23:48:47)

R00KIE · 2014-02-08 02:05:38

High Seek Error Rate, Raw Read Error Rate, and Hardware ECC Recovered values might be normal depending on the drive's manufacturer. However, if Reallocated_Event_Count or Reallocated_Sector_Ct are not zero I'd say it's a good idea to start doing good backups.

From personal experience, if you have doubts about the hard disk, let badblocks try to do 10 passes without errors on the disk (badblocks -w -p10). I've seen 2 disks (same manufacturer and model) fail intermittently, at random locations, without triggering any big increment in any of the smart values one would expect to see increasing on a bad disk. This could happen right on the first pass or on the 2nd or 3rd (or any other, the key word is random) and then do a few good passes and then spit out a lot of errors, rinse and repeat.

alphaniner · 2014-02-08 02:26:08

+1 @ROOKIE.

Today I had a disk (Seagate 7200.11 series) pass the manufacturer's short test, a single badblocks write pass, and a smartctl long test, only to then have the manufacturer's long test find sectors needing re-allocation.

nomorewindows · 2014-02-08 02:32:40

R00KIE wrote:

High Seek Error Rate, Raw Read Error Rate, and Hardware ECC Recovered values might be normal depending on the drive's manufacturer. However, if Reallocated_Event_Count or Reallocated_Sector_Ct are not zero I'd say it's a good idea to start doing good backups.

Reallocated_Sector_Ct got a couple just after I replaced the disk with the one I'm using now, and it hasn't increased since then. Just some sort of fluke I guess.

R00KIE wrote:

From personal experience, if you have doubts about the hard disk, let badblocks try to do 10 passes without errors on the disk (badblocks -w -p10). I've seen 2 disks (same manufacturer and model) fail intermittently, at random locations, without triggering any big increment in any of the smart values one would expect to see increasing on a bad disk. This could happen right on the first pass or on the 2nd or 3rd (or any other, the key word is random) and then do a few good passes and then spit out a lot of errors, rinse and repeat.

I'll have to give a try I guess, but it seems as if ext4 is doing things to ext3 that is the result of some bug. When I'm using e2fsck on it, it acts as if it is a totally foreign filesystem. It starts reporting strange messages such as the inodes are in use (unmounted), and that certain features are flagged on the filesystem that haven't been adjusted for the filesystem, using something like tune2fs or just being formatted ext4.

nomorewindows · 2014-02-08 09:52:13

Ran badblocks -w -p10 with no reported bad blocks and no significant change in smarts.

nomorewindows · 2014-02-08 14:05:22

After mounting and umounting and running e2fsck it's allowing me to download again. Strange I haven't fixed anything.

R00KIE · 2014-02-08 14:17:18

Then I guess you should give ext4 (or some other filesystem) a go for a while in that partition and see if it breaks. Any special reason why you are using ext3 and not ext4?

On the other hand, 10 passes of badblocks went quite fast for you so I suppose the partition is quite small. The test I mentioned earlier was a full 1TB disk test, with one pass meaning 4 full writes and 4 full reads (average speed maybe around 100MB/s), this means that in some runs I saw errors only after 24H of continuous testing.

@alphaniner
As far as I could see so far, when doing a full test with the manufacturer's tool it just does a smart full test. I suppose the difference is that the manufacturer's tool knows exactly how to interpret the smart values and may be able to query for some extra data.

nomorewindows · 2014-02-08 16:21:03

R00KIE wrote:

Then I guess you should give ext4 (or some other filesystem) a go for a while in that partition and see if it breaks. Any special reason why you are using ext3 and not ext4?

I probably had no particular reason. I had thought that while I was trying to install FreeBSD under MBR it had clobbered my partition in an attempt to get it installed. I had to convert over to GPT and apparently that wasn't the problem. If this is a problem with ext3/4, then maybe it needs a bug report.

R00KIE wrote:

On the other hand, 10 passes of badblocks went quite fast for you so I suppose the partition is quite small. The test I mentioned earlier was a full 1TB disk test, with one pass meaning 4 full writes and 4 full reads (average speed maybe around 100MB/s), this means that in some runs I saw errors only after 24H of continuous testing.

It's small because laptops don't have the capacity to begin with. And for everything else on the network package updates utilize pacserve.

WorMzy · 2014-02-08 17:40:16

Have you checked your RAM? I've seen fsck tools repeatedly reporting a filesystem as damaged, 'fixing' the problem, then complaining again at the next boot. A quick memtest86+ run in these scenarios usually finds the problem. Not saying it will here, but it's worth checking.

nomorewindows · 2014-02-08 18:33:22

WorMzy wrote:

Have you checked your RAM? I've seen fsck tools repeatedly reporting a filesystem as damaged, 'fixing' the problem, then complaining again at the next boot. A quick memtest86+ run in these scenarios usually finds the problem. Not saying it will here, but it's worth checking.

I just swapped out the memory in this box to maximize use of memory in other boxes, and I think I ran a complete test on the memory putting it in. Nothing was wrong with the memory I took out.

nomorewindows · 2014-02-10 14:36:41

Oops! My bad. I didn't have the partition mounted, and now it won't mount. That's the only reason ATM that the pacman was working. I have no option now but to hose the partition with mkfs.ext4.

nomorewindows · 2014-02-19 16:19:24

Wasn't sure the browser took.

Last edited by nomorewindows (2014-02-19 16:21:15)

nomorewindows · 2014-02-19 16:20:35

It's been awhile since I've seen these messages, but plugging and unplugging the drive seems to trigger these and after powering up here they are again:

ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0xe frozen
ata3.00: irq_stat 0x00400000, PHY RDY changed
ata3: SError: { PHYRdyChg CommWake }
ata3.00: failed command: SET FEATURES
ata3.00: cmd ef/05:fe:00:00:00/00:00:00:00:00/40 tag 0
         res 50/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
ata3.00: status: { DRDY }
ata3: hard resetting link
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata3.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out
ata3.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out
ata3.00: configured for UDMA/100
ata3: EH complete

alphaniner · 2014-03-05 19:41:23

R00KIE wrote:

let badblocks try to do 10 passes without errors on the disk

The disk I mentioned has been running for over 500 hours like this.

Obviously it's toast, but I'm curious to see if it will ever complete...

R00KIE · 2014-03-05 21:19:51

alphaniner wrote:

R00KIE wrote:
let badblocks try to do 10 passes without errors on the disk
The disk I mentioned has been running for over 500 hours like this.
Obviously it's toast, but I'm curious to see if it will ever complete...

Now that is something, I usually canceled the tests after 2 or 3 days if I saw that errors kept coming randomly (if I had a drive that did 10 full passes without errors and smart parameters were ok, I would assume the drive to be ok).

I don't remember if I ever let the drive "work" for longer before it was eventually replaced. I did test that drive in a different computer and errors still kept coming so I suppose chipset/mobo/cable/psu problems were ruled out.

Now I'm just waiting for the day when 2 other identical drives start throwing a fit too (faulty drive was 1 of 3 identical drives, same manufacturer, same model and probably same manufacturing batch judging by the serial numbers).

nomorewindows · 2014-03-05 23:16:38

I still hear my drive beeping from time to time. My partition is now full with formatted ext4 and no indication that I'm aware of for corruption.

Last edited by nomorewindows (2014-03-05 23:18:33)

alphaniner · 2014-03-06 14:41:24

Your drive beeps?

nomorewindows · 2014-03-06 15:50:24

alphaniner wrote:

Your drive beeps?

It is a hardware diagnostic located on the drive itself if it knows about some kind of problem. It's a seagate and they've had this feature for quite some time.

alphaniner · 2014-03-06 16:12:21

I'm familiar with several families of Seagate drives, and I've never seen one with an onboard beeper.

nomorewindows · 2014-03-06 16:54:11

alphaniner wrote:

I'm familiar with several families of Seagate drives, and I've never seen one with an onboard beeper.

I haven't actually seen the beeper myself, but I know I had a hold of one that didn't pass the spin up cycle and I could hear it beeping (just like a BIOS beep).

Arch Linux

#1 2014-02-07 23:17:10

Partition becomes corrupted?

#2 2014-02-07 23:25:28

Re: Partition becomes corrupted?

#3 2014-02-07 23:39:52

Re: Partition becomes corrupted?

#4 2014-02-08 02:05:38

Re: Partition becomes corrupted?

#5 2014-02-08 02:26:08

Re: Partition becomes corrupted?

#6 2014-02-08 02:32:40

Re: Partition becomes corrupted?

#7 2014-02-08 09:52:13

Re: Partition becomes corrupted?

#8 2014-02-08 14:05:22

Re: Partition becomes corrupted?

#9 2014-02-08 14:17:18

Re: Partition becomes corrupted?

#10 2014-02-08 16:21:03

Re: Partition becomes corrupted?

#11 2014-02-08 17:40:16

Re: Partition becomes corrupted?

#12 2014-02-08 18:33:22

Re: Partition becomes corrupted?

#13 2014-02-10 14:36:41

Re: Partition becomes corrupted?

#14 2014-02-19 16:19:24

Re: Partition becomes corrupted?

#15 2014-02-19 16:20:35

Re: Partition becomes corrupted?

#16 2014-03-05 19:41:23

Re: Partition becomes corrupted?

#17 2014-03-05 21:19:51

Re: Partition becomes corrupted?

#18 2014-03-05 23:16:38

Re: Partition becomes corrupted?

#19 2014-03-06 14:41:24

Re: Partition becomes corrupted?

#20 2014-03-06 15:50:24

Re: Partition becomes corrupted?

#21 2014-03-06 16:12:21

Re: Partition becomes corrupted?

#22 2014-03-06 16:54:11

Re: Partition becomes corrupted?

Board footer