[SOLVED] Bad Storage -- md5 fails

fukawi2 · 2010-07-26 10:14:55

OK, so I've obviously got something dodgy going on here which highly concerns me

fukawi2@phil-desktop /media/Torrents/Complete  $ md5sum LARGE_3.5G_FILE.iso 
a0b2dd3bc198702c60208dab37190dc4  LARGE_3.5G_FILE.iso
fukawi2@phil-desktop /media/Torrents/Complete  $ md5sum LARGE_3.5G_FILE.iso 
bb9fa9dba8c6e0bdfa3aa05bd9bd18e4  LARGE_3.5G_FILE.iso
fukawi2@phil-desktop /media/Torrents/Complete  $ md5sum LARGE_3.5G_FILE.iso 
9a81917842e5a82e8883b68ccbc8b061  LARGE_3.5G_FILE.iso

So what are your thoughts? Bad Disk? Bad Controller? Bad Mobo?

This particular partition is ext4 on a software RAID-1 of 2 x Seagate 1tb SATA drives.

Same disks, different LVM volume testing with a 1gb file:

fukawi2@phil-desktop /home/fukawi2  $ md5sum ryla2010-2010.07.09_22-23-05.m2t 
a5cc1e489ad086eeb6146586d90c96ba  ryla2010-2010.07.09_22-23-05.m2t
fukawi2@phil-desktop /home/fukawi2  $ md5sum ryla2010-2010.07.09_22-23-05.m2t 
a5cc1e489ad086eeb6146586d90c96ba  ryla2010-2010.07.09_22-23-05.m2t
fukawi2@phil-desktop /home/fukawi2  $ md5sum ryla2010-2010.07.09_22-23-05.m2t 
a5cc1e489ad086eeb6146586d90c96ba  ryla2010-2010.07.09_22-23-05.m2t
fukawi2@phil-desktop /home/fukawi2  $ md5sum ryla2010-2010.07.09_22-23-05.m2t 
a5cc1e489ad086eeb6146586d90c96ba  ryla2010-2010.07.09_22-23-05.m2t

Makes me think it might just be a corrupt file system...?

Last edited by fukawi2 (2010-07-30 22:55:30)

karol · 2010-07-26 10:24:39

> So what are your thoughts? Bad Disk? Bad Controller? Bad Mobo?
Yes ;P

I see that the file is in '/media/Torrents/Complete' but maybe it's not complete, it's still growing.

fukawi2 · 2010-07-26 10:26:39

karol wrote:

I see that the file is in '/media/Torrents/Complete' but maybe it's not complete, it's still growing.

Nope -- Deluge says it's finished, and even if that's not the case, deluge is definitely not running, so it cant be touching the file

mikesd · 2010-07-26 10:45:41

Ouch!

I imagine this is what happens when the individual RAID devices get out of sync or your filesystem has serious problems. Only a guess.

I would be interested in seeing the output of:

# echo check > /sys/block/md1/md/sync_action

where md1 is replaced with your software RAID device. A fsck would be an idea too.

To be honest I would be wary of doing anything to that filesystem or its underlying devices at the moment. Might be best to mount it read only and try to copy as much data off as possible, or dig out your backups.

There was a similar problem mentioned in one of the linux.conf.au talks. Don't think they worked out the problem unfortunately.
http://2009.r2.co.nz/20100118/mfc-tu-2.htm

Good luck!

fukawi2 · 2010-07-26 10:52:36

hmm, didn't think to check that... The RAID is currently rebuilding, but I dunno if that's going to be scrooing it.

I'll let that resync finish, retest and if it's still failing then I'll keep debugging

EDIT: I don't think I got to see that pres @ LCA. Thx for the link

Last edited by fukawi2 (2010-07-26 10:53:00)

knopwob · 2010-07-26 13:00:27

i guess you did this already, but have you checked with

 lsof | grep LARGE_3.5G_FILE.iso

that there's no other program having this file open?

If it's caused by the RAID rebuild, then there's something broken. AFAIK RAID should make sure, that only the correct data is shown, even if the disks go out of sync. At least that's what my understandment of RAID is.

fukawi2 · 2010-07-26 23:02:45

lsof definitely shows nothing has the file open.

The RAID rebuild is complete, but it's still FUBAR

root@phil-desktop /media/Torrents/Complete # for X in 1 2 3 ; do md5sum LARGE_3.5G_FILE.iso ; done
9e33a0a525da0cbdf99cfcb5c7e5f68c  LARGE_3.5G_FILE.iso
95d94bb513dd3bd0ed0ed4b080e79a8a  LARGE_3.5G_FILE.iso
93e33864f9985a3d2d13ed901c81ec8d  LARGE_3.5G_FILE.iso

I'm running a check over md2 as per mikesd suggestion now; 120min to go....

knopwob · 2010-07-26 23:14:21

just found this http://osdir.com/ml/gnu.textutils.bugs/ … 00014.html

I guess, this doesn't help you very much, but it looks very similar to your problem. So maybe it's a problem with md5sum itself

Last edited by knopwob (2010-07-26 23:14:43)

karol · 2010-07-26 23:18:49

> maybe it's a problem with md5sum itself

funsum () { sudo < /dev/urandom tr -dc a-z0-9 | dd bs=8 count=4 2>/dev/null && echo "  $1"; }

Unfortunately, fukawi2's md5sum is reported to work on another LVM volume so it's not the case.

Last edited by karol (2010-07-26 23:19:33)

fukawi2 · 2010-07-26 23:20:53

knopwob wrote:

just found this http://osdir.com/ml/gnu.textutils.bugs/ … 00014.html

Thanks for the link, but same problem with sha1sum

root@phil-desktop /media/Torrents/Complete # for X in 1 2 3 ; do sha1sum LARGE_3.5G_FILE.iso  ; done
ef501fa0955277420ae52abddba41916eaa514ce  LARGE_3.5G_FILE.iso 
365423ed59d0c07e3a4561d6d706e1764c6a7828  LARGE_3.5G_FILE.iso 
365423ed59d0c07e3a4561d6d706e1764c6a7828  LARGE_3.5G_FILE.iso

karol · 2010-07-26 23:25:06

But the last two are the same!

fukawi2 · 2010-07-26 23:30:56

karol wrote:

But the last two are the same!

Fluke? Really confusing me this is

root@phil-desktop /media/Torrents/Complete # for X in `seq 1 10` ; do sha1sum LARGE_3.5G_FILE.iso ; done
2b1134d8c7438c552d7b94695d8bfe02c23cd1d9  LARGE_3.5G_FILE.iso
ef501fa0955277420ae52abddba41916eaa514ce  LARGE_3.5G_FILE.iso
e4bff3d91dae9eb9ad1fcaa95cc32997a664bb27  LARGE_3.5G_FILE.iso
c2e9dbdbdad95ae0e739376b64b31b9a61473305  LARGE_3.5G_FILE.iso
365423ed59d0c07e3a4561d6d706e1764c6a7828  LARGE_3.5G_FILE.iso
9e230292fd39ccd30ae77ecba109fbfbac812ba1  LARGE_3.5G_FILE.iso
954394e26c04787195a7273389b2dd0fa1a08879  LARGE_3.5G_FILE.iso
ef501fa0955277420ae52abddba41916eaa514ce  LARGE_3.5G_FILE.iso
ef501fa0955277420ae52abddba41916eaa514ce  LARGE_3.5G_FILE.iso
ef501fa0955277420ae52abddba41916eaa514ce  LARGE_3.5G_FILE.iso

knopwob · 2010-07-26 23:35:44

have you tried copying the file to another harddrive, or usb-stick or what else you have around? just to get a hint to what the problem would be? If it still behaves the same on another drive (preferably formatted with another filesystem) you can at least sort out, that it's not a problem with that particular drive.

karol · 2010-07-26 23:41:40

To handle this case one needs a black belt in voodoo I suppose.

http://serverfault.com/questions/36966/ … 751#149751

knopwob · 2010-07-27 00:09:27

when it's not a problem with the filesystem, maybe something like this would help to get some conculsions:

mount the file and let something like this run over it:

#!/bin/bash - 
for file in $(find .); do                                                                                                                                                                                                     
    if [ -d $file ]; then
        continue
    fi
    echo "testing $file"
    first_sum=$( sha1sum $file)
    for X in $(seq 1 10); do
        another_sum=$(sha1sum $file)
        if [ "$first_sum" != "$another_sum" ]; then
            echo "different sums in $file"
        fi
    done
done

(skript untested)

fukawi2 · 2010-07-27 01:24:34

Good idea knopwob

Lots of files match OK, but lots of files fail too:

different sums in ./Billy_Joel-Greatest_Hits_Volumes_I_And_II-(1973-1985)-2CD-1985-EOS/102-billy_joel-captain_jack-eos.mp3
different sums in ./Billy_Joel-Greatest_Hits_Volumes_I_And_II-(1973-1985)-2CD-1985-EOS/102-billy_joel-captain_jack-eos.mp3
different sums in ./Billy_Joel-Greatest_Hits_Volumes_I_And_II-(1973-1985)-2CD-1985-EOS/102-billy_joel-captain_jack-eos.mp3
different sums in ./Andre Rieu/In Wonderland/CD 1/13 The Impossible Dream.mp3
different sums in ./Andre Rieu/In Wonderland/CD 1/13 The Impossible Dream.mp3
different sums in ./Andre Rieu/In Wonderland/CD 1/13 The Impossible Dream.mp3
different sums in ./Andre Rieu/In Wonderland/CD 1/13 The Impossible Dream.mp3
different sums in ./Hip Hop RnB 2008 50 minutes club REMIX/Hip Hop RnB 2008 50 minutes club REMIX (Feat Missy Elliot, Ludacris, JayZ, 2pac, Busta Rhymes, 112 Etc).mp3
different sums in ./VA-Hip_Hop_The_Collection_The_Classics-3CD-2008-WRE/217-ludacris-stand_up.mp3
different sums in ./Michael Jackson - Discography (320kbps)/[1991] Dangerous/Michael Jackson - 07 - Heal The World.mp3
different sums in ./Michael Jackson - Discography (320kbps)/[1991] Dangerous/Michael Jackson - 07 - Heal The World.mp3
different sums in ./Michael Jackson - Discography (320kbps)/[xxxx] Unreleased & Other/Michael Jackson - We Are The World (Demo).mp3
different sums in ./Michael Jackson - Discography (320kbps)/[xxxx] Unreleased & Other/Michael Jackson - We Are The World (Demo).mp3
different sums in ./Michael Jackson - Discography (320kbps)/[xxxx] Unreleased & Other/Michael Jackson - We Are The World (Demo).mp3
different sums in ./Michael Jackson - Discography (320kbps)/[xxxx] Unreleased & Other/Michael Jackson - Tease Me.mp3
different sums in ./Michael Jackson - Discography (320kbps)/[xxxx] Unreleased & Other/Michael Jackson - Tease Me.mp3
different sums in ./Michael Jackson - Discography (320kbps)/[xxxx] Unreleased & Other/Michael Jackson - Tease Me.mp3
different sums in ./Trace_Adkins-Dangerous_Man-2006-RNS/02-trace_adkins-ladies_love_country_boys.mp3

I think I may have damaged something on the mobo when I built this computer because I put some wrong RAM in (2.1v RAM but mobo doesn't support RAM that high) and it took me a while to realize. This machine locks up "randomly" (espeically under high disk I/O) and performance of the on-board GB NIC is shocking (2MB/s when everything else on the LAN can get up to 40 - 50MB/s).

I'm going to buy a replacement CPU, mobo and RAM at lunch time

Last edited by fukawi2 (2010-07-27 01:24:58)

karol · 2010-07-27 08:46:31

> I'm going to buy a replacement CPU, mobo and RAM at lunch time.
One weird diet you have down under. Are you having fries with that? ;-)

Google said it's either a failing drive or memory, but some people said they run all the test and come out empty.

fukawi2 · 2010-07-27 22:43:13

karol wrote:

> I'm going to buy a replacement CPU, mobo and RAM at lunch time.
One weird diet you have down under. Are you having fries with that? ;-)

Well I did with maccas
Apparently socket 775 is redundant already so I can't just replace the mobo and RAM which makes my nice Q8400 quad-core useless I'm getting a core i5 instead. Upgrading my system drive to a SSD too (why the hell not? )
They were out of stock of the mobo and RAM I want, so will have to wait a few days before I can finish the upgrade.

karol wrote:

Google said it's either a failing drive or memory, but some people said they run all the test and come out empty.

The RAID check returned OK so I dont think it's the drives. I'm sure I ran memtest overnight after I'd finished building this machine last time, but with 8gb RAM it took forever so can't remember exactly what the outcome was or if it finished. I'm sure I wouldn't have left it if it had shown errors though. I guess we'll find out after I rebuild with the i5!

Thanks everyone for you input

karol · 2010-07-27 22:52:04

fukawi2 wrote:

I guess we'll find out after I rebuild with the i5!

It reminds me of a Steve and Bill joke:
Gates: Yesterday I lost keys to my house.
Jobs: Really? What did you do then?
Gates: I bough a new one.

fukawi2 · 2010-07-27 22:54:06

Yeah, it might be a bit of overkill, but the "random" freezes on I/O are driving me insane, especially when I'm trying to rip RAW HD footage from my video camera and it dies half-way through a tape
Plus, I can write it off on tax

R00KIE · 2010-07-27 23:57:05

I don't know how much ram you have in that box but if the 3.5GB file can get cached it may point to bad ram.

Why do I say this? Because you have checked the same file 10 times in a row, if your machine has enough ram, it would get cached in the ram the first time you did it and then just get read back from the ram the other times.

The only strange this is that it works from another filesystem, but just for the sake of curiosity I would run memtest. It might be just a failing mobo, I've had that happen to me with strange hard disk detection problems and hangups.

fukawi2 · 2010-07-28 00:12:31

R00KIE wrote:

I don't know how much ram you have in that box but if the 3.5GB file can get cached it may point to bad ram.
Why do I say this? Because you have checked the same file 10 times in a row, if your machine has enough ram, it would get cached in the ram the first time you did it and then just get read back from the ram the other times.

8gb RAM so you raise a good point... But doesn't seem to be the case I wish it was, that would make me feel much better!

root@phil-desktop /media/Torrents/Complete # for X in `seq 1 10` ; do md5sum LARGE_3.5G_FILE.iso
 ; echo 1 > /proc/sys/vm/drop_caches; done
0aea3871413ac5c2418faf6ad6ad662f  LARGE_3.5G_FILE.iso
de277bd2bed06e131e3532fda2fe7cf4  LARGE_3.5G_FILE.iso
588d5043e2539860949462b2598d0053  LARGE_3.5G_FILE.iso
81e89c8f04ef7ef784e31b17bc3543d8  LARGE_3.5G_FILE.iso
caa55a3d3e4f75a8ab741b2bf9a12510  LARGE_3.5G_FILE.iso
81d10b8f881ef91aa67696023f03f48b  LARGE_3.5G_FILE.iso
f9ea40bce6eb55f2cb83338b56c2b683  LARGE_3.5G_FILE.iso

karol · 2010-07-28 00:20:18

A bit OT, but do you really use 16 GB swap? Did I miss / misread sth?
http://fukawi2.nl/index.php?page=proc-home-desktop

fukawi2 · 2010-07-28 00:22:48

karol wrote:

A bit OT, but do you really use 16 GB swap? Did I miss / misread sth?
http://fukawi2.nl/index.php?page=proc-home-desktop

No, just a habit I have of creating 2 x RAM as swap. I have 120gb for my system drives so plenty of space to spare, but I've decided to drop swap all together when I put the SSD in.

R00KIE · 2010-07-28 15:47:13

I would still give memtest a (quick) go and see what the tests that take less time to complete have to say.

Another thing I would look at is the output of 'smartctl -HA /dev/sda', check the output for all disks and check for anything suspicious.

Apart from this I can't remember anything else and I can't provide much help with LVM and RAID because I don't have much experience with that.

Arch Linux

#1 2010-07-26 10:14:55

[SOLVED] Bad Storage -- md5 fails

#2 2010-07-26 10:24:39

Re: [SOLVED] Bad Storage -- md5 fails

#3 2010-07-26 10:26:39

Re: [SOLVED] Bad Storage -- md5 fails

#4 2010-07-26 10:45:41

Re: [SOLVED] Bad Storage -- md5 fails

#5 2010-07-26 10:52:36

Re: [SOLVED] Bad Storage -- md5 fails

#6 2010-07-26 13:00:27

Re: [SOLVED] Bad Storage -- md5 fails

#7 2010-07-26 23:02:45

Re: [SOLVED] Bad Storage -- md5 fails

#8 2010-07-26 23:14:21

Re: [SOLVED] Bad Storage -- md5 fails

#9 2010-07-26 23:18:49

Re: [SOLVED] Bad Storage -- md5 fails

#10 2010-07-26 23:20:53

Re: [SOLVED] Bad Storage -- md5 fails

#11 2010-07-26 23:25:06

Re: [SOLVED] Bad Storage -- md5 fails

#12 2010-07-26 23:30:56

Re: [SOLVED] Bad Storage -- md5 fails

#13 2010-07-26 23:35:44

Re: [SOLVED] Bad Storage -- md5 fails

#14 2010-07-26 23:41:40

Re: [SOLVED] Bad Storage -- md5 fails

#15 2010-07-27 00:09:27

Re: [SOLVED] Bad Storage -- md5 fails

#16 2010-07-27 01:24:34

Re: [SOLVED] Bad Storage -- md5 fails

#17 2010-07-27 08:46:31

Re: [SOLVED] Bad Storage -- md5 fails

#18 2010-07-27 22:43:13

Re: [SOLVED] Bad Storage -- md5 fails

#19 2010-07-27 22:52:04

Re: [SOLVED] Bad Storage -- md5 fails

#20 2010-07-27 22:54:06

Re: [SOLVED] Bad Storage -- md5 fails

#21 2010-07-27 23:57:05

Re: [SOLVED] Bad Storage -- md5 fails

#22 2010-07-28 00:12:31

Re: [SOLVED] Bad Storage -- md5 fails

#23 2010-07-28 00:20:18

Re: [SOLVED] Bad Storage -- md5 fails

#24 2010-07-28 00:22:48

Re: [SOLVED] Bad Storage -- md5 fails

#25 2010-07-28 15:47:13

Re: [SOLVED] Bad Storage -- md5 fails

Board footer