[SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

graysky · 2012-10-24 19:00:33

Ted released a patch in this post which he believes may fix the issue. He goes on to write, "...we know that my patch definitely restores the behaviour previous to commit eeecef0af5, so it can't hurt, but we do want to make 100% sure that it really fixes the problem. "

I have patched this into 3.6.3 just fine.

Last edited by graysky (2012-10-24 19:06:38)

headkase · 2012-10-24 19:10:49

graysky wrote:

Ted released a patch in this post which he believes may fix the issue. He goes on to write, "...we know that my patch definitely restores the behaviour previous to commit eeecef0af5, so it can't hurt, but we do want to make 100% sure that it really fixes the problem. "
I have patched this into 3.6.3 just fine.

Do you mean you patched it into your Linux-CK kernel repo you maintain?

graysky · 2012-10-24 19:15:40

@headkase - No. I haven't updated the AUR or the repo with Ted's restore commit eeecef0af5 patch.

It should be fine since, in Ted's own words, "...we know that my patch definitely restores the behaviour previous to commit eeecef0af5, so it can't hurt, but we do want to make 100% sure that it really fixes the problem. " But I am still reluctant to push it since this seems to be an evolving situation... perhaps I'll change my mind. I dunno.

Last edited by graysky (2012-10-24 19:26:34)

headkase · 2012-10-24 19:16:26

Ok, thank you for the clarification graysky.

graysky · 2012-10-24 19:46:58

Consulting higher powers

https://bugs.archlinux.org/task/32204

Janarto · 2012-10-24 19:48:51

https://lkml.org/lkml/2012/10/23/690

Ted Tso is pushing a new patch for upgrade but beware with the lastest kernel and changes that were backported

karol · 2012-10-24 19:50:17

Please search before posting: https://bbs.archlinux.org/viewtopic.php?id=151341

graysky · 2012-10-24 20:11:16

--Deleted due to thread merge--

Last edited by graysky (2012-10-24 23:18:46)

Morn · 2012-10-24 20:22:35

Is 3.5.6 also affected by this bug or did the kernel developers backport it to 3.5.7 only? If 3.5.6 is also bad, I guess I'd need to downgrade even further.

Inxsible · 2012-10-24 20:25:29

Janarto's thread merged.

graysky · 2012-10-24 20:45:30

Inxsible wrote:

Janarto's thread merged.

Ah, now it looks like I double posted ;p

jrussell · 2012-10-24 20:47:15

Sorry for a maybe obvious question but - what would the symptoms be? Would you just not be able to boot? Or would the checking fail? Or would you not even realise what has happened untill later?

headkase · 2012-10-24 20:49:13

jrussell wrote:

Sorry for a maybe obvious question but - what would the symptoms be? Would you just not be able to boot? Or would the checking fail? Or would you not even realise what has happened untill later?

Your filesystem would be corrupted with unpredictable results. It might be recovered and you might have lost your high-resolution scan of the Mona Lisa for good.

Janarto · 2012-10-24 20:54:56

Inxsible wrote:

Janarto's thread merged.

Thanks, sorry for the inconvenience

mcover · 2012-10-24 21:06:56

Morn wrote:

Is 3.5.6 also affected by this bug or did the kernel developers backport it to 3.5.7 only? If 3.5.6 is also bad, I guess I'd need to downgrade even further.

3.5.6 does not have said commit "jbd2: don't write superblock when if its empty", therefore I would assume it is is fine. If you look at the git log [1], the tag for 3.5.6 is before that commit.

[1] http://git.kernel.org/?p=linux/kernel/g … 3.5.y;pg=1

Morn · 2012-10-24 21:40:18

Thanks, mcover! I suppose I'll stick with 3.5.6 in that case until 3.7 comes out. I had originally downgraded from 3.6 because of network issues, which may have been entirely unrelated to the 3.6 kernel of course. But I'm still glad I reverted early. Better safe than sorry where the filesystem is concerned. I've seen quite a few hard drives that were totaled by ReiserFS 3, so I was really hoping EXT4 would prove rock-solid by comparison.

ontobelli · 2012-10-25 02:49:16

It looks like my original analysis may not have been correct. At least, Eric and I haven't been able to figure out a way to trigger the problem based on my hypothesis of what had been going wrong. Still, the commit in question *does* change things, and so it's still the most likely culprit. (There were no ext4-related changes between v3.6..v3.6.1 and v3.6.2..v3.6.3, and I've looked at all of the changes between v3.6.2 and v3.6.3; all of the other changes look innocuous.) I have a patch (sent around 1:23 am Eastern on Wed., Oct. 24th to the ext4 list on the relevant mail thread) which should revert the problematic change in behavior, as well as put it a check which looks for the original conditions which might have triggered the patch, and prints a warning plus a stack trace so we can really understand what is going on. I don't want to consider this fixed until we have a reproduction case, so we can state with 100% certainty that we understand how it was triggered, and so we know that the proposed patch really does fix things.
That being said, please note that Fedora 17 is apparently on 3.6.2, and so far we only have two users who have reported the problem (or more specifically, both have reproduced file system corruptions with very similar symptoms, one running v3.6.2 and one running v3.6.3). The fact that they have reported the problem on very different hardware (one using a USB stick, the other using a Software RAID-5 setup), means it's not likely a hardware induced problem. However, this could potentially just be bad luck, since the fs corruption that was reported could have been explained by a random hardware glitch. With two users reporting it, though we have to treat it as potentially a real bug, and so I've gone back and re-audited all of the ext4 related commits that went into the v3.6.x stable kernel series.
If you think you have a related, similar bug, please check which kernel version you are using, and get the EXT4 error messages from the syslogs, and report it to me and the ext4 list. And if you can reproduce it reliably, I definitely want to hear from you. :-)
Thanks!!
-- Ted
http://phoronix.com/forums/showthread.p … post293374

FYI.

JLloyd13 · 2012-10-25 03:05:17

will this (possable) bug affect all ext4 filesystems mounted or just root? I say this because my / is btrfs but my /home is ext4.. others might be in a likewise place. time to backup.

dolby · 2012-10-25 05:22:31

When i woke up my system was unresponsive for the most part and when i tried to log in in a vt i got some I/O error for /dev/sda.
A hard rebooot and the recovery of the journal at startup made the above go away, but i think it must be cause of this.
As there is nothing in logs, how do you check for corrupted data?

felixonmars · 2012-10-25 07:38:02

I'm lucky sticking at 3.5.6 because of the compcache(zram) issue
Just waiting for the two bugs being solved.

blackout23 · 2012-10-25 08:50:34

I don't power down my laptop anyway. Tried btrfs a month ago but the rollback feature did not work quite right following the wiki.

punkrockguy318 · 2012-10-25 12:28:56

switching to linux-lts was a simple workaround for the time being for me

jrussell · 2012-10-25 12:44:19

punkrockguy318 wrote:

switching to linux-lts was a simple workaround for the time being for me

My shutdowns with systemd and linux-lts are much much slower than with 3.6.2, any chance you are on systemd and have slower shutdowns with the lts?

89c51 · 2012-10-25 13:32:05

As it seems its not as scary as we first thought.

https://plus.google.com/117091380454742 … cc5tMiCgq7

Grinch · 2012-10-25 15:13:10

89c51 wrote:

As it seems its not as scary as we first thought.
https://plus.google.com/117091380454742 … cc5tMiCgq7

Yes, filesystem corruption is a really nasty prospect and it's a good thing this problem has been identified, thankfully though it seems very hard to get bitten by this.

Arch Linux

#26 2012-10-24 19:00:33

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#27 2012-10-24 19:10:49

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#28 2012-10-24 19:15:40

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#29 2012-10-24 19:16:26

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#30 2012-10-24 19:46:58

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#31 2012-10-24 19:48:51

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#32 2012-10-24 19:50:17

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#33 2012-10-24 20:11:16

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#34 2012-10-24 20:22:35

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#35 2012-10-24 20:25:29

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#36 2012-10-24 20:45:30

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#37 2012-10-24 20:47:15

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#38 2012-10-24 20:49:13

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#39 2012-10-24 20:54:56

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#40 2012-10-24 21:06:56

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#41 2012-10-24 21:40:18

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#42 2012-10-25 02:49:16

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#43 2012-10-25 03:05:17

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#44 2012-10-25 05:22:31

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#45 2012-10-25 07:38:02

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#46 2012-10-25 08:50:34

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#47 2012-10-25 12:28:56

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#48 2012-10-25 12:44:19

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#49 2012-10-25 13:32:05

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

#50 2012-10-25 15:13:10

Re: [SOLVED] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3

Board footer