jbd2 process (ext4 journal) is writing on disk all the time

jlindgren · 2011-06-28 19:55:12

mkkot wrote:

Please look at this as well: https://bbs.archlinux.org/viewtopic.php?pid=903779

Were we supposed to find a solution in that thread, or what?

mkkot · 2011-06-28 20:09:46

I have just found that half of my blinking led issues are caused by sata dvd burner pooling. Anyway, it doesn't solve jdb2 problem.

Lars Stokholm · 2011-07-04 07:48:52

Am I wrong or isn't this practically making it impossible to spin down a hard drive?

Even with commit=600 as an option the drive will spin up every 10 minutes, even if nothing has changed at all. Can someone explain to me why this isn't a huge deal? It's like going to the front door every 5 seconds to check if someone was knocking even though you didn't hear anything. What's the rationale behind this behavior?

Last edited by Lars Stokholm (2011-07-04 08:07:12)

mkkot · 2011-07-04 08:35:24

Okay, has anyone reported this to kernel bugzilla?

Lars Stokholm · 2011-07-07 04:54:19

mkkot wrote:

Okay, has anyone reported this to kernel bugzilla?

I don't think so: https://bugzilla.kernel.org/buglist.cgi?content=jbd2.
But I don't understand the problem well enough to report it. It would make more sense for me to ask some questions about it in some kernel mailing list.

Lars Stokholm · 2011-07-07 19:27:18

I just spent a couple of hours looking into it. I started without any deamons and to my surprise my hard drive was able to spin down and stay that way for a long period of time (I caused it to spin up myself after 15 minutes or so).

I was even more surprised when I saw that jbd2 was still "writing to the disk" every five seconds, according to lm-profiler. I guess there's a lot I don't understand.

So I tried starting up daemons, one at a time. syslog-ng first, no problem, hard drive was still able to stay spinned down. Then net-auto-wireless, no problem. But with ntpd it was like the hard drive had a hard time getting a chance to spin down, so i killed ntpd again and tried crond instead. With crond and syslog-ng both started it seemed impossible to get a spin down lasting for more than 30 seconds or so. So I killed crond aswell and started tor. In the upstart of tor there was "a lot" of disk activity, but after a couple of minutes it was possible for the hard drive to sleep for at least 10 minutes. So I tried polipo aswell, but either I was getting tired or it seemed to me to be the same problem as with ntpd.

So that's where I'm at. Did I learn anything? I don't know - maybe that things are even more complicated than I thought. I wonder if anyone is having success letting their hard drives spin down for a long period of time. If so, I'd like to learn how their doing it.

mkkot · 2011-07-08 01:26:48

Okay, case closed. This doesn't allow drive spin-down but it's another thing to improve in system's configuration, I think. Anyway, here's a solution:

mount

Find your ext4 partition. If you found something like:

/dev/sdx on / type ext4 (rw,noatime,commit=300,commit=0)

then read further. As you can see there is a doubled commit value. The first one comes from /etc/fstab, the second one is an effect of stupid pm-utils which are another subsystem to make your life miserable.

FIX:

su
touch /etc/pm/power.d/journal-commit
chown root.0  /etc/pm/power.d/journal-commit
chmod 755  /etc/pm/power.d/journal-commit
reboot

(I'm not sure if this chown is needed but it's the way other config have)

How does it work? /etc/pm/power.d/journal-commit replaces default script from /usr/lib/pm-utils/power.d/journal-commit. Actually file in /etc is empty as this stupid script isn't needed at all but you may find useful to do a cp instead of touch and adjust it as you like (especially if running a laptop).
Hope this helps.

Last edited by mkkot (2011-07-08 01:28:09)

archman-cro · 2011-07-08 07:35:17

And what if there's

/dev/sda1 on / type ext4 (rw,noatime,barrier=1,data=ordered)
/dev/sda3 on /home type ext4 (rw,noatime)

?

mkkot · 2011-07-08 09:23:41

Then you haven't set commit interval manually which makes jdb2 journal writes with default value which is 5 seconds. Please add commit to /etc/fstab like this:

/dev/sda1 / ext4 defaults,noatime,commit=60    0   1

and use the solution from my previous post

or

cp /usr/lib/pm-utils/power.d/journal-commit /etc/pm/power.d/journal-commit

and then edit /etc/pm/power.d/journal-commit and change this value there. Of course if you have pm-utils installed at all, but I guess it's some opendesktop depedency.

archman-cro · 2011-07-08 10:03:34

Thanks! Nope, I don't have pm-utils installed.

Lars Stokholm · 2011-07-08 10:44:03

mkkot wrote:

Okay, case closed. This doesn't allow drive spin-down but it's another thing to improve in system's configuration, I think. Anyway, here's a solution

I'm not using pm-utils, so my commit is 5 seconds. But I don't consider increasing the commit time to 10 minutes a solution. Your hard drive will probably still spin up at least every 10 minutes.

I think I've come to the conclusion that jbd2 writes to disk as often as it should. It doesn't write anything if there's nothing to write (although lm-profiler seems to think so). If jbd2 often causes a spin up its because some other process will cause it to do so. Increasing the commit time is only a work-arround. And it has been presented before in this thread. The real fix would be to find the processes causing frequent jbd2 writes and make some changes there.

Last edited by Lars Stokholm (2011-07-08 10:44:44)

archman-cro · 2011-07-08 12:37:07

Amazing! I've set commit to 120 and now it's silent. Nice tip!

mkkot · 2011-07-08 18:09:39

Lars Stokholm wrote:

I think I've come to the conclusion that jbd2 writes to disk as often as it should. It doesn't write anything if there's nothing to write (although lm-profiler seems to think so). If jbd2 often causes a spin up its because some other process will cause it to do so. Increasing the commit time is only a work-arround. And it has been presented before in this thread. The real fix would be to find the processes causing frequent jbd2 writes and make some changes there.

I think jdb2 writes journal every 5 seconds, just that. Look at the documentation:

Ext4 can be told to sync all its data and metadata every 'nrsec' seconds. The default value is 5 seconds. This means that if you lose your power, you will lose as much as the latest 5 seconds of work (your filesystem will not be damaged though, thanks to the journaling).This default value (or any low value) will hurt performance, but it's good for data-safety. Setting it to 0 will have the same effect as leaving it at the default (5 seconds). Setting it to very large values will improve performance.

There are other processes which make disk spin-up, like pdflush, syslog or kde stuff. IMO journaling problem can be solved only this way. Spin-ups are as normal as it's crazy idea to spin-down a system disc. Possible, though. Try it on xfs, I doubt if it would be easier.

Last edited by mkkot (2011-07-08 18:12:25)

Lars Stokholm · 2011-07-08 22:41:10

mkkot wrote:

I think jdb2 writes journal every 5 seconds, just that. Look at the documentation:

I've already read the documentation, but as I said earlier I managed to make my hard drive stay spinned down indefinately in spite of a commit time of only 5 seconds. So it may be true when you say jbd2 writes every 5 seconds, but it depends on what you mean by "write". It definately does not cause the disk to spin up every time (as I thought earlier). An indication that jbd2 is not the culprit here, if you ask me. But you've chosen your solution - I don't believe your right, though.

mkkot · 2011-07-09 12:11:27

Fine, I'll be glad if you find anything. Still, increasing the commit serves my HDD :>
Suprisingly, despite the situation in my PC, in my laptop there are also pm-utils but no doubled commit value. I will be looking into it as well, maybe we get some facts at last.
BTW: https://bugs.launchpad.net/ubuntu/+sour … bug/607560

mkkot · 2011-07-10 19:20:48

Reported this. I hope you could provide some feedback as well:
https://bugzilla.kernel.org/show_bug.cgi?id=39072

Lars Stokholm · 2011-07-10 20:07:37

Thanks mkkot. I will keep a close eye on the bug.

Last edited by Lars Stokholm (2011-07-10 20:12:07)

mkkot · 2011-07-11 09:34:13

Saw your output. I guess I'll have to repeat this test on my laptop where syslog-ng isn't running and jbd2 still writes. Did you make a package with this patched kernel for x86 so I could install it? Compiled kernel once but it was like 7 years ago and my optical devices didn't work
(and tomorrow is my graduation exam so I shouldn't have time to play with this at all...)

Last edited by mkkot (2011-07-11 09:35:07)

Lars Stokholm · 2011-07-11 10:01:00

Sure. I've just used my hostname in the naming. Nothing else than this patch has been applied though. Here it is.

Just build with makepkg. Add this to /boot/grub/menu.lst:

# (2) Arch Linux
title  Arch Linux X201
root   (hd0,0)
kernel /vmlinuz26-x201 root=/dev/sda2 ro
initrd /kernel26-x201.img

Good luck with your exam.

Edit: Whoops. You asked for a package. I'm on 64bit. I don't know how to build for another architecture, but I'll have a look into it.
Edit2: Well I did look into it and I can safely say that it's probably not going to happen. Sorry. But it's fairly easy building it with makepkg.

Last edited by Lars Stokholm (2011-07-11 10:33:06)

mkkot · 2011-07-11 10:11:51

Thanks, I guess I'm going to learn kernel building anyway :>
And I have to use failsafe image to boot this stupid laptop...

slytux · 2011-07-11 17:02:19

Thanks for opening a bug report. The other awesome thing is that the ext4 maintainer replied. I recompiled the Arch kernel with his patch and I don't see anything unusual (at least every 5 sec) in my trace. You have to mount debugfs first.

  mount -t debugfs nodev /sys/kernel/debug

slytux · 2011-07-11 17:19:52

I didn't see anything for syslog-ng in my trace although I left it running. Also, the log source in the newer versions of syslog-ng has been changed to unix-dgram instead of unix-stream. I'm not sure if it affects anything but check your system for .pacsave files if you installed Arch before the change was made. https://bugs.archlinux.org/task/22153

Lars Stokholm · 2011-07-11 17:28:10

slytux wrote:

I didn't see anything for syslog-ng in my trace although I left it running. Also, the log source in the newer versions of syslog-ng has been changed to unix-dgram instead of unix-stream. I'm not sure if it affects anything but check your system for .pacsave files if you installed Arch before the change was made. https://bugs.archlinux.org/task/22153

I've merged all pacsave files I've ever had, but thanks.

I think the reason why my syslog-ng is so active is because my crond is very active. I have a command that gets run every minute. But even that doesn't explain everything, which is why I posted to the syslog-ng mailing list: https://lists.balabit.hu/pipermail/sysl … 16893.html

But are you having the same trouble with the harddrive spinning up, slytux? And if so, don't you get anything from trace_pipe? One of the points I've been trying to make in this topic is that there's "nothing" going on every 5 seconds as lm-profiler (/proc/sys/vm/block_dump) says. So if you're looking for something with 5 second intervals, you're on a wild goose chase.

slytux · 2011-07-11 20:36:27

I'm following this to see if it is specific to his system or not. What I did earlier was log out of my DE and I did see things in trace_pipe but it doesn't occur every 5 secs. I'm using an SSD with ext4 + discard. I'll test on some boxes at work that have real hard drives and let you know if I see anything particular.

Last edited by slytux (2011-07-11 20:57:29)

Lars Stokholm · 2011-07-12 05:36:56

Well. My problems are solved, thanks to the replies I got to my mail on the syslog-ng mailing list. Long story short, it turned out to be crond causing syslog-ng to write to /var/log/crond.log, /var/log/everything.log and /var/log/auth.log. This was again caused by a command being run from my crontab very frequently.

For those of you who might not be satisfied I suggest you try out Theodore's kernel patch, which will definately lead you toward the culprit. In my experience strace and lsof will take you the rest of the way (have a look at the syslog-ng replies I got).

Here is the ARCH kernel with Theodore's patch. All you have to do is build it with makepkg, install it and add this to /boot/grub/menu.lst:

# (2) Arch Linux
title  Arch Linux X201
root   (hd0,0)
kernel /vmlinuz26-x201 root=/dev/sda2 ro
initrd /kernel26-x201.img

After booting that kernel, running these commands will get you going:

mount -t debugfs none /sys/kernel/debug
echo 1 >/sys/kernel/debug/tracing/events/ext4/ext4_journal_start/enable
echo 1 >/sys/kernel/debug/tracing/events/jbd2/jbd2_run_stats/enable
cat /sys/kernel/debug/tracing/trace_pipe

That's about it. If you want to fix this properly, without the risks involved in increasing the commit time from fstab, you'll have to put some work into it. Please post to this topic if you do.

Last edited by Lars Stokholm (2011-07-12 05:39:09)

Arch Linux

#26 2011-06-28 19:55:12

Re: jbd2 process (ext4 journal) is writing on disk all the time

#27 2011-06-28 20:09:46

Re: jbd2 process (ext4 journal) is writing on disk all the time

#28 2011-07-04 07:48:52

Re: jbd2 process (ext4 journal) is writing on disk all the time

#29 2011-07-04 08:35:24

Re: jbd2 process (ext4 journal) is writing on disk all the time

#30 2011-07-07 04:54:19

Re: jbd2 process (ext4 journal) is writing on disk all the time

#31 2011-07-07 19:27:18

Re: jbd2 process (ext4 journal) is writing on disk all the time

#32 2011-07-08 01:26:48

Re: jbd2 process (ext4 journal) is writing on disk all the time

#33 2011-07-08 07:35:17

Re: jbd2 process (ext4 journal) is writing on disk all the time

#34 2011-07-08 09:23:41

Re: jbd2 process (ext4 journal) is writing on disk all the time

#35 2011-07-08 10:03:34

Re: jbd2 process (ext4 journal) is writing on disk all the time

#36 2011-07-08 10:44:03

Re: jbd2 process (ext4 journal) is writing on disk all the time

#37 2011-07-08 12:37:07

Re: jbd2 process (ext4 journal) is writing on disk all the time

#38 2011-07-08 18:09:39

Re: jbd2 process (ext4 journal) is writing on disk all the time

#39 2011-07-08 22:41:10

Re: jbd2 process (ext4 journal) is writing on disk all the time

#40 2011-07-09 12:11:27

Re: jbd2 process (ext4 journal) is writing on disk all the time

#41 2011-07-10 19:20:48

Re: jbd2 process (ext4 journal) is writing on disk all the time

#42 2011-07-10 20:07:37

Re: jbd2 process (ext4 journal) is writing on disk all the time

#43 2011-07-11 09:34:13

Re: jbd2 process (ext4 journal) is writing on disk all the time

#44 2011-07-11 10:01:00

Re: jbd2 process (ext4 journal) is writing on disk all the time

#45 2011-07-11 10:11:51

Re: jbd2 process (ext4 journal) is writing on disk all the time

#46 2011-07-11 17:02:19

Re: jbd2 process (ext4 journal) is writing on disk all the time

#47 2011-07-11 17:19:52

Re: jbd2 process (ext4 journal) is writing on disk all the time

#48 2011-07-11 17:28:10

Re: jbd2 process (ext4 journal) is writing on disk all the time

#49 2011-07-11 20:36:27

Re: jbd2 process (ext4 journal) is writing on disk all the time

#50 2011-07-12 05:36:56

Re: jbd2 process (ext4 journal) is writing on disk all the time

Board footer