jbd2 process (ext4 journal) is writing on disk all the time

Mad Fish · 2011-02-13 22:45:14

File system is ext4.
I've tried to shutdown everything, but journaling process is still constantly accessing the disk (once in every few seconds).
This looks like a bug.
Looking for solution.

Last edited by Mad Fish (2011-02-13 22:46:10)

some-guy94 · 2011-02-13 23:02:46

Are you using noatime?

Mad Fish · 2011-02-13 23:03:34

I've tried adding noatime in fstab. No difference.

karol · 2011-02-13 23:50:11

https://bbs.archlinux.org/viewtopic.php … 17#p801017 ?

Mad Fish · 2011-02-14 00:44:46

karol wrote:

https://bbs.archlinux.org/viewtopic.php … 17#p801017 ?

The "solution" mentioned there was to stop HAL. I don't run HAL.

ejmarkow · 2011-02-14 09:38:48

This kernel patch just may resolve the issue:

http://git.kernel.org/?p=linux/kernel/g … 62532d8e5f

------------------------------------------------------------------
jbd2: call __jbd2_log_start_commit with j_state_lock write locked

author Theodore Ts'o <tytso@mit.edu>
Sat, 12 Feb 2011 13:18:24 +0000 (08:18 -0500)

On an SMP ARM system running ext4, I've received a report that the
first J_ASSERT in jbd2_journal_commit_transaction has been triggering:

J_ASSERT(journal->j_running_transaction != NULL);

While investigating possible causes for this problem, I noticed that
__jbd2_log_start_commit() is getting called with j_state_lock only
read-locked, in spite of the fact that it's possible for it might
j_commit_request. Fix this by grabbing the necessary information so
we can test to see if we need to start a new transaction before
dropping the read lock, and then calling jbd2_log_start_commit() which
will grab the write lock.

Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
fs/jbd2/journal.c
fs/jbd2/transaction.c
------------------------------------------------------------------

archman-cro · 2011-02-14 11:31:21

I have ext4 without HAL too and it accesses the disk all the time too.
Anyone tried the above patch? This is kinda critical, no?

fredre · 2011-02-15 00:23:10

I just installed Arch on a new laptop, and am getting this problem as well. It sounds like the hard drive is spinning down and back up each time, in addition to the clicks and jdb2 access. Should I be worried about this damaging the hard drive?

I've tried adding commit=60 to my fstab, but it hasn't made a difference.

Mad Fish · 2011-02-15 00:26:42

Does the problem exist on ext3, btrfs (and others)?

DarksideEE7 · 2011-02-15 10:40:14

I would like to add to this thread by mentioning that I see the journaling process active often by running iotop on several of my partitions even though noatime is used in fstab. This occurs on ext4 filesystems. My /var is reiserfs and I don't see any journal process activity in iotop for it.

This is especially troubling considering that one of these partitions is on an OCZ Revodrive (RAID0 PCI-E SSD), and I don't want this to cause performance degradation.

After watching iotop for a little while I noticed that the only two active journaling processes writing are for dm-2 (home) and sdd3 (Torrents, downloads).

I'm going to read the other thread over and search some, but I just wanted to add my 2 cents and subscribe to this thread in case anything is discovered.

I also have not run hal since KDE 4.6 no longer requires it.

I ran:

#ps -ef|grep jbd2
root       609     2  0 03:57 ?        00:00:00 [jbd2/dm-1-8]
root      2674     2  0 03:57 ?        00:00:00 [jbd2/dm-2-8]
root      2682     2  0 03:57 ?        00:00:00 [jbd2/sdd3-8]
root      4226     2  0 04:10 ?        00:00:00 [jbd2/sdc1-8]

So it looks like noatime isn't working on any of the entries in fstab. I have them setup like so:

/dev/mapper/sil_bgbgdjaddicbp1 / ext4 defaults,noatime 0 1
/dev/mapper/sil_bgbgdjaddicbp2 /home ext4 defaults,noatime 0 2
UUID=31061fe9-f3de-40d3-93da-c0fce7ed5014 /boot ext2 defaults 0 0
LABEL=var /var reiserfs defaults,noatime 0 2
LABEL=Torrents /home/l33/Torrents ext4 defaults,noatime 0 2
#LABEL=DLs /home/l33/DLs ext4 defaults,noatime 0 0
192.168.0.120:/mnt/Green /mnt/Green nfs defaults 0 0

none    /tmp    tmpfs   nodev,nosuid,noatime,size=1000M,mode=1777       0 0
shm     /dev/shm        tmpfs   nodev,nosuid,size=6G 0 0

Should I try something other than defaults,noatime?

Also, here is a plot of disk throughput vs time in KDE's system monitor. The red shows the constant write activity on dm-0
http://i.imgur.com/i4keW.jpg

Last edited by DarksideEE7 (2011-02-15 10:52:19)

archman-cro · 2011-02-15 13:39:12

Here's my logs on jbd2:

iotop -obtqqq | grep jbd2:

TIME  TID  PRIO  USER           DISK READ  DISK WRITE  SWAPIN   IO    COMMAND
13:57:35  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  1.68 % [jbd2/sda3-8]
13:57:37   531 be/3 root        0.00 B/s   23.53 K/s  0.00 %  3.24 % [jbd2/sda1-8]
13:58:09  2712 be/3 root        0.00 B/s   11.75 K/s  0.00 %  4.91 % [jbd2/sda3-8]
13:58:10   531 be/3 root        0.00 B/s   11.76 K/s  0.00 %  2.90 % [jbd2/sda1-8]
13:58:14  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.03 % [jbd2/sda3-8]
14:00:08  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  4.36 % [jbd2/sda3-8]
14:00:15  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.62 % [jbd2/sda3-8]
14:04:09  2712 be/3 root        0.00 B/s   11.58 K/s  0.00 %  4.51 % [jbd2/sda3-8]
14:04:14  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.58 % [jbd2/sda3-8]
14:06:09  2712 be/3 root        0.00 B/s   11.76 K/s  0.00 %  4.57 % [jbd2/sda3-8]
14:06:14  2712 be/3 root        0.00 B/s    7.84 K/s  0.00 %  3.28 % [jbd2/sda3-8]
14:08:09  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  3.92 % [jbd2/sda3-8]
14:08:14  2712 be/3 root        0.00 B/s   27.48 K/s  0.00 %  2.85 % [jbd2/sda3-8]
14:08:15   531 be/3 root        0.00 B/s   11.76 K/s  0.00 %  4.06 % [jbd2/sda1-8]
14:08:38  2712 be/3 root        0.00 B/s   35.32 K/s  0.00 %  2.22 % [jbd2/sda3-8]
14:08:49   531 be/3 root        0.00 B/s   15.70 K/s  0.00 %  3.29 % [jbd2/sda1-8]
14:09:02   531 be/3 root        0.00 B/s   39.16 K/s  0.00 %  5.70 % [jbd2/sda1-8]
14:09:09   531 be/3 root        0.00 B/s    3.92 K/s  0.00 %  8.89 % [jbd2/sda1-8]
14:09:09  2712 be/3 root        0.00 B/s   15.69 K/s  0.00 %  4.95 % [jbd2/sda3-8]
14:10:08  2712 be/3 root        0.00 B/s   11.52 K/s  0.00 %  4.64 % [jbd2/sda3-8]
14:10:15  2712 be/3 root        0.00 B/s    7.84 K/s  0.00 %  2.57 % [jbd2/sda3-8]
14:12:09  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  4.21 % [jbd2/sda3-8]
14:12:14  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.07 % [jbd2/sda3-8]
14:14:09  2712 be/3 root        0.00 B/s   11.56 K/s  0.00 %  4.34 % [jbd2/sda3-8]
14:14:14  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.66 % [jbd2/sda3-8]
14:16:09  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  4.69 % [jbd2/sda3-8]
14:16:14  2712 be/3 root        0.00 B/s    7.84 K/s  0.00 %  2.16 % [jbd2/sda3-8]
14:18:09  2712 be/3 root        0.00 B/s   11.76 K/s  0.00 %  4.98 % [jbd2/sda3-8]
14:18:14  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.87 % [jbd2/sda3-8]
14:18:36  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.17 % [jbd2/sda3-8]
14:18:37  2712 be/3 root        0.00 B/s    3.91 K/s  0.00 %  1.74 % [jbd2/sda3-8]
14:19:09  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  1.54 % [jbd2/sda3-8]
14:20:08  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  5.10 % [jbd2/sda3-8]
14:20:15  2712 be/3 root        0.00 B/s    7.83 K/s  0.00 %  2.41 % [jbd2/sda3-8]
14:22:09  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  4.19 % [jbd2/sda3-8]
14:22:14  2712 be/3 root        0.00 B/s    7.84 K/s  0.00 %  1.91 % [jbd2/sda3-8]
14:28:09  2712 be/3 root        0.00 B/s   11.77 K/s  0.00 %  5.04 % [jbd2/sda3-8]
14:28:14  2712 be/3 root        0.00 B/s    7.85 K/s  0.00 %  2.30 % [jbd2/sda3-8]
14:28:15   531 be/3 root        0.00 B/s    7.84 K/s  0.00 % 11.05 % [jbd2/sda1-8]
14:28:36  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.28 % [jbd2/sda3-8]
14:28:37  2712 be/3 root        0.00 B/s    7.84 K/s  0.00 %  2.16 % [jbd2/sda3-8]
14:29:09  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.98 % [jbd2/sda3-8]
14:30:45  2712 be/3 root        0.00 B/s    3.92 K/s  0.00 %  2.23 % [jbd2/sda3-8]
14:31:15  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  1.01 % [jbd2/sda3-8]
14:31:54  2712 be/3 root        0.00 B/s   52.93 K/s  0.00 %  2.40 % [jbd2/sda3-8]
14:32:23  2712 be/3 root        0.00 B/s    0.00 B/s  0.00 %  1.99 % [jbd2/sda3-8]
14:32:43   531 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.27 % [jbd2/sda1-8]

df -h:

Filesystem            Size  Used Avail Use% Mounted on
udev                   10M  156K  9.9M   2% /dev
/dev/sda1             9.2G  4.6G  4.2G  53% /
shm                   625M     0  625M   0% /dev/shm
/dev/sda3              45G   39G  4.2G  91% /home

Hmm, the frequency of journaling is kinda proportional to the individual partition occupation.

Btw, I use 'noatime' options on both partitions ('sda1' is /, 'sda3' is /home).

fredre · 2011-02-15 16:08:10

I just reinstallled on all ext3 partitions in hopes of alleviating the problem, but the same issue occurs. The only change is now it happens due to kjournald accessing it every few seconds. I still get the clicking and the constant spin-downs. I had been using ext4 prior to this. Once again, noatime and commit=60 make no difference.

Last edited by fredre (2011-02-15 16:08:49)

archman-cro · 2011-02-15 16:56:57

fredre, you can reduce clicking and spindowns with

hdparm -B 255 /dev/sdX

if your hdd supports SMART. Also see 'man hdparm'.

fredre · 2011-02-16 01:47:31

archman-cro wrote:

fredre, you can reduce clicking and spindowns with
hdparm -B 255 /dev/sdX
if your hdd supports SMART. Also see 'man hdparm'.

archman-cro, you are awesome.

archman-cro · 2011-02-16 22:57:03

The fix is present in 2.6.38-rc5! I can't wait for the stable one to arrive. So much improvements.

DarksideEE7 · 2011-02-16 23:48:23

Great....hopefully it fixes the problem for all of us.

archman-cro · 2011-02-22 10:56:04

I've tried killing conky as suggested in a similar thread, but it doesn't matter. Journalling still occurs every few seconds.

adr3nal1n · 2011-02-22 13:17:36

Same issue here too using ext4, disk is access every 2 seconds.

If I format the disk using ext3 or XFS the issue goes away.

This is on a clean Arch i686 install. I am not running X Windows, HAL or ACPI.

Does anyone know if the default options changed recently in mkfs.ext4?

archman-cro · 2011-03-02 15:41:42

Using 2.6.38-rc7:

16:03:06  2690 be/3 root        0.00 B/s   11.52 K/s  0.00 %  1.92 % [jbd2/sda3-8]
16:03:09   527 be/3 root        0.00 B/s   23.11 K/s  0.00 %  3.02 % [jbd2/sda1-8]
16:03:43   527 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.28 % [jbd2/sda1-8]
16:03:44   527 be/3 root        0.00 B/s   11.50 K/s  0.00 %  2.83 % [jbd2/sda1-8]
16:14:56   527 be/3 root        0.00 B/s    7.72 K/s  0.00 %  4.19 % [jbd2/sda1-8]
16:19:03   527 be/3 root        0.00 B/s   45.88 K/s  0.00 %  5.77 % [jbd2/sda1-8]
16:19:10   527 be/3 root        0.00 B/s    7.74 K/s  0.00 %  5.35 % [jbd2/sda1-8]
16:34:58   527 be/3 root        0.00 B/s    7.72 K/s  0.00 %  4.10 % [jbd2/sda1-8]

Using 2.6.37-ARCH:

16:46:03   524 be/3 root        0.00 B/s   23.63 K/s  0.00 %  3.62 % [jbd2/sda1-8]
16:46:37   524 be/3 root        0.00 B/s   11.81 K/s  0.00 %  3.28 % [jbd2/sda1-8]
17:02:57   524 be/3 root        0.00 B/s    7.88 K/s  0.00 %  5.78 % [jbd2/sda1-8]
17:19:02   524 be/3 root        0.00 B/s   43.28 K/s  0.00 %  6.44 % [jbd2/sda1-8]
17:19:08   524 be/3 root        0.00 B/s    7.88 K/s  0.00 %  6.10 % [jbd2/sda1-8]

Seems like the problem I thought I was having is conky, with companions like dzen2, who pull off various commands every few seconds. I'll be testing it further, though.

Last edited by archman-cro (2011-03-02 16:21:15)

rahenri · 2011-05-20 08:57:19

For some reason that goes beyond my understanding, enabling UEFI in the BIOS fixed the problem for me.
I was having a minor issue booting which was fixed by UEFI and luckily also fixed this problem.
My hard drive is 2TB, I think it shouldn't be working at all to begin with without UEFI.

I hope it helps someone.

cheers!

mentat · 2011-06-14 08:58:19

adding commit=60 to my fstab work for me.
If doesn't work maybe you have to look in

/etc/laptop-mode/laptop-mode.conf

CONTROL_NOATIME=0
...
USE_RELATIME=1

stripwax · 2011-06-16 09:24:20

Has anyone confirmed that the newer kernel actually fixes the underlying issue? (being that journalling shouldn't occur continually when disk i/o activity is otherwise idle, which is what prevents disk from spinning down).

DarksideEE7 · 2011-06-16 18:07:17

stripwax wrote:

Has anyone confirmed that the newer kernel actually fixes the underlying issue? (being that journalling shouldn't occur continually when disk i/o activity is otherwise idle, which is what prevents disk from spinning down).

I still get I/O activity on both my laptop and desktop with SSD's on both. My desktop has an OCZ Revodrive (PCI-E) and my laptop has an OCZ Vertex 2.

Both of these have ext4 filesystems running fully updated, non-testing repos.

jlindgren · 2011-06-17 12:12:17

stripwax wrote:

Has anyone confirmed that the newer kernel actually fixes the underlying issue? (being that journalling shouldn't occur continually when disk i/o activity is otherwise idle, which is what prevents disk from spinning down).

Still a problem with 2.6.39 here.

mkkot · 2011-06-28 19:41:30

Please look at this as well: https://bbs.archlinux.org/viewtopic.php?pid=903779

Arch Linux

#1 2011-02-13 22:45:14

jbd2 process (ext4 journal) is writing on disk all the time

#2 2011-02-13 23:02:46

Re: jbd2 process (ext4 journal) is writing on disk all the time

#3 2011-02-13 23:03:34

Re: jbd2 process (ext4 journal) is writing on disk all the time

#4 2011-02-13 23:50:11

Re: jbd2 process (ext4 journal) is writing on disk all the time

#5 2011-02-14 00:44:46

Re: jbd2 process (ext4 journal) is writing on disk all the time

#6 2011-02-14 09:38:48

Re: jbd2 process (ext4 journal) is writing on disk all the time

#7 2011-02-14 11:31:21

Re: jbd2 process (ext4 journal) is writing on disk all the time

#8 2011-02-15 00:23:10

Re: jbd2 process (ext4 journal) is writing on disk all the time

#9 2011-02-15 00:26:42

Re: jbd2 process (ext4 journal) is writing on disk all the time

#10 2011-02-15 10:40:14

Re: jbd2 process (ext4 journal) is writing on disk all the time

#11 2011-02-15 13:39:12

Re: jbd2 process (ext4 journal) is writing on disk all the time

#12 2011-02-15 16:08:10

Re: jbd2 process (ext4 journal) is writing on disk all the time

#13 2011-02-15 16:56:57

Re: jbd2 process (ext4 journal) is writing on disk all the time

#14 2011-02-16 01:47:31

Re: jbd2 process (ext4 journal) is writing on disk all the time

#15 2011-02-16 22:57:03

Re: jbd2 process (ext4 journal) is writing on disk all the time

#16 2011-02-16 23:48:23

Re: jbd2 process (ext4 journal) is writing on disk all the time

#17 2011-02-22 10:56:04

Re: jbd2 process (ext4 journal) is writing on disk all the time

#18 2011-02-22 13:17:36

Re: jbd2 process (ext4 journal) is writing on disk all the time

#19 2011-03-02 15:41:42

Re: jbd2 process (ext4 journal) is writing on disk all the time

#20 2011-05-20 08:57:19

Re: jbd2 process (ext4 journal) is writing on disk all the time

#21 2011-06-14 08:58:19

Re: jbd2 process (ext4 journal) is writing on disk all the time

#22 2011-06-16 09:24:20

Re: jbd2 process (ext4 journal) is writing on disk all the time

#23 2011-06-16 18:07:17

Re: jbd2 process (ext4 journal) is writing on disk all the time

#24 2011-06-17 12:12:17

Re: jbd2 process (ext4 journal) is writing on disk all the time

#25 2011-06-28 19:41:30

Re: jbd2 process (ext4 journal) is writing on disk all the time

Board footer