Periodic write spikes with btrfs

vkumar · 2013-02-09 23:52:55

I'm experiencing periodic write spikes of up to 6 megabytes with btrfs when my computer is completely idle. I've confirmed that this is an issue by comparing my iostat output with someone else's on IRC: while idling, their system performed no writes that were significant in the megabytes range.

According to iotop, the main culprits are;

btrfs-endio-wri
btrfs-dealloc-
btrfs-transaction-

The main culprit is btrfs-endio-wri. There are often four such threads that show up on iotop. I've been trying to dig into this a little bit -- I think it could be a kernel problem. There appear to be a few places where "endio-write" is scheduled.

In linux/fs/btrfs/disk-io.c around line 680, we have;

        if (bio->bi_rw & REQ_WRITE) {
                if (end_io_wq->metadata == 1)
                        btrfs_queue_worker(&fs_info->endio_meta_write_workers,
                                           &end_io_wq->work);
                else if (end_io_wq->metadata == 2)
                        btrfs_queue_worker(&fs_info->endio_freespace_worker,
                                           &end_io_wq->work);
                else 
                        btrfs_queue_worker(&fs_info->endio_write_workers,
                                           &end_io_wq->work);
        } else {
                if (end_io_wq->metadata)
                        btrfs_queue_worker(&fs_info->endio_meta_workers,
                                           &end_io_wq->work);
                else 
                        btrfs_queue_worker(&fs_info->endio_workers,
                                           &end_io_wq->work);
        }

Also, in linux/fs/btrfs/inode.c on line 2051, the kernel runs;

        if (btrfs_is_free_space_inode(inode))
                workers = &root->fs_info->endio_freespace_worker;
        else 
                workers = &root->fs_info->endio_write_workers;
        btrfs_queue_worker(workers, &ordered_extent->work);

I don't know of any other places where "endio-write" is scheduled. So if there is a bug, it has to be around here somewhere.

This problem exists on 3.7.5-1 and on linux-git.

Any advice on how I could prevent these write spikes from happening?

edit: grammar, and here's an image;
http://i.imgur.com/oWulMZj.png

Last edited by vkumar (2013-02-09 23:55:36)

vkumar · 2013-02-10 23:09:48

I pulled -rc7 yesterday and added two printk() calls to the suspicious regions I mentioned above. I then left iostat running and waited for the kernel output.

The results were really interesting. Every time a flood of text originated from my printk()'s inside of btrfs, iostat showed a 2mb or 6mb write spike. Here's an excerpt from them printk output;

[ 143.852904] Selecting endio_write_workers in inode.c.
start = 16314368 end=16318463uptodate=1 inode=1707352
inode-writecount=1
[ 143.858982] Selecting endio_write_workers in inode.c.
start = 6885376 end=6889471uptodate=1 inode=1707352
inode-writecount=1
[ 143.863056] Selecting endio_write_workers in inode.c.
start = 16809984 end=16814079uptodate=1 inode=1707352
inode-writecount=1
[ 143.869149] Selecting endio_write_workers in inode.c.
start = 19202048 end=19206143uptodate=1 inode=1707352
inode-writecount=1
[ 143.873212] Selecting endio_write_workers in inode.c.
start = 18890752 end=18894847uptodate=1 inode=1707352
inode-writecount=1
[ 143.879299] Selecting endio_write_workers in inode.c.
start = 18444288 end=18448383uptodate=1 inode=1707352
inode-writecount=1
[ 143.883363] Selecting endio_write_workers in inode.c.
start = 19308544 end=19312639uptodate=1 inode=1707352
inode-writecount=1
[ 143.889449] Selecting endio_write_workers in inode.c.
start = 19763200 end=19767295uptodate=1 inode=1707352
inode-writecount=1
[ 143.893508] Selecting endio_write_workers in inode.c.
start = 19607552 end=19611647uptodate=1 inode=1707352
inode-writecount=1
...

Note that none of these messages came from disk-io.c, they're all from the second excerpt in btrfs_writepage_end_io_hook (inode.c). Does anyone know what this function does, or why page->mapping->host->i_ino (the inode number) is always 1707352? Is btrfs constantly rewriting metadata regarding "inode 1707352"?

WonderWoofy · 2013-02-10 23:35:32

I hope you are able to get help here, but I honestly think that you should be asking this stuff on the #btrfs irc channel or something.

Arch Linux

#1 2013-02-09 23:52:55

Periodic write spikes with btrfs

#2 2013-02-10 23:09:48

Re: Periodic write spikes with btrfs

#3 2013-02-10 23:35:32

Re: Periodic write spikes with btrfs

Board footer