lots of hdd chatter on boot (slow btrfs)

jrussell · 2013-07-13 12:04:26

I get a lot of hdd chatter from my laptop after my system appears to have fully booted in terms of services, multi-user.target is reached which is then followed by 10 - 15 seconds of hdd chatter during which my laptop is very unresponsive, it takes a while to login even from the tty. I have managed to open up iotop while this happens and I see systemd-journald up at the top of the list, which a couple of btrfs things which pop up every now and again.

jason@jason-laptop ~ % sudo systemd-analyze blame
[sudo] password for jason: 
          3.568s netctl-auto@wlan0.service
          1.978s avahi-daemon.service
          1.974s systemd-logind.service
           808ms sys-kernel-debug.mount
           729ms systemd-tmpfiles-setup-dev.service
           723ms systemd-vconsole-setup.service
           700ms sys-fs-fuse-connections.mount
           597ms systemd-udev-trigger.service
           530ms dev-mqueue.mount
           527ms sys-kernel-config.mount
           527ms dev-hugepages.mount
           489ms tmp.mount
           414ms accounts-daemon.service
           322ms systemd-sysctl.service
           316ms lightdm.service
           310ms systemd-remount-fs.service
           307ms systemd-fsck-root.service
           208ms systemd-tmpfiles-setup.service
           179ms alsa-restore.service
           175ms polkit.service
           158ms systemd-random-seed-load.service
           153ms udisks2.service
           137ms systemd-update-utmp.service
           115ms upower.service
            65ms systemd-udevd.service
            59ms systemd-user-sessions.service
            46ms systemd-journal-flush.service
            27ms boot.mount

* I have no idea why lightdm is in that list, its not enabled. I just started it manually with systemctl.
Edit: Ok lightdm and some other things pop into that list when I manually start lightdm from the tty with systemctl.

jason@jason-laptop ~ % sudo systemd-analyze      
Startup finished in 6.702s (kernel) + 9.607s (userspace) = 16.310s

Google hasnt retuned much, any ideas? Its very annoying.

Last edited by jrussell (2013-09-10 13:55:12)

karol · 2013-07-13 12:13:40

jrussell wrote:

I have no idea why lightdm is in that list, its not enabled. I just started it manually with systemctl.

That's normal. If you run 'systemd-analyze blame' before starting them, they won't be on the list. You can run 'systemd-analyze blame' after hours of uptime, after running some backup services, pkgstats.service etc. and they all will be on the list.

jrussell · 2013-07-13 15:15:10

karol wrote:

jrussell wrote:
I have no idea why lightdm is in that list, its not enabled. I just started it manually with systemctl.
That's normal. If you run 'systemd-analyze blame' before starting them, they won't be on the list. You can run 'systemd-analyze blame' after hours of uptime, after running some backup services, pkgstats.service etc. and they all will be on the list.

Cool, thanks for clarifying

jrussell · 2013-07-14 22:19:36

Seems to be getting worse, login prompts appear at consistent times from boot, with no overflow from systemd, so multi-user is reached, followed by a lot of hdd chatter.

If I remember correctly this did not happen after the clean installation, I'll work my way backwards and test more tomorrow, I really want to get to the bottom of this. I have a feeling it's a static service causing trouble, as I've tried disabling everything I can from systemctl, still get the chatter.

Last edited by jrussell (2013-07-14 22:19:55)

WonderWoofy · 2013-07-14 22:34:59

Instead of humoring the idea of a clean installation, since you use btrfs, why not just make a new subvol in which you can create a new base installation. Then you can just tell the bootloader to boot this different subvolume instead. That way, you don't lose your current working system, and deletion is as simple as one command.

There is a devtools script to create a chroot with one command, which might be quite convenient. I think they are intended for clean package building, but basically result in a full base system.

jrussell · 2013-07-16 15:12:55

Ok so Im not crazy, this doesnt happen after a fresh install of just the base group, one difference with my new install so far is that its not on a LUKS partition (did I say that correctly?)

jrussell · 2013-07-18 19:28:49

I'm now fairly certain its not a service or running app, been dissecting my laptop for the past two days, reinstalling and fiddling. and:

After a fresh install, for a couple of boots there is no issue, as soon as login prompts pop up, the hdd is silent and the light stops flickering and login is instant, so all is good.

So I have used a fresh install and installed my usual packages and then the problem occurred and I started working my way to back to just base group, now even with just base group there is still a lot of chatter.

So, I'm gonna try just keeping a base install and rebooting it a couple of times ( I think the excessive chatter might be btrfs related, haven't done any googling yet) and see what happens,
Ill also try experiment on different hardware ( Haven't had the will or time for this yet.)

possibly related:
http://lists.opensuse.org/opensuse-bugs … 02418.html

My setup as of now is
2G /boot btrfs
100G / btrfs compress=lzo

Last edited by jrussell (2013-07-18 19:34:28)

jrussell · 2013-09-10 13:54:59

Just came across this http://www.mail-archive.com/linux-btrfs … 20547.html

WonderWoofy · 2013-09-10 14:01:50

I am going to venture to guess that this is defragmentation pure and simple. I think you should try running a manual defrag on everything, which is not quite as straight forward as it should be using the btrfs filesystem defragment tool. But look in the btrfs wiki and you should be able to find the necessary info. After that, start mounting with autodefrag. IMO, autodefrag should be part of the default mount options, since this situation is by no means rare, and atuodefrag seems to not only solve the problem, but also seems to be non-problematic, if user reported errors on the mailing list are any indication.

Just be aware that there is a bug in 3.11 that might prevent you from properly defragging. There is a fix in btrfs-next, and I have applied that particular commit to my git kernel to confirm it indeed works. But for now, just make sure you use 3.10.* to do your defrag.

Edit: BTW, I think that if you have a whole shit load of data, it might be easier and faster to simply rsync the entire filesystem to another filesystem, recreate the btrfs filesystem and subvol structure, then rsync it back. This will definitely result in a defragged filesystem. Make sure too that you are either using WorMzy's btrfs-progs-git package from the AUR, or that you are using tomegun's btrfs-progs from the [testing] repo. The one in [core] is pretty out of date and doens't properly match up with the capabilities of the current kernel.

Last edited by WonderWoofy (2013-09-10 14:04:16)

qft · 2013-09-19 19:43:44

WonderWoofy wrote:

I am going to venture to guess that this is defragmentation pure and simple. I think you should try running a manual defrag on everything, which is not quite as straight forward as it should be using the btrfs filesystem defragment tool. But look in the btrfs wiki and you should be able to find the necessary info. After that, start mounting with autodefrag. IMO, autodefrag should be part of the default mount options, since this situation is by no means rare, and atuodefrag seems to not only solve the problem, but also seems to be non-problematic, if user reported errors on the mailing list are any indication.
Just be aware that there is a bug in 3.11 that might prevent you from properly defragging. There is a fix in btrfs-next, and I have applied that particular commit to my git kernel to confirm it indeed works. But for now, just make sure you use 3.10.* to do your defrag.

Edit: BTW, I think that if you have a whole shit load of data, it might be easier and faster to simply rsync the entire filesystem to another filesystem, recreate the btrfs filesystem and subvol structure, then rsync it back. This will definitely result in a defragged filesystem. Make sure too that you are either using WorMzy's btrfs-progs-git package from the AUR, or that you are using tomegun's btrfs-progs from the [testing] repo. The one in [core] is pretty out of date and doens't properly match up with the capabilities of the current kernel.

Hi WonderWoffy, do you mind telling me which commit you used to patch the kernel in order to get defrag working in 3.11? Thanks a log.

jrussell · 2013-10-29 16:59:24

Some more info http://cd34.com/blog/scalability/btrfs- … most-full/

Arch Linux

#1 2013-07-13 12:04:26

lots of hdd chatter on boot (slow btrfs)

#2 2013-07-13 12:13:40

Re: lots of hdd chatter on boot (slow btrfs)

#3 2013-07-13 15:15:10

Re: lots of hdd chatter on boot (slow btrfs)

#4 2013-07-14 22:19:36

Re: lots of hdd chatter on boot (slow btrfs)

#5 2013-07-14 22:34:59

Re: lots of hdd chatter on boot (slow btrfs)

#6 2013-07-16 15:12:55

Re: lots of hdd chatter on boot (slow btrfs)

#7 2013-07-18 19:28:49

Re: lots of hdd chatter on boot (slow btrfs)

#8 2013-09-10 13:54:59

Re: lots of hdd chatter on boot (slow btrfs)

#9 2013-09-10 14:01:50

Re: lots of hdd chatter on boot (slow btrfs)

#10 2013-09-19 19:43:44

Re: lots of hdd chatter on boot (slow btrfs)

#11 2013-10-29 16:59:24

Re: lots of hdd chatter on boot (slow btrfs)

Board footer