You are not logged in.

#1 2024-08-27 19:50:34

Wild Penguin
Member
Registered: 2015-03-19
Posts: 347

btrfs and lots of writes from btrfs-transaction

Hi!

For some reason, btrfs-transaction writes a lot on my system. I don't know why, and any tips on why/how to find out why it does the writes, would be welcome.

The writes amount up to 0.1TB/week. After running iotop -a for 30 minutes:

Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     189.19 K/s                                                                                                                                               
    TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND                                                                                                                                           
    372 be/4 root       1168.00 K    287.69 M  ?unavailable?  [btrfs-transaction]                                                                                                                                  
   3922 be/4 428679        2.15 M     79.91 M  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [RenderManager-0]
   3923 be/4 428679        2.01 M     78.15 M  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [RenderManager-0]
    933 be/4 root       1152.00 K     53.32 M  ?unavailable?  monitorix.pid                                                                                                                                        
   3087 be/4 428679       25.14 M     24.05 M  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~ags.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [RegionFile I/O]
    485 be/4 root        688.00 K     15.55 M  ?unavailable?  [kworker/u64:11-events_power_efficient]
   4179 be/4 root        208.00 K     11.80 M  ?unavailable?  [kworker/u64:0-events_unbound]
    119 be/4 root        384.00 K     11.72 M  ?unavailable?  [kworker/u64:2-flush-btrfs-1]    
   4540 be/4 root        512.00 K     11.36 M  ?unavailable?  [kworker/u64:7-btrfs-endio-write]
   4387 be/4 root         48.00 K     10.45 M  ?unavailable?  [kworker/u64:4-btrfs-endio-write]
   4502 be/4 root          0.00 B     10.38 M  ?unavailable?  [kworker/u64:6-btrfs-endio-write]
    483 be/4 root         48.00 K     10.20 M  ?unavailable?  [kworker/u64:9-btrfs-endio-write] 
   5939 be/4 root        256.00 K      8.59 M  ?unavailable?  [kworker/u64:10-btrfs-endio-write]    
   5937 be/4 root        352.00 K      7.02 M  ?unavailable?  [kworker/u64:3-events_power_efficient]                                                                                                               
   3071 be/4 428679      180.00 K      5.59 M  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~lags.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [Server thread]
   3800 be/4 428679     1872.00 K      3.42 M  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [spark-async-sam]
    423 be/4 root          0.00 B      2.76 M  ?unavailable?  systemd-journald                                                                                                                                     
   3910 be/4 428679        0.00 B      2.24 M  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [BlueMap-Plugin-]
   3909 be/4 428679        0.00 B   1260.00 K  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~/mcflags.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [Thread-15]
   3771 be/4 428679        0.00 B   1008.00 K  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [DimensionDataIO]
   6583 be/4 root         16.00 K    608.00 K  ?unavailable?  [kworker/u64:1-btrfs-endio-write]
   6487 be/4 ville         0.00 B    516.00 K  ?unavailable?  -zsh                                                                                                                                                 
   3778 be/4 428679        0.00 B    504.00 K  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [DimensionDataIO]
   3780 be/4 428679        0.00 B    504.00 K  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [DimensionDataIO]
   6342 be/4 ville         0.00 B     88.00 K  ?unavailable?  -zsh              
    746 be/4 ntp           0.00 B     84.00 K  ?unavailable?  ntpd -g -u ntp:ntp
   5622 be/4 ville         0.00 B     72.00 K  ?unavailable?  -zsh                                                                                                                                                 
   5904 be/4 428679        0.00 B     28.00 K  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [BlueMap-RegionF]
   3022 be/4 428679        0.00 B      8.00 K  ?unavailable?  java -Xms5G -Xmx10G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPaus~gs.emc.gs -Daikars.new.flags=true -jar ./paper.jar nogui [Log4j2-AsyncApp]
    967 be/4 minecraf      0.00 B      4.00 K  ?unavailable?  conmon --api-version 1 -c 7ffbdef99434e49996067392b1170b6ac76c59faa209642e~mmand-arg 7ffbdef99434e49996067392b1170b6ac76c59faa209642e4120bfa83c918b8d
   2983 be/4 428679        0.00 B      4.00 K  ?unavailable?  cat 

It seems btrfs-transactions rises with approximately 6M every 30 seconds, which is the commit interval. I consider this exessive.

Currently, the computer is running a minecraft server in a podman container and monitorix - and not much else.

Writes done by btrfs-transaction are more than everything else combined (by roughly eyeballing it). Monitorix writes quite a lot into it's rrd databases (I could put these in tmpfs and sync them with anything-sync-daemon, I suppose)

My btrfs mounts are:

❯ mount  | grep -i btrfs
/dev/nvme0n1p2 on / type btrfs (rw,relatime,lazytime,ssd,discard=async,space_cache=v2,subvolid=272,subvol=/@)
/dev/nvme0n1p2 on /home type btrfs (rw,noatime,lazytime,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@home)
/dev/nvme0n1p2 on /mnt/fatrace type btrfs (rw,relatime,lazytime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)

I've mounted "/" subvol in /mnt/fatrace exactly to find out what is doing the writes, I get the same behavior without "/" mounted (fatrace breaks if btrfs "/" subvolume i.e. subvolid=5 is not mounted).

Cheers!

Offline

#2 2024-09-04 17:22:21

ua4000
Member
Registered: 2015-10-14
Posts: 487

Re: btrfs and lots of writes from btrfs-transaction

>>relatime,lazytime,
I propose to start here: switch to noatime
Read: https://wiki.archlinux.org/title/Fstab#atime_options

>>discard=async
I propose to switch to nodiscard, and use instead periodic TRIM.
Read: https://wiki.archlinux.org/title/Solid_ … iodic_TRIM

and

https://wiki.archlinux.org/title/Improv … ads/writes

Offline

#3 2024-10-17 17:16:16

Wild Penguin
Member
Registered: 2015-03-19
Posts: 347

Re: btrfs and lots of writes from btrfs-transaction

Hi,

Thanks ua4000 for your reply.

Your solutions have negligible effect in my case; noatime was already enabled for /home, and root has negligible writes anyways EDIT: I brainfarted, not really relevant here. I also believe discard should not affect the number of writes, however I will disable here (and report if it has any effect).

I've red the section in the last link. iotop is not useful as it just shows the btrfs-transactions doing the writes. Is the observation I'm having just a feature of btrfs? If so, why/what is it actually writing to the disk?

I also understand my SSD is probably fine. It's still a bit weird to have so much writes for no reason.

EDIT: Preliminary test seem to point out it was indeed the problem with iops_limit and setting nodiscard is a workaround. I'm not sure why, I didn't (yet) read the upstream bug report.

EDIT: Setting nodiscard for the mounts and noatime had no effect, btrfs-transactions is still trashing away on the disc...

Last edited by Wild Penguin (2024-10-17 20:43:28)

Offline

Board footer

Powered by FluxBB