[SOLVED] systemd writing too much on ssd

erfanjoker · 2020-12-20 01:59:34

I was wondering why does this guy writing to disk that much ! Below is the image :

https://i.imgur.com/2gPj1Up.png

Also here is my pstree which ofcourse arch uses systemd, but do all writes under systemd are considered as systemd writes ? if so, then why not all applications obey this rule ? some show their own read/writes even when they are under systemd :

systemd─┬─NetworkManager───3*[{NetworkManager}]
        ├─accounts-daemon───2*[{accounts-daemon}]
        ├─colord───2*[{colord}]
        ├─dbus-daemon
        ├─gdm─┬─gdm-session-wor─┬─gdm-x-session─┬─Xorg───4*[{Xorg}]
        │     │                 │               ├─gnome-session-b───3*[{gnome-+
        │     │                 │               └─2*[{gdm-x-session}]
        │     │                 └─2*[{gdm-session-wor}]
        │     └─2*[{gdm}]
        ├─geoclue───2*[{geoclue}]
        ├─gnome-keyring-d───3*[{gnome-keyring-d}]
        ├─lvmetad
        ├─mount.ntfs
        ├─nm-dispatcher───2*[{nm-dispatcher}]
        ├─nm-openconnect-─┬─openconnect
        │                 └─2*[{nm-openconnect-}]
        ├─polkitd───7*[{polkitd}]
        ├─rtkit-daemon───2*[{rtkit-daemon}]
        ├─snapd───9*[{snapd}]
        ├─systemd─┬─(sd-pam)
        │         ├─at-spi-bus-laun─┬─dbus-daemon
        │         │                 └─3*[{at-spi-bus-laun}]
        │         ├─at-spi2-registr───2*[{at-spi2-registr}]
        │         ├─bamfdaemon───3*[{bamfdaemon}]
        │         ├─blueman-tray───3*[{blueman-tray}]
        │         ├─chromium─┬─chromium───chromium─┬─chromium
        │         │          │                     └─6*[{chromium}]
        │         │          ├─chromium───chromium─┬─4*[chromium───10*[{chromi+
        │         │          │                     ├─7*[chromium───9*[{chromiu+
        │         │          │                     └─2*[chromium───11*[{chromi+
        │         │          ├─chromium───9*[{chromium}]
        │         │          └─29*[{chromium}]
        │         ├─dbus-daemon
        │         ├─dconf-service───2*[{dconf-service}]
        │         ├─evolution-addre───5*[{evolution-addre}]
        │         ├─evolution-calen───8*[{evolution-calen}]
        │         ├─evolution-sourc───3*[{evolution-sourc}]
        │         ├─gjs───6*[{gjs}]
        │         ├─gnome-session-b─┬─blueman-applet───3*[{blueman-applet}]
        │         │                 ├─evolution-alarm───5*[{evolution-alarm}]
        │         │                 ├─gnome-software───8*[{gnome-software}]
        │         │                 ├─gsd-disk-utilit───2*[{gsd-disk-utilit}]
        │         │                 ├─plank───4*[{plank}]
        │         │                 ├─run.sh───python3─┬─python3
        │         │                 │                  ├─python3───2*[{python3+
        │         │                 │                  └─2*[{python3}]
        │         │                 └─3*[{gnome-session-b}]
        │         ├─gnome-session-c───{gnome-session-c}
        │         ├─gnome-shell─┬─gnome-system-mo───3*[{gnome-system-mo}]
        │         │             └─13*[{gnome-shell}]
        │         ├─gnome-shell-cal───5*[{gnome-shell-cal}]
        │         ├─gnome-terminal-─┬─fish─┬─pstree
        │         │                 │      └─{fish}
        │         │                 └─4*[{gnome-terminal-}]
        │         ├─goa-daemon───3*[{goa-daemon}]
        │         ├─goa-identity-se───2*[{goa-identity-se}]
        │         ├─gsd-a11y-settin───3*[{gsd-a11y-settin}]
        │         ├─gsd-color───3*[{gsd-color}]
        │         ├─gsd-datetime───3*[{gsd-datetime}]
        │         ├─gsd-housekeepin───3*[{gsd-housekeepin}]
        │         ├─gsd-keyboard───3*[{gsd-keyboard}]
        │         ├─gsd-media-keys───4*[{gsd-media-keys}]
        │         ├─gsd-power───3*[{gsd-power}]
        │         ├─gsd-print-notif───2*[{gsd-print-notif}]
        │         ├─gsd-printer───2*[{gsd-printer}]
        │         ├─gsd-rfkill───2*[{gsd-rfkill}]
        │         ├─gsd-screensaver───2*[{gsd-screensaver}]
        │         ├─gsd-sharing───3*[{gsd-sharing}]
        │         ├─gsd-smartcard───4*[{gsd-smartcard}]
        │         ├─gsd-sound───3*[{gsd-sound}]
        │         ├─gsd-usb-protect───3*[{gsd-usb-protect}]
        │         ├─gsd-wacom───3*[{gsd-wacom}]
        │         ├─gsd-xsettings───3*[{gsd-xsettings}]
        │         ├─gvfs-afc-volume───3*[{gvfs-afc-volume}]
        │         ├─gvfs-goa-volume───2*[{gvfs-goa-volume}]
        │         ├─gvfs-gphoto2-vo───2*[{gvfs-gphoto2-vo}]
        │         ├─gvfs-mtp-volume───2*[{gvfs-mtp-volume}]
        │         ├─gvfs-udisks2-vo───3*[{gvfs-udisks2-vo}]
        │         ├─gvfsd─┬─gvfsd-dnssd───2*[{gvfsd-dnssd}]
        │         │       ├─gvfsd-network───3*[{gvfsd-network}]
        │         │       ├─gvfsd-trash───2*[{gvfsd-trash}]
        │         │       └─2*[{gvfsd}]
        │         ├─gvfsd-fuse───5*[{gvfsd-fuse}]
        │         ├─gvfsd-metadata───2*[{gvfsd-metadata}]
        │         ├─mission-control───3*[{mission-control}]
        │         ├─obexd
        │         ├─pipewire─┬─pipewire-media-───{pipewire-media-}
        │         │          └─{pipewire}
        │         ├─pulseaudio─┬─gsettings-helpe───3*[{gsettings-helpe}]
        │         │            └─4*[{pulseaudio}]
        │         ├─tracker-miner-f───4*[{tracker-miner-f}]
        │         ├─xdg-desktop-por───6*[{xdg-desktop-por}]
        │         ├─xdg-desktop-por───3*[{xdg-desktop-por}]
        │         ├─xdg-document-po─┬─fusermount
        │         │                 └─6*[{xdg-document-po}]
        │         └─xdg-permission-───2*[{xdg-permission-}]
        ├─systemd-journal
        ├─systemd-logind
        ├─systemd-timesyn───{systemd-timesyn}
        ├─systemd-udevd
        ├─tor
        ├─udisksd───4*[{udisksd}]
        ├─upowerd───2*[{upowerd}]
        └─wpa_supplicant

Does systemd write anything other than journal ? And also can i make systemd journal globally unverbose ? I mean can i reduce the log traces at all (systemd, dmesg, etc) ? This is just in "1 day" :

^^>>> sudo journalctl --since "1 day ago" -x | wc -l                  05:31:30 
36520

In just 6 hours ! :

^^>>> neofetch --off | grep -i uptime                             (1) 05:37:01 
Uptime: 6 hours, 28 mins

moderator edit -- replaced oversized image with link.
Pasting pictures and code

Last edited by erfanjoker (2020-12-21 23:44:36)

Scimmia · 2020-12-20 02:01:40

I would assume it's the journal

erfanjoker · 2020-12-20 02:05:54

Scimmia wrote:

I would assume it's the journal

I suppose so, but what log could take 5GB in 6 hours uptime !

Trilby · 2020-12-20 03:31:54

I asked my crystal ball, but given that you've only counted the lines in the journal and not given any indication as to it's content, even the crystal ball was clueless.

Read the logs - I'd guess some process is screaming error messages at you that are being ignored.

Last edited by Trilby (2020-12-20 04:03:35)

loqs · 2020-12-20 03:44:04

Was the graphical process monitor for all IO activity including to the virtual file-systems?
36520 lines may not be an issue:

journalctl --since "1 day ago" -x | wc
  29976  231878 1901660
journalctl --since "1 day ago" | wc
  11771  139447 1099885

Last edited by loqs (2020-12-20 03:45:06)

seth · 2020-12-20 09:47:54

sudo iotop -a -p $(pgrep -d' -p ' systemd)

And please replace the oversized screenshot w/ a thumbnail/link.

graysky · 2020-12-20 11:11:22

seth wrote:

sudo iotop -a -p $(pgrep -d' -p ' systemd)
And please replace the oversized screenshot w/ a thumbnail/link.

I think the OP could emulate the graphical with a more streamlined:

sudo iotop -Pao

I have been using that for years. Probably should re-read the man page to make sure.

Last edited by graysky (2020-12-20 11:14:51)

seth · 2020-12-20 13:04:33

Sorry, this was maybe misleading. Removing the image is one part.
The other one is to actually run that command, not to emulate the GUI (I don't even know that actually does) but to collect some data on what "systemd" is causing the IO.

2ManyDogs · 2020-12-20 13:33:36

seth wrote:

And please replace the oversized screenshot w/ a thumbnail/link.

erfanjoker, I replaced your oversized image. You have been asked to read the Code of Conduct before. Read it now, especially the How to Post section. This is your final warning.

https://wiki.archlinux.org/index.php/Code_of_conduct

erfanjoker · 2020-12-20 17:05:22

Trilby wrote:

I asked my crystal ball, but given that you've only counted the lines in the journal and not given any indication as to it's content, even the crystal ball was clueless.
Read the logs - I'd guess some process is screaming error messages at you that are being ignored.

Hi, here is my journal : https://mega.nz/file/n7gETZTb#PoW9pikUW … 1a6czDojaA
and the only thing i can see is a strange nautilus connect() error, the only addon i installed for nautilus is megasync addon, and it is not running right now, maybe it would be for that
But why ?! Does just nautlius error generate Gigabytes of log in 6 hours ?!

erfanjoker · 2020-12-20 17:09:29

seth wrote:

sudo iotop -a -p $(pgrep -d' -p ' systemd)
And please replace the oversized screenshot w/ a thumbnail/link.

Total DISK READ :       0.00 B/s | Total DISK WRITE :       3.99 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
    224 be/4 root          0.00 B      4.00 K  0.00 %  0.00 % systemd-journald
      1 be/4 root          0.00 B      0.00 B  0.00 %  0.00 % init
    239 be/4 root          0.00 B      0.00 B  0.00 %  0.00 % systemd-udevd
    527 be/4 systemd-      0.00 B      0.00 B  0.00 %  0.00 % systemd-timesyncd
    535 be/4 root          0.00 B      0.00 B  0.00 %  0.00 % systemd-logind
    877 be/4 erfan         0.00 B      0.00 B  0.00 %  0.00 % systemd --user

erfanjoker · 2020-12-20 17:13:56

loqs wrote:

Was the graphical process monitor for all IO activity including to the virtual file-systems?
36520 lines may not be an issue:
journalctl --since "1 day ago" -x | wc
  29976  231878 1901660
journalctl --since "1 day ago" | wc
  11771  139447 1099885

Not for sure, i have rebooted since that time and now it doesn't any
But FYI i have snap package which have created 4 loopbacks at /dev/loopX , though here is lsblk, you can see all my FS and VFS's :

[erfan@erfan ~]$ lsblk -ab
NAME   MAJ:MIN RM         SIZE RO TYPE MOUNTPOINT
loop0    7:0    0     13201408  1 loop /var/lib/snapd/snap/mojave-themes/2
loop1    7:1    0     58052608  1 loop /var/lib/snapd/snap/core18/1932
loop2    7:2    0     58073088  1 loop /var/lib/snapd/snap/core18/1944
loop3    7:3    0     32571392  1 loop /var/lib/snapd/snap/snapd/10492
loop4    7:4    0            0  0 loop 
loop5    7:5    0            0  0 loop 
loop6    7:6    0            0  0 loop 
loop7    7:7    0            0  0 loop 
sda      8:0    0 240057409536  0 disk 
├─sda1   8:1    0      1031168  0 part 
├─sda2   8:2    0 204010946560  0 part /
├─sda3   8:3    0    104857600  0 part 
├─sda4   8:4    0     16777216  0 part 
├─sda5   8:5    0  35398848000  0 part 
└─sda6   8:6    0    522190848  0 part 
sdb      8:16   0 500107861504  0 disk 
├─sdb1   8:17   0 491516854272  0 part /run/media/erfan/Files
└─sdb2   8:18   0   8589934592  0 part [SWAP]
sdc      8:32   1            0  0 disk

loqs · 2020-12-20 17:15:23

erfanjoker wrote:

and the only thing i can see is a strange nautilus connect() error, the only addon i installed for nautilus is megasync addon, and it is not running right now, maybe it would be for that
But why ?! Does just nautlius error generate Gigabytes of log in 6 hours ?!

The journal you posted is three megabytes including catalogue messages which are not written to disk, which is not six gigabytes.

erfanjoker · 2020-12-20 17:19:44

graysky wrote:

seth wrote:
sudo iotop -a -p $(pgrep -d' -p ' systemd)
And please replace the oversized screenshot w/ a thumbnail/link.
I think the OP could emulate the graphical with a more streamlined:
sudo iotop -Pao
I have been using that for years. Probably should re-read the man page to make sure.

Here is 1 min gui emulation of iotop : https://mega.nz/file/uqYgVZ6J#1khObZTNe … mUYD5Agiwg

erfanjoker · 2020-12-20 17:20:55

loqs wrote:

erfanjoker wrote:
and the only thing i can see is a strange nautilus connect() error, the only addon i installed for nautilus is megasync addon, and it is not running right now, maybe it would be for that
But why ?! Does just nautlius error generate Gigabytes of log in 6 hours ?!
The journal you posted is three megabytes including catalogue messages which are not written to disk, which is not six gigabytes.

Yes, and thats why im confused ! journal has been 3Mb since yesterday, then what is writing this amount of data to my disk using systemd !

seth · 2020-12-20 20:20:54

erfanjoker wrote:

Here is 1 min gui emulation of iotop : https://mega.nz/file/uqYgVZ6J#1khObZTNe … mUYD5Agiwg

Why is that a video?
"-a" puts iotop into accumulation, ie. the numbers are only growing and the final one is what has been read/written over the passed time.
And we'll likely need that for more than one minute, keep it running for about at least an hour.

Nautilus seems to yell that message once every second, so even if it was stored in the output format that would be 55 bytes what gets you ~194kB/h …

erfanjoker · 2020-12-20 22:31:32

seth wrote:

erfanjoker wrote:
Here is 1 min gui emulation of iotop : https://mega.nz/file/uqYgVZ6J#1khObZTNe … mUYD5Agiwg
Why is that a video?
"-a" puts iotop into accumulation, ie. the numbers are only growing and the final one is what has been read/written over the passed time.
And we'll likely need that for more than one minute, keep it running for about at least an hour.

I have been monitoring for 2 hours, and i wanted to copy the result to share with you, i forgot it is cli and i pressed god damn it CTRL + C to copy and the process exited

The result i saw was completely normal, no huge writes ! the sum of total writes in 2 hours was around 150Mb which is normal i suppose

The problem is somewhere else i suppose, iotop monitored around 150Mb writes in 2 hours (using command : sudo iotop -Pao)

But look at this image : https://mega.nz/file/miY2FBaL#8ayesOSzt … jojCJ9ccu4

When i started iotop monitoring, GUI Montior was showing 1.2GB Disk Writes, Now after 2 hours it is showing 3.5GB Disk Writes while iotop has just monitored 150Mb Writes in this duration !

1.2 + 0.15 != 3.5

Something is not right here, maybe iotop just cant monitor every process that is writing to my disk, or maybe the gnome-system-monitor is having a monitoring bug or etc, maybe knowing its monitoring mechanism and getting its verbose cli output could give us a clue

Nautilus seems to yell that message once every second, so even if it was stored in the output format that would be 55 bytes what gets you ~194kB/h …

I have uninstalled nautilus-megasync and i have to wait and recheck the journal to see if connect() error was from this guy or not, but as you said, even yelling this msg 100 times per second wont be more than 10Mb in hour, I just cant imagine what in the world could systemd be writing to my disk that takes these Gigabytes

Last edited by erfanjoker (2020-12-20 22:40:34)

loqs · 2020-12-20 23:03:18

You can check if the IO is actually being sent to the disk by looking at the seventh field of /sys/block/<dev>/stat and multiplying it by sector size (512).

https://www.kernel.org/doc/Documentation/block/stat.txt

Last edited by loqs (2020-12-20 23:10:49)

GSMiller · 2020-12-21 19:48:21

erfanjoker, please do not use videos or images, show us text!
We would love to take a look at your journal.

erfanjoker · 2020-12-21 23:44:15

Trilbyfan wrote:

erfanjoker, please do not use videos or images, show us text!
We would love to take a look at your journal.

Thanks, I will,

Seems like the problem got fixed by uninstalling nautilus-megasync, the connect() error it was yelling was due to megasync was not running and it was not able to communicate with it's daemon (or service or etc),
Now my systemd is working fine and usual writes

But the unanswered thing is, the log it was generating was just 2-3 Megabytes, Why it was showing Gigabytes

icar · 2020-12-22 00:08:41

Should I be worried?

 ~ % sudo journalctl --since "1 day ago" | wc -l
372587

I might open a new thread if that's more appropiate.

Last edited by icar (2020-12-22 00:09:01)

Ropid · 2020-12-22 00:10:31

erfanjoker wrote:

[...]
But the unanswered thing is, the log it was generating was just 2-3 Megabytes, Why it was showing Gigabytes

I guess those I/O numbers were for things like pipes, where it's processes communicating with each other and the work only happening in memory. You can for example start this here in a terminal window:

cat < /dev/urandom > /dev/null

All its work will happen in the CPU but you will see that it will get counted as bytes reading and writing in a process monitor tool.

Last edited by Ropid (2020-12-22 00:11:08)

erfanjoker · 2020-12-22 12:56:59

Ropid wrote:

erfanjoker wrote:
[...]
But the unanswered thing is, the log it was generating was just 2-3 Megabytes, Why it was showing Gigabytes
I guess those I/O numbers were for things like pipes, where it's processes communicating with each other and the work only happening in memory. You can for example start this here in a terminal window:
cat < /dev/urandom > /dev/null
All its work will happen in the CPU but you will see that it will get counted as bytes reading and writing in a process monitor tool.

Thanks for response, I think thats the explanation.

erfanjoker · 2020-12-22 12:57:59

icar wrote:

Should I be worried?
 ~ % sudo journalctl --since "1 day ago" | wc -l
372587
I might open a new thread if that's more appropiate.

You can start a new thread containing your full log

Trilby · 2020-12-22 14:39:48

Where did this "wc -l" nonsense start? Determining if your linux system has a problem by counting the number of lines in the journal is like a mechanic trying to determine if your car is running well by measuring it's weight. 372 thousand lines in 1 day seems like a very large number of lines to me - but it's indicative of nothing. READ the fucking journal, don't count it.

Arch Linux

#1 2020-12-20 01:59:34

[SOLVED] systemd writing too much on ssd

#2 2020-12-20 02:01:40

Re: [SOLVED] systemd writing too much on ssd

#3 2020-12-20 02:05:54

Re: [SOLVED] systemd writing too much on ssd

#4 2020-12-20 03:31:54

Re: [SOLVED] systemd writing too much on ssd

#5 2020-12-20 03:44:04

Re: [SOLVED] systemd writing too much on ssd

#6 2020-12-20 09:47:54

Re: [SOLVED] systemd writing too much on ssd

#7 2020-12-20 11:11:22

Re: [SOLVED] systemd writing too much on ssd

#8 2020-12-20 13:04:33

Re: [SOLVED] systemd writing too much on ssd

#9 2020-12-20 13:33:36

Re: [SOLVED] systemd writing too much on ssd

#10 2020-12-20 17:05:22

Re: [SOLVED] systemd writing too much on ssd

#11 2020-12-20 17:09:29

Re: [SOLVED] systemd writing too much on ssd

#12 2020-12-20 17:13:56

Re: [SOLVED] systemd writing too much on ssd

#13 2020-12-20 17:15:23

Re: [SOLVED] systemd writing too much on ssd

#14 2020-12-20 17:19:44

Re: [SOLVED] systemd writing too much on ssd

#15 2020-12-20 17:20:55

Re: [SOLVED] systemd writing too much on ssd

#16 2020-12-20 20:20:54

Re: [SOLVED] systemd writing too much on ssd

#17 2020-12-20 22:31:32

Re: [SOLVED] systemd writing too much on ssd

#18 2020-12-20 23:03:18

Re: [SOLVED] systemd writing too much on ssd

#19 2020-12-21 19:48:21

Re: [SOLVED] systemd writing too much on ssd

#20 2020-12-21 23:44:15

Re: [SOLVED] systemd writing too much on ssd

#21 2020-12-22 00:08:41

Re: [SOLVED] systemd writing too much on ssd

#22 2020-12-22 00:10:31

Re: [SOLVED] systemd writing too much on ssd

#23 2020-12-22 12:56:59

Re: [SOLVED] systemd writing too much on ssd

#24 2020-12-22 12:57:59

Re: [SOLVED] systemd writing too much on ssd

#25 2020-12-22 14:39:48

Re: [SOLVED] systemd writing too much on ssd

Board footer