[SOLVED] System very slow after long builds, issue with drive?

Phantz_ · 2026-01-28 04:45:31

My system has gotten extremely slow suddenly. It happened after building firefox (the browser) from source a couple times. I do development work on SpiderMonkey (the JS shell) and have been building the browser lately and after some building over a few days - my laptop is now extremely slow. It boots much slower, even opening the terminal (xterm on xmonad) takes awhile.

After opening up things, they seem to work... fine. There's no real hiccups - it's really "loading resources" (e.g loading the terminal initially, loading thumbnails in youtube etc) that takes longer.

This has happened once before, trying to solve it myself last time resulted in this: https://bbs.archlinux.org/viewtopic.php?id=311053

That time, it happened after the same thing - building firefox (a huge C++ project with rust and JS components). That time, I thought it was because the filesystem was too full (Firefox needs a lot of storage for artifacts). So I deleted stuff and ran fstrim..... which ended up corrupting my drive despite first ensuring my NVMe ssd supports fstrim....

I use a WD770. Here's the fdisk output:

Disk /dev/nvme0n1: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: WD_BLACK SN770 500GB                    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AE328858-62B6-AA46-9D1D-8799158202D9

Device             Start       End   Sectors   Size Type
/dev/nvme0n1p1      2048   1026047   1024000   500M EFI System
/dev/nvme0n1p2   1026048  66562047  65536000  31.3G Linux root (x86-64)
/dev/nvme0n1p3 147007488 252684287 105676800  50.4G Linux filesystem
/dev/nvme0n1p4 252684288 976773134 724088847 345.3G Linux home

Here's my `df -h` for that drive:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2   31G   22G  7.2G  76% /
/dev/nvme0n1p3   50G   35G   12G  76% /nix
/dev/nvme0n1p4  339G   98G  225G  31% /home
/dev/nvme0n1p1  500M   69M  431M  14% /boot

This time, it isn't so full. So I suspect being full wasn't the cause last time either.

Here's the output of `hdparm -Tt /dev/nvme0n1`:

/dev/nvme0n1:
 Timing cached reads:   3332 MB in  2.00 seconds = 1669.01 MB/sec
 Timing buffered disk reads: 2898 MB in  3.00 seconds = 965.74 MB/sec

How can I diagnose the cause?

Last edited by Phantz_ (2026-01-30 18:42:37)

wtx · 2026-01-28 09:56:04

A few questions:
1. Have you run fsck on the filesystem? I'd try this from some live linux CD/USB drive.
2. What shows: "df -i"?
3. Have you done any system upgrade in the meantime?
4. Output of this command: "vmstat 1 50"?

seth · 2026-01-28 13:55:33

So I deleted stuff and ran fstrim..... which ended up corrupting my drive despite first ensuring my NVMe ssd supports fstrim....

Corrupted how and did you fix that?
Is the device encrypted?

lsblk -f

Phantz_ · 2026-01-28 16:26:56

Couple updates first: I shutdown my laptop last night and let it rest. Note that I had already tried rebooting a couple times by now (with no luck). This morning, it's certainly faster but not quite at the same speed it used to be. Here's the new `hdparm -tT`:

/dev/nvme0n1:
 Timing cached reads:   14264 MB in  2.00 seconds = 7144.36 MB/sec
 Timing buffered disk reads: 6478 MB in  3.00 seconds = 2158.97 MB/sec

It's obviously faster. Note that I did run fsck just before this but fsck reported "clean" and only trimmed a small portion (0.2% was non-contiguous) of /dev/nvme0n1p2 after running it with `-f`. Again, my laptop is still not quite at normal speed yet and I'm not sure what the cause of this all is still.

Now, I'll answer the questions:

wtx wrote:

A few questions:
1. Have you run fsck on the filesystem? I'd try this from some live linux CD/USB drive.
2. What shows: "df -i"?
3. Have you done any system upgrade in the meantime?
4. Output of this command: "vmstat 1 50"?

1. I just did from a live arch ISO. It initially just reported everything as clean (each ext4 partition under /dev/nvme0n1), then I ran it with `-f` and it trimmed a small portion of `/dev/nvme0n1p2` (my root).
2.

Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/nvme0n1p2  2048000  571929  1476071   28% /
/dev/nvme0n1p3  3309568  687574  2621994   21% /nix
/dev/nvme0n1p4 22634496 1051453 21583043    5% /home
/dev/nvme0n1p1        0       0        0     - /boot

3. I have not done any upgrades during this whole thing. So the package state before this started happening and after (now) is the same.
4. http://0x0.st/PNYn.txt

seth wrote:

So I deleted stuff and ran fstrim..... which ended up corrupting my drive despite first ensuring my NVMe ssd supports fstrim....
Corrupted how and did you fix that?
Is the device encrypted?
lsblk -f

I essentially had to delete all the corrupted stuff and reinstall everything. I did it as per your instructions in that thread actually!: https://bbs.archlinux.org/viewtopic.php?id=311053

seth · 2026-01-28 20:31:09

I'm not sure what the cause of this all is still.

I doubt that you'll notice reading slowdowns at 2GB/s

seth wrote:

Is the device encrypted?

Phantz_ · 2026-01-29 16:24:50

seth wrote:

I'm not sure what the cause of this all is still.
I doubt that you'll notice reading slowdowns at 2GB/s
seth wrote:
Is the device encrypted?

My drive is not encrypted. Sorry for missing this question last time.

seth · 2026-01-29 19:45:21

strace -o /tmp/strace.log -tt -f "load_the_terminal_initially"
cat /tmp/strace.log | curl -F 'file=@-' 0x0.st

Phantz_ · 2026-01-29 21:19:54

seth wrote:

strace -o /tmp/strace.log -tt -f "load_the_terminal_initially"
cat /tmp/strace.log | curl -F 'file=@-' 0x0.st

http://0x0.st/PqHz.txt

For the record, it seems like every reboot (not back to back reboots) restores my laptop's speed a bit more..... and now it's harder to debug probably.

Any idea _why_ it'd slowly fix itself with time and reboots? One reboot fixing ephemeral issues makes sense but this is very weird behaviour to me

seth · 2026-01-29 21:41:42

My immediate gut feeling would have been trimming but your OP suggested
a) you had been running fstrim
b) it ended up breaking stuff

Stupid question: is this btrfs?
https://wiki.archlinux.org/title/Solid_state_drive#TRIM

wtx · 2026-01-29 22:29:16

@Phantz_:
1. Do you use any swap file on this disk?

cat /proc/swaps

2. For graphical view of you system boot up you can run:

systemd-analyze plot > /tmp/1.svg

and then open /tmp/1.svg with for example "inkscape". It will show the timings of systemd services
3. Is it possible that cpu is overheated? Try to run

sensors

and check temperatures
4. Any cpu frequency throttling?

cat /proc/cpuinfo |grep Hz

sudo dmesg | grep -i throttling

Phantz_ · 2026-01-30 18:41:52

seth wrote:

My immediate gut feeling would have been trimming but your OP suggested
a) you had been running fstrim
b) it ended up breaking stuff
Stupid question: is this btrfs?
https://wiki.archlinux.org/title/Solid_state_drive#TRIM

I did not run fstrim this time due to prior concerns about corruption. I also don't have the periodic fstrim systemd service enabled (again, due to concerns). Also I did verify that my ssd supports TRIM (by following the arch wiki instructions) before running it last time.

I'm using ext4 across the board, no brtfs.

wtx wrote:

@Phantz_:
1. Do you use any swap file on this disk?
cat /proc/swaps
2. For graphical view of you system boot up you can run:
systemd-analyze plot > /tmp/1.svg
and then open /tmp/1.svg with for example "inkscape". It will show the timings of systemd services
3. Is it possible that cpu is overheated? Try to run
sensors
and check temperatures
4. Any cpu frequency throttling?
cat /proc/cpuinfo |grep Hz
sudo dmesg | grep -i throttling

1. I do have a swap under my /home partition:

Filename                                Type            Size            Used            Priority
/home/swapfile                          file            8388604         0               -2

2. The dev-nvme0n1p2.device seems to have taken the longest amongst the top level stuff under systemd (252ms). Further down the line, there's also lvm2-monitor (69ms), systemd-udev-trigger (126ms), systemd-fsck@dev-disk-by\x2duuid-279be40e\x2d64a8\x2d4f66\x2d962b\x2d218191c6d1b0.service (714ms) are noteworthy.

3. I know the cpu was not overheated during the initial slowdown periods. I checked both physically and through sensors. I also had been monitoring the cpu usage and it was consistently stable/low.

4. Similar to the point above. I don't think it was throttling during the slowdown periods. The dmesg command yields nothing right now but it's probably not terribly illustrative at this point since my laptop has mostly returned back to normal...

Alright, thanks for the help trying to sniff out the cause guys. Unfortunately, I don't think further debugging attempts will clue us onto anything now because my laptop seems to have mostly returned to normal all on its own. The "solution", though certainly an indirect one, was "time" (and some shutdowns and restarts?). Throughout this time, I rarely used my laptop, I did not upgrade my packages, and as mentioned before I don't have much (if any) background stuff going on under my nose anyway - so all I can think of is "time" being the fix.

Still no clue what problem it could possibly be that time alone (not just one or two reboots) can fix.

I'm going to tentatively mark this as solved as I don't think any monitoring commands now will help figure things out (the problem no longer shows)

Arch Linux

#1 2026-01-28 04:45:31

[SOLVED] System very slow after long builds, issue with drive?

#2 2026-01-28 09:56:04

Re: [SOLVED] System very slow after long builds, issue with drive?

#3 2026-01-28 13:55:33

Re: [SOLVED] System very slow after long builds, issue with drive?

#4 2026-01-28 16:26:56

Re: [SOLVED] System very slow after long builds, issue with drive?

#5 2026-01-28 20:31:09

Re: [SOLVED] System very slow after long builds, issue with drive?

#6 2026-01-29 16:24:50

Re: [SOLVED] System very slow after long builds, issue with drive?

#7 2026-01-29 19:45:21

Re: [SOLVED] System very slow after long builds, issue with drive?

#8 2026-01-29 21:19:54

Re: [SOLVED] System very slow after long builds, issue with drive?

#9 2026-01-29 21:41:42

Re: [SOLVED] System very slow after long builds, issue with drive?

#10 2026-01-29 22:29:16

Re: [SOLVED] System very slow after long builds, issue with drive?

#11 2026-01-30 18:41:52

Re: [SOLVED] System very slow after long builds, issue with drive?

Board footer