You are not logged in.

Pages: 1

#1 2014-08-24 00:04:48

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

How best to diagnose cause of slow down as uptime increases

My system slows gradually as uptime increases. Rebooting results in a return to a responsive system. The slowness increases over hours and days.

What would be the best way to figure out the cause of this and/or to avoid it?

The system is a laptop with 8G RAM. I frequently sleep the laptop when not in use. On reboot, no memory is swapped out. Slowing correlates with increased use of swap although I am not sure whether this increase is the cause of the slowing or due to a common cause. Right now, for example, I have

$ free -h
             total       used       free     shared    buffers     cached
Mem:          7.7G       5.7G       2.0G       2.0G        40M       2.6G
-/+ buffers/cache:       3.1G       4.6G
Swap:         9.0G       1.3G       7.7G

That means, as I understand it, that 4.6G RAM is available (since stuff cached can be dropped to free memory) and 2G is entirely free. But obviously if stuff has been swapped out, it would still (I think) slow things down when that is accessed and 1.3G is a lot to be swapped. This is up from 1.1G a couple of hours ago.

I know that it is possible to decrease swappiness but I am not sure whether that is a good idea or not. Right now, I'm using the default settings and have resisted the temptation to try fiddling with this. (That is, I read about how to change it and about some pros and cons and decided I was not really sure whether it would improve things and so left well alone.)

Typical usage is expected to be reasonably memory intensive as I use KDE and typically have Firefox, Thunderbird, Konsole, Kile and Okular open plus other things as required. I'm also running DropBox and SpiderOak. Kile is particularly badly affected by the slow downs. Compiling the same code on this machine is noticeably slower than doing so on another machine with an older, slower processor and less RAM. (That machine is running Fedora but has a similar set up except that it does not have DropBox and never sleeps as it is not a laptop and is running a ssh server.)

I do not think the following is likely to be connected but mention it in case. I use subversion to manage most of my documents. I have three main projects within the repo, which is just accessed using file:// on the machine I'm using. For one of these projects, svn operations are noticeably slower than the others. The repo is mirrored (via SpiderOak) on the Fedora box. On that machine, there is no difference in responsiveness. So either it is a difference between the working copies on my Arch machine or it is some other difference in configuration between the two machines. As I say, this is probably an unrelated issue and I only mention it because it also involves slowness, really.

I can't see anything relevant in the logs (but I'm not entirely sure what to look for), and top does not point to any obvious culprit. (This surprised me: in the past, the culprit has been easy to spot in top - usually Firefox or Thunderbird. But they don't *look* problematic. Again, I might be missing something.)

I realise this is not much information to go on. I'm really asking what I should look for or what I should look at in order to narrow down the problem. (Unless, of course, anything stands out to anybody.)

Last edited by cfr (2014-08-24 00:06:46)

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#2 2014-08-24 00:28:28

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,804

Re: How best to diagnose cause of slow down as uptime increases

It is probably a memory leak as you seem to suspect.   Fire up htop and look at the memory use.  Perhaps sort by mem%, resident memory or swapped memory.  Watch what happens over time.

Thunderbird is a big user of memory on my system, but it does not seem to leak.

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#3 2014-08-24 00:44:24

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

Re: How best to diagnose cause of slow down as uptime increases

OK. Thanks. I've got htop running sorted by MEM right now. I'm not really sure what I'm looking for but is it normal to have a gazillion (approximate) firefox processes? The two top ones are xorg.bin and then the rest of the screen is firefox. I didn't realise that it used that many distinct processes...

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#4 2014-08-24 00:59:33

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,804

Re: How best to diagnose cause of slow down as uptime increases

cfr wrote:

...is it normal to have a gazillion (approximate) firefox processes? The two top ones are xorg.bin and then the rest of the screen is firefox.

More or less.  +/- a bajillion.  Each of those are really a thread.  They probably all have the same the same memory stats because the memory pool they all use is the same if they are threads running in the same instance of the application.

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#5 2014-08-24 01:44:13

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: How best to diagnose cause of slow down as uptime increases

What's the output of "smem -kt"?

Offline

#6 2014-08-25 18:04:57

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

Re: How best to diagnose cause of slow down as uptime increases

I've been watching the output from htop sorted in different ways but suspect I don't know what I'm looking for. This is the output from smem -kt which I didn't know about before it was suggested by lucke:

  PID User     Command                         Swap      USS      PSS      RSS 
 2647 username   kwrapper4 ksmserver            88.0K     4.0K    13.0K  1016.0K 
 2228 root     /sbin/agetty --noclear tty1   104.0K     4.0K    18.0K     1.4M 
 2284 username   /bin/sh /usr/bin/startkde     460.0K     4.0K    18.0K     1.4M 
 2737 username   /bin/bash                       1.3M     4.0K    19.0K     1.6M 
 1010 root     /usr/bin/avahi-dnsconfd -s    120.0K     8.0K    23.0K     1.4M 
  859 root     /usr/bin/lvmetad -f           236.0K     8.0K    24.0K     1.5M 
 1016 root     /usr/bin/mcelog --ignorenod   124.0K     8.0K    24.0K     1.6M 
 2897 username   /bin/bash                       1.2M    12.0K    28.0K     1.6M 
 1086 avahi    avahi-daemon: chroot helpe    172.0K     4.0K    30.0K     1.1M 
 2264 root     -:0                           744.0K     8.0K    35.0K     2.1M 
 3727 username   /usr/lib/at-spi2-core/at-sp   708.0K    16.0K    39.0K     1.9M 
 2617 root     /usr/lib/kde4/libexec/start    48.0K    40.0K    46.0K   616.0K 
 1020 root     /usr/bin/ifplugd -i eth0 -r    76.0K    40.0K    53.0K     1.3M 
 2229 root     /usr/bin/kdm -nodaemon        148.0K    48.0K    66.0K     1.7M 
 2191 root     wpa_actiond -p /run/wpa_sup    28.0K    72.0K    85.0K     1.4M 
 2558 username   dbus-launch --sh-syntax --e   184.0K    72.0K    92.0K     1.4M 
 2572 username   dbus-launch --sh-syntax --e   188.0K    72.0K    92.0K     1.4M 
 2230 rpc      /usr/bin/rpcbind -w           296.0K    80.0K    93.0K     1.4M 
 1026 root     /usr/bin/acpid -f              84.0K   108.0K   123.0K     1.6M 
 2559 username   /usr/bin/dbus-daemon --fork   240.0K    56.0K   140.0K     1.6M 
 2282 username   (sd-pam)                        2.5M     8.0K   170.0K   732.0K 
 1006 root     /usr/bin/crond -n -s -m off   568.0K   156.0K   171.0K     1.7M 
 1019 avahi    avahi-daemon: running [Cyfr   124.0K   208.0K   260.0K     2.2M 
 3821 username   /usr/lib/GConf/gconfd-2       536.0K   224.0K   261.0K     2.2M 
 2746 username   /bin/bash                       1.1M   268.0K   287.0K     1.9M 
17738 root     dhclient -4 -q -e TIMEOUT=3     6.5M   220.0K   287.0K     1.8M 
31437 root     dhclient -4 -q -e TIMEOUT=3     6.5M   220.0K   287.0K     1.8M 
 2256 root     dhclient -4 -q -e TIMEOUT=3     6.5M   220.0K   289.0K     1.9M 
 2603 username   /usr/bin/gpg-agent -s --ena   152.0K   252.0K   311.0K     1.7M 
 2695 colord   /usr/lib/colord/colord          2.0M   288.0K   343.0K     3.1M 
27060 username   scdaemon --multi-server       440.0K   268.0K   354.0K     2.1M 
 2821 username   /bin/bash                       1.0M   396.0K   413.0K     2.0M 
 1022 root     /usr/lib/systemd/systemd-lo   132.0K   420.0K   436.0K     2.1M 
 2688 username   ksysguardd                     80.0K   420.0K   440.0K     2.1M 
 1004 root     /usr/bin/smartd -n            332.0K   440.0K   463.0K     2.3M 
17630 username   sed -e s/username/username/g         0   352.0K   482.0K     2.0M 
 1011 root     /usr/sbin/thermald --no-dae   464.0K   448.0K   507.0K     3.0M 
 2762 username   /bin/bash                     924.0K   500.0K   517.0K     2.1M 
 2188 root     wpa_supplicant -B -P /run/w   408.0K   444.0K   522.0K     2.7M 
 2618 username   kdeinit4: kdeinit4 Running.     4.4M   320.0K   556.0K     5.7M 
 2941 username   /bin/bash                     860.0K   580.0K   598.0K     2.2M 
  293 root     /usr/lib/systemd/systemd-ud   136.0K   664.0K   683.0K     2.5M 
 3020 username   /bin/bash                     616.0K   716.0K   732.0K     2.3M 
 2733 username   /usr/bin/gwenview -session      6.0M   572.0K   742.0K     7.4M 
 2714 username   kdeinit4: kaccess [kdeinit]     7.5M   524.0K   756.0K     8.1M 
 2728 username   /usr/bin/gwenview -session      5.9M   588.0K   756.0K     7.5M 
  275 root     /usr/lib/systemd/systemd-jo   120.0K   736.0K   760.0K     2.3M 
 2281 username   /usr/lib/systemd/systemd --   304.0K   564.0K   794.0K     2.8M 
 2724 username   /usr/bin/krandrtray -sessio     4.7M   692.0K   853.0K     7.1M 
 3334 username   /usr/lib/kde4/libexec/polki     4.9M   696.0K   858.0K     7.2M 
 1031 dbus     /usr/bin/dbus-daemon --syst   328.0K   756.0K   861.0K     2.7M 
 3123 username   /bin/bash                     520.0K   960.0K   977.0K     2.6M 
 2637 root     /usr/lib/upower/upowerd       708.0K   732.0K   978.0K     4.7M 
 2190 clamav   /usr/bin/freshclam -d -p /r     1.1M   700.0K     1.1M     4.1M 
 2619 username   kdeinit4: klauncher [kdeini     4.5M   840.0K     1.4M     7.2M 
 2791 username   /bin/bash                     824.0K     1.2M     1.5M     3.8M 
    1 root     /sbin/init                      1.1M     1.3M     1.7M     3.7M 
 3057 username   /bin/bash                     496.0K     1.5M     1.8M     4.0M 
 3621 username   /bin/bash                     508.0K     1.6M     1.8M     4.0M 
 2665 root     /usr/lib/udisks2/udisksd --   552.0K     1.7M     1.9M     4.5M 
17629 root     sudo smem -kt                      0     1.7M     2.0M     4.9M 
32532 root     /usr/bin/syslog-ng -F              0     2.1M     2.2M     5.5M 
 2573 username   /usr/bin/dbus-daemon --fork   224.0K     2.4M     2.5M     4.1M 
 2734 username   /usr/bin/gwenview -session      4.2M     2.4M     2.5M     9.2M 
 2648 username   kdeinit4: ksmserver [kdeini     6.1M     2.3M     2.9M    13.4M 
 2624 username   kdeinit4: kglobalaccel [kde     6.4M     2.7M     3.0M    10.8M 
 2694 root     /usr/bin/cupsd -f               1.4M     2.9M     3.1M     6.8M 
 2636 username   /usr/bin/kactivitymanagerd      3.3M     3.3M     3.5M     9.9M 
 6246 username   /opt/SpiderOak/lib/inotify_        0     3.3M     3.5M     5.5M 
 2718 username   /usr/bin/gtk-kde4 -session     14.0M     4.8M     5.0M    12.2M 
 2640 polkitd  /usr/lib/polkit-1/polkitd -     2.8M     6.0M     6.1M     9.0M 
14211 root     dhclient -4 -q -e TIMEOUT=3        0     6.7M     6.7M     8.2M 
23464 root     dhclient -4 -q -e TIMEOUT=3        0     6.7M     6.8M     8.3M 
 2720 username   kdeinit4: kmix [kdeinit] -s    15.2M     6.5M     6.9M    15.8M 
 3342 username   /usr/bin/knotify4               7.1M     6.9M     7.1M    14.0M 
 2231 ntp      /usr/bin/ntpd -g -u ntp:ntp        0    10.4M    10.8M    15.4M 
 2691 username   kdeinit4: krunner [kdeinit]    10.2M    13.0M    13.8M    24.2M 
 2621 username   kdeinit4: kded4 [kdeinit]      13.6M    13.3M    14.3M    26.1M 
 2722 username   kdeinit4: konsole [kdeinit]    10.6M    14.2M    15.6M    29.3M 
 3339 username   kdeinit4: klipper [kdeinit]     6.3M    15.5M    15.9M    24.8M 
 2727 username   /usr/bin/okular -session 10    20.6M    18.6M    18.9M    27.8M 
 2735 username   /usr/bin/okular -session 10    25.3M    20.5M    20.9M    29.6M 
 2701 username   /usr/bin/kuiserver              3.1M    20.7M    21.0M    27.5M 
 2760 username   /usr/bin/okular -session 10    46.8M    21.5M    21.8M    30.8M 
 3209 username   /usr/bin/okular /home/share    37.9M    21.0M    22.2M    35.7M 
17631 root     python2 /usr/bin/smem -kt          0    29.0M    29.1M    31.0M 
 3185 username   /usr/bin/okular /home/share    28.9M    23.5M    29.2M    51.8M 
 2452 username   /usr/bin/okular /home/mine   316.0K    33.9M    34.4M    43.1M 
 8765 username   /usr/bin/okular /usr/local/        0    35.3M    35.6M    43.3M 
 8724 username   /usr/bin/okular /usr/local/        0    38.2M    38.5M    45.8M 
 2674 username   kwin -session 101561a614111    15.7M    40.0M    41.6M    54.3M 
 2725 username   /usr/bin/okular -session 10     9.0M    40.2M    45.9M    69.1M 
 6241 username   /opt/SpiderOak/lib/SpiderOa        0    65.8M    66.9M    70.9M 
 2680 username   kdeinit4: plasma-desktop [k    33.0M    69.4M    71.7M    89.6M 
 3603 username   /usr/bin/kile                  18.3M    69.1M    71.9M    89.4M 
11550 username   /usr/bin/okular prawf3.pdf         0    88.6M    89.5M   100.7M 
 3316 username   /opt/dropbox/dropbox           56.4M    91.4M    92.9M    99.0M 
 6245 username   /opt/SpiderOak/lib/SpiderOa        0   122.5M   123.6M   127.2M 
 3861 username   /usr/bin/okular timeline.pd    12.6M   124.5M   127.7M   147.6M 
 2739 username   /usr/bin/okular -session 10    11.3M   136.2M   136.5M   145.6M 
 2278 clamav   /usr/bin/clamd                764.0K   294.1M   294.4M   297.1M 
 2726 username   /usr/bin/okular -session 10    15.0M   355.3M   355.7M   364.7M 
 3604 username   /usr/local/bin/firefox         54.9M   690.6M   693.5M   703.0M 
 2245 root     /usr/bin/Xorg.bin :0 vt7 -n    12.5M  1012.9M  1014.9M  1020.1M 
-------------------------------------------------------------------------------
  104 9                                      586.4M     3.5G     3.6G     4.1G

[Off topic but I thought X now ran as non-root by default so I must have misunderstood what pacman's recent message meant.]

I'm not sure why gwenview is running. ps confirms it is but I'd finished with the application and closed it yet I still seem to have at least 3 gwenviews active. On the other hand, that can hardly explain the amount of memory I'm using:

$ free -h
             total       used       free     shared    buffers     cached
Mem:          7.7G       7.2G       520M       1.4G        83M       4.0G
-/+ buffers/cache:       3.1G       4.6G
Swap:         9.0G       1.8G       7.2G

To be honest, I'm not sure how much memory it is reasonable to expect things to be using. For example, Okular seems to be using quite a bit in total but perhaps that is not unreasonable for a PDF viewer when I have multiple windows open.

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#7 2014-08-25 22:56:28

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: How best to diagnose cause of slow down as uptime increases

You don't seem to be constrained by memory.

Okular can be set to use less or more memory, the more memory it uses, the more pages it keeps in memory.

Your X server is using a lot of memory. Perhaps it is because of many okular processes? What's the output of "xrestop"?

As the news item mentions, X still runs as root when started from a display manager.

Please run "dstat -cdnpmgs --top-bio --top-cpu" for a while and post the output to pastebin.

How does this slowdown manifest? Is compiling the code generally slower than on your other machine or does it gradually get to that point?

Offline

#8 2014-08-25 23:13:34

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,804

Re: How best to diagnose cause of slow down as uptime increases

Iucke, I am not sure I agree.  It looks like he is almost 2GB into the swap space.  I am not sure, because half of the physical RAM is cached, so there should be need to be into the swap.  I have a feeling that something is very 'peaky' in its demand for memory asking for gigabytes at a time.

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#9 2014-08-25 23:44:23

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: How best to diagnose cause of slow down as uptime increases

That stuff in swap might actually be mainly from tmpfs. smem doesn't show some extraordinary swap usage (although there was enough memory pressure and/or uptime for unused stuff to be put there). What's the output of "df -hT", cfr? Maybe you have a lot of data in /tmp?

Offline

#10 2014-08-26 21:23:03

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

Re: How best to diagnose cause of slow down as uptime increases

lucke wrote:

Your X server is using a lot of memory. Perhaps it is because of many okular processes? What's the output of "xrestop"?

Didn't know about this either. Yes, Okular does seem to be the culprit here (though plasma-desktop and kwin are also quite hungry):

xrestop - Display: localhost:0
          Monitoring 46 clients. XErrors: 0
                    Pixmaps: 1596275K total, Other:     900K total, All: 1597176K total

                    res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier
                    3000000    13    2    0 1952 1982   647498K     46K 647545K  2726 /usr/local/texlive/2013/texmf-dist/doc/generic/pgf/pgfmanual.pdf ~@~S Oku
                    3c00000    15    2    0 2371 2403   391183K     56K 391240K  2739 The Comprehensive LaTeX Symbol List ~@~S Okular
                    2000000    40   26    0 1060 1216   110956K     30K 110986K  2680 plasma-desktop
                    2e00000    15    2    0 1417 1449    71998K     34K  72033K  2725 The Biblatex Package ~@~S Okular
                    1e00000   309    4    0  922 3718    67975K     94K  68070K  2674 kwin
                    5600000    13    2    0  550  584    56712K     14K  56727K  2452 Paper ~@~S Okular
                    3e00000    13    2    0  754  789    39010K     18K  39029K  2760 Using common PostScript fonts with LaTeX ~@~S Okular
                    5400000    15    2    0 1122 1151    38460K     27K  38488K  3185 /home/dir/symbols-
                    4e00000    13    2    0  290  324    32836K      7K  32844K  8765 The xparse package Document command parser ~@~S Okular
                    5200000    13    2    0  155  187    21444K      4K  21448K  8724 /usr/local/texlive/2014/texmf-dist/doc/latex/enumitem/enumitem.pdf ~@~S O
                    3a00000    13    2    0  325  357    18708K      8K  18717K  2735 /usr/local/texlive/2013/texmf-dist/doc/support/arara/arara-usermanual.pdf
                    3200000    13    2    0  155  187    18552K      4K  18557K  2727 biber ~@~S Okular
                    4c00000    83   56    1  723 1242    16253K     33K  16286K  3604 Title - Mozilla Firefox
                    5800000    13    2    0 1335 1362    13224K     32K  13257K  3209 /home/dir/symbols-
                    1800000    75 4073    0 5751 5861    11797K    234K  12032K  3603 filename1.tex ~@~S Kile
                    5000000   321    3    0 3302 3407    11387K     87K  11474K  3861 /home/username/Documents/dir/filename1.pdf ~
                    4000000    34    2    0 2386 2431     7223K     57K   7280K 11550 /home/username/dir/filename2.pdf ~@~S Okular
                    4800000     2    2    1  756  781     5699K     19K   5719K 27380 SpiderOak
                    2a00000     7    4    0 1015 1055     4496K     24K   4521K  2722 username : xrestop ~@~S Konsole
                    0000000     2    0    2    0  511     3940K     14K   3954K   ?   <unknown>
                    4600000     7    2    0  146  165     1643K      4K   1647K  3339 klipper
                    2600000     6    1    0  229  248     1007K      5K   1013K  2718 gtk-kde4
                    4a00000     7   43    1   31   39      531K      3K    534K  3316 dropbox
                    1000000     9    4    0   60   94      517K      2K    520K  2621 kded4
                    1a00000     4    1    0   43   56      367K      1K    369K  2648 Qt-subapplication
                    2200000     5    1    0   61  114      353K      2K    356K  2691 krunner
                    2800000    17    3    0   71   97      337K      2K    339K  2720 kmix
                    1200000     4    1    0   43   57      264K      1K    265K  2636 kactivitymanagerd
                    0c00000     4    1    0   43   57      264K      1K    265K  2714 kaccess
                    1400000     3    1    0   41  493      232K     11K    243K  2624 kglobalaccel
                    4400000     3    1    0   41   53      232K      1K    233K  3334 polkit-kde-authentication-agent-1
                    3800000     3    1    0   41   53      232K      1K    233K  2734 gwenview
                    3600000     3    1    0   41   53      232K      1K    233K  2733 gwenview
                    3400000     3    1    0   41   53      232K      1K    233K  2728 gwenview
                    2c00000     3    1    0   41   53      232K      1K    233K  2724 krandrtray
                    1600000     3    1    0   41   53      232K      1K    233K  3342 knotify4
                   2400000     3    3    1    1   11      256B      1K      1K  2701 kuiserver
                    0200000     0    1    1    0    0        0B      1K      1K   ?   <unknown>
                    5a00000     1    1    0    0    0        0B     48B     48B   ?   xrestop
                    4200000     1    1    0    0    0        0B     48B     48B   ?   <unknown>
                    0e00000     1    1    0    0    0        0B     48B     48B   ?   <unknown>

As the news item mentions, X still runs as root when started from a display manager.

Oops missed that in the snippet I read. (Though that was my guess after I thought about it.)

Please run "dstat -cdnpmgs --top-bio --top-cpu" for a while and post the output to pastebin.

How long (roughly) is a while? (I want to know if this means minutes, hours or days, for example, just so I don't collect far too little or far too much.)

How does this slowdown manifest? Is compiling the code generally slower than on your other machine or does it gradually get to that point?

Login is slower on the laptop than the other machine but that's just because I have more stuff start automatically. After that, the laptop is more responsive than the other machine (as I'd expect - faster processor, twice as much RAM). This is generally - not just compiling code. The slowdown is gradual.

ewaller wrote:

It looks like he is almost 2GB into the swap space.  I am not sure, because half of the physical RAM is cached, so there should be need to be into the swap.  I have a feeling that something is very 'peaky' in its demand for memory asking for gigabytes at a time.

That's what is puzzling me. I can't figure out why it is using swap at all or why the usage increases with increased uptime. If something is 'peaky', would it need to be fairly regularly so to produce consistent slowdown of the system? Or could it be something which ran only once a day, say?

lucke wrote:

That stuff in swap might actually be mainly from tmpfs. smem doesn't show some extraordinary swap usage (although there was enough memory pressure and/or uptime for unused stuff to be put there). What's the output of "df -hT", cfr? Maybe you have a lot of data in /tmp?

I don't think so:

$ df -hT | grep tmpfs
dev                                  devtmpfs  3.9G     0  3.9G   0% /dev
run                                  tmpfs     3.9G   10M  3.9G   1% /run
tmpfs                                tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs                                tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                                tmpfs     3.9G  593M  3.3G  15% /tmp
tmpfs                                tmpfs     791M  4.0K  791M   1% /run/user/1000

At least, 593M is quite a lot but it is not all that much relative to 8G of RAM.

Just discovered that if I close all Okular windows, I regain a bunch of space:

$ free -h
             total       used       free     shared    buffers     cached
Mem:          7.7G       5.7G       2.0G       591M        85M       3.5G
-/+ buffers/cache:       2.2G       5.5G
Swap:         9.0G       770M       8.2G

Unfortunately, when I restart the application, I get processes but no windows. (Probably I need to restart.)

Last edited by cfr (2014-08-26 21:39:25)

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#11 2014-08-26 22:09:44

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: How best to diagnose cause of slow down as uptime increases

What's the output of "smem -kt" and "df -h /tmp" without okular processes?

One minute of dstat output should give me some overview.

Offline

#12 2014-08-26 22:58:56

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

Re: How best to diagnose cause of slow down as uptime increases

Here's output from dstat --cdnpmgs -top-bio -top-cpu -top-mem. (I don't know the -top-mem is useful but I threw it in anyway.)

I just restarted the machine. (That's not why I couldn't get Okular to work earlier - it was just very, very slow.) But the system update updated systemd so it seemed best to restart. As a result, I'm now using no swap so I don't think the output of anything is likely to be very useful until things start slowing down again. However, I'll get some more data when that happens and post it back here. The dstat output linked above I collected before rebooting.

The current state, after restart with Firefox (2 windows, many tabs), Thunderbird (was not running at all when I posted earlier - it is usually, but it wasn't then), Kile (1 window, 5 tabs), Konsole (1 window, ~10 tabs) and Okular (4 PDFs open), with the usual running in the background (SpiderOak, DropBox etc.):

$ free -h
             total       used       free     shared    buffers     cached
Mem:          7.7G       6.2G       1.6G       400M       478M       3.3G
-/+ buffers/cache:       2.4G       5.3G
Swap:         9.0G         0B       9.0G

$ df -hT | grep tmpfs
dev                                  devtmpfs  3.9G     0  3.9G   0% /dev
run                                  tmpfs     3.9G   10M  3.9G   1% /run
tmpfs                                tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs                                tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                                tmpfs     3.9G   48M  3.9G   2% /tmp
tmpfs                                tmpfs     791M  4.0K  791M   1% /run/user/1000

$ smem -kt
  PID User     Command                         Swap      USS      PSS      RSS 
 1651 username   kwrapper4 ksmserver                0   104.0K   116.0K     1.1M 
 1577 username   dbus-launch --sh-syntax --e        0   256.0K   303.0K     2.1M 
 1563 username   dbus-launch --sh-syntax --e        0   256.0K   309.0K     2.1M 
 1564 username   /usr/bin/dbus-daemon --fork        0   252.0K   373.0K     1.4M 
 1387 username   /bin/sh /usr/bin/startkde          0   468.0K   586.0K     3.3M 
 1695 username   ksysguardd                         0   600.0K   622.0K     2.2M 
 1608 username   /usr/bin/gpg-agent -s --ena        0   616.0K   635.0K     1.9M                                                            
 2496 username   /usr/lib/at-spi2-core/at-sp        0   716.0K   976.0K     5.3M                                                            
 1578 username   /usr/bin/dbus-daemon --fork        0   944.0K     1.1M     2.8M                                                            
 2784 username   /usr/lib/GConf/gconfd-2            0   912.0K     1.1M     4.9M                                                            
 1384 username   /usr/lib/systemd/systemd --        0   668.0K     1.2M     3.8M                                                            
 1774 username   /bin/bash                          0     1.3M     1.4M     4.4M                                                            
 1761 username   /bin/bash                          0     1.3M     1.4M     4.5M 
 1829 username   /bin/bash                          0     1.3M     1.4M     4.5M 
 1746 username   /bin/bash                          0     1.3M     1.4M     4.4M 
 1918 username   /bin/bash                          0     1.3M     1.4M     4.5M 
 2147 username   /bin/bash                          0     1.3M     1.4M     4.5M 
 2505 username   /bin/bash                          0     1.3M     1.4M     4.5M 
 1783 username   /bin/bash                          0     1.3M     1.4M     4.4M 
 1748 username   /bin/bash                          0     1.3M     1.4M     4.5M 
 2137 username   /bin/bash                          0     1.3M     1.5M     4.6M 
 1994 username   /bin/bash                          0     1.9M     2.0M     5.1M 
 1624 username   kdeinit4: klauncher [kdeini        0     1.1M     2.5M    24.8M 
 2432 username   /opt/SpiderOak/lib/inotify_        0     3.3M     3.6M     5.7M 
 1623 username   kdeinit4: kdeinit4 Running.        0   724.0K     5.3M    33.5M 
 1722 username   kdeinit4: kaccess [kdeinit]        0     3.7M     5.4M    45.8M 
 2338 username   kdeinit4: klipper [kdeinit]        0     3.7M     5.6M    47.0M 
 1653 username   kdeinit4: ksmserver [kdeini        0     3.7M     5.7M    46.9M 
 1629 username   kdeinit4: kglobalaccel [kde        0     4.9M     6.7M    46.8M 
 1732 username   /usr/bin/krandrtray -sessio        0     5.5M     6.9M    41.3M 
 1708 username   /usr/bin/kuiserver                 0     5.5M     7.0M    42.3M 
 2333 username   /usr/lib/kde4/libexec/polki        0     5.7M     7.4M    44.8M 
 1641 username   /usr/bin/kactivitymanagerd         0     6.3M     7.9M    43.0M 
 2353 username   /usr/bin/knotify4                  0     6.5M     8.1M    45.1M 
 2164 username   /usr/bin/gwenview -session         0     6.4M     9.5M    52.6M 
 1733 username   /usr/bin/gwenview -session         0     6.5M     9.6M    52.8M 
 2084 username   /usr/bin/gwenview -session         0     6.5M     9.7M    52.7M 
 3407 username   python2 /usr/bin/smem -kt          0    12.5M    12.5M    14.7M 
 1626 username   kdeinit4: kded4 [kdeinit]          0    12.3M    15.6M    61.4M 
 1726 username   /usr/bin/gtk-kde4 -session         0    18.7M    20.7M    60.1M 
 1730 username   kdeinit4: konsole [kdeinit]        0    18.6M    21.8M    69.1M 
 1728 username   kdeinit4: kmix [kdeinit] -s        0    18.5M    22.1M    69.9M 
 1698 username   kdeinit4: krunner [kdeinit]        0    22.8M    27.2M    78.2M 
 1681 username   kwin -session 101561a614111        0    52.7M    60.6M   114.2M 
 2458 username   /usr/bin/kile                      0    70.1M    75.7M   127.4M 
 2321 username   /opt/SpiderOak/lib/SpiderOa        0    72.7M    77.5M    99.0M 
 1688 username   kdeinit4: plasma-desktop [k        0    89.4M    97.0M   155.9M 
 2416 username   /opt/SpiderOak/lib/SpiderOa        0   108.6M   112.9M   120.1M 
 2320 username   /opt/dropbox/dropbox               0   144.9M   148.4M   174.3M 
 3153 username   /usr/bin/okular /usr/local/        0   193.6M   197.3M   244.5M 
 2456 username   /usr/bin/thunderbird               0   288.4M   295.2M   341.6M 
 2457 username   /usr/local/bin/firefox             0   742.4M   750.2M   797.1M 
-------------------------------------------------------------------------------
   52 1                                           0     1.9G     2.0G     3.2G

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#13 2014-08-27 00:36:23

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: How best to diagnose cause of slow down as uptime increases

You need to use double hyphen-minuses before "top" arguments.

In that log, stuff was removed from swap sometimes, and near the end, it apparently caused dstat to skip ten seconds, which means it had strong impact on your system. Such swapping in should not ordinarily have such impact.

How does this slowdown manifest, except for slower compilation? Is changing windows or opening menus or programs slow, do you get stalls, etc.? Is it constant or intermittent (once your system gets slow)?

Both smem and dstat provide better output when run as root.

Offline

#14 2014-08-27 01:07:10

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

Re: How best to diagnose cause of slow down as uptime increases

lucke wrote:

How does this slowdown manifest, except for slower compilation? Is changing windows or opening menus or programs slow, do you get stalls, etc.? Is it constant or intermittent (once your system gets slow)?

Compilation was just an example. Yes, other things are noticeably slower, too. Once the system gets slow, everything is pretty slow pretty constantly but there are times when it is much worse and the system more-or-less stops responding for a brief time.

If something is 'peaking', I don't really understand why the system doesn't stop using swap once resources are free. I understand it won't swap stuff back in until it is needed, but that doesn't seem to explain why accessing the same document, say, should remain slow after accessing it once, even when nothing then 'peaks' again.

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#15 2014-08-27 01:39:24

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: How best to diagnose cause of slow down as uptime increases

It is strange. If you look at the swapped out part of the document you last looked at 24 hours ago, you could get a slight pause (because of swapping in), but then it'd be back in memory and looking at it should be without any slowdowns. You shouldn't get a constant slowdown, and nothing in the dstat output points to a constant resource starvation.

If you don't use hibernation, you could disable swap, just to see if you still get constant slowdowns. Maybe X server is at fault? But then, slower compilation sounds as if something else was using more resources than before. When did you first notice this problem anyway?

Offline

#16 2014-08-27 22:58:09

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,140

Re: How best to diagnose cause of slow down as uptime increases

I suspect it is something I'm doing. Or, rather, I've already discovered one thing I was doing which involved a cron job which I set up ages ago. At that point, its use of tmpfs would have been negligible. Now it is not. So I've switched that to using hard disk. However, I'm still eating swap but not as dramatically which makes me suspect I'm missing something else I've forgotten doing.

I'm reluctant to disable swap since I'd rather not lose data and I worry that I might. (After all, the system thinks it *needs* to use swap. Can I be sure it won't just kill processes which might have unsaved data?)

I don't remember when I first noticed the problem but it has gotten gradually worse. So it is hard to pinpoint anything precise. (I guess a Windows user might assume it was time to reinstall but I don't come from that world. Not, at least, at all recently.)

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

Pages: 1

Board footer

Powered by FluxBB