[SOLVED] Script to identify unused packages based on atime

vladms · 2021-08-22 16:58:48

Does anybody know/have a script that lists packages unused for a long time based on atime?

Some observations, to avoid comments I have seen for the same question:

I understand that ideally you would install only what you need, but:
- I occasionally install software to try it out (ex: new text editor) and after a couple of months either old or new software might be unused while I will forget about one of them
- sometimes I install development libraries while working for a project, but if in a couple of months I stop working on a project, it's hard to keep track what I installed for which project
my filesystem has relatime set, which should give a good enough estimate of "is this file read"
for utility packages that are rarely used, considering the internet speed, I would rather install them when I need them, rather than updating them many times, "just in case I need them". There is also a possibility to have a whitelist of packages to exclude from the script analysis (ex: I always want wireshark installed even if I never used it)

Don't think such a script is very complex to make, but rather than reinventing the wheel thought to ask first if someone knows one. I have seen one script, but it did not seem user friendly (ex: have option to set the time for packages unused, have option to check only packages installed or all packages, have option to exclude packages, etc.)

Last edited by vladms (2021-08-26 17:11:51)

lmn · 2021-08-22 18:33:26

vladms wrote:

Don't think such a script is very complex to make...

except for running into some limitations of reality.

Consider the case when you update a package, then the access times will be updated too. So at best you will have the difference between the last update and now.

What I do on my system is looking at the pacman update list and `pacman -Qii` the update candidates (sometimes you learn a thing or two).
Not all of them every time just those that peek my interest, but you can be more thorough of course. Over time this cuts down on packages (at least for me).
Another system that can be leveraged is your shell history. You could enable timestamps for the history and use a simple script to filter for that.
This of course misses all the packages that you don't call explicitly (when did you last call `m4` for example?)
In this case atime can then be used in conjunction with the above list to eliminate some packages.

Edit: `Qii` for reverse dependencies

Last edited by lmn (2021-08-22 18:38:48)

Trilby · 2021-08-22 18:44:49

Given the above conerns I just look for leaf packages and confirm that I want everything on the list:

#!/bin/bash

comm -13 <(pacman -Qgq base-devel) <(pacman -Qettq)

vladms · 2021-08-22 21:18:54

lmn wrote:

Consider the case when you update a package, then the access times will be updated too. So at best you will have the difference between the last update and now.

I tend to update at 3-4 months intervals (maybe is too much, but works for me). The moment I would run such a script would be before an update. In fact it is a good idea to run the script by default with the period since last update (guess I can just check pacman.log for last update).

I do sometimes look through lists of packages (either explicitly installed or the list of updates) and it is indeed occasionally instructive. I just want to have an automated solution where the system could just say "hey, in last 3 months nothing read packages X, Y" rather than me looking through a long list and saying "oh, I use A, B, C, D, E, F, etc. hmm did I use X, Y?". I currently have 124 packages in the list of leaf packages, checking through that list each time seems inefficient and error prone.

lmn · 2021-08-23 12:52:07

I used this little script

#!/bin/sh

now=$(date +%s)

pacman -Qlq "$1" | while IFS="" read -r file
do
	[ -d "$file" ] && continue

	prev=$(stat -c %X "$file")
	[ $(($now-$prev)) -lt 86400 ] && echo "$file" 
done

but this revealed another wrench in works of using atime: automated system timers/units. For example the indexer for man-db scans through the installed man pages and therefore accesses them.
While you could try to work around those, it cements my feeling that atime alone is just not the right avenue for this.

NB 86400 is the number of seconds in a day

Trilby · 2021-08-23 15:32:01

That takes a package name - so presumably the whole script would be used within a loop over package names. Then within the script it loops over files, and runs stat on each one individually. It'd be far more efficient to run stat on all files (or all managed files) once, sort the list to get files with an atime more recent than a threshold, and run pacman -Qo on that file list to get package names to keep. Then check what packages are installed that are not on the to-keep list. E.g.,

#!/bin/bash

before=$(date +%s -d 2021-08-01)

pacman -Qettlq \
	| grep -v '/$' \
	| xargs -n 500 stat -c "%X %n" 2>/dev/null \
	| awk -v T=$before '$1 > T { print $2; }' \
	| xargs -n 500 pacman -Qqo \
	| sort -u >| /tmp/recently_used

pacman -Qettq | comm -13 /tmp/recently_used -

But beware of some remaining false positives for meta-packages (e.g., base) that don't actually contain files.

There are actually loops in there implemented by xargs - but that's only due to limits on the number of command line arguments. This still processes 500 files a time in each place this is needed rather than 1 at a time. The 500 is also arbitrary - I don't use xargs much and I don't know the best way to set the flags to optimize efficiency without going over OS limits for number of command line arguments and / or total command line length. I just tried replacing 500 with 1000 and it worked fine for me: the higher the number the faster the total script will complete, but at some level it will result in an error due to too many arguments being passed (this is why xargs is there in the first place).

EDIT: I've jacked that number up to obscenely high values, and they stop making the script any faster, but there are no errors - so I gather xargs imposes it's own internal limits based on the OS limits. So really that 500 should be replaced with a huge number, and just leave it to xargs. I don't know if there is an option to xargs to tell it just to do this without giving it a huge number ... there should be, but like I said, I don't use it much.

Last edited by Trilby (2021-08-24 00:30:31)

lmn · 2021-08-23 18:03:36

Trilby wrote:

...so presumably the whole script would be used within a loop over package names.

That was not my intention, but your right that I should have been more focused on performance. In my mind this was just a one off, and I didn't want to over complicate things.

My intention was to show that there are some fundamental flaws in the proposed concept (there is a good intention behind it).
And in order to do this I just wanted to point out that some files are accessed by more processes than are first expected.

PS: precomputing the difference in $before was a very nice touch, I didn't think of that. Thanks Trillby

vladms · 2021-08-23 22:27:25

Thanks, looks good! It did not work for files with spaces (ex: I had a file named "/usr/lib/atom/node_modules/language-ruby/grammars/html (ruby - erb).json"), so I changed it a bit see below (as many times when I get a script that does what I want I end up thinking: "I should have done this in a proper language"). With version below still does not work with files containing a newline in the name, but files with newlines seem much more exceptional than files with spaces in the name. I also made it automatically use the date when I last did a system update.

The fact that some files can be accessed by automatic process will make this script less than perfect, but it can still be useful. Adding files that are ignored when looking at atime is not very complex (another issue probably are .desktop files).

before=$(grep "pacman -Syu" /var/log/pacman.log | \
           tail -n 1 | \
           awk -FT '{print substr($1,2)}' | \
           xargs -I{} date +%s -d "{} +2 day")

pacman -Qettlq \
    | grep -v '/$' \
    | xargs -d \\n stat -c "%X %n" 2>/dev/null \
    | awk -v T=$before '$1 > T { $1=""; print substr($0,2); }' \
    | tr '\n' '\0'\
    | xargs -0 pacman -Qqo \
    | sort -u >| /tmp/recently_used

pacman -Qettq | sort | comm -13 /tmp/recently_used -

Trilby · 2021-08-24 00:33:20

I'm a bit surprised that there are any packages that include files with spaces in the names. I'm far more skeptical that there are packages with files with newlines in the name.

vladms · 2021-08-24 08:16:32

Some more changes: for me it does not makes sense to analyze only explicitly installed packages. Some packages in groups can be never used and linger around. Of course, this introduces a problem of meta-packages (ex: you would not want to remove base). So I added just in case exclusion for groups and package groups. Also I made the date just relative to now, it will miss packages updated (because pacman changes them), but nothing that can be done about that.

I will mark as solved tomorrow if I do not have extra ideas/there are no comments.

#!/bin/bash

before=$(date +%s -d "now -2 months")

exclude="base grub"
exclude_groups="base-devel[ ]|xfce4[ ]"

pacman -Qttlq \
    | grep -v '/$' \
    | xargs -d \\n stat -c "%X %n" 2>/dev/null \
    | awk -v T=$before '$1 > T { $1=""; print substr($0,2); }' \
    | tr '\n' '\0'\
    | xargs -0 pacman -Qqo \
    | sort -u >| /tmp/recently_used

pacman -Qttq | sort | comm -13 /tmp/recently_used - \
    | grep -Ev $(echo $exclude | tr ' ' '|' ) \
    | grep -vFf <(pacman -Qg | grep -E "^($exclude_groups)" | cut -f2 -d' ')

Trilby · 2021-08-24 11:38:09

Instead of all the grepping at the end, just add all the "excludes" to the "recently_used" list from the start:

echo base grub >| /tmp/recently_used
pacman -Qgq base-devel xfce4 >> /tmp/recently_used

pacman -Qttlq \
   # ...
   | sort -u >> /tmp/recently_used

pacman -Qttq | sort | comm -13 /tmp/recently_used -

Arch Linux

#1 2021-08-22 16:58:48

[SOLVED] Script to identify unused packages based on atime

#2 2021-08-22 18:33:26

Re: [SOLVED] Script to identify unused packages based on atime

#3 2021-08-22 18:44:49

Re: [SOLVED] Script to identify unused packages based on atime

#4 2021-08-22 21:18:54

Re: [SOLVED] Script to identify unused packages based on atime

#5 2021-08-23 12:52:07

Re: [SOLVED] Script to identify unused packages based on atime

#6 2021-08-23 15:32:01

Re: [SOLVED] Script to identify unused packages based on atime

#7 2021-08-23 18:03:36

Re: [SOLVED] Script to identify unused packages based on atime

#8 2021-08-23 22:27:25

Re: [SOLVED] Script to identify unused packages based on atime

#9 2021-08-24 00:33:20

Re: [SOLVED] Script to identify unused packages based on atime

#10 2021-08-24 08:16:32

Re: [SOLVED] Script to identify unused packages based on atime

#11 2021-08-24 11:38:09

Re: [SOLVED] Script to identify unused packages based on atime

Board footer