Howto list files not known by pacman (-Ql) and not in /home

ibendiben · 2008-02-07 12:56:55

Imagine I want to list all files, except those listed by pacman -Ql, and except those stored in /home.
How would I do that simple? I want to see if I've got many leftovers from previous scripts I made, or uninstalled packages, or installed packages pacman doesn't know of.

tomk · 2008-02-07 13:01:32

Roughly like this - run find on your system (except /home), run pacman -Qo on the results, write unowned filenames to a file.

I'll leave the specifics up to you.

shining · 2008-02-07 13:24:08

Another equivalent way : build a filelist of all files on your file system except /home with find.
build an excludelist (~ output of pacman -Ql)
then :
grep -v -f excludelist filelist

Gilneas · 2008-02-07 13:32:25

use find all the way:

find / \( -path '/sys' -o -path '/proc' -o -path '/home' \) -prune -o -type f -a -exec pacman -Qo '{}' \; | less

edit:
probably add /var in there too (and a 2>&1)
find / $ -path '/var' -o -path '/sys' -o -path '/proc' -o -path '/home' $ -prune -o -type f -a -exec pacman -Qo '{}' \; 2>&1 | less

Last edited by Gilneas (2008-02-07 13:39:56)

ibendiben · 2008-02-07 15:35:22

Okay,
This is what I ended up doing:

find /etc /bin /sbin /opt /lib /usr -type f >/tmp/files
pacman -Ql | cut -d' ' -f2 | grep -v '/$' >/tmp/exclude
cat /tmp/files /tmp/exclude | sort | uniq -u >/tmp/orphans

It is pretty interesting really....
6334 results

EDIT: grep -v -f wouln't work, I don't quite understand

Last edited by ibendiben (2008-02-07 15:36:38)

ibendiben · 2008-02-07 15:43:36

I'm gonna try pacman -Qo now, seems better option, because -Ql doesn't know:
/bin/awk
/bin/bunzip2
/bin/bzcat
/bin/compress
/bin/dnsdomainname
/bin/domainname
/bin/nisdomainname
/bin/pidof
/bin/sh
/bin/ypdomainname

Cerebral · 2008-02-07 16:01:34

Wait, pacman -Ql doesn't list /bin/awk, but pacman -Qo /bin/awk can find an owner for it? That seems odd to me.

ibendiben · 2008-02-07 16:25:40

I feel the same actually... but try for yourself!
Another question, I can't get

pacman -Qo `cat /tmp/orphans` | grep 'error' >/tmp/orphans2

to work...
tried different things already, but every time the results are displayed in stead of being written to the file.

/maybe it's the newlines, let's see
/nope, still the same

Last edited by ibendiben (2008-02-07 16:29:48)

tomk · 2008-02-07 16:30:12

pacman -Ql gawk shows /bin/awk for me.

ibendiben · 2008-02-07 16:32:32

hey, yeah it does for me too...
failure somewhere else then srry

shining · 2008-02-07 16:32:54

ibendiben wrote:

Okay,
This is what I ended up doing:
find /etc /bin /sbin /opt /lib /usr -type f >/tmp/files
pacman -Ql | cut -d' ' -f2 | grep -v '/$' >/tmp/exclude
cat /tmp/files /tmp/exclude | sort | uniq -u >/tmp/orphans
It is pretty interesting really....
6334 results
EDIT: grep -v -f wouln't work, I don't quite understand

Forget it, it was a bad idea. Apparently, every line needs to be quoted (simple sed line). Then it is still complaining about /usr/bin/[
(can be changed to /usr/bin/\[), but then the complexity explodes. It uses 100% cpu and makes the system swap like hell.
Your method is nicer and WAY more efficient, very nice find (uniq -u).

A small note : it should be -f2- instead of -f2 for dealing with filenames with space :
pacman -Ql | cut -d' ' -f2 | grep -v '/$' >/tmp/exclude

ibendiben · 2008-02-07 16:35:59

Have you tried this though?

find /etc /bin /sbin /opt /lib /usr -type f >/tmp/files
pacman -Ql | cut -d' ' -f2 | grep -v '/$' >/tmp/exclude
cat /tmp/files /tmp/exclude | sort | uniq -u >/tmp/orphans

What could be wrong?

nj · 2008-02-07 16:40:36

ibendiben wrote:

Another question, I can't get
pacman -Qo `cat /tmp/orphans` | grep 'error' >/tmp/orphans2
to work...
tried different things already, but every time the results are displayed in stead of being written to the file.

The error lines are printed to stderr.

pacman -Qo `cat /tmp/orphans` 2> /tmp/orphans2

ibendiben · 2008-02-07 16:42:58

aha, learned something again...
can't cope with all that yet thanks!

Cerebral · 2008-02-07 16:50:43

ibendiben wrote:

Have you tried this though?

find /etc /bin /sbin /opt /lib /usr -type f >/tmp/files
pacman -Ql | cut -d' ' -f2 | grep -v '/$' >/tmp/exclude
cat /tmp/files /tmp/exclude | sort | uniq -u >/tmp/orphans

What could be wrong?

Well, first, find out which file /bin/awk is missing from:

$ grep '/bin/awk' /tmp/files
$ grep '/bin/awk' /tmp/exclude

If it's in /tmp/exclude but not in /tmp/files, then somehow your find command isn't seeing /bin/awk. When you run find, do you have read/execute permissions on /bin?

shining · 2008-02-07 16:58:32

/bin/awk is a symlink. Your find command doesn't keep symlink.

Here are the commands I just used :

time sudo find /etc /bin /sbin /opt /lib /usr /home/httpd -type f -o -type l >/tmp/files
pacman -Ql | cut -d' ' -f2- | grep -v '/$' >/tmp/exclude
cat /tmp/files /tmp/exclude | sort | uniq -u >/tmp/orphans

Your last command doesn't deal with filename space either :
pacman -Qo `cat /tmp/orphans` 2> /tmp/orphans2

ibendiben · 2008-02-07 17:54:23

time -p (sudo find /etc /bin /sbin /opt /lib /usr -type f -o -type l >/tmp/files
pacman -Ql | cut -d' ' -f2- | grep -v '/$' >/tmp/exclude
cat /tmp/files /tmp/exclude | sort | uniq -u >/tmp/orphans)

real 3.82
user 3.05
sys 0.89

Thanks a lot.
I thought about symlinks at first, only forget about it at second
I like this. Gives me 3203 lines to investigate.
-A lot of .pacsave files of course, and the old manpages (which should have been deleted),
-a lot of fonts (good?),
-also numbers of /gconf/gconf.xml.defaults/%gconf-tree...(what's that??),
-there's a whole range of /share/texmf-dist/tex/.. files too, and that's old leftovers from a wrong install of xemacs -(not enough free space) if I'm correct.
-/usr/bin/... and /usr/lib show A LOT .

How safe would it be to just delete everything??

Last edited by ibendiben (2008-02-07 18:08:15)

Gilneas · 2008-02-07 18:09:00

Dangerous!
.install files can produce additional files. Think about /usr/share/fonts/*/fonts.{dir,scale} for instance.

ibendiben · 2008-02-07 18:12:55

you mean those 'additional' files aren't register by pacman?

ibendiben · 2008-02-07 18:18:36

I'd be interested to know how much files other users would have...
Can someone post some of his numbers?

shining · 2008-02-07 18:54:58

ibendiben wrote:

you mean those 'additional' files aren't register by pacman?

No these files are not tracked (just like any other files created as package installation or at runtime, as opposed to package creation)

shining · 2008-02-07 19:03:12

ibendiben wrote:

I'd be interested to know how much files other users would have...
Can someone post some of his numbers?

I get 1381 files. But since you didn't include all directories in the find command, there are many legit files there (some are even critical).
For example, there are the kernel and grub files in /boot/, the web files in /home/httpd, etc.
But there are also some files that every user added manually that should stay, like /etc/X11/xorg.conf for example.

Anyway, removing these files will likely break your system. Even if you are careful and don't remove any critical ones, you can still mess up something. And all this for 0 benefit.

A little problem with the cat | uniq -u way is that we don't know which files are only in "files", in which files are only in "exclude".

Cerebral · 2008-02-07 19:56:48

shining wrote:

A little problem with the cat | uniq -u way is that we don't know which files are only in "files", in which files are only in "exclude".

I would think that a diff would be more useful than a uniq:

sudo find /bin /boot /etc /lib /opt /sbin /srv /usr /var -type f -o -type l | sort >/tmp/files
pacman -Ql | cut -d' ' -f2- | grep -v '/$' | sort >/tmp/exclude
diff /tmp/files /tmp/exclude >/tmp/orphans

Last edited by Cerebral (2008-02-07 20:17:08)

shining · 2008-02-07 23:18:37

That's much better indeed. Seems like this gives a decent result now.
To find all files that should be installed as part of a package, but were not on the filesystem, I then did :

 grep ">" /tmp/orphans

This gave me a lot of files I think my local database was a bit broken (I remember now that I must have messed up something once )
After removing / installing some packages, I only have some files from ttf-ms-fonts and zsh packages, because these are modified by their scriptlets.

ibendiben · 2008-02-08 10:13:56

I'm not yet content with the output of the diff command.
I do not like it, I tried using it before (also this time before I decided to use uniq) but it is difficult to understand. There are definitely lines (custom scripts in /usr/bin for example) that aren't displayed with a grep ">".
I see there are some folders, uncluding /usr/bin, displayed though. (what does that mean?)
As I understand it, diff has its own way of understanding whether a line should be added (>) or one should be deleted (<) to sync lines between the two files it compares. But then again I'm quite new to linux and bash, and stuff... so I'd like to be corrected.

There's a point though in the folder selection. If I don't select those folders outputted by a pacman -Ql, those folders-->files will be uniq....and therefor the uniq -u isn't perfect either. I think I can make up something though.
I could make a first column containing the names: "files" or "exlude", and then let sort and uniq ignore this first column.

shining wrote:

ibendiben wrote:
you mean those 'additional' files aren't register by pacman?
No these files are not tracked (just like any other files created as package installation or at runtime, as opposed to package creation)

Can you tell me how pacman succeeds in removing those packages/files then? If pacman can find out (when removing a package) which files belonged to a package, including those 'additional' files.... shouldn't there be a way to make a list of those files... including all 'additional' files?
As you see, I don't quite understand Maybe the simple question is: Doesn't pacman -Ql foo list all files it would remove with pacman -R foo?

Last edited by ibendiben (2008-02-08 10:16:10)

Arch Linux

#1 2008-02-07 12:56:55

Howto list files not known by pacman (-Ql) and not in /home

#2 2008-02-07 13:01:32

Re: Howto list files not known by pacman (-Ql) and not in /home

#3 2008-02-07 13:24:08

Re: Howto list files not known by pacman (-Ql) and not in /home

#4 2008-02-07 13:32:25

Re: Howto list files not known by pacman (-Ql) and not in /home

#5 2008-02-07 15:35:22

Re: Howto list files not known by pacman (-Ql) and not in /home

#6 2008-02-07 15:43:36

Re: Howto list files not known by pacman (-Ql) and not in /home

#7 2008-02-07 16:01:34

Re: Howto list files not known by pacman (-Ql) and not in /home

#8 2008-02-07 16:25:40

Re: Howto list files not known by pacman (-Ql) and not in /home

#9 2008-02-07 16:30:12

Re: Howto list files not known by pacman (-Ql) and not in /home

#10 2008-02-07 16:32:32

Re: Howto list files not known by pacman (-Ql) and not in /home

#11 2008-02-07 16:32:54

Re: Howto list files not known by pacman (-Ql) and not in /home

#12 2008-02-07 16:35:59

Re: Howto list files not known by pacman (-Ql) and not in /home

#13 2008-02-07 16:40:36

Re: Howto list files not known by pacman (-Ql) and not in /home

#14 2008-02-07 16:42:58

Re: Howto list files not known by pacman (-Ql) and not in /home

#15 2008-02-07 16:50:43

Re: Howto list files not known by pacman (-Ql) and not in /home

#16 2008-02-07 16:58:32

Re: Howto list files not known by pacman (-Ql) and not in /home

#17 2008-02-07 17:54:23

Re: Howto list files not known by pacman (-Ql) and not in /home

#18 2008-02-07 18:09:00

Re: Howto list files not known by pacman (-Ql) and not in /home

#19 2008-02-07 18:12:55

Re: Howto list files not known by pacman (-Ql) and not in /home

#20 2008-02-07 18:18:36

Re: Howto list files not known by pacman (-Ql) and not in /home

#21 2008-02-07 18:54:58

Re: Howto list files not known by pacman (-Ql) and not in /home

#22 2008-02-07 19:03:12

Re: Howto list files not known by pacman (-Ql) and not in /home

#23 2008-02-07 19:56:48

Re: Howto list files not known by pacman (-Ql) and not in /home

#24 2008-02-07 23:18:37

Re: Howto list files not known by pacman (-Ql) and not in /home

#25 2008-02-08 10:13:56

Re: Howto list files not known by pacman (-Ql) and not in /home

Board footer