You are not logged in.

#1 2014-09-13 16:26:32

nstgc
Member
Registered: 2014-03-17
Posts: 393

[solved] Questions about pacman -Qk and -Qkk output.

I was looking through the pacman manual today to see if there was anything I could add to a some Arch maintenance scripts I've been working on and came across -Qk which, among other things, lists missing files. I ran pacman -Qk in terminal and found a few package installations were missing files. I reinstalled them manually check again and the files were no longer missing. The first package I saw that was missing a file was gnome-command-center for instance.

So I really have two questions.
1) Is it normal for packages to have missing files?
2) Is it a good idea to reinstall these packages to replace the missing files.

I then ran pacman -Qkk, and got even more disturbing information such as modification time mismatches and size mismatches.

$ pacman -Qkk |grep warning
warning: avahi: /etc/avahi/avahi-daemon.conf (Modification time mismatch)
warning: avahi: /etc/avahi/hosts (Modification time mismatch)
warning: avahi: /etc/avahi/services/sftp-ssh.service (Modification time mismatch)
warning: avahi: /etc/avahi/services/ssh.service (Modification time mismatch)
warning: avahi: /usr/lib/avahi/service-types.db (Modification time mismatch)
warning: avahi: /usr/share/avahi/service-types (Modification time mismatch)
warning: bash: /etc/bash.bash_logout (Modification time mismatch)
warning: bash: /etc/bash.bashrc (Modification time mismatch)
warning: bash: /etc/skel/.bash_logout (Modification time mismatch)
warning: bash: /etc/skel/.bash_profile (Modification time mismatch)
warning: bash: /etc/skel/.bashrc (Modification time mismatch)
warning: bluez: /etc/bluetooth/main.conf (Modification time mismatch)
warning: bluez: /etc/dbus-1/system.d/bluetooth.conf (Modification time mismatch)
warning: ca-certificates-java: /etc/default/cacerts (Modification time mismatch)
warning: cdrtools: /etc/default/cdrecord (Modification time mismatch)
warning: cdrtools: /etc/default/rscsi (Modification time mismatch)
warning: chromium: /etc/chromium/default (Modification time mismatch)
warning: chromium: /etc/chromium/default (Size mismatch)
warning: colord: /var/lib/colord (UID mismatch)
warning: colord: /var/lib/colord (GID mismatch)
warning: dhcpcd: /etc/dhcpcd.conf (Modification time mismatch)
warning: e2fsprogs: /etc/mke2fs.conf (Modification time mismatch)
warning: filesystem: /etc/crypttab (Modification time mismatch)
warning: filesystem: /etc/fstab (Modification time mismatch)
warning: filesystem: /etc/fstab (Size mismatch)
warning: filesystem: /etc/group (Modification time mismatch)
warning: filesystem: /etc/group (Size mismatch)
warning: filesystem: /etc/gshadow (Permissions mismatch)
warning: filesystem: /etc/gshadow (Modification time mismatch)
warning: filesystem: /etc/gshadow (Size mismatch)
warning: filesystem: /etc/host.conf (Modification time mismatch)
warning: filesystem: /etc/hosts (Modification time mismatch)
warning: filesystem: /etc/issue (Modification time mismatch)
warning: filesystem: /etc/ld.so.conf (Modification time mismatch)
warning: filesystem: /etc/motd (Modification time mismatch)
warning: filesystem: /etc/mtab (Modification time mismatch)
warning: filesystem: /etc/mtab (Symlink path mismatch)
warning: filesystem: /etc/nsswitch.conf (Modification time mismatch)
warning: filesystem: /etc/passwd (Modification time mismatch)
warning: filesystem: /etc/passwd (Size mismatch)
warning: filesystem: /etc/profile (Modification time mismatch)
warning: filesystem: /etc/resolv.conf (Modification time mismatch)
warning: filesystem: /etc/resolv.conf (Size mismatch)
warning: filesystem: /etc/securetty (Modification time mismatch)
warning: filesystem: /etc/shadow (Permissions mismatch)
warning: filesystem: /etc/shadow (Modification time mismatch)
warning: filesystem: /etc/shadow (Size mismatch)
warning: filesystem: /etc/shells (Modification time mismatch)
warning: flashplugin: /etc/adobe/mms.cfg (Modification time mismatch)
warning: flashplugin: /etc/adobe/mms.cfg (Size mismatch)
warning: glibc: /etc/locale.gen (Modification time mismatch)
warning: glibc: /etc/locale.gen (Size mismatch)
warning: gnome-control-center: /usr/share/polkit-1/rules.d/gnome-control-center.rules (Permission denied)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/GnoMenu (Modification time mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/GnoMenu (Size mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/GnoMenu.conf (Modification time mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/GnoMenu.conf (Size mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/auto-load.conf (Permissions mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/auto-load.conf (Modification time mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/auto-load.conf (Size mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/icon (Modification time mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/icon (Size mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/preview (Permissions mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/preview (Modification time mismatch)
warning: gnomenu: /usr/share/cairo-dock/plug-ins/Dbus/third-party/GnoMenu/preview (Size mismatch)
warning: grub: /boot/grub/grub.cfg (Permissions mismatch)
warning: grub: /boot/grub/grub.cfg (Modification time mismatch)
warning: grub: /boot/grub/grub.cfg (Size mismatch)
warning: iproute2: /etc/iproute2/ematch_map (Modification time mismatch)
warning: iproute2: /etc/iproute2/rt_dsfield (Modification time mismatch)
warning: iproute2: /etc/iproute2/rt_protos (Modification time mismatch)
warning: iproute2: /etc/iproute2/rt_realms (Modification time mismatch)
warning: iproute2: /etc/iproute2/rt_scopes (Modification time mismatch)
warning: iproute2: /etc/iproute2/rt_tables (Modification time mismatch)
warning: jack: /etc/security/limits.d/99-audio.conf (Modification time mismatch)
warning: java-common: /etc/profile.d/jre.csh (Modification time mismatch)
warning: java-common: /etc/profile.d/jre.sh (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/calendars.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/content-types.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/flavormap.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/fontconfig.bfc (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/fontconfig.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/logging.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/net.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/psfont.properties.ja (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/psfontj2d.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/sound.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/tz.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/amd64/jvm.cfg (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/images/cursors/cursors.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/management/jmxremote.access (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/management/jmxremote.password (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/management/management.properties (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/management/snmp.acl (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/security/java.policy (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/security/java.security (Modification time mismatch)
warning: jre7-openjdk-headless: /etc/java-7-openjdk/security/nss.cfg (Modification time mismatch)
warning: krb5: /etc/krb5.conf (Modification time mismatch)
warning: krb5: /var/lib/krb5kdc/kdc.conf (Modification time mismatch)
warning: libcups: /etc/cups/client.conf (Modification time mismatch)
warning: libirman: /etc/irman.conf (Modification time mismatch)
warning: libnl: /etc/libnl/classid (Modification time mismatch)
warning: libnl: /etc/libnl/pktloc (Modification time mismatch)
warning: libreoffice-still-common: /etc/libreoffice/bootstraprc (Modification time mismatch)
warning: libreoffice-still-common: /etc/libreoffice/psprint.conf (Modification time mismatch)
warning: libreoffice-still-common: /etc/libreoffice/sofficerc (Modification time mismatch)
warning: libreoffice-still-common: /etc/profile.d/libreoffice-still-common.csh (Modification time mismatch)
warning: libreoffice-still-common: /etc/profile.d/libreoffice-still-common.sh (Modification time mismatch)
warning: linux: /etc/mkinitcpio.d/linux.preset (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.alias (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.alias (Size mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.alias.bin (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.alias.bin (Size mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.builtin.bin (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.dep (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.dep (Size mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.dep.bin (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.dep.bin (Size mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.devname (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.softdep (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.symbols (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.symbols (Size mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.symbols.bin (Modification time mismatch)
warning: linux: /usr/lib/modules/3.16.2-1-ARCH/modules.symbols.bin (Size mismatch)
warning: lirc-utils: /etc/lirc/lirc_options.conf (Modification time mismatch)
warning: lirc-utils: /etc/lirc/lircd.conf (Modification time mismatch)
warning: lirc-utils: /etc/lirc/lircmd.conf (Modification time mismatch)
warning: lvm2: /etc/lvm/lvm.conf (Modification time mismatch)
warning: mdadm: /etc/mdadm.conf (Modification time mismatch)
warning: mime-types: /etc/mime.types (Modification time mismatch)
warning: mkinitcpio: /etc/mkinitcpio.conf (Modification time mismatch)
warning: mkinitcpio: /etc/mkinitcpio.conf (Size mismatch)
warning: mplayer: /etc/mplayer/codecs.conf (Modification time mismatch)
warning: mplayer: /etc/mplayer/input.conf (Modification time mismatch)
warning: nano: /etc/nanorc (Modification time mismatch)
warning: networkmanager: /etc/NetworkManager/NetworkManager.conf (Modification time mismatch)
warning: openresolv: /etc/resolvconf.conf (Modification time mismatch)
warning: openssl: /etc/ssl/openssl.cnf (Modification time mismatch)
warning: pacman: /etc/makepkg.conf (Modification time mismatch)
warning: pacman: /etc/makepkg.conf (Size mismatch)
warning: pacman: /etc/pacman.conf (Modification time mismatch)
warning: pacman: /etc/pacman.conf (Size mismatch)
warning: pacman-mirrorlist: /etc/pacman.d/mirrorlist (UID mismatch)
warning: pacman-mirrorlist: /etc/pacman.d/mirrorlist (GID mismatch)
warning: pacman-mirrorlist: /etc/pacman.d/mirrorlist (Modification time mismatch)
warning: pacman-mirrorlist: /etc/pacman.d/mirrorlist (Size mismatch)
warning: pam: /etc/environment (Modification time mismatch)
warning: pam: /etc/default/passwd (Modification time mismatch)
warning: pam: /etc/security/access.conf (Modification time mismatch)
warning: pam: /etc/security/group.conf (Modification time mismatch)
warning: pam: /etc/security/limits.conf (Modification time mismatch)
warning: pam: /etc/security/namespace.conf (Modification time mismatch)
warning: pam: /etc/security/namespace.init (Modification time mismatch)
warning: pam: /etc/security/pam_env.conf (Modification time mismatch)
warning: pam: /etc/security/time.conf (Modification time mismatch)
warning: polkit: /etc/polkit-1/rules.d/50-default.rules (Permission denied)
warning: pulseaudio-alsa: /etc/asound.conf (Modification time mismatch)
warning: rhash: /etc/rhashrc (Modification time mismatch)
warning: rhash: /etc/rhashrc (Size mismatch)
warning: rsync: /etc/rsyncd.conf (Modification time mismatch)
warning: rsync: /etc/xinetd.d/rsync (Modification time mismatch)
warning: samba: /etc/conf.d/samba (Modification time mismatch)
warning: samba: /etc/logrotate.d/samba (Modification time mismatch)
warning: samba: /etc/pam.d/samba (Modification time mismatch)
warning: sudo: /etc/sudoers (Modification time mismatch)
warning: sudo: /etc/sudoers (Size mismatch)
warning: systemd: /etc/dbus-1/system.d/org.freedesktop.hostname1.conf (Modification time mismatch)
warning: systemd: /etc/dbus-1/system.d/org.freedesktop.locale1.conf (Modification time mismatch)
warning: systemd: /etc/dbus-1/system.d/org.freedesktop.login1.conf (Modification time mismatch)
warning: systemd: /etc/dbus-1/system.d/org.freedesktop.machine1.conf (Modification time mismatch)
warning: systemd: /etc/dbus-1/system.d/org.freedesktop.systemd1.conf (Modification time mismatch)
warning: systemd: /etc/dbus-1/system.d/org.freedesktop.timedate1.conf (Modification time mismatch)
warning: systemd: /etc/pam.d/systemd-user (Modification time mismatch)
warning: systemd: /etc/systemd/bootchart.conf (Modification time mismatch)
warning: systemd: /etc/systemd/coredump.conf (Modification time mismatch)
warning: systemd: /etc/systemd/journald.conf (Modification time mismatch)
warning: systemd: /etc/systemd/logind.conf (Modification time mismatch)
warning: systemd: /etc/systemd/resolved.conf (Modification time mismatch)
warning: systemd: /etc/systemd/system.conf (Modification time mismatch)
warning: systemd: /etc/systemd/timesyncd.conf (Modification time mismatch)
warning: systemd: /etc/systemd/timesyncd.conf (Size mismatch)
warning: systemd: /etc/systemd/user.conf (Modification time mismatch)
warning: systemd: /etc/udev/udev.conf (Modification time mismatch)
warning: systemd: /var/log/journal/remote (GID mismatch)
warning: texlive-core: /etc/texmf/web2c/updmap.cfg (Modification time mismatch)
warning: texlive-core: /etc/texmf/web2c/updmap.cfg (Size mismatch)
warning: util-linux: /etc/pam.d/chfn (Modification time mismatch)
warning: util-linux: /etc/pam.d/chsh (Modification time mismatch)
warning: util-linux: /etc/pam.d/login (Modification time mismatch)
warning: util-linux: /etc/pam.d/su (Modification time mismatch)
warning: util-linux: /etc/pam.d/su-l (Modification time mismatch)
warning: vim-runtime: /etc/vimrc (Modification time mismatch)
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (Modification time mismatch)
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (Size mismatch)
warning: xorg-xinit: /etc/X11/xinit/xinitrc (Modification time mismatch)
warning: xorg-xinit: /etc/X11/xinit/xserverrc (Modification time mismatch)
warning: xorg-xinit: /etc/skel/.xinitrc (Modification time mismatch)
warning: xorg-xinit: /etc/skel/.xsession (Modification time mismatch)
warning: xorg-xinit: /etc/skel/.xsession (Size mismatch)
warning: yaourt: /etc/yaourtrc (Modification time mismatch)

I'm assuming all of those are bad. "Warning" rarely sounds friendly, though for those files I've modified myself, such as passwd, shadow, and group, its okay. I assume I can fix msot of these by running # pacman -S {package}?

Last edited by nstgc (2014-09-14 01:26:04)

Offline

#2 2014-09-13 16:57:06

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: [solved] Questions about pacman -Qk and -Qkk output.

Generally pacman -Qk reporting missing files would be a problem.  If that recurs, you may want to look into what is happening to those files.

But -Qkk is very verbose.  A vast majority of those warnings (if not all) are normal and expected.  Reinstalling the package to just change modification times would be really wasteful.

I did see a permission mismatch in there which might not be normal - but overall I don't think scruitinizing -Qkk output is a productive maintenance step.  If something is wrong it *might* help track down the cause of a problem - but if it aint broke, don't fix it.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2014-09-13 17:30:01

nstgc
Member
Registered: 2014-03-17
Posts: 393

Re: [solved] Questions about pacman -Qk and -Qkk output.

Thank you. I'll have to think up some code that will point out which packages have missing files later then, and I'll just ignore pacman -Qkk.

Offline

#4 2014-09-13 18:29:05

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: [solved] Questions about pacman -Qk and -Qkk output.

What code do you have to think up?  Pacman does this itself: `pacman -Qkq`


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#5 2014-09-13 20:52:14

nstgc
Member
Registered: 2014-03-17
Posts: 393

Re: [solved] Questions about pacman -Qk and -Qkk output.

Trilby wrote:

What code do you have to think up?  Pacman does this itself: `pacman -Qkq`

A lot less now. I didn't notice that -q had that effect on -k. Thank you. I still want to write a bit to strip off the file so I can run "pacman -S $(something that gives package names only)" after it tells me what it is that is missing so that I can keep track of it.

Offline

#6 2014-09-13 20:59:17

emeres
Member
Registered: 2013-10-08
Posts: 1,570

Re: [solved] Questions about pacman -Qk and -Qkk output.

 Something like this?

pacman -Qkq | awk '{print $1 | "sort"}' | uniq
pacman -S $(pacman -Qkq | awk '{print $1 | "sort"}' | uniq )

Offline

#7 2014-09-13 21:47:00

nstgc
Member
Registered: 2014-03-17
Posts: 393

Re: [solved] Questions about pacman -Qk and -Qkk output.

emeres wrote:

 Something like this?

pacman -Qkq | awk '{print $1 | "sort"}' | uniq
pacman -S $(pacman -Qkq | awk '{print $1 | "sort"}' | uniq )

Thank you! I don't have time right now to think through what all of that does since I really only have a working knowledge of bash, which much of that is not a part of since my my working knowledge is basic, however once I do I certainly will look to incorporating that. My current knowledge barely extends past the script in this post which is something that is also a work in progress, but a less critical work in progress.

But yeah, once I can figure out how that works I'll definitely include it. I just don't like running commands as root unless I know exactly how they work, and my maintenance scripts (daily, weekly, monthly) have to be run as root.

I came asking questions about pacman -Qk and -Qkk, and came back with commands and code. Its like Christmas.

Offline

#8 2014-09-13 22:20:56

emeres
Member
Registered: 2013-10-08
Posts: 1,570

Re: [solved] Questions about pacman -Qk and -Qkk output.

 Awk processes the output from pacman, splits it using [white]space[⁣s] as delimiter and prints only first field $1, which is the package name. Those reoccur if several files are involved, so just to be sure, I also used awks sort, maybe that could be skipped or alternatively sort command outside awk used instead. This sorted output gets fed into uniq, which outputs only unique strings coming from input. And that leaves you with the package list.

Edit: I think those are whitespaces actually, which if I recall correctly include space.

Edit2: So this might be better and certainly a little faster:

pacman -S $(pacman -Qkq | awk '{print $1}' | uniq)
# And this should be even faster, since the output pacman 
# delivers seems to be sorted anyway. Should you parse
# something else, use my first suggestion instead.
pacman -Qkq | awk 'BEGIN{ y=0; }; {x=$1; if (x == y){ next; }; print $1; y=x;}'

Edit3: Out of curiosity I tested different methods measuring execution time with a input file consisting out of 1443 lines from pacman -Qkkq. Here are the results and the bottom is the test script.

Sun 14 Sep 15:14:26 CEST 2014, Generating input file
Sun 14 Sep 15:15:22 CEST 2014, Loops: 10
	0	:	/tmp/awk-adv.log
	0.01	:	/tmp/awk-sort.log
	0	:	/tmp/awk-sort-outside.log
	0.002	:	/tmp/awk-uniq.log
Seconds: 0
Sun 14 Sep 15:15:22 CEST 2014, Loops: 20
	0.001	:	/tmp/awk-adv.log
	0.0095	:	/tmp/awk-sort.log
	0.0005	:	/tmp/awk-sort-outside.log
	0	:	/tmp/awk-uniq.log
Seconds: 1
Sun 14 Sep 15:15:23 CEST 2014, Loops: 50
	0	:	/tmp/awk-adv.log
	0.0096	:	/tmp/awk-sort.log
	0.0002	:	/tmp/awk-sort-outside.log
	0	:	/tmp/awk-uniq.log
Seconds: 2
Sun 14 Sep 15:15:25 CEST 2014, Loops: 100
	0.0003	:	/tmp/awk-adv.log
	0.0092	:	/tmp/awk-sort.log
	0	:	/tmp/awk-sort-outside.log
	0.0003	:	/tmp/awk-uniq.log
Seconds: 5
Sun 14 Sep 15:15:30 CEST 2014, Loops: 200
	0.00045	:	/tmp/awk-adv.log
	0.00955	:	/tmp/awk-sort.log
	0.0001	:	/tmp/awk-sort-outside.log
	0.0006	:	/tmp/awk-uniq.log
Seconds: 8
Sun 14 Sep 15:15:38 CEST 2014, Loops: 500
	0.00042	:	/tmp/awk-adv.log
	0.0094	:	/tmp/awk-sort.log
	0.0004	:	/tmp/awk-sort-outside.log
	0.00026	:	/tmp/awk-uniq.log
Seconds: 21
Sun 14 Sep 15:15:59 CEST 2014, Loops: 1000
	0.00034	:	/tmp/awk-adv.log
	0.00942	:	/tmp/awk-sort.log
	0.00042	:	/tmp/awk-sort-outside.log
	0.00025	:	/tmp/awk-uniq.log
Seconds: 42
Sun 14 Sep 15:16:41 CEST 2014, Loops: 2000
	0.00038	:	/tmp/awk-adv.log
	0.009355	:	/tmp/awk-sort.log
	0.000285	:	/tmp/awk-sort-outside.log
	0.00042	:	/tmp/awk-uniq.log
Seconds: 83
Sun 14 Sep 15:18:04 CEST 2014, Loops: 5000
	0.000388	:	/tmp/awk-adv.log
	0.009274	:	/tmp/awk-sort.log
	0.00034	:	/tmp/awk-sort-outside.log
	0.000366	:	/tmp/awk-uniq.log
Seconds: 208
Sun 14 Sep 15:21:32 CEST 2014, Loops: 10000
        0.00036	:	/tmp/awk-adv.log
	0.009286	:	/tmp/awk-sort.log
	0.000346	:	/tmp/awk-sort-outside.log
	0.000358	:	/tmp/awk-uniq.log
Seconds: 419
Sun 14 Sep 15:28:31 CEST 2014, Loops: 20000
	0.00029	:	/tmp/awk-adv.log
	0.009115	:	/tmp/awk-sort.log
	0.0002725	:	/tmp/awk-sort-outside.log
	0.000299	:	/tmp/awk-uniq.log
Seconds: 828
Sun 14 Sep 15:42:19 CEST 2014, Loops: 50000
	0.0003174	:	/tmp/awk-adv.log
	0.0093014	:	/tmp/awk-sort.log
	0.0002422	:	/tmp/awk-sort-outside.log
	0.000292	:	/tmp/awk-uniq.log
Seconds: 2171

 The awk internal sorting is the slowest method, but it seems it initializes longer, since there appears to be a constant difference between internal and external sorting. Uniq fed by awk is faster with smaller loops, but gets slower if repeated more often, yet overall remain fastest with the unique function I used within awk. It appears the more lines the input has, the slower uniq in comparison uniq in awk gets. Here is the testing script:

#!/bin/bash
LOOPS=('10' '20' '50' '100' '200' '500' '1000' '2000' '5000' '10000' '20000' '50000');
rm /tmp/results*.log
echo $(date), Generating input file;
pacman -Qkkq > /tmp/input.txt

for LOOP in ${LOOPS[@]}; do
START=$(date +%s);
echo $(date), Loops: $LOOP;

for i in $(seq $LOOP); do env TIME='%e:%U:%S' time awk '{print $1 | "sort" }' /tmp/input.txt | uniq; done 2>&1 >/dev/null | grep : > /tmp/awk-sort.log
for i in $(seq $LOOP); do env TIME='%e:%U:%S' time awk '{print $1}' /tmp/input.txt | sort | uniq; done 2>&1 >/dev/null | grep : > /tmp/awk-sort-outside.log
for i in $(seq $LOOP); do env TIME='%e:%U:%S' time awk '{print $1}' /tmp/input.txt | uniq; done 2>&1 >/dev/null | grep : > /tmp/awk-uniq.log
for i in $(seq $LOOP); do env TIME='%e:%U:%S' time awk 'BEGIN{ y=0; }; {x=$1; if (x == y){ next; }; print $1; y=x;}' /tmp/input.txt; done 2>&1 >/dev/null | grep : > /tmp/awk-adv.log

for FILE in /tmp/awk-*.log; do awk 'BEGIN{x=0; y=0; FS=":"}; {x++; y=y+$1;}; END{print "\t"y/x"\t:\t"FILENAME;}' "$FILE"; done | sort >> /tmp/results-sorted.log
for FILE in /tmp/awk-*.log; do awk 'BEGIN{x=0; y=0; FS=":"}; {x++; y=y+$1;}; END{print "\t"y/x"\t:\t"FILENAME;}' "$FILE"; done | tee -a /tmp/results.log

END=$(date +%s);
DIFF=$(( $END - $START ));
echo Seconds: $DIFF;

done | tee /tmp/awk.log

Last edited by emeres (2014-09-14 14:38:18)

Offline

#9 2014-09-14 00:53:55

nstgc
Member
Registered: 2014-03-17
Posts: 393

Re: [solved] Questions about pacman -Qk and -Qkk output.

emeres wrote:

 Awk processes the output from pacman, splits it using [white]space[⁣s] as delimiter and prints only first field $1, which is the package name. Those reoccur if several files are involved, so just to be sure, I also used awks sort, maybe that could be skipped or alternatively sort command outside awk used instead. This sorted output gets fed into uniq, which outputs only unique strings coming from input. And that leaves you with the package list.

Edit: I think those are whitespaces actually, which if I recall correctly include space.

Edit2: So this might be better and certainly a little faster:

pacman -S $(pacman -Qkq | awk '{print $1}' | uniq)
# And this should be even faster, since the output pacman 
# delivers seems to be sorted anyway. Should you parse
# something else, use my first suggestion instead.
pacman -Qkq | awk 'BEGIN{ y=0; }; {x=$1; if (x == y){ next; }; print $1; y=x;}'

Thank you once again. You easily saved me at least an hour of work.

Offline

#10 2014-09-14 00:56:12

emeres
Member
Registered: 2013-10-08
Posts: 1,570

Re: [solved] Questions about pacman -Qk and -Qkk output.

 You are welcome. Should you consider this issue solved, then please mark the thread as such.

Offline

Board footer

Powered by FluxBB