a menu to list ALL apps (not only gui) - like debian menu?

ondoho · 2016-05-19 06:24:28

edit: updated script as github repo. also see this thread.

I am looking for a comprehensive listing of all apps and utilities installed.
something like the debian menu, which does not use .desktop files, but has its own structure and integration into debian's package management.
Its main advantage: It lists way more apps, and also those that run in a terminal.
Since Archlinux does not have this afaik, how could I approach the task?
Maybe something similar or a partial solution already exists?

to elaborate:

in order of priority:
* All executables that are not listed in any menu that gets .desktop files from /usr/share/applications or ~/.local/share/applications
* A short description of what it does
* Since the list will become large, some submenu system might be required
* Preferably filter out some stuff (coreutils?)
* launching the application (would require to seperate graphical and non-graphical apps. how?)
* create output suitable for dmenu, openbox menu etc.

I can already think of some scripting that simply goes through `/usr/bin`, then uses various `pacman -Q...` options, but i don't want to re-invent the wheel, hence this thread.
I'm sure someone must have thought of (something like) this before.

Last edited by ondoho (2016-10-02 11:48:50)

AbaddonOrmuz · 2016-05-19 16:27:44

On your terminal

pacman -Qs

Check pacman man page for more options, like -q if you only need the package name and -l (query options) to list all files provided by the package.

For example

pacman -Ql | grep -E -e '/(sbin|bin)/.'

Will show something like

...
pacman /usr/bin/bacman
pacman /usr/bin/checkupdates
pacman /usr/bin/cleanupdelta
pacman /usr/bin/makepkg
pacman /usr/bin/makepkg-template
pacman /usr/bin/paccache
pacman /usr/bin/pacdiff
pacman /usr/bin/paclist
pacman /usr/bin/paclog-pkglist
pacman /usr/bin/pacman
pacman /usr/bin/pacman-db-upgrade
pacman /usr/bin/pacman-key
pacman /usr/bin/pacman-optimize
pacman /usr/bin/pacscripts
pacman /usr/bin/pacsearch
pacman /usr/bin/pacsort
pacman /usr/bin/pactree
pacman /usr/bin/pkgdelta
pacman /usr/bin/rankmirrors
pacman /usr/bin/repo-add
pacman /usr/bin/repo-elephant
pacman /usr/bin/repo-remove
pacman /usr/bin/testpkg
pacman /usr/bin/updpkgsums
pacman /usr/bin/vercmp
...

*Edit

There's also a package (pkgfile) for that

- https://wiki.archlinux.org/index.php/Pkgfile
- https://github.com/falconindy/pkgfile/b … README.pod

Last edited by AbaddonOrmuz (2016-05-19 18:27:41)

TheChickenMan · 2016-05-19 17:48:37

#!/bin/bash
for FILE in /usr/bin/*; do
	pacman -Qo $FILE | awk -F\/ {'print $4'}
done

ondoho · 2016-05-21 07:32:17

so it looks like i will have to write this, if i want it.

i had a look at menumaker and it finds more applications than the usual .desktop entries! but still not all.

i have yet to fully understand the xdg menu specifications; maybe i will find a way to "fill the gaps" left by a standard xdg desktop menu.

right now the most difficult task seems to be to programmatically find out whether an app runs graphically or in a terminal.

Trilby · 2016-05-21 11:10:50

I have no idea what you mean by "app". That is a mobile phone buzzword that is completely meaningless in the context of a linux desktop.

Are you referring to binary programs? If so, the list is `ls /usr/bin/`

Why do you need to know if they are a graphical program or not? Earlier in this thread (and in your title), you said you wanted to include both. Most graphical programs do ship their own .desktop files.

Have you tried dmenu / dmenu_run?

ondoho · 2016-10-01 12:17:06

the idea is to have an informative 1-liner for each program.
as a quick reminder what is installed and available.
the goal is that it also works as an actual launcher.

parsing the info for installed packages with pacman seemed a little slow, so i decided to access the files and directories in /var/lib/pacman/local directly, which, afaics, contains a list of all installed packages.

the script creates a list of all installed packages with a short description, and a second list has corresponding entries for executables available for each package.
so via dmenu i first choose a package, then an executable to execute.
it's better than having thousands of executables to choose from in one big menu.
it also filters out packages that have nothing in /usr/bin.

this is fun to hack, but i'm not completely happy with the functionality yet.

the biggest problem still is to know if the program is a command line program or not.
right now i just open everything in a terminal; not satisfactory in some cases (e.g. launching a panel or pager).

anyhow, here's the WIP:
edit: don't use this! updated script as github repo.

#!/bin/bash

me="$(basename $0)"
dir="/tmp/$me.cache.d"
update=0
menu=0
lines=10

version="0.0.1"

######################### FUNCTIONS #############################

usage() {
	[[ "x$1" != "x" ]] && echo "$1"
	echo -e "
Usage:	$me [-U] [-M] [-D directory]

	-M	will show the menu. This is the default if no options are given.
		Defaults to $dir if not specified with -D

	-U	will update the cache.
		Defaults to $dir if not specified with -D

	-D	specify directory for cached files.
		Applies to both -M and -U options.

Version: $version"
	exit 1
}
update() {
	mkdir -p "$dir"
	# directory not writeable? abort!
	[[ ! -w "$dir" ]] && usage "Directory not writeable!"
	cd "$dir"
	dir="$PWD"
	rm -f "$dir/headers" "$dir/binaries"
	cd /var/lib/pacman/local/
	for package in  *
	do
		if [[ -d "$package" ]]
		then
			binaries="$(sed '/usr\/bin\/[^$]/!d; s/usr\/bin\///' "$package/files" )"
			if [[ "x$binaries" != "x" ]]
			then
				echo "${package%%-[0-9]*} - $(sed '8!d' "$package/desc")" >> "$dir/headers"
				echo "${binaries//$'\n'/ }" >> "$dir/binaries"
			fi
		fi
	done
	unset package binaries
	cd
}
menu() {
	[[ ! -f "$dir/headers" ]] && usage "File $dir/headers does not exist!"
	
	choice="$(cat "$dir/headers" | dmenu -l $lines)"
	
	[[ ! -f "$dir/binaries" ]] && usage "File $dir/binaries does not exist!"

	if [[ "x$choice" != "x" ]]
	then
		choice="$(grep -n -F "$choice" "$dir/headers")"
		choice="$(sed "${choice%%:*}!d; s/ /\n/g" "$dir/binaries" | dmenu -l "$lines")"
		exec $TERMINAL -e sh -c "$choice;$SHELL" & disown
	fi
}

######################### MAIN ######################################

while getopts "vhUMD:" opt; do
    case "$opt" in
    U)  update=1
        ;;
    D)  dir="$OPTARG"
				;;
		M)	menu=1
				;;
    h|v)  usage
        ;;
    *)	usage
				;;
    esac
done

# default action is to show the menu.
[[ "$update" == "0" ]] && menu=1

[[ "$update" == "1" ]] && update

[[ "$menu" == "1" ]] && menu

exit 0

i have seperated the cache generation from the menu because it takes quite a few seconds.
my idea is to have a pacman hook do that whenever something gets un/installed:

[Trigger]
Type = File
Operation = Install
Operation = Remove
Target = *

[Action]
Description = Updating pacmenu
When = PostTransaction
Exec = /usr/bin/sudo -u <username> /home/<username>/bin/pacmenu -U

Last edited by ondoho (2016-10-02 12:04:34)

Trilby · 2016-10-01 13:05:43

Note that `[[ "x$variable" != "x" ]]` is better as `[[ -n $variable ]]`. But in this case the whole construction of the "binaries" variable and testing of it can be replaced:

if $(grep -qm1 'usr/bin/[^$]' $package/files); then

You should also determine whether you want to use the more efficient bash builtin '[[' or the POSIX-compliant binary '['. As is, you are switching between them. As your shebang is bash anyways, use '[[' for all conditionals, it will be more efficient.

EDIT:

You may also be interested in expac. I'm not going to translate your whole script to use expac, but I suspect it could do almost everything you are doing manually with the /var/ files, eg:

expac -Q "%n: %d"

Or maybe I can translate a majority of that script - show package name and description of packages with files under /usr/bin:

 expac -Q '%n\t%d\t%F' | awk -F"\t" '/ usr\/bin\/[^$]/ { printf "%s: %s\n", $1, $2; }'

(edit: oops I quoted myself rather than edited by accident. I've cleaned up and removed the duplicated portion of the post).

Last edited by Trilby (2016-10-01 18:56:16)

ondoho · 2016-10-01 16:10:38

thanks for telling me about expac!
i'm sure it is precisely what i need.

i made a few small alterations to the script 2 posts up, and will return when i got more familiar with expac.

any thoughts about seperating gui/cli apps?

ondoho · 2016-10-02 11:39:30

yes!
expac helped a lot.
i still need to grep the database in /var/lib/pacman/local directly, to find which packages install files in /usr/bin.
with expac, this would require first listing ALL installed files (files! not packages!), a HUGE list. in that grep is much faster.

i have whittled it down to 1 expac command and 1 grep command, the rest is just juggling arrays in loops...

executes in 0.2s on my system, so no caching is necessary anymore.

it is now very close to the standard dmenu_run, with added package description.

i have created a github repo.

Trilby · 2016-10-02 11:42:15

ondoho wrote:

I still need to grep the database in /var/lib/pacman/local directly, to find which packages install files in /usr/bin.
with expac, this would require first listing ALL installed files (files! not packages!)

Did you read my second example? It does this. Either approach will require scanning an enormous list of files. But in neither case should you construct a full list then grep it. Just pipe the output of expac through awk as in my example.

Your approach scans through every "files" file and greps all of the files, creates a huge bash array, then loops through the bash array calling a *new* instance of expac for every single package. My approach gets the same output with a single instance of expac and a single instance of awk.

ondoho · 2016-10-02 12:03:16

Trilby wrote:

Did you read my second example? It does this.

you mean this?

expac -Q '%n\t%d\t%F' | awk -F"\t" '/ usr\/bin\/[^$]/ { printf "%s: %s\n", $1, $2; }'

it's a neat oneliner; i think i discarded it out of tiredness... sorry.

anyhow, my version is still 3x as fast! i don't know how and why, but there it is:

=>$ time expac -Q '%n\t%d\t%F' | awk -F"\t" '/ usr\/bin\/[^$]/ { printf "%s: %s\n", $1, $2; }' >/dev/null
real	0m0.355s
user	0m0.373s
sys	0m0.060s
=>$ time testy >/dev/null
real	0m0.117s
user	0m0.103s
sys	0m0.010s
=>$ cat testy
#!/bin/bash

array_bin=($(cd /var/lib/pacman/local/ && /usr/bin/grep -ls 'usr/bin/' */files))

oldifs="$IFS"
IFS=$'\n'
array=($(expac '%n %d'))
IFS="$oldifs"

count=0
countarraycount=0
# the countarray will hold indices of packages that install files in /usr/bin

# 1st loop: loop through list of packages that install files in /usr/bin
while [[ "${array_bin[countarraycount]}" != "" ]]
do
	if [[ "${array_bin[countarraycount]%%/*}" =~ "${array[count]%% *}" ]]
	# if the packagename equals one from the array of ALL packages...
	then
		countarray[((countarraycount++))]=$count
		# ... then add the index to the countarray
	fi
	((count++))
done
unset array_bin

for (( count=0 ; count<countarraycount ; count++ ))
do
	echo "${array[countarray[count]]}"
done

(the output of both commands is identical)

Your approach scans through every "files" file and greps all of the files, creates a huge bash array, then loops through the bash array calling a *new* instance of expac for every single package.

no it doesn't.
please look at the updated script on github; the one from a few posts up is not it anymore.

Trilby · 2016-10-02 12:37:59

I just looked at it, that's exactly what it does. Anyhow, you may be interested in this:

expac -Q '%n\t%d\t%F' | awk -F"\t" '/ usr\/bin\/[^$]/ { printf "%s: %s\n", $1, $2; exit; }'

Adding the exit makes it 12 times faster which is something I overlooked when I posted it the first time.

EDIT: Ah, I see you don't run expac on *every* package, but you still run far more processes.

ondoho · 2016-10-02 13:02:28

Trilby wrote:

you may be interested in this:
expac -Q '%n\t%d\t%F' | awk -F"\t" '/ usr\/bin\/[^$]/ { printf "%s: %s\n", $1, $2; exit; }'
Adding the exit makes it 12 times faster which is something I overlooked when I posted it the first time.

it exits after the first result??? no wonder it's 12x faster. removing the last "exit;" obviously returns it to the previously described behavior. (GNU Awk 4.1.4)

EDIT: Ah, I see you don't run expac on *every* package, but you still run far more processes.

not sure what you mean by "processes", but my script gets by with 1 call to grep and 1 call to expac to arrive at the same result you get with 1 call to expac and 1 call to awk.
the rest, all the way until the first dmenu call, is pure bash.

i wanted to add some formatting (see screenshot), which results in some extra for/while looping.
my guess is that this happens in practically no time in bash, compared to launching external utilities.

Last edited by ondoho (2016-10-02 13:05:15)

ondoho · 2016-10-02 21:23:05

i am not familiar with awk at all, sorry.
if you are correct, i would very much like to use something even faster for my script?

ondoho · 2016-10-14 05:44:41

that expac|awk pipe is really neater, i can even add formatting like this:

expac -Q '%n\t%d\t%F' | awk -F"\t" '/ usr\/bin\/[^$]/ { printf "%-30s %s\n", $1, $2; }'

BUT the bottleneck isn't the processing (after the pipe with awk, or inside the script with bash, whichever way), but the expac command itself:
the first time after a reboot (?) it takes a looooong time to complete (something like 15s).
after that it is near instant (about 0.2s).

anybody have an idea what causes this difference in performance?

bstaletic · 2016-10-14 08:44:25

Maybe it's getting cached. Drop all caches with

echo 3 | sudo tee /proc/sys/vm/drop_caches

ondoho · 2016-10-22 07:35:34

^ indeed it does.

i will have to separate out the generation of a cached file then.

Trilby · 2016-10-22 11:57:12

Or just run a random expac command that queries the list of packages at startup. Even just `expac -Q ""`.

ondoho · 2016-11-02 21:47:11

every time i drop the cache, the next execution would take ages. a random expac command would not cure this.
i did have to re-introduce cache generation, added some config options... https://bbs.archlinux.org/viewtopic.php?id=217769

thanks for your input.

Arch Linux

#1 2016-05-19 06:24:28

a menu to list ALL apps (not only gui) - like debian menu?

#2 2016-05-19 16:27:44

Re: a menu to list ALL apps (not only gui) - like debian menu?

#3 2016-05-19 17:48:37

Re: a menu to list ALL apps (not only gui) - like debian menu?

#4 2016-05-21 07:32:17

Re: a menu to list ALL apps (not only gui) - like debian menu?

#5 2016-05-21 11:10:50

Re: a menu to list ALL apps (not only gui) - like debian menu?

#6 2016-10-01 12:17:06

Re: a menu to list ALL apps (not only gui) - like debian menu?

#7 2016-10-01 13:05:43

Re: a menu to list ALL apps (not only gui) - like debian menu?

#8 2016-10-01 16:10:38

Re: a menu to list ALL apps (not only gui) - like debian menu?

#9 2016-10-02 11:39:30

Re: a menu to list ALL apps (not only gui) - like debian menu?

#10 2016-10-02 11:42:15

Re: a menu to list ALL apps (not only gui) - like debian menu?

#11 2016-10-02 12:03:16

Re: a menu to list ALL apps (not only gui) - like debian menu?

#12 2016-10-02 12:37:59

Re: a menu to list ALL apps (not only gui) - like debian menu?

#13 2016-10-02 13:02:28

Re: a menu to list ALL apps (not only gui) - like debian menu?

#14 2016-10-02 21:23:05

Re: a menu to list ALL apps (not only gui) - like debian menu?

#15 2016-10-14 05:44:41

Re: a menu to list ALL apps (not only gui) - like debian menu?

#16 2016-10-14 08:44:25

Re: a menu to list ALL apps (not only gui) - like debian menu?

#17 2016-10-22 07:35:34

Re: a menu to list ALL apps (not only gui) - like debian menu?

#18 2016-10-22 11:57:12

Re: a menu to list ALL apps (not only gui) - like debian menu?

#19 2016-11-02 21:47:11

Re: a menu to list ALL apps (not only gui) - like debian menu?

Board footer