[Solved] Bash script to install packages above given popularity

NSA_Spies_You · 2022-01-10 18:37:43

This script receives a number, and it's supposed to return all the packages from https://pkgstats.archlinux.de/api/packages with a popularity above the given number. I can only get a few packages at a time from the api, so I have to do multiple calls. In every call I compare the popularity of the first package to the given popularity, and if the popularity of the package is lower then I stop getting more packages. Right now I'm getting an error in this line

if [ $(jq ".packagePopularity[0].popularity" <<<"$pkgs") -lt $popularity ]; then

I want to be able to use this script from a different one like this

pkgs=$(source get_popular_packages.sh 90)
for pkg in "${pkgs[@]}" ; do
    sudo pacman -S --noconfirm --needed $pkg
    yay -S --noconfirm -needed $pkg
done

I'm having difficulty feeding the list of packages that I get from https://pkgstats.archlinux.de/api/packages to pacman. I need some help getting the packages from the function get_popular_packages_with_popularity_above() to install_packages().

get_popular_packages.sh

#!/usr/bin/env bash

function usage() {
    cat <<EOF
Usage: get_popular_packages.sh popularity
EOF
    exit 1
}

function get_popularity() {
    if [ $# -gt 0 ]; then
        if (($1 >= 0 && $1 <= 100)); then
            echo "$1"
        else
            echo "Popularity must be between 0 and 100"
            exit 1
        fi
    else
        usage
    fi
}

function get_total_packages() {
    curl -sX "GET" -H "accept: application/json" \
        "https://pkgstats.archlinux.de/api/packages?limit=1&offset=0" \
        | jq ".total"
}

function get_packages() {
    curl -sX "GET" -H "accept: application/json" \
        "https://pkgstats.archlinux.de/api/packages?limit=$2&offset=$3" \
         | jq ".packagePopularities[]" \
         | jq "select( .popularity >= $1 )" \
         | jq ".name"
}

function get_fixed_number_of_popular_packages() {
    local res x total limit
    popularity=$(get_popularity "$@")
    total=2000
    x=0
    limit=1000
    res=()
    while [ $x -le $total ]; do
        res+=$(get_packages $popularity $limit $x)
        x=$(($x+$limit))
    done
    echo $res
}

function get_popular_packages_with_popularity_above() {
    local total limit pkgs res offset
    popularity=$(get_popularity "$@")
    echo "Get packages with a popularity greater than $popularity"
    total=$(get_total_packages)
    echo "There are $total total packages"
    offset=0
    limit=200
    res=()
    pkgs=$(get_packages $popularity $limit $offset)

    # Get 'limit' number of packages each time
    while [ $offset -le $total ]; do
        pkgs=$(get_packages $popularity $limit $offset)
        res+=$pkgs
        if [ $(jq ".packagePopularity[0].popularity" <<<"$pkgs") -lt $popularity ]; then
          break
        fi
        offset=$(($offset+$limit))
    done
    echo $res
}

function install_packages() {
    pkgs=$(get_popular_packages_with_popularity_above 90)

    for pkg in "${pkgs[@]}" ; do
        sudo pacman -S --noconfirm --needed $pkg
        yay -S --noconfirm -needed $pkg
    done
}

get_popular_packages_with_popularity_above "$@"
#install_packages

Last edited by NSA_Spies_You (2022-01-10 19:47:45)

Trilby · 2022-01-10 18:58:56

Your second code block with the loop doesn't make sense for a few reasons. First, there's no reason at all to loop - pacman can install many packages in a single transaction, so it should just be:

pacman -S --noconfirm --needed $(source get_popular_packages.sh 90)

Second, I have no idea what yay is doing in that loop. Why do you reinstall the package again with yay right after installing it with pacman ... especially when yay is functioning there just as a pacman wrapper anyways.

I've yet to dig into the get_popular_packages script - but I'll look at that momentarily.

EDIT: I can't really edit that script as it has a lot more complexity that I think it needs. Wouldn't this do what you want:

#!/bin/sh

apicmd() {
	curl -sX "GET" -H "accept: application/json" \
		"https://pkgstats.archlinux.de/api/packages?limit=${1}&offset=${2}"
}

total=$(apicmd 1 0 | jq '.total')
offset=0
limit=100
popularity=90

while [ $offset -lt $total ]; do
	pkgs=$(apicmd $limit $offset | jq -r ".packagePopularities[] | select(.popularity>=$popularity) | .name")
	let offset+=limit
	[ -z "$pkgs" ] && break
	pacman -S --needed --noconfirm $pkgs
done

Note in particular, jq commands can be chained in a single invocation. For this script, adjust "popularity" as desired, or set it from a script argument, I just used 90 in order to have a relatively manageable number of results.

Last edited by Trilby (2022-01-10 19:35:25)

NSA_Spies_You · 2022-01-10 19:31:32

Trilby wrote:

I have no idea what yay is doing in that loop. Why do you reinstall the package again with yay right after installing it with pacman ... especially when yay is functioning there just as a pacman wrapper anyways.

I've assumed that some of the packages are from the Arch User Repository.

Trilby wrote:

First, there's no reason at all to loop - pacman can install many packages in a single transaction.

If I passed the packages to pacman and yay directly, and one package failed, I thought it would fail installing the rest. Now that you've mentioned it, I've tried it and you are right, it can handle all of them even if one fails.

Trilby wrote:

can't really edit that script as it has a lot more complexity that I think it needs. Wouldn't this do what you want:

Yes, it does, thanks. You made it look really easy.

Last edited by NSA_Spies_You (2022-01-10 19:40:51)

Trilby · 2022-01-10 19:33:59

The AUR has it's own api, this is just for repo packages isn't it? Besides, even if it included AUR packages too, you'd still install every repo package twice - you'd at least need an "||" in there to run yay as a backup if pacman fails. Or just use only yay as that will install repo packages just fine itself (as it is, again, a pacman wrapper).

progandy · 2022-01-10 20:05:33

Trilby wrote:

The AUR has it's own api, this is just for repo packages isn't it?

pkgstats does not have anything to do with the repositories, it only collects package names of installed packages submitted by users. It can contain AUR packages as well as any random package that is installed from an unknown source.

Last edited by progandy (2022-01-10 20:06:04)

Arch Linux

#1 2022-01-10 18:37:43

[Solved] Bash script to install packages above given popularity

#2 2022-01-10 18:58:56

Re: [Solved] Bash script to install packages above given popularity

#3 2022-01-10 19:31:32

Re: [Solved] Bash script to install packages above given popularity

#4 2022-01-10 19:33:59

Re: [Solved] Bash script to install packages above given popularity

#5 2022-01-10 20:05:33

Re: [Solved] Bash script to install packages above given popularity

Board footer