You are not logged in.

#1 2022-01-10 18:37:43

NSA_Spies_You
Member
Registered: 2022-01-10
Posts: 4

[Solved] Bash script to install packages above given popularity

This script receives a number, and it's supposed to return all the packages from https://pkgstats.archlinux.de/api/packages with a popularity above the given number. I can only get a few packages at a time from the api, so I have to do multiple calls. In every call I compare the popularity of the first package to the given popularity, and if the popularity of the package is lower then I stop getting more packages. Right now I'm getting an error in this line

if [ $(jq ".packagePopularity[0].popularity" <<<"$pkgs") -lt $popularity ]; then

I want to be able to use this script from a different one like this

pkgs=$(source get_popular_packages.sh 90)
for pkg in "${pkgs[@]}" ; do
    sudo pacman -S --noconfirm --needed $pkg
    yay -S --noconfirm -needed $pkg
done

I'm having difficulty feeding the list of packages that I get from https://pkgstats.archlinux.de/api/packages to pacman. I need some help getting the packages from the function get_popular_packages_with_popularity_above() to install_packages().

get_popular_packages.sh

#!/usr/bin/env bash

function usage() {
    cat <<EOF
Usage: get_popular_packages.sh popularity
EOF
    exit 1
}

function get_popularity() {
    if [ $# -gt 0 ]; then
        if (($1 >= 0 && $1 <= 100)); then
            echo "$1"
        else
            echo "Popularity must be between 0 and 100"
            exit 1
        fi
    else
        usage
    fi
}

function get_total_packages() {
    curl -sX "GET" -H "accept: application/json" \
        "https://pkgstats.archlinux.de/api/packages?limit=1&offset=0" \
        | jq ".total"
}

function get_packages() {
    curl -sX "GET" -H "accept: application/json" \
        "https://pkgstats.archlinux.de/api/packages?limit=$2&offset=$3" \
         | jq ".packagePopularities[]" \
         | jq "select( .popularity >= $1 )" \
         | jq ".name"
}

function get_fixed_number_of_popular_packages() {
    local res x total limit
    popularity=$(get_popularity "$@")
    total=2000
    x=0
    limit=1000
    res=()
    while [ $x -le $total ]; do
        res+=$(get_packages $popularity $limit $x)
        x=$(($x+$limit))
    done
    echo $res
}

function get_popular_packages_with_popularity_above() {
    local total limit pkgs res offset
    popularity=$(get_popularity "$@")
    echo "Get packages with a popularity greater than $popularity"
    total=$(get_total_packages)
    echo "There are $total total packages"
    offset=0
    limit=200
    res=()
    pkgs=$(get_packages $popularity $limit $offset)

    # Get 'limit' number of packages each time
    while [ $offset -le $total ]; do
        pkgs=$(get_packages $popularity $limit $offset)
        res+=$pkgs
        if [ $(jq ".packagePopularity[0].popularity" <<<"$pkgs") -lt $popularity ]; then
          break
        fi
        offset=$(($offset+$limit))
    done
    echo $res
}

function install_packages() {
    pkgs=$(get_popular_packages_with_popularity_above 90)

    for pkg in "${pkgs[@]}" ; do
        sudo pacman -S --noconfirm --needed $pkg
        yay -S --noconfirm -needed $pkg
    done
}

get_popular_packages_with_popularity_above "$@"
#install_packages

Last edited by NSA_Spies_You (2022-01-10 19:47:45)

Offline

#2 2022-01-10 18:58:56

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 26,197
Website

Re: [Solved] Bash script to install packages above given popularity

Your second code block with the loop doesn't make sense for a few reasons.  First, there's no reason at all to loop - pacman can install many packages in a single transaction, so it should just be:

pacman -S --noconfirm --needed $(source get_popular_packages.sh 90)

Second, I have no idea what yay is doing in that loop.  Why do you reinstall the package again with yay right after installing it with pacman ... especially when yay is functioning there just as a pacman wrapper anyways.

I've yet to dig into the get_popular_packages script - but I'll look at that momentarily.

EDIT: I can't really edit that script as it has a lot more complexity that I think it needs.  Wouldn't this do what you want:

#!/bin/sh

apicmd() {
	curl -sX "GET" -H "accept: application/json" \
		"https://pkgstats.archlinux.de/api/packages?limit=${1}&offset=${2}"
}

total=$(apicmd 1 0 | jq '.total')
offset=0
limit=100
popularity=90

while [ $offset -lt $total ]; do
	pkgs=$(apicmd $limit $offset | jq -r ".packagePopularities[] | select(.popularity>=$popularity) | .name")
	let offset+=limit
	[ -z "$pkgs" ] && break
	pacman -S --needed --noconfirm $pkgs
done

Note in particular, jq commands can be chained in a single invocation.  For this script, adjust "popularity" as desired, or set it from a script argument, I just used 90 in order to have a relatively manageable number of results.

Last edited by Trilby (2022-01-10 19:35:25)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2022-01-10 19:31:32

NSA_Spies_You
Member
Registered: 2022-01-10
Posts: 4

Re: [Solved] Bash script to install packages above given popularity

Trilby wrote:

I have no idea what yay is doing in that loop.  Why do you reinstall the package again with yay right after installing it with pacman ... especially when yay is functioning there just as a pacman wrapper anyways.

I've assumed that some of the packages are from the Arch User Repository.

Trilby wrote:

First, there's no reason at all to loop - pacman can install many packages in a single transaction.

If I passed the packages to pacman and yay directly, and one package failed, I thought it would fail installing the rest. Now that you've mentioned it, I've tried it and you are right, it can handle all of them even if one fails.

Trilby wrote:

can't really edit that script as it has a lot more complexity that I think it needs.  Wouldn't this do what you want:

Yes, it does, thanks. You made it look really easy.

Last edited by NSA_Spies_You (2022-01-10 19:40:51)

Offline

#4 2022-01-10 19:33:59

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 26,197
Website

Re: [Solved] Bash script to install packages above given popularity

The AUR has it's own api, this is just for repo packages isn't it?  Besides, even if it included AUR packages too, you'd still install every repo package twice - you'd at least need an "||" in there to run yay as a backup if pacman fails.  Or just use only yay as that will install repo packages just fine itself (as it is, again, a pacman wrapper).


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#5 2022-01-10 20:05:33

progandy
Member
Registered: 2012-05-17
Posts: 4,767

Re: [Solved] Bash script to install packages above given popularity

Trilby wrote:

The AUR has it's own api, this is just for repo packages isn't it?

pkgstats does not have anything to do with the repositories, it only collects package names of installed packages submitted by users. It can contain AUR packages as well as any random package that is installed from an unknown source.

Last edited by progandy (2022-01-10 20:06:04)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

Board footer

Powered by FluxBB