AUR helper/assistent

qinohe · 2022-08-18 16:41:26

Using a homebrew shell script as AUR-helper for git-AUR that does a good job for me.
The existence of a package git dir is done using 'find'
However, I have discovered that packages can contain multiple package git directories.
As an example I choose 'noisetorch' it contains 3 package git dirs. 'c-ringbuf' 'noisetorch' & 'rnnoise'
So when updating, the git directory that should be chosen by the script is 'noisetorch'
A 'git fetch' to download refs and objects
HEAD is checked before and after the fetch.
Basically this method works for all git AUR packages, static and git, in a customrepo or not.
I have it working for 'noisetorch' too using an extra for loop, this is fragile at least.
In the long run I have a feeling this wont work for all packages like 'noisetorch' because package names are not used consistent...
An example: 'dwm-git', the package git dir is named 'dwm'
Or vim-plug-git, the package git dir is named 'vim-plug-git'
These two packages only have one package git dir and don't have the problem packages like 'noisetorch' have but just to show what I mean with 'not consistent...
Do you have some advice or a better way to check this for packages like noisetorch?
The part that is important is 'updchk' function, first for loop.

Could be there are some faults in the script...still working on it;)

#!/bin/sh

# Depends on:
# git
# jq
# vim

set -eu

# where your projects are stored
buildir="$HOME/build"

# directory for include & exclude list
# include lists can be multiple
# 'exclude' is the only exclude list !!
lists="$HOME/bin-data/repo-lists"

# AUR API URL - type=search
search_api="https://aur.archlinux.org/rpc/?v=5&type=search&arg="

# AUR API URL / local json-file - type=info
local_json="file://$HOME/bin-data/concat-lists/packages-meta-ext-v1.json"
#remote_api="https://aur.archlinux.org/rpc/?v=5&type=info&arg[]="
# local AUR json read: https://lists.archlinux.org/pipermail/aur-general/2021-November/036659.html

#        curl -s "${remote_api}${prog}" | jq -r '.results[] |
info() {
    if [ -n "$(pacman -Qqm "${prog}" 2>/dev/null)" ]; then
        pacman -Qi "${prog}"
    else
        curl -s "$local_json" | jq --arg prog "${prog}" '.[] | select(.Name==$prog) | . |
            "Name            : " + .Name,
            "Version         : " + .Version,
            "Description     : " + .Description,
            "URL             : " + .URL,
            "License         : " + .License[]?,
            "Provides        : " + .Provides[]?,
            "Depends         : " + .Depends[]?,
            "OptDepends      : " + .OptDepends[]?,
            "MakeDepends     : " + .MakeDepends[]?,
            "Conflicts       : " + .Conflicts[]?,
            "Replaces        : " + .Replaces[]?,
            "Maintainer      : " + .Maintainer,
            "ID              : " + (.ID|tostring),
            "FirstSubmitted  : " + (.FirstSubmitted|tostring),
            "LastModified    : " + (.LastModified|tostring),
            "Popularity      : " + (.Popularity|tostring),
            "OutOfDate       : " + (.OutOfDate|tostring),
            "URLPath         : " + .URLPath' | sed 's/"//g'
        printf '%s\n' ""
    fi && exit
}

search() {
    aurfile=$(curl -s "${search_api}${prog}" | jq '.results[] | .Name' | sed 's/"//g')

    [ -z "${aurfile}" ] && printf '%s\n' "${prog} not found" && exit

    while :; do
        printf '%s\n' "${aurfile}" "" "'Enter' to exit:" "to clone one of the results, enter one of the above package name/s"

        read -r prog
        case ${prog-} in
            '') exit;;
             *) clone; return;;
        esac
    done
}

clone() {
    if [ -d "${buildir}"/"${prog}" ]; then
        printf '%s\n' "no reason to clone ${prog}, dir excists" && return
    fi

    cd -- "${buildir}" || return

    git clone https://aur.archlinux.org/"${prog}"

    return
}

bldinst() {
    [ ! -d "${buildir}"/"${prog}" ] && printf '%s\n' "${buildir}/${prog} does not excist" && return

    [ "${edit}" = 'on' ] && vim -d "${buildir}"/"${prog}"

    cd -- "${buildir}"/"${prog}" || return

    for zst in *.zst; do
        if [ -e "${zst}" ]; then
            rename -- .zst .zst.bak *.zst
            break
        fi
    done

    if [ "${noconfrm}" = 'on' ] && [ "${customrepo}" = 'off' ]; then
        makepkg --noconfirm -sic
    elif [ "${noconfrm}" = 'off' ] && [ "${customrepo}" = 'off' ]; then
         makepkg -sic
    elif [ "${customrepo}" = 'on' ] && [ "${noconfrm}" = 'on' ]; then
         makepkg --noconfirm -sc
    elif [ "${customrepo}" = 'on' ] && [ "${noconfrm}" = 'off' ]; then
        makepkg -sc
    fi
}

updchk() {
    if [ ! -d "${buildir}"/"${prog}"/.git ] && [ "${input}" != 'on' ]; then
        return
    fi

    if [ -d "${buildir}"/"${prog}" ]; then
        cd -- "${buildir}"/"${prog}" || return
        gitdir=$(find . -maxdepth 1 ! -iname ".*" -type d | awk -F'/' '{print $NF}')
    fi

    for  p in $gitdir; do
        if [ "${prog}" = "${p}" ]; then
            gitdir="${prog}"
            break
        fi
    done

    if [ -d "${gitdir}" ]; then
        cd -- "${buildir}"/"${prog}"/"${gitdir}" || return
    fi

    if [ -f "${lists}"/exclude ] && [ "${input}" = 'off' ] ; then
        excl=$(< "${lists}/exclude" awk -F'/' '{print $NF}') || return
        for p in $excl; do
            [ "${p}" = "${prog}" ] && return
        done
    fi

    locgitcom=$(git rev-parse --short HEAD)
    git fetch
    remgitcom=$(git rev-parse --short HEAD)

    [ "${message}" = 'on' ] && printf '%s\n' "update check for '${prog}':"

    if [ "${locgitcom}" != "${remgitcom}" ] && [ "${message}" = 'on' ]; then
        printf '%s\n' "installed ${locgitcom}" "remote    ${remgitcom}" \
        "commits do not match, rebuild/update ${prog}" ""
    elif [ "${locgitcom}" = "${remgitcom}" ]; then
        [ "${message}" = 'on' ] &&  printf '%s\n' "installed ${locgitcom}" "remote    ${remgitcom} - commits match" ""
        return
    fi

    bldinst
}

chkout() {
    if printf '%s' "${prog}" | grep -q -e '^[-]$'; then
        printf '%s\n' "wrong or incomplete option: '${prog}'  (see) '${app_this} -h'" && exit 1
    fi

    [ "${clone}" = 'on' ] && clone
    [ "${bldinst}" = 'on' ] && bldinst
    [ "${info}" = 'on' ] && info
    [ "${search}" = 'on' ] && search

    if [ "${input}" = 'on' ]; then
        progchk="${prog}"
    elif [ "${list}" = 'on' ]; then
        progchk=$(cat "${lists}"/"${prog}")
    elif [ "${customrepo}" = 'on' ]; then
        progchk=$(pacman -Slq "${prog}")
    elif [ "${updchk}" = 'on' ]; then
        progchk=$(pacman -Qqm)
    else
        return
    fi

    for prog in ${progchk} ; do
        updchk
    done
}

verbose() {
    set -vex
}

usage() {
cat <<USAGE

${app_this}:
      Search, clone, get info, edit & update AUR packages

Options:
      the order of used options and arguments is important
      -b  <pkgname>  - build/rebuild & install a single package in '$buildir/*'
      -c  <pkgname>  - clone a package from AUR
      -i  <pkgname>  - get info about an AUR package
      -k  <reponame> - update packages in a customrepo - no install!!
      -l  <listname> - update/install packages from a list
      -s  <pkgname>  - search and optionally clone a result
      -u  <pkgname>  - update/rebuild single local & AUR packages"
      -U  <>         - mass update AUR packages not in a list or customrepo

      combi = only in combination with above options
      possible options - see end of each line
      -e  combi      - edit files before makepkg  -  '-ue' 'be' '-Ue'
      -m  combi      - get info while updating (default=off) - '-Um' '-Uml'
      -n  combi      - turn off makepkg 'noconfirm' -  '-un' or '-in'
      -v  combi      - verbose, if needed - combi all other options

Examples:
      $ ${app_this} -Uem             -  edit files before updating & get messages
      $ ${app_this} -l   listname    -  update from a predefined list
      $ ${app_this} -ek  customrepo  -  update a customrepo & edit files before building
      $ ${app_this} -u   pkgname     -  update one package
      $ ${app_this} -sei pkgname     -  search clone edit & build/install a package

Lists:
      the exclude file should always be named 'exclude', it should not be an emty list.
      exclude:  list of packages that should always! be excluded from mass updating '-U' '-l' -k'
      include-lists:  file with list of packages to update (together) - one list per run! '-Ul listname'

Tip:
      Uses local json file or remote(AUR) 'API-info'
      this can changed swapping currently commented lines

USAGE
}

# no need to change these variables
app_this="${0##*/}"
bldinst=''
clone=''
customrepo='off'
edit=''
info=''
input='off'
list=''
locgitcom=''
message=''
noconfrm='on'
prog=''
remgitcom=''
search=''
updchk=''

for prog; do true; done
if [ -n "${prog}" ]; then
    while getopts 'bcehiklimnrsuUv' opt; do
        case "${opt}" in
            b) bldinst='on';;
            c) clone='on';;
            e) edit='on';;
            h) usage; exit;;
            i) info='on';;
            k) customrepo='on' && updchk='on';;
            l) list='on';;
            m) message='on';;
            n) noconfrm='off';;
            s) search='on';;
            u) input='on'; message='on'; updchk='on';;
            U) updchk='on';;
            v) verbose;;
            *) printf '%s\n' "Wrong option, see '${app_this} -h' for help" && exit 1;;
        esac
    done
else
    usage
    printf '%s\n' "Note:" "      No arguments at all were passed to ${app_this}, see above help for examples" ""
    exit 1
fi

chkout

Trilby · 2022-08-18 16:45:46

I haven't reviewed the script yet (I will try to do so shortly) but you seem to be conflating the AUR git repos (for which there is one and only one for every package) and the upstream source material which is sometimes on git, sometimes a tarball, sometimes something else. There's no reason for you to ever manually clone / fetch / pull the upstream source material: leave this to makepkg.

EDIT: okay, I looked it over - and I gather you are referring to the updchk function - but I just dont understand what the goals are there. What is that function supposed to do? It doesn't seem to check for updates as it ignores any non -git package (or really packages without a .git directory). So "updchk" will do nothing with most packages even if they have updates - including all other VCSs.

SIDENOTE: you should use jq's -r flag rather than just using sed to indescriminately remove all quotes after the fact.

Last edited by Trilby (2022-08-18 16:53:41)

qinohe · 2022-08-18 17:12:01

Thanks for your preview Trilby.
Yes, the tool is (for now) only meant for git, I will add others like VCS along the way...
BTW. all my AUR packages are using git?!, git AUR.

Checks are done in the root dir if the package is static and done in the package git dir(one level deeper) if the package is a git repo, hope this makes sense.
It simply;) uses 'git fetch' to 'update' the tree.
It has done a good job for me until I discovered the multiple package git dirs in 'noisetorch'....
I'll check the '-r' flag for jq!
Thanks!

edit: that I missed the -r flag .. works nice thanks;)
If you look at the commented curl line, I did use it there, oh well..

Last edited by qinohe (2022-08-18 17:26:14)

Trilby · 2022-08-18 17:44:27

qinohe wrote:

Checks are done in the root dir if the package is static and done in the package git dir(one level deeper) if the package is a git repo, hope this makes sense.

It makes sense in that I know what you are saying. But it doesn't make sense to do. Just run makepkg in the VCS dirs - then it will work for *all* VCS packages regardless of whether they are git/hg/svn/fossil/whatever. Why jump through all the hoops to try to rewrite what makepkg already does for you ... especially if you end up with a limited implementation?

qinohe · 2022-08-18 17:50:46

Simple, your right..this is what came up first and I kept working that way
It will take some time to reorder things.
BTW. what do you think of the rest of the script, is it okay?

Trilby · 2022-08-18 18:11:23

I didn't look through much of the rest - but it seems quite long (i.e., complicated) for the tasks it actually does. I'm not sure what local_json is for as it is only used once, but is never (re)set ... how is that updated? And why? It looks like you may periodically update the local_json file which is a info list for lots of packages. I don't know why you'd do this: how do you time the udpates to the local file? If they're too fast, they're wasteful, or too slow and you have outdated info. Just skip all that redirection and retrieve the info just for the package in question direct from the AUR in the info function.

I'm not sure what these lines are for at the main entry point of the script:

prog=''
for prog; do true; done
if [ -n "${prog}" ]; then
    # ... actual relevant getopts code here
else
   # error code here that will never run
fi

I can only suspect that the variable 'prog' might be intended to be set in some user configuration file - but that still doesn't add up. If a future revision of the script includes a user config file that might redefine 'prog' your current code will result in it bypassing getopts all-togther. I imagine you'd want to respond to command line flags even if there is a user config setting 'prog'. Or perhaps I'm just completely missing the point of that variable and conditional test.

On a partly stylistic level, I consider the format strings used for tools like 'jq' to be "data" rather than "code" and like to have them apart from the script / program logic as this makes it easier to read and understand what the code is doing. While reading through such code to understand the workings, the details of the output formating isn't important, it's a given that the jq string defines the output format, that's all that one needs to know at that point. E.g., in my aur script I have something like the following:

#early in the script in a purely "data" region setting variables:
info_fmt='.results[]|'\
# long multiline string here with spacing, alignment, and asci-codes for color, etc

#later in the actual "workings" of the script where the arg array has been loaded with the relevant package names:
-i|--info)
   curl -s "${url}type=info$(printf '&arg[]=%s' "$@")" \
      | jq -r "$info_fmt" \
      | column -tLl 2

This is much easier to read than having the long complicated multiline string in place of info_fmt there. See the Rule of Representation. My script is 17 lines of code, and 22 lines of "data" - it searches, pretty-prints info queries, checks for updates, and downloads/clones AUR packages. However I don't have options in it for some of the local repo options that yours does.

Last edited by Trilby (2022-08-18 18:19:00)

qinohe · 2022-08-18 21:52:07

Well, that local_json file is a download indeed, mostly a new one is not needed for you only need it for 'info' like dependencies optdepends e.d. If you need actual info of the installed package 'pacman -Qi' would do the job, but you may be right I could just 'ask' AUR API for that info...
I read the mail about this info and decided at that moment; it may be enough to use a local file instead of making an API requests 'every time' it's this mail:
https://lists.archlinux.org/pipermail/a … 36659.html

If you look at the commented lines above the info function you see you can swap 'local_json' for 'remote_api' and swap the commented curl lines in the info function.
Sorry the script is without comments ATM. It was actually not ready to publish I just needed some info I asked in OP.
I got more from you than I could bargain for..and am very happy with it even if that means I need to change a lot of it

Trilby wrote:

I'm not sure what these lines are for at the main entry point of the script:

That's actually quite simple, I have 'set -eu' checks for non zero exit and unset variables.
If you remove "prog=''" you will have an unset variable...
'for prog; do true; done' only takes the last argument 'prog' and is used throughout the script.
You could also use 'OPTARG' but the whole getops and script must be changed if I did.
I have other scripts where using 'OPTARG' is more convenient than in this one.
It can be one program or a customrepo or a list but is also used for search, info and clone and is a name inputted by the user.
if [ -n "${prog}" ]; then (etc) this checks if the user gave an argument and print a line if none was found, shows help and exit.

The 'jq' part, I'm no master in it and 'copied' an example you gave and made it what it is now, the relevant post is:
https://bbs.archlinux.org/viewtopic.php … 3#p2024393
This was my starting point and I worked it out to what it is now;)

If I have a spare hour I will read 'The Art of Unix Programming' I'll get back on this!

Trilby · 2022-08-18 23:24:19

qinohe wrote:

I read the mail about this info and decided at that moment; it may be enough to use a local file instead of making an API requests 'every time'

Downloading tens of thousands of package's data once is not likely better than occasionally downloading one packages data.

qinohe wrote:

you can swap 'local_json' for 'remote_api' and swap the commented curl lines in the info function.

No, that'd be very bad as written as the url gets all info, then you use jq to filter for the relevant package. You should just request info for the relevant package(s).

qinohe wrote:

IIf you remove "prog=''" you will have an unset variable...

Only because you have a loop ... that does nothing.

qinohe wrote:

'for prog; do true; done' only takes the last argument 'prog' and is used throughout the script.

Oh ... okay. So this is just a way of doing:

eval prog='$'$#

However, I'd also question the assumption / requirement that "prog" must be the last argument (and that there must be only one!). It also makes a help flag (-h) impossible as you skip parsing flags if no "prog" is set. All in all, it looks like you should use this:

while getopts '...' opt; do
   case "$opt" in
      b) ...
      ...
   esac
done
shift $OPTIND
prog=$1

This would also open up the possiblility of having more than one 'prog' acted on at the same time (by changing $1 to $@).

EDIT: sidenote, I didn't mention anything about sparcity of code comments as comments suck. Adding comments to code is like adding sugar to coffee. The more you need, the worse the coffee itself must be. Strive to write black coffee code. The intent and purpose of perfect code would be self-evident and need no comments. But as perfect code does not exist, some degree of comments are necessary - but only to the degree our code is falling short. In a vast majority of cases, spending time improving the code would go much further in providing clarity than spending time adding comments.

Last edited by Trilby (2022-08-18 23:57:35)

qinohe · 2022-08-19 01:27:49

Trilby wrote:

Downloading tens of thousands of package's data once is not likely better than occasionally downloading one packages data.

I'm still using my first list downloaded april-9th, but I tend to agree to your statement!

No, that'd be very bad as written as the url gets all info, then you use jq to filter for the relevant package. You should just request info for the relevant package(s).

Then this is not completely clear to me because the data curled should only be a few 100 bytes, I will try your example !
Just for clarity, this code is bad??

prog=some-app
remote_api="https://aur.archlinux.org/rpc/?v=5&type=info&arg[]="
curl -s "${remote_api}${prog}" | jq -r '.results[] |
            "Name            : " + .Name,
            "Version         : " + .Version,
            etc. etc.

Only because you have a loop ... that does nothing.
Oh ... okay. So this is just a way of doing:
eval prog='$'$#
However, I'd also question the assumption / requirement that "prog" must be the last argument (and that there must be only one!). It also makes a help flag (-h) impossible as you skip parsing flags if no "prog" is set. All in all, it looks like you should use this:
while getopts '...' opt; do
   case "$opt" in
      b) ...
      ...
   esac
done
shift $OPTIND
prog=$1
This would also open up the possiblility of having more than one 'prog' acted on at the same time (by changing $1 to $@).

Of course not, so you say '-h' is not an argument! this is not true; '' or empty that's no argument.
Sure '-h' is an option ... but it is also seen as an argument to '[ -n "${prog}" ]'
BTW. your right I didn't need to set an empty prog, it became a habbit I guess, that one is removed;)
Still, I will 'later' have a shot at the way you suggest using 'shift' and '$@'..

EDIT: sidenote, I didn't mention anything about sparcity of code comments as comments suck. Adding comments to code is like adding sugar to coffee. The more you need, the worse the coffee itself must be. Strive to write black coffee code. The intent and purpose of perfect code would be self-evident and need no comments. But as perfect code does not exist, some degree of comments are necessary - but only to the degree our code is falling short. In a vast majority of cases, spending time improving the code would go much further in providing clarity than spending time adding comments.

Yeah, well this very hard to just do for a hobbyist like me, but, I will try to read and understand that 'Rule of Representation' and see if I manage to use that way in my script..no guarantees

Trilby · 2022-08-19 01:57:12

qinohe wrote:

Then this is not completely clear to me because the data curled should only be a few 100 bytes... Just for clarity, this code is bad?

Nope, that's great. Sorry I overlooked the commented line that did that as the replacement for the local data. I thought you were suggesting just dropping in the remote url you downloaded the whole bundle from in place of the local file and then using jq to effectively just "grep" for the one relevant package - that would be bad, but I had misunderstood.

qinohe wrote:

Sure '-h' is an option ... but it is also seen as an argument to '[ -n "${prog}" ]'

I stand corrected on that. But that's due to the confusion of an empty loop bing used with the intent of getting the last command line argument to set a variable intended to hold package-names / targets which ends up holding a flag.

qinohe wrote:

Yeah, well this very hard to just do for a hobbyist like me

I'm also just a hobbiest. I've had no formal training in computers (other than moving the "turtle" around in Logo on an Apple ][ back in the 80s).

qinohe · 2022-08-19 02:12:33

Thank you Trilby for walking trough this with me very much appreciated and very helpful!;)

Trilby wrote:

I'm also just a hobbiest. I've had no formal training in computers (other than moving the "turtle" around in Logo on an Apple ][ back in the 80s).

I know you're also 'just' a hobbyist but you do know more about programming than I do(fact)!
So I will read it and try to use it for the benefit of this script and hope I learn something from it
About that turtle ... I hope you abandoned that beast penguins are much more likeable and faster

Alad · 2022-08-19 10:59:13

I suggest avoiding looping over all your AUR packages with git-fetch to check for updates. This results in 2*N requests to the AUR which is inefficient (and you'll get rate limited for a sufficiently high amount of packages).

qinohe · 2022-08-19 11:16:02

Alad wrote:

I suggest avoiding looping over all your AUR packages with git-fetch to check for updates. This results in 2*N requests to the AUR which is inefficient (and you'll get rate limited for a sufficiently high amount of packages).

Hmm, I haven't thought about that, but I don't understand how you get '2*N' for one fetch per static! package?
Most of my own packages are a git repo, they don't fetch from AUR but from GitHub or Gitlab.
Far as I know you can fetch from GitHub much as you like.

To get to the point, Trilby already convinced me to use makepkg for this.
At the moment I'm short on time but soon as I have a few hours I will get to it;)

Alad · 2022-08-19 11:26:22

I mean, if you have N AUR packages, you have a total of 2*N http requests - because git-fetch does 2.

To get to the point, Trilby already convinced me to use makepkg for this.

makepkg only handles this for VCS packages, and even then if you're only interested in pkgver() updates.

Last edited by Alad (2022-08-19 11:28:01)

qinohe · 2022-08-19 11:41:42

Alad wrote:

I mean, if you have N AUR packages, you have a total of 2*N http requests - because git-fetch does 2.

Do you have a link for me where I can read this? I couldn't find it in 'man git-fetch' (or I missed it)

makepkg only handles this for VCS packages, and even then if you're only interested in pkgver() updates.

So, do you suggest Trilby is not right in #4? If so do you have a suggestion for me for a better way to do this.
And no the point was not to use pkgver(), this is a bad deal for packages that are a git repo and would only work for static pkgs.

Trilby · 2022-08-19 12:28:01

qinohe wrote:

So, do you suggest Trilby is not right in #4?

We're talking about different points. As far as I could tell, your script didn't do anything at all with non -git packages in checking for updates. At the time I reviewed it, you were drawing only from the upstream source (not AUR) to check if there were updates in -git packages, and all other packages were ignored (as I understood it, which could well be wrong).

But one single AUR RPC request is sufficient to check for updates to all non-vcs packages in one go.

qinohe wrote:

And no the point was not to use pkgver(), this is a bad deal for packages that are a git repo and would only work for static pkgs.

`makepkg -o` will get all VCS updates and rerun pkgver() and set pkgver in the PKGBUILD. So you can run `makepkg -o` followed by grabbing the pkgver variable from the PKGBUILD or running `makepkg --printsrcinfo` or `makepkg --packagelist` to get the new pkgver(s). I'd recommend using one of the latter two options as this will be much more robust for split packages and some other cases. Of course this is only relevant if you want to check for updates without building them - personally I'd not see a point in this and I'd just run `makepkg -si` in each directory. If there's nothing to do, makepkg will exit cleanly after the equivalent of the above steps, if there is something to do, it will do it (build / install).

Last edited by Trilby (2022-08-19 12:33:01)

Alad · 2022-08-19 12:28:27

Do you have a link for me where I can read this? I couldn't find it in 'man git-fetch' (or I missed it)

I don't think it's documented - it's just something that was observed in the devops channel, after discussion rate limiting.

So, do you suggest Trilby is not right in #4? If so do you have a suggestion for me for a better way to do this.

For VCS packages you can use makepkg, though at some point the PKGBUILD may change to address upstream changes (e.g. new dependencies or build options).

And no the point was not to use pkgver(), this is a bad deal for packages that are a git repo and would only work for static pkgs.

I'm confused by this statement. A "static" package is, to me, a package without pkgver function. Packages that "are" a git repo would have a pkgver() function.

edit: it seems I read the "updcheck" wrong - this indeed has no benefit over using makepkg itself to update the source git repo.

Last edited by Alad (2022-08-19 12:31:26)

qinohe · 2022-08-19 15:13:34

Trilby wrote:

We're talking about different points. As far as I could tell, your script didn't do anything at all with non -git packages in checking for updates. At the time I reviewed it, you were drawing only from the upstream source (not AUR) to check if there were updates in -git packages, and all other packages were ignored (as I understood it, which could well be wrong).

AUR static packages (example: gdmap) that are cloned ' git clone https://aur.archlinux.org/gdmap' are a git repo but the package itself is not a git repo! Another example: archiso,
' git clone https://aur.archlinux.org/archiso-git' is a git repo but the package also is a git repo, hope that makes sense. They are updated just fine the way the script is now. I'm using the script for a while now and all my packages (static or git packages) are updated just fine;) All my packages from AUR are 'git clone' I don't use another method to get packages.
Of course, I listen to advise and do understand parts need to be rewritten!

But one single AUR RPC request is sufficient to check for updates to all non-vcs packages in one go.
`makepkg -o` will get all VCS updates and rerun pkgver() and set pkgver in the PKGBUILD. So you can run `makepkg -o` followed by grabbing the pkgver variable from the PKGBUILD or running `makepkg --printsrcinfo` or `makepkg --packagelist` to get the new pkgver(s). I'd recommend using one of the latter two options as this will be much more robust for split packages and some other cases. Of course this is only relevant if you want to check for updates without building them - personally I'd not see a point in this and I'd just run `makepkg -si` in each directory. If there's nothing to do, makepkg will exit cleanly after the equivalent of the above steps, if there is something to do, it will do it (build / install).

Thank you for this example I will try to figure out all in the coming weeks ( time is not on my side ATM:( ) but I'm sure I need to rewrite significant parts! I'd say try if you dare;) put 'set -x' on top or run with '-v'(verbose) and you'll see it works ... that is if you clone everything with git..., you wont fuck-up your system as that should have happened and that didn't;)

qinohe · 2022-08-19 15:19:20

Alad wrote:

I don't think it's documented - it's just something that was observed in the devops channel, after discussion rate limiting.

Hmm, okay I can't enter devops channel so I can't read it, so be it...

For VCS packages you can use makepkg, though at some point the PKGBUILD may change to address upstream changes (e.g. new dependencies or build options).

Thanks, I need to start reading again because I know little about all these methods!

I'm confused by this statement. A "static" package is, to me, a package without pkgver function. Packages that "are" a git repo would have a pkgver() function.
edit: it seems I read the "updcheck" wrong - this indeed has no benefit over using makepkg itself to update the source git repo.

A static package like gdmap, a git package like archiso. See above example in my answer to Trilby, thanks;)

Trilby · 2022-08-19 15:53:01

qinohe wrote:

They are updated just fine the way the script is now.

Oh ... yes, I did completely misunderstand the upchk function. It is confusing as you batch together the AUR git repos that hold the PKGBUILDs and associated files together with any upstream source repos and treat them the same (in the same variable). This is very hard to follow in the code. But in any case, you are indeed fetching updates from the AUR git repo for every single package that function runs on (whether it is VCS or not). In this case, Alad's point applies and your following reply to him is incorrect:

qinohe wrote:

Most of my own packages are a git repo, they don't fetch from AUR but from GitHub or Gitlab.

Every single package, whether VCS or what you are calling 'static', will each fetch from the AUR every time the updchk function is called. Then ${pkgbase}-git packages will also fetch from the upstream git repos as well in that function (while other VCS systems are ignored). But the important point is that you are making a git fetch request of the AUR for every single package that goes through the upchk function. While this is not an AUR RPC request, it's still an AUR request.

Last edited by Trilby (2022-08-19 15:55:08)

Alad · 2022-08-19 15:54:54

Hmm, okay I can't enter devops channel so I can't read it, so be it...

https://gitlab.archlinux.org/archlinux/ … quests/611

Of course, I listen to advise and do understand parts need to be rewritten!

If anything I'd advise to refactor the code for readability. You can indeed run it with set -x, but that can hardly be the goal...

Last edited by Alad (2022-08-19 15:55:08)

qinohe · 2022-08-19 16:08:38

Trilby wrote:

In this case, Alad's point applies and your following reply to him is incorrect:
qinohe wrote:
Most of my own packages are a git repo, they don't fetch from AUR but from GitHub or Gitlab.
Every single package, whether VCS or what you are calling 'static', will each fetch from the AUR every time the updchk function is called. Then ${pkgbase}-git packages will also fetch from the upstream git repos as well in that function (while other VCS systems are ignored). But the important point is that you are making a git fetch request of the AUR for every single package that goes through the upchk function. While this is not an AUR RPC request, it's still an AUR request.

I beg to differ, Let me give an example of the git config of archiso

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
[remote "origin"]
        url = https://gitlab.archlinux.org/archlinux/archiso.git
        fetch = +refs/*:refs/*
        mirror = true

and the gdmap git config

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://aur.archlinux.org/gdmap
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

I cd into one of both above examples dirs, the first one fetches from Gitlab the second one(gdmap) fetches from AUR, correct me if I'm wrong. And if I'm wrong why, the config clearly shows, where it fetches from!?

qinohe · 2022-08-19 16:13:57

Alad wrote:

https://gitlab.archlinux.org/archlinux/ … quests/611

Thanks, I didn't think I could..;)

If anything I'd advise to refactor the code for readability. You can indeed run it with set -x, but that can hardly be the goal...

Sure, I just mentioned this so you can see it working, of course I don't put this in the script, I have '-v'(verbose) as an option in getops.
If you need it? it's there and shows everything from the point getops runs!

Trilby · 2022-08-19 16:23:08

I have no idea what those examples of git configs are supposed to illustrate. Archiso is not even an aur package. Gdmap is an aur package, but not a VCS package - it's what you've been referring to as a "static" package, right. And that "static" package shows that it fetches from aur.archlinux.org to get the PKGBUILD and associated files. This is also true of VCS packages (e.g., those with names ending in -git, or -hg, or -fossil, etc). ALL aur packages fetch from aur.archlinuxorg. ALL of them.

EDIT: if you are referring to archiso-git, that is an AUR package, and it will have multiple .git folders: one at the top level that fetches from aur.archlinux.org (as does every single other aur package), then it will also have upstream source repos that fetch from elsewhere. Showing a config for one of those other repos demonstrates nothing. Here it is:

$ git clone https://aur.archlinux.org/archiso-git.git
$ cd archiso-git
$ cat .git/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = https://aur.archlinux.org/archiso-git.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master

Last edited by Trilby (2022-08-19 16:27:59)

qinohe · 2022-08-19 16:36:07

Trilby wrote:

I have no idea what those examples of git configs are supposed to illustrate. Archiso is not even an aur package.

My mistake, I meant archiso-git;)

Gdmap is an aur package, but not a VCS package - it's what you've been referring to as a "static" package, right.

Correct, this is what meant as static package!

And that "static" package shows that it fetches from aur.archlinux.org to get the PKGBUILD and associated files. This is also true of VCS packages (e.g., those with names ending in -git, or -hg, or -fossil, etc). ALL aur packages fetch from aur.archlinuxorg. ALL of them.

Correct, gdmap fetches from AUR, but, archiso-git doesn't it shouldn't touch AUR but Gitlab!
I don't completely understand why you say 'ALL aur packages fetch from aur.archlinuxorg. ALL of them' while fetching is a git process and I decent into the git dir so should fetch from the git repo and in the case of archiso-git that's Gitlab ... not AUR. Remember we're using 'git fetch'! not makepkg...
If I'm really wrong, and I'm still not convinced then I don't understand where my thinking goes awry

edit: about your edit, I don't touch the root dir. I decent into the package git dir! (which is named 'archiso')

Last edited by qinohe (2022-08-19 16:42:59)

Arch Linux

#1 2022-08-18 16:41:26

AUR helper/assistent

#2 2022-08-18 16:45:46

Re: AUR helper/assistent

#3 2022-08-18 17:12:01

Re: AUR helper/assistent

#4 2022-08-18 17:44:27

Re: AUR helper/assistent

#5 2022-08-18 17:50:46

Re: AUR helper/assistent

#6 2022-08-18 18:11:23

Re: AUR helper/assistent

#7 2022-08-18 21:52:07

Re: AUR helper/assistent

#8 2022-08-18 23:24:19

Re: AUR helper/assistent

#9 2022-08-19 01:27:49

Re: AUR helper/assistent

#10 2022-08-19 01:57:12

Re: AUR helper/assistent

#11 2022-08-19 02:12:33

Re: AUR helper/assistent

#12 2022-08-19 10:59:13

Re: AUR helper/assistent

#13 2022-08-19 11:16:02

Re: AUR helper/assistent

#14 2022-08-19 11:26:22

Re: AUR helper/assistent

#15 2022-08-19 11:41:42

Re: AUR helper/assistent

#16 2022-08-19 12:28:01

Re: AUR helper/assistent

#17 2022-08-19 12:28:27

Re: AUR helper/assistent

#18 2022-08-19 15:13:34

Re: AUR helper/assistent

#19 2022-08-19 15:19:20

Re: AUR helper/assistent

#20 2022-08-19 15:53:01

Re: AUR helper/assistent

#21 2022-08-19 15:54:54

Re: AUR helper/assistent

#22 2022-08-19 16:08:38

Re: AUR helper/assistent

#23 2022-08-19 16:13:57

Re: AUR helper/assistent

#24 2022-08-19 16:23:08

Re: AUR helper/assistent

#25 2022-08-19 16:36:07

Re: AUR helper/assistent

Board footer