[solved] gitlab, no anonymous access?

Grogan · 2023-05-22 19:01:10

I build my own packages and I'm trying to clone the packages repo. I'd been doing this for a long time using the svntogit repo, and updating my packages and community trees to have the latest PKGBUILDs.

[grogan@getstuffed:arch_sources_new]$ git clone https://gitlab.archlinux.org/archlinux/packaging/packages.git
Cloning into 'packages'...
Username for 'https://gitlab.archlinux.org': 
Password for 'https://gitlab.archlinux.org': 
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.archlinux.org/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.archlinux.org/archlinux/packaging/packages.git/'

I tried my gitlab login (I only have an account for reporting issues) but that's not it.

Apparently I need a personal access token with read_repository permission now?

I'm not looking for any kind of commit access, I just want to be able to work with my own local copy of the trees.

What should I do?

Last edited by Grogan (2023-05-26 01:51:28)

loqs · 2023-05-22 20:13:25

Each package is now in its own repository so for each package you need to do

$ git clone https://gitlab.archlinux.org/archlinux/packaging/packages/$pkgname.git

or to get all packages using pkgctl

$ pkgctl repo clone --unprivileged --universe

Grogan · 2023-05-22 22:00:41

Thanks, that's actually better to be able to clone individual PKGBUILDs.

However, for now, I want the whole PKGBUILD trees so I installed the arch devtools and using pkgctrl to get them now.

Grogan · 2023-05-22 23:01:23

No, I still don't have access permissions for this. It seems this uses ssh. I figured out why it was stalling (on the first directory) I noticed in my process list there were a bunch of ssh processes.

So I did a manual ssh command to gitlab.archlinux.org just so I could say 'yes' to accepting the key, and got further, but it's just rolling through the directories:

[grogan@getstuffed arch_sources_new]$ pkgctl repo clone --unprivileged --universe
==> Query all released packages...........................................................done
0% 0:11486=0s a2ps                                                                                                                                                                                                 ==> Cloning 0ad-data ...
Cloning into '0ad-data'...
git@gitlab.archlinux.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
==> ERROR: failed to clone 0ad-data
0% 1:11485=0s a52dec                                                                                                                                                                                               ==> Cloning 0ad ...
Cloning into '0ad'...
git@gitlab.archlinux.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
==> ERROR: failed to clone 0ad
0% 2:11484=0s aalib                                                                                                                                                                                                ==> Cloning 389-ds-base ...
Cloning into '389-ds-base'...
git@gitlab.archlinux.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

... and so on. I need an ssh key

P.S. I should say that I can clone individual packages, but that's just using https

Last edited by Grogan (2023-05-23 00:16:34)

loqs · 2023-05-23 01:47:46

Have you configured PACKAGER in makepkg.conf? I ask because of https://gitlab.archlinux.org/archlinux/ … issues/132

Locally I get

$ pkgctl repo clone --unprivileged --universe
==> Query all released packages...........................................................done
0% 0:11486=0s acorn

Where it then hangs (rate limit reached?)

Grogan · 2023-05-23 03:35:24

I did try dropping in a default arch makepkg.conf for testing and it still tried to use SSH vs. read only https.

I just looked and I only have a commented out

#PACKAGER="John Doe <john@doe.com>"

in my /etc/makepkg.conf as does the arch default file.

The reason I got past that hang was, because I accepted their SSH key so my client would communicate. But then the result is permission denied because I don't have the credential. That doesn't actually matter, I really shouldn't have that, I shouldn't be running into SSH here in the first place.

New reports since you posted that issue link for me though. It isn't just me. According to the man page for pkgctl-repo-clone this also runs "configure" (pkgctrl repo configure) which does NOT recognize the --unprivileged option, according to a recent post in that issue.

Grogan · 2023-05-23 04:08:30

I thank you for pointing me to that last piece of the puzzle. It leads to understanding that unless/until they re-implement --unprivileged, this method is simply not viable for getting off the ground without SSH access.

While I would want to get back to being able to having a local tree that can be updated, for my uses I suppose it doesn't matter that much. I was already copying PKGBUILD directories into my own build dirs, so it simply moves the git operations to right before building (as needed) now that they can be cloned individually. It's just a few kb, so should take but a second.

loqs · 2023-05-23 10:53:39

You might want to add a comment to https://gitlab.archlinux.org/archlinux/ … issues/132 about the current behavior of `pkgctl repo clone --unprivileged --universe`

Grogan · 2023-05-23 20:49:21

Thanks, I should, especially in light of the last comment that showed up there, but I've got some registration hoops to sort. Linking with my github account wants a mobile phone scan of a QR code, and trying to register the normal way says my email address is already in use. I'm registered at gitlab.freedesktop.org for the Mesa project (because I needed to report an issue), that's the only way that gitlab would even know about my email address. I'll have to create another I guess.

P.S. Actually the registration process still asks for a QR code scan after I gave it a new email address. I don't have anything that can do that at this time. So I click "unable to scan" and my request timed out and it made me register again, and NOW, the new email address I created is "in use". Argh.

Last edited by Grogan (2023-05-23 21:40:12)

seth · 2023-05-23 21:13:24

https://wiki.archlinux.org/title/Initia … entication
~/bin/qrdecode

#!/bin/sh
ogg123 $HOME/.local/share/sounds/myscheme/camera_focus.ogg &
rm '/tmp/.qrdecode.png'
scrot --line style=dash,width=1,color='#1793d0' -s -q85 '/tmp/.qrdecode.png'
ogg123 -q $HOME/.local/share/sounds/myscheme/camera.ogg &
qr_content="$(zbarimg -q /tmp/.qrdecode.png)"
printf "$qr_content" | xsel -ib
dunstify -I /tmp/.qrdecode.png -u low -a qrdecode "QR decoded" "$qr_content"

loqs · 2023-05-23 21:22:51

https://gitlab.archlinux.org/archlinux/ … quests/160 improves the situation for me as the *magic* detects I am not a maintainer although even with -j 1 --universe triggers rate limiting 429 responses. I imagine for you Grogan it would have no impact leaving things still broken.

Grogan · 2023-05-23 21:42:42

loqs wrote:

https://gitlab.archlinux.org/archlinux/ … quests/160 improves the situation for me as the *magic* detects I am not a maintainer although even with -j 1 --universe triggers rate limiting 429 responses. I imagine for you Grogan it would have no impact leaving things still broken.

Thanks, I'll patch and give it a try!

Grogan · 2023-05-23 23:54:15

I generated a patch for "160" from the web interface, rolled up a PKGBUILD from git sources using the patch, and by golly, it started to work

$ pkgctl repo clone --universe
==> Collecting packager identity from makepkg.conf
==> Query all released packages...........................................................done
0% 0:11488=0s a2ps                                                                                                                                                                                                 ==> Collecting packager identity from makepkg.conf
==> Cloning a2jmidid ...
==> Collecting packager identity from makepkg.conf
==> Configuring a2jmidid
  -> name    : undefined
  -> email   : undefined
  -> gpg-key : undefined
  -> protocol: https
Cloning into 'a2jmidid'...

and so on. However, yes, it will intermittently start failing because of rate limiting... it throws a 429 and keeps running through the directories like that. After a period of time, it will start succeeding again.

However, yes, that's still viable for me because I can just keep running through it until it's all "warning, skipping cloning..." etc. I'm on my second run through right now lol

P.S. It looks like it's banning me for longer now. Also, while I'm blocked, I can't access anything at gitlab, it's got my number heheh

So no, this is not appropriate. It's abusive too and I'm going to stop doing it. Play stupid games, win stupid prizes.

Last edited by Grogan (2023-05-24 00:00:17)

loqs · 2023-05-24 11:08:27

pkgctl repo clone --protocol=https $pkgname

Added in devtools 1:1.0.1-1 by https://gitlab.archlinux.org/archlinux/ … 78c46ae2b4 with 160 being deferred until later also improves the situation (also applies to --universe with no change in rate limits).

Grogan · 2023-05-24 18:35:17

I rerolled from current master (without 160 patch now of course) and tested. It still seems to be doing the PACKAGER check

$ pkgctl repo clone --protocol=https -j 1 --universe

... good until ...

==> Cloning ambix ...
Cloning into 'ambix'...
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 22 (delta 6), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (22/22), 6.00 KiB | 6.00 MiB/s, done.
Resolving deltas: 100% (6/6), done.
==> Collecting packager identity from makepkg.conf
  -> name    : undefined
  -> email   : undefined
  -> gpg-key : undefined
  -> protocol: https
==> Configuring ambix
==> Cloning amdvlk ...
Cloning into 'amdvlk'...
remote: Retry later
fatal: unable to access 'https://gitlab.archlinux.org/archlinux/packaging/packages/amdvlk.git/': The requested URL returned error: 429
==> ERROR: failed to clone amdvlk

Yeah, it's still not viable for --universe even with -j 1. However, it aborts on the first 429 now. (I think that's better behaviour instead of continuing to flood the rate limit)

P.S. Something like a --sleep=numsecs (delay inserted between each request) would be good here. I'm not sure 1s would be enough, but it would be a starting point. At least that way one could start it before going to bed in the morning or something.

Last edited by Grogan (2023-05-24 18:48:18)

Grogan · 2023-05-25 01:35:23

I stuck some sleep in the loop there to test,

/usr/share/devtools/lib/repo/clone.sh

for pkgbase in "${pkgbases[@]}"; do
                if [[ ! -d ${pkgbase} ]]; then
                        sleep 5
                        msg "Cloning ${pkgbase} ..."
                        project_path=$(gitlab_project_name_to_path "${pkgbase}")
                        remote_url="${GIT_REPO_BASE_URL}/${project_path}.git"
                        if ! git clone --origin origin "${remote_url}" "${pkgbase}"; then
                                die 'failed to clone %s' "${pkgbase}"
                        fi
                else
                        warning "Skip cloning ${pkgbase}: Directory exists"
                fi

                pkgctl_repo_configure "${CONFIGURE_OPTIONS[@]}" "${pkgbase}"

                if [[ -n "${VERSION}" ]]; then
                        pkgctl_repo_switch "${VERSION}" "${pkgbase}"
                fi
        done
}

So far I think 2 seconds is enough (it's been going for a while now), but I'm going to abort this in a few minutes and start it before bed in the morning and we'll see then if the whole thing completes.

P.S. No! 2 seconds of sleep is not enough. Not so much because of the rate of requests, but it seems that it's a certain number of requests in a certain unit of time. I started over and 5 seconds got me much further past that point, so it's probably enough that it expires.

Last edited by Grogan (2023-05-25 02:06:20)

laichiaheng · 2023-05-25 02:22:55

loqs wrote:

pkgctl repo clone --protocol=https $pkgname
Added in devtools 1:1.0.1-1 by https://gitlab.archlinux.org/archlinux/ … 78c46ae2b4 with 160 being deferred until later also improves the situation (also applies to --universe with no change in rate limits).

After adding --protocol=https, it works.

sudo pkgctl repo clone --protocol=https linux

Grogan · 2023-05-25 16:00:00

pkgctl repo clone --protocol=https -j 1 --universe

I umm, kind of mis-anticipated the math here. Tens of thousands of clone operations, sequentially, with 5 seconds of delay between them.

It's still going, more than 14 hours later. It's on python-g* now. It's going to complete though and I'll have the whole tree. Even if it fails and aborts, where I stuck the sleep in the loop, it should just run through and skip all the existing directories quickly when run again (the "else"). I'd just delete the last one in case it's short, before re-issuing the command.

The only good thing about that delay is that the i/o from this was completely unnoticeable (hence, I didn't have to stop it at all)

Once I get the tree, I'll just update them individually as needed before copying and editing. This takes so long that I'll bet PKGBUILDs are out of date since I started the operation last night heheh

Last edited by Grogan (2023-05-25 16:10:35)

Grogan · 2023-05-26 01:42:56

OK, my operation completed and I'll consider this "solved" for me, so I"ll mark it. I'll still be here if anyone wants to continue it though.

The bottom line is, this is not practical for a non-project dev user to clone --universe. It took 20+ hours.

However, I'm trying it with 3 seconds and so far it hasn't bombed.

For my personal use, I did this::

--- clone.sh_orig	2023-05-24 14:16:54.000000000 -0400
+++ clone.sh	2023-05-25 21:28:19.511962551 -0400
@@ -116,6 +116,12 @@
 				jobs=$2
 				shift 2
 				;;
+                        --sleep)
+                                (( $# <= 1 )) && die "missing argument for %s" "$1"
+                                SLEEPCMD="sleep"
+                                SECS="$2"
+                                shift 2
+				;;
 			--)
 				shift
 				break
@@ -180,6 +186,7 @@
 
 	for pkgbase in "${pkgbases[@]}"; do
 		if [[ ! -d ${pkgbase} ]]; then
+                        $SLEEPCMD $SECS
 			msg "Cloning ${pkgbase} ..."
 			project_path=$(gitlab_project_name_to_path "${pkgbase}")
 			remote_url="${GIT_REPO_BASE_URL}/${project_path}.git"

So I don't have to edit the file to adjust the sleep value. I'm not the fanciest of scripters, it was the only way I could figure to get it to do nothing (and not throw any errors like invalid command) if the --sleep option is not given, and what I want it to do if it is :-)

pkgctl repo clone --protocol=https -j 1 --sleep 3 --universe

Last edited by Grogan (2023-05-26 01:54:06)

stripwax · 2023-05-27 09:57:55

Same problem, and the --protocol=https fix worked for me too.

Do the docs need to be updated to recommend (or point out the existence of) --protocol=https ?
https://wiki.archlinux.org/title/Arch_build_system

constantchangesforwhat · 2023-06-11 03:08:03

Having the same issue...although doing what has/is suggested does not work, this also does not happen to other packages for some reason:

+-[$]> pkgctl repo clone --protocol=https alsa-card-profiles
==> Cloning alsa-card-profiles ...
Cloning into 'alsa-card-profiles'...
Username for 'https://gitlab.archlinux.org':

does not happen to this randomly chosen package:

+-[$]> pkgctl repo clone --protocol=https aalib
==> Cloning aalib ...
Cloning into 'aalib'...
remote: Enumerating objects: 72, done.
remote: Total 72 (delta 0), reused 0 (delta 0), pack-reused 72
Receiving objects: 100% (72/72), 14.35 KiB | 14.35 MiB/s, done.
Resolving deltas: 100% (14/14), done.
==> Collecting packager identity from makepkg.conf
  -> name    : undefined
  -> email   : undefined
  -> gpg-key : undefined
  -> protocol: https
==> Configuring aalib

Anything else that can be tried?

Last edited by constantchangesforwhat (2023-06-11 03:09:25)

Allan · 2023-06-11 03:55:15

alsa-card-profiles is part of the pipewire PKGUBILD.

https://archlinux.org/packages/extra/x86_64/pipewire/

constantchangesforwhat · 2023-06-11 04:58:56

Would there be a way to inform the pkgcontrol of this? When "asp" was around before it got deprecated it would simply state that XXX was part of the package XXX and then download the main one. In this case "asp" does it but, since it is deprecated... What should I do now to build a package while not knowing it may belong to another?

+-[$]> asp checkout alsa-card-profiles
==> alsa-card-profiles is part of package pipewire
remote: Enumerating objects: 1168, done.
remote: Counting objects: 100% (271/271), done.
remote: Compressing objects: 100% (254/254), done.
remote: Total 1168 (delta 163), reused 45 (delta 13), pack-reused 897
Receiving objects: 100% (1168/1168), 238.71 KiB | 740.00 KiB/s, done.
Resolving deltas: 100% (263/263), done.
From https://github.com/archlinux/svntogit-packages
 * branch            packages/pipewire -> FETCH_HEAD
 * [new branch]      packages/pipewire -> packages/packages/pipewire
Cloning into 'pipewire'...
done.

Allan · 2023-06-11 05:13:37

I think a feature request needs made:
https://gitlab.archlinux.org/archlinux/ … s/-/issues

constantchangesforwhat · 2023-06-11 05:40:29

Is there some way I could find out which package something may belong to? I looked at the https://gitlab.archlinux.org/archlinux/ … g/packages site but, when I search for "alsa-card-profiles" to determine it's main package it informs me it couldn't find anything. So I did a "pacman -Qi alsa-card-profiles" and it does not provide or show anything about a main package. Thanks in advance.

Arch Linux

#1 2023-05-22 19:01:10

[solved] gitlab, no anonymous access?

#2 2023-05-22 20:13:25

Re: [solved] gitlab, no anonymous access?

#3 2023-05-22 22:00:41

Re: [solved] gitlab, no anonymous access?

#4 2023-05-22 23:01:23

Re: [solved] gitlab, no anonymous access?

#5 2023-05-23 01:47:46

Re: [solved] gitlab, no anonymous access?

#6 2023-05-23 03:35:24

Re: [solved] gitlab, no anonymous access?

#7 2023-05-23 04:08:30

Re: [solved] gitlab, no anonymous access?

#8 2023-05-23 10:53:39

Re: [solved] gitlab, no anonymous access?

#9 2023-05-23 20:49:21

Re: [solved] gitlab, no anonymous access?

#10 2023-05-23 21:13:24

Re: [solved] gitlab, no anonymous access?

#11 2023-05-23 21:22:51

Re: [solved] gitlab, no anonymous access?

#12 2023-05-23 21:42:42

Re: [solved] gitlab, no anonymous access?

#13 2023-05-23 23:54:15

Re: [solved] gitlab, no anonymous access?

#14 2023-05-24 11:08:27

Re: [solved] gitlab, no anonymous access?

#15 2023-05-24 18:35:17

Re: [solved] gitlab, no anonymous access?

#16 2023-05-25 01:35:23

Re: [solved] gitlab, no anonymous access?

#17 2023-05-25 02:22:55

Re: [solved] gitlab, no anonymous access?

#18 2023-05-25 16:00:00

Re: [solved] gitlab, no anonymous access?

#19 2023-05-26 01:42:56

Re: [solved] gitlab, no anonymous access?

#20 2023-05-27 09:57:55

Re: [solved] gitlab, no anonymous access?

#21 2023-06-11 03:08:03

Re: [solved] gitlab, no anonymous access?

#22 2023-06-11 03:55:15

Re: [solved] gitlab, no anonymous access?

#23 2023-06-11 04:58:56

Re: [solved] gitlab, no anonymous access?

#24 2023-06-11 05:13:37

Re: [solved] gitlab, no anonymous access?

#25 2023-06-11 05:40:29

Re: [solved] gitlab, no anonymous access?

Board footer