Git migration completed

Foxboron · 2023-05-21 11:22:00

We are proud to announce that the migration to Git packaging succeeded! ?

Thanks to everyone who has helped during the migration!

Package sources are now available on GitLab. Note that the bugtracker is still flyspry and that merge requests are not accepted as of now.
We intend to open the issue tracker and merge requests on the Gitlab package repos in the near future.

Mirrors are syncing again, but it may take a bit of time until your mirror of choice has caught up.

For users

Update your system and merge the pacman pacnew `/etc/pacman.conf.pacnew` file. This is required as we have moved the `[community]` repository into `[extra]`.

    $ pacman -Syu "pacman>=6.0.2-7"`

For users of the now deprecated `asp` tool, you will need to switch to `pkgctl`:

    $ pacman -Syu "devtools>=1:1.0.0-1"
    $ pkgctl repo clone linux

For some more detailed instructions on how to obtain PKGBUILDs see the corresponding wiki article.

For packagers

Before starting, first uninstall `devtools-git-poc` and remove any repos from your filesystem that you cloned during the git proof-of-concept testing.

Make sure you have both, an updated devtools and pacman on your system:

    $ pacman -Syu "devtools>=1:1.0.0-1" "pacman>=6.0.2-7"`

Additionally clean up old chroots in `/var/lib/archbuild/`

    $ rm -rf /var/lib/archbuild/
    # or optionally, use the --clean option for pkgctl build *ONCE*
    $ pkgctl build --clean

For instruction on how to use `pkgctl`, please take a look at the "How to be a packager" wiki article and also consult the man page of each subcommand for further information:

    $ man pkgctl-build
    $ man pkgctl-repo-clone

hcjl · 2023-05-21 11:49:04

Thanks a lot for the great work to all involved. I watched the runbook regularly and I am really impressed how fast and professional everything was executed! Cheers!!

ratcheer · 2023-05-21 12:07:38

When I run the command

pacman -Syu "pacman>=6.0.2-7"

I get

error: target not found: pacman>=6.0.2-7

Yet, when I search the repository, the new package is clearly there. What is wrong?

ratcheer · 2023-05-21 12:12:34

Never mind. I see on reddit that the repos are not necessarily done synching. I will wait a while.

Trilby · 2023-05-21 12:18:33

ratcheer wrote:

the ~~repos~~ mirrors are not necessarily done synching

Foxboron wrote:

Mirrors are syncing again, but it may take a bit of time until your mirror of choice has caught up.

mccurly · 2023-05-21 12:31:23

ratcheer wrote:

When I run the command
pacman -Syu "pacman>=6.0.2-7"
I get
error: target not found: pacman>=6.0.2-7
Yet, when I search the repository, the new package is clearly there. What is wrong?

Hello, I am having the same issue as you are.

Maybe this is just a choice of repositories, since it was advertised that syncing repositories might take a while.

What makes me wonder which repositories are up to sync right now...

Thanks for your hard work all of you folks that run this wonderful distro that is Arch! (My choice for a while now and I am certain that will be as long as I'll be able to choose)!!! Cheers!

Trilby · 2023-05-21 12:37:25

You can check the mirror statuses. As of a few moments ago only one US https mirror was actually synced (all others had last-syncs from prior to the migration), and that one only synced a couple hours ago.

Other regions may have different results (I'm only tracking USA https mirrors as those are what I'd consider using where I am).

Again, wait for your preferred mirrors to sync, there is nothing wrong here.

amish · 2023-05-21 12:51:42

Can someone please tell me if Mirror status page is up-to-date?

Under "Successfully Syncing Mirrors", when I sort by Delay (hh:mm), the first mirror it shows me is https://mirror.osbeck.com/archlinux/ with 0:00 as delay.

That mirror also has 100% Completion. Duration and Score is also good.

So I expect that, that mirror has synced now with Tier 0 mirror.

But when I go to Community packages page of that mirror, it still has tons of packages listed.

I was expecting that there will be no package listed on that page.

So I am just wondering, if I am interpreting Mirror status page wrongly?
OR Mirror status page is not showing correct results? And still showing old results.

Last edited by amish (2023-05-21 12:53:33)

Trilby · 2023-05-21 12:54:26

FWIW, after my last post I checked that mirror status webpage, and while I didn't do thorough checks, it did not seem to represent the data I was getting from the json API. The latter seems/seemed more accurate. I'm not sure what to make of this but thought I'd share my data point.

EDIT: and I can confirm the below observation on the one US mirror reporting a recent sync: it has a 56 byte pacman "pkg"

Last edited by Trilby (2023-05-21 13:11:01)

fredbezies · 2023-05-21 12:58:15

I noticed something weird after looking at "top" mirrors. A lot of them - all? - are listing this:

pacman-6.0.2-7-x86_64.pkg.tar.zst                  20-May-2023 14:17                  56

Pacman 6.0.2-7 only 56 bytes? Pacman 6.0.2-6 is far bigger:

pacman-6.0.2-6-x86_64.pkg.tar.zst                  18-Mar-2023 15:24              944412

Weird...

Trilby · 2023-05-21 13:17:45

The 56byte "package" files apparently are not tarballs but just html 404 error pages including the following:

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

I gather this is the result of those mirrors trying to sync pre-emptively when their source was not ready.

Last edited by Trilby (2023-05-21 13:18:42)

Allan · 2023-05-21 13:18:43

> ls -l /var/cache/pacman/pkg/pacman-6.0.2-7-x86_64.pkg.tar.zst
-rw-r--r-- 1 alpm alpm 943852 May 20 08:46 /var/cache/pacman/pkg/pacman-6.0.2-7-x86_64.pkg.tar.zst

wait for mirrors to properly sync.

BS86 · 2023-05-21 15:26:42

Allan wrote:

wait for mirrors to properly sync.

well, it looks like the mirrors (at least the osbeck one) think that they are already done syncing but they all have only synced 404 files in the new repos and in the packages that changed with the migration (most importantly pacman).
The last update timestamp is updated regularly, currently I see:
https://mirror.osbeck.com/archlinux/lastsync

1684682640

which is 1 minute ago, 17:24 CEST - but the files don't change anymore. It looks like their database thinks they already have that file and don't check if the file actually has the correct checksum or size.

Edit: I have now received the new mesa updates that arrived after the migration, but pacman that changed during the migration still is not available for download. I still think there is something broken in the sync process for the mirrors that tried to sync during migration, affecting all packages that were updated during the migration.
Even Tier 1 mirrors that report 100% done are only offering 56 byte pacman but are already offering packages that were updated after the migration:
https://archlinux.org/mirrors/status/tier/1/

Last edited by BS86 (2023-05-21 16:11:35)

Trilby · 2023-05-21 16:27:31

Note that the "tier 0" mirror is currently responding only with a 404 for these packages. So the mirrors are accurately mirroring the archlinux.org content. The problem is the packages are not available from archlinux.org.

Edit: sorry, this was a misunderstanding of the "download" link on the package description pages - I didn't know that link directed to something other than the "tier 0". But yet there still seems to be some form of systemic problem that all the tier 1 mirrors are getting a 404 error page content in place of packages.

Last edited by Trilby (2023-05-21 16:47:21)

Foxboron · 2023-05-21 16:44:37

Trilby wrote:

Note that the "tier 0" mirror is currently responding only with a 404 for these packages. So the mirrors are accurately mirroring the archlinux.org content. The problem is the packages are not available from archlinux.org.

The tier 0 mirror has all the packages available, however it's not publicly accessible.

Staff and Tier 1 can sync from it, and the packages are available there as expected.

Trilby wrote:

Edit: sorry, this was a misunderstanding of the "download" link on the package description pages - I didn't know that link directed to something other than the "tier 0". But yet there still seems to be some form of systemic problem that all the tier 1 mirrors are getting a 404 error page content in place of packages.

It's going to take several hours before stuff is properly synced. We are talking about 40 GB of download pr Tier 1 mirror syncing and a fully saturated 1 GB/s uplink. The issue will repeat itself downwards when Tier 2 syncs from Tier 1 and so on.

Last edited by Foxboron (2023-05-21 16:49:56)

BS86 · 2023-05-21 17:23:20

Foxboron wrote:

It's going to take several hours before stuff is properly synced. We are talking about 40 GB of download pr Tier 1 mirror syncing and a fully saturated 1 GB/s uplink. The issue will repeat itself downwards when Tier 2 syncs from Tier 1 and so on.

while I understand that the saturated uplink will definitely be a bottleneck, I still have a hard time understanding why the mirrors (or at least one that is configured on my system) are already offering updates that were done after the migration (namely bash, curl, mesa, pipewire to name a few) in the new testing repositories but still don't offer the updated pacman and devtools - packages. I edited my pacman.conf manually to match the new repository layout. Do the new testing repositories use a separate uplink?

Last edited by BS86 (2023-05-21 17:25:07)

Foxboron · 2023-05-21 17:30:08

BS86 wrote:

Foxboron wrote:
It's going to take several hours before stuff is properly synced. We are talking about 40 GB of download pr Tier 1 mirror syncing and a fully saturated 1 GB/s uplink. The issue will repeat itself downwards when Tier 2 syncs from Tier 1 and so on.
while I understand that the saturated uplink will definitely be a bottleneck, I still have a hard time understanding why the mirrors (or at least one that is configured on my system) are already offering updates that were done after the migration (namely bash, curl, mesa, pipewire to name a few) in the new testing repositories but still don't offer the updated pacman and devtools - packages. I edited my pacman.conf manually to match the new repository layout. Do the new testing repositories use a separate uplink?

Output of pacman -Q pacman bash

BS86 · 2023-05-21 17:42:50

 pacman -Q pacman bash
pacman 6.0.2-6
bash 5.1.016-4

I am not rebooting because the system appears to be in a partial update situation due to the mirror problem.

Last edited by BS86 (2023-05-21 17:44:17)

Foxboron · 2023-05-21 17:52:55

BS86 wrote:

 pacman -Q pacman bash
pacman 6.0.2-6
bash 5.1.016-4
I am not rebooting because the system appears to be in a partial update situation due to the mirror problem.

We don't necessarily control how mirrors sync, so the case here is that `[core-testing]` was synces before `[core]` and thus the package availability is different.

progandy · 2023-05-21 18:03:50

BS86 wrote:

well, it looks like the mirrors (at least the osbeck one) think that they are already done syncing but they all have only synced 404 files in the new repos and in the packages that changed with the migration (most importantly pacman).

osbeck has synced the repositories, but the package pool was not. Now there are dangling symlinks that result in 404s. Maybe the rsync command synced the lastupdate file first and failed afterwards? The mirror will probably recover as soon as the tier0 has a newer lastupdate timestamp and it is not overloaded.

Even Tier 1 mirrors that report 100% done are only offering 56 byte pacman but are already offering packages that were updated after the migration:
https://archlinux.org/mirrors/status/tier/1/

Some of the tier1 mirrors seem to be synced properly.

Last edited by progandy (2023-05-21 18:04:50)

amish · 2023-05-22 03:31:19

Hello all. Sorry for the noise.

I just wanted to know if this 404 problem will take a long time to resolve.

Because I had postponed some upgrades by 3 days. And it was scheduled today i.e. Monday.

Now I need to tell my company management that it needs to be postponed again. So I need some estimate.

I am trying to avoid a situation that I tell them 2 more days right now and again after 2 days, I need to ask them to postpone again.

Is there any other thread (or bug report page) where I can track the progress of this issue?

I do hope that root cause has been found. I am interested in knowing why this is happening for knowledge purpose. Because if mirrors use rsync then file that does not exist will not exist in mirror as well instead of files containing 404 error text.

None the less, thank you very much to all the people involved in this migration. Great efforts have been put on. Appreciate them all.

Allan · 2023-05-22 06:09:00

The issue is likely fixed already... I use a Tier-2 mirror and there was no issues for at least 12 hours.

amish · 2023-05-22 06:47:04

Yes, I can confirm that it is now fixed for me. It was still giving 404 error when I posted the previous message.

silent · 2023-05-22 10:26:36

Foxboron wrote:

$ pacman -Syu "pacman>=6.0.2-7"`

What is the role of the closing character here after the double quote? I just came here because it was difficult to understand the line in the Arch-announce E-mail:

Arch-announce wrote:

$ pacman -Syu "pacman>=6.0.2-7"`

JoeCool · 2023-05-22 12:26:48

I am pretty sure that backtick was a typo. It's not there any longer on the homepage.

I am guessing it is safe and recommended to delete

/var/lib/pacman/sync/community.*

now. Or is there a reason not to?

Last edited by JoeCool (2023-05-22 12:27:26)

Arch Linux

#1 2023-05-21 11:22:00

Git migration completed

#2 2023-05-21 11:49:04

Re: Git migration completed

#3 2023-05-21 12:07:38

Re: Git migration completed

#4 2023-05-21 12:12:34

Re: Git migration completed

#5 2023-05-21 12:18:33

Re: Git migration completed

#6 2023-05-21 12:31:23

Re: Git migration completed

#7 2023-05-21 12:37:25

Re: Git migration completed

#8 2023-05-21 12:51:42

Re: Git migration completed

#9 2023-05-21 12:54:26

Re: Git migration completed

#10 2023-05-21 12:58:15

Re: Git migration completed

#11 2023-05-21 13:17:45

Re: Git migration completed

#12 2023-05-21 13:18:43

Re: Git migration completed

#13 2023-05-21 15:26:42

Re: Git migration completed

#14 2023-05-21 16:27:31

Re: Git migration completed

#15 2023-05-21 16:44:37

Re: Git migration completed

#16 2023-05-21 17:23:20

Re: Git migration completed

#17 2023-05-21 17:30:08

Re: Git migration completed

#18 2023-05-21 17:42:50

Re: Git migration completed

#19 2023-05-21 17:52:55

Re: Git migration completed

#20 2023-05-21 18:03:50

Re: Git migration completed

#21 2023-05-22 03:31:19

Re: Git migration completed

#22 2023-05-22 06:09:00

Re: Git migration completed

#23 2023-05-22 06:47:04

Re: Git migration completed

#24 2023-05-22 10:26:36

Re: Git migration completed

#25 2023-05-22 12:26:48

Re: Git migration completed

Board footer