You are not logged in.

#1 2019-02-26 09:22:49

blochl
Member
Registered: 2018-08-31
Posts: 45

[Solved] How packages are being built if their source is unavailable?

Hi,

I'm trying to rebuild a subset of about 1000 packages from the official repositories. But in a significant number of cases (5~10%), the build fails because the upstream sources fail to download. In some cases the server is just not available temporarily, and in some cases the link in the PKGBUILD looks obsolete.

I wonder, how such cases are handled on the official build server?

Thanks!

Last edited by blochl (2019-02-26 18:16:25)

Offline

#2 2019-02-26 09:29:13

Slithery
Forum Moderator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 3,386

Re: [Solved] How packages are being built if their source is unavailable?


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Online

#3 2019-02-26 09:51:41

blochl
Member
Registered: 2018-08-31
Posts: 45

Re: [Solved] How packages are being built if their source is unavailable?

Thanks, yeah, I've seen these. But not all of the packages are there. Just a random example - jq seems not to be there, although it is in the community repository. Also, OK, how are the sources downloaded to create the src.tar.gz archives, if they are not available upstream? I'm aware of makepkg's "-S" option, but it does still try to fetch the sources from the URL in the PKGBUILD. What happens if this URL is unreachable? And apparently some PKGBUILDs do have unreachable URLs in them.

Offline

#4 2019-02-26 12:50:02

Muflone
Trusted User (TU)
From: Italy
Registered: 2013-10-08
Posts: 73
Website

Re: [Solved] How packages are being built if their source is unavailable?

Hi

Please file a bug for each broken source URLs, the maintainer could not be aware as during the rebuild if the source was already downloaded it wouldn't be downloaded again.

Offline

#5 2019-02-26 15:55:34

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,853

Re: [Solved] How packages are being built if their source is unavailable?

As muflone says -- a package with missing sources is a bug we need to fix. In some cases, we will self-host the latest version we have cached, in other cases we need to adapt to upstream changes, and we don't always know this because the maintainer might still have a locally downloaded copy.

The .src.tar.gz which you were pointed to is created for a large subset of packages that require one who distributes binaries, to also distribute sources and not just build scripts.
It is created shortly after the initial upload, so at the time it is created, the sources presumably still worked -- but errrors for the sourceballs service are not necessarily closely watched either.

...

Some downstream projects like archlinux32, archlinuxarm, and the Debian-sponsored https://tests.reproducible-builds.org CI will rebuild our entire repository, and in the process catch an eclectic collection of errors, including:
- missing sources
- checkdepends that should be makedepends
- packages that no longer build after their build dependencies were updated

We are always grateful for these sort of bug reports, although in some cases if there are a lot of errors we can make a TODO list for it instead, for example https://www.archlinux.org/todo/source-retirement/


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#6 2019-02-26 18:12:35

blochl
Member
Registered: 2018-08-31
Posts: 45

Re: [Solved] How packages are being built if their source is unavailable?

Hi,

Thanks for the reply. I've opened one such bug: https://bugs.archlinux.org/task/61867
However, it will be very time consuming to manually open such bugs, for packages which I essentially don't need - they are just dependencies of other packages which I need.
Since my scripts run automatically, will it be helpful if I just post a list of packages the sources for which fail to download?

By the way, I see that some URLs which were not accessible last week, are accessible now. For example the one for alsa-utils. Must be a server downtime on their side.

eschwartz - thanks for your explanation! That's clear now, so marking as "solved".

Last edited by blochl (2019-02-26 18:14:21)

Offline

#7 2019-02-26 18:24:43

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,853

Re: [Solved] How packages are being built if their source is unavailable?

If you would like to post such a list here, I can help turn it into a mass rebuild list.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#8 2019-02-27 10:30:27

blochl
Member
Registered: 2018-08-31
Posts: 45

Re: [Solved] How packages are being built if their source is unavailable?

Opened bugs for 4 more packages:

https://bugs.archlinux.org/task/61875
https://bugs.archlinux.org/task/61876
https://bugs.archlinux.org/task/61877
https://bugs.archlinux.org/task/61878

The issue with alsa-related stuff, as I wrote earlier, fixed itself.

Question: the bug I've opened yesterday, about cracklib, was downgraded from "severity: high" to "severity: low". The further bugs I've opened with "severity: low" initially, to comply, but isn't the fact that the package doesn't build deserves a higher severity?

Offline

#9 2019-02-27 13:11:29

Muflone
Trusted User (TU)
From: Italy
Registered: 2013-10-08
Posts: 73
Website

Re: [Solved] How packages are being built if their source is unavailable?

blochl wrote:

isn't the fact that the package doesn't build deserves a higher severity?

I don't think so.
The package should be fixed but it isn't a severe issue, as long as Arch Linux repositories will provide the binary packages.

This is a so low severity issue that doesn't deserve even a pkgrel bump, as only the sources were moved, while the binary package is still valid.

Offline

#10 2019-02-27 13:58:29

blochl
Member
Registered: 2018-08-31
Posts: 45

Re: [Solved] How packages are being built if their source is unavailable?

Muflone wrote:
blochl wrote:

isn't the fact that the package doesn't build deserves a higher severity?

I don't think so.
The package should be fixed but it isn't a severe issue, as long as Arch Linux repositories will provide the binary packages.

This is a so low severity issue that doesn't deserve even a pkgrel bump, as only the sources were moved, while the binary package is still valid.

I see your point. However this can be a major issue for people/projects who rely on building the packages locally, or even caching the sources for possible re-building at a later time. True, not as much as it would be with Gentoo, but still.
But hey, I'm not arguing about that, just saying. smile

Also, if the checksum of a downloaded source (of the correct version) does not match the one in the PKGBUILD (as is the case in several of the bugs I've opened) how severe is that? I mean it may indicate some inconsistency in the sources...

Offline

#11 2019-02-27 14:09:09

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 22,042
Website

Re: [Solved] How packages are being built if their source is unavailable?

blochl wrote:

Also, if the checksum of a downloaded source (of the correct version) does not match the one in the PKGBUILD (as is the case in several of the bugs I've opened) how severe is that? I mean it may indicate some inconsistency in the sources...

It may, but there are also countless other things that could result in the same outcome.  For example, if the source actually has moved, it could be quite likely that a README or similar would document why the source was moved - even if there was no change in the source code.  As that README would be part of he tarball or source repo, the checksum would be expected to change.

Last edited by Trilby (2019-02-27 14:09:35)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#12 2019-02-28 02:04:56

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,853

Re: [Solved] How packages are being built if their source is unavailable?

Trilby wrote:
blochl wrote:

Also, if the checksum of a downloaded source (of the correct version) does not match the one in the PKGBUILD (as is the case in several of the bugs I've opened) how severe is that? I mean it may indicate some inconsistency in the sources...

It may, but there are also countless other things that could result in the same outcome.  For example, if the source actually has moved, it could be quite likely that a README or similar would document why the source was moved - even if there was no change in the source code.  As that README would be part of he tarball or source repo, the checksum would be expected to change.

It should not be expected to be modified in the archives of an old release, only added to the development version in order to appear in the next upcoming release.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB