[FEATURE REQUEST] Ability to obtain sources dynamically

bibies · 2024-01-25 19:29:15

Two observations:
1. All the current PKGBUILD packaging functions called after source array was read and acted upon.
2. The entire PKGBUILD script is processing twice, second go happens in fakeroot environment with re-read and package() call.

The current model allows either retrieving sources from VCS or oriented on manual

pkgver

var changing by hand.

The idea is to have a routine to dynamically define source array before its processing. That is needed especially for foreign sources/binaries, when AUR package maintainer has no control nor notifications about version changes. What I mean is easiest to explain as the Obtainium html source scan.

Current situation is that if the source array points to a version that was removed the makepkg script will throw the error.

Maybe a new PKGBUILD function getsrc() or initsource() will do the trick?

Scimmia · 2024-01-25 19:40:21

So short version, you want auto-updating PKGBUILDs, so that stable package updates happen with no testing. Not a good idea.

bibies · 2024-01-25 19:49:37

Scimmia wrote:

So short version, you want auto-updating PKGBUILDs, so that stable package updates happen with no testing. Not a good idea.

1. If you have only signed binaries for AUR from a trusted dev (BTW "https://${_down_url}/${_app}-${_src_ver}-${_arch}.${_base_ext}{,.${_sign_ext}}" doesn't resolve) what kind of testing we supposed to have?

2. Besides VCS updates pkgver why that is not good idea for stable versions, how the VCS sources are more trustworthy?

Last edited by bibies (2024-01-25 19:53:40)

Lone_Wolf · 2024-01-26 11:21:57

(AUR) package maintainers should at the very least
- run a build test
- install the new package with pacman
- run a basic functionality test (example for mesa trunk glxinfo -B and/or eglinfo -B are suitable for that)
BEFORE uploading a new PKGBUILD

This is needed regardless of where the sources come from and also applicable to -bin packages.

Last edited by Lone_Wolf (2024-01-26 11:23:04)

bibies · 2024-01-26 12:13:06

Lone_Wolf wrote:

(AUR) package maintainers should at the very least
- run a build test ... BEFORE uploading a new PKGBUILD

The thing is that when old version that PKGBUILD source array is pointing to is removed and no longer available, the entire PKGBUILD is invalid until a maintainer notice[d|s] and come to rectify the version number by hand. For a reputable dev the basic functionality of new stable version should be self-evident. In such cases a maintenance comes to changing version numbers and maybe checking integrity hashes that is more a futile burden than anything else.

This is needed regardless of where the sources come from and also applicable to -bin packages.

Exactly my point - if VCS sources allowed to have dynamic version tag changes, there is no reason to deny the same functionality from stable packages, especially when maintainer has no control over sources and version update notifications.

Lone_Wolf · 2024-01-27 13:26:29

The thing is that when old version that PKGBUILD source array is pointing to is removed and no longer available, the entire PKGBUILD is invalid until a maintainer notice[d|s] and come to rectify the version number by hand.

This is very rare for publicly available software[1] .

For a reputable dev the basic functionality of new stable version should be self-evident.

archlinux rolling release model makes it a very fast moving target. The majority of opensource devs develop on slow moving distros like debian stable, ubuntu LTS, fedora and redhat .

Our packages and versions are often months and sometimes years ahead of the versions devs develop & test against,
Over time this has led to arch users often being the first people that notice bugs in new releases.

There is another aspect of PKGBUILDS that is important and hasn't been mentioned yet.
They are designed so everyone that runs them on an uptodate archlinux installation gets the same results as the maintainer.

In order to achieve this, the user needs to use the same sources as the maintainer. The sources + *sum arrays are very important for that.

(For VCS packages this is harder to achieve , but using commit hashes in pkgver() helps a lot.)

Especially for AUR packages where the user puts their trust in the maintainer(s) that submit packages this aspect is very important.

[1] Source code maintenance systems like git, svn, cvs , subversion etc and the platforms (gitlab, githib, sourcehut etc) have no problem keeping commit history for decades.
Even if released tarballs are deleted (unusual, but possible) they can be recreated from the commit history.

Some developers may not use versioned naming and overwrite release tarballs when a new one is made, but those are a very small minority in my experience.

Last edited by Lone_Wolf (2024-01-27 13:27:01)

bibies · 2024-01-27 17:39:16

The thing is that when old version that PKGBUILD source array is pointing to is removed and no longer available, the entire PKGBUILD is invalid until a maintainer notice[d|s] and come to rectify the version number by hand.
This is very rare for publicly available software[1] .
[1] Source code maintenance systems like git, svn, cvs , subversion etc and the platforms (gitlab, githib, sourcehut etc) have no problem keeping commit history...

The perception Archlinux is a small distribution for an independent peeps. The assumption of having accounts with big git distributors is kind of contradictory to that. Most small devs don't need MS github or unfriendly to Tor network gitlab. Many don't need VCS at all, for what they do or try to accomplish. So the references to 'majority' is kind of void.

And when the traffic and space aren't free it's quite normal to delete old versions except maybe the most recent one. Arch currently doesn't cover that. That's also why we have this topic.

archlinux rolling release model makes it a very fast moving target. The majority of opensource devs develop on slow moving distros like debian stable, ubuntu LTS, fedora and redhat .
Our packages and versions are often months and sometimes years ahead of the versions devs develop & test against,
Over time this has led to arch users often being the first people that notice bugs in new releases.

Looks like that exactly what I'm proposing here - reduce a cycle round-time between release of new version and the feedback to it, That's one of immediate benefits of rolling releases model - update & test as soon as it issued!

There is another aspect of PKGBUILDS that is important and hasn't been mentioned yet.
They are designed so everyone that runs them on an uptodate archlinux installation gets the same results as the maintainer.
In order to achieve this, the user needs to use the same sources as the maintainer.

If you mean packages reproducibility you'll want to move to Nixos. I see no point in reproducing slowly maintained versions over up to date ones pushed by the SW devs. So the goal is to start 'testing' released versions as soon as they pushed.

The sources + *sum arrays are very important for that.
(For VCS packages this is harder to achieve , but using commit hashes in pkgver() helps a lot.)
Especially for AUR packages where the user puts their trust in the maintainer(s) that submit packages this aspect is very important.

I'd like to address several points here:
1. Integrity checks in the form of hashes played much bigger role 30 years ago due to unreliable transfers. Nowadays checksum does NOT ensure trust nor reliability so much. At least when both are compared with dev's signature verification. Maintainer of package is less relevant here but obviously could have the role when he IS presenting the dev side. Should Archlinux have separate requirement for matching both SW developers AND maintainers signatures to proceed with an installation is another good question.
2. VCS commit hashes are just a workaround for filling pkgver variable and have no impact on trust.
3. Limiting version updates to VCS backends only via updating pkgver but marking that undesirable for stable releases, even when signed by devs, is irrational and limiting. Instead making official statement that even current ability to parse html pages in PKGBUILDs and proceeding with installations in (at least) THAT case would be desirable. Unless we have more serious counter argumentation than simple VCS have better responsibility thus reputability & trust than lonely wolves around random self-hosted sites.

Allan · 2024-01-28 00:44:39

bibies wrote:

[
1. Integrity checks in the form of hashes played much bigger role 30 years ago due to unreliable transfers. Nowadays checksum does NOT ensure trust nor reliability so much. At least when both are compared with dev's signature verification. Maintainer of package is less relevant here but obviously could have the role when he IS presenting the dev side. Should Archlinux have separate requirement for matching both SW developers AND maintainers signatures to proceed with an installation is another good question.

You assume software developers actually sign stuff... A fair portion of critical packages are still distributed without methods for verification.

progandy · 2024-01-28 08:19:48

By the way, you are free to develop some AUR helper that can use a custom variable or function defined in the package to autoupdate the sources and warns the user when it made some changes to verify the untested version.
I have no idea how well received that would be by AUR packagers (or the archlinux Package Maintainers responsible for the AUR management) , though.

especially when maintainer has no control over sources and version update notifications.

What do you mean by that? If the maintainer wants to be notified about version updates, there are a number of tools that implement scanning of release pages like nvchecker or urlwatch
https://wiki.archlinux.org/title/Arch_p … h_upstream

Last edited by progandy (2024-01-28 08:26:30)

Allan · 2024-01-28 08:33:00

Makepkg features drop in source modules. You could create an "auto" module that would allow sources like "auto+https://..." which in combination with pkgver() may achieve what you want.

This will not officially become part of makepkg.

bibies · 2024-01-30 23:05:41

there are a number of tools that implement scanning of release pages

I'd like to see a package maintainer has control over that, including built-in source array pre-scan. We don't need external programs/tools when there is dedicated makepkg functionality. It needs a function that could initialize the script BEFORE acting on the source array.

Last edited by bibies (2024-01-30 23:06:19)

bibies · 2024-01-30 23:12:58

an "auto" module that would allow sources like "auto+https://..."

Yes, that's probably a start... Although I think a function would be much cleaner when implemented properly as now source array is practically limited to just a couple of variables (pkgname and pkgver).

This will not officially become part of makepkg

...but this kills any attempts as is will remain UNSUPPORTED and second-class stuff for freaks for makepkg function fromAUR... Not good at all.

Arch Linux

#1 2024-01-25 19:29:15

[FEATURE REQUEST] Ability to obtain sources dynamically

#2 2024-01-25 19:40:21

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#3 2024-01-25 19:49:37

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#4 2024-01-26 11:21:57

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#5 2024-01-26 12:13:06

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#6 2024-01-27 13:26:29

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#7 2024-01-27 17:39:16

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#8 2024-01-28 00:44:39

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#9 2024-01-28 08:19:48

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#10 2024-01-28 08:33:00

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#11 2024-01-30 23:05:41

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

#12 2024-01-30 23:12:58

Re: [FEATURE REQUEST] Ability to obtain sources dynamically

Board footer