A more distributed AUR?

Lone_Wolf · Yesterday 10:16:59

seth wrote:

Mass deployment - it's essentially what the chaotic-aur repo is doing for you as well.

I wouldn't call 3 systems mass deployment, but there are similarities with unofficial repos that are used by their creator. .

you hunt down the dependecies yourself and keep everything up-to-date

Not sure what you mean by hunting deps, but I am subscribed to aur packages I use and also use nvchecker for non-vcs packages to keep an eye on upstream.
My local custom repo has been a valuable asset to me and my workflow for atleast 15 years now.

I have tried a few aur helpers in the past (including yaourt) but found I had less control and more problems with them then without.

cryptearth · Yesterday 12:07:58

@LW
recursive aur deps: I want to build and use aur package A - but it depends on another aur-only package B (repeat)
a local repo doesn't help unless you do quite some work that a helper can do for you
if I learned one thing from LinuxFromScratch: I don't want to deal with that manually if there're tools doing that for me

Lone_Wolf · Yesterday 12:39:15

That just means I have to add B to my local repo before I can build A.

Several of the aur packages I maintain have a more complicated setup. Building packages and putting them in my local repo makes it easy for pacman/makepkg to use them.

The unix philosophy of

Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

fits well with me.
Maybe you prefer a tool that can do many jobs ?

seth · Yesterday 13:08:50

If you're gonna throw unix philosophies at each other I'll briefly remind everyone that they're currently using systemd (and some even try to excuse that w/ a disclaimer )

Whether you want/need something to auto-resolve and build dependency chains for you kinda depends on the package.
If you're interested in an AUR package that depends on an entire tree of other AUR packages that need rebuilding, it's not unreasonable to not doing this manually.
https://aur.archlinux.org/packages/aurutils might intersect either opinions here.

plp · Yesterday 20:19:07

Lone_Wolf wrote:

A custom local repository that is added to pacman.conf solves that.

I didn't know about local repositories. Serves me right for not meticulously reading the wiki...

So, a simple script that converts the entire AUR GitHub mirror into a local repository would be all that's needed to keep going whenever aur.archlinux.org goes down?

Or will using such a large repo cause performance or storage space issues?

Last edited by plp (Yesterday 20:19:18)

seth · Yesterday 20:23:55

The repos would host actual packages, you'd have to build constantly everything from the AUR - which isn't realistic nor reasonable.
https://aur.chaotic.cx/ does something like that for a limited set of packages (and implies that you need to trust the repo next to the individual packages it builds)

When aur.archlinux.org goes down (and in general) use the github mirror.

plp · Yesterday 20:33:04

seth wrote:

The repos would host actual packages, you'd have to build constantly everything from the AUR - which isn't realistic nor reasonable.

Oh, I hadn't realised that.

We're back to my old plan for a "utility service" that just provides the PKGBUILDs and associated resource files then. And the RPC.

BTW, does anyone know how large the AUR exactly is? Would it even be feasible to host a local copy of the GitHub mirror on a average-sized SSD?

Last edited by plp (Yesterday 20:33:26)

seth · Yesterday 20:48:54

The github mirror currently has 141620 branches, many just some 3-4 4kB inodes (the AUR doesn't host any sources) so ~5GB?
(Plus the git "overhead", ie. the history if you don't just want a snapshot) - are you genuinely worried that github will drop out?

plp · Yesterday 22:05:18

seth wrote:

are you genuinely worried that github will drop out?

No, I was simply not aware of the actual size. 5GB is nothing, 5TB on the other hand would be infeasible for many people. Even though GitHub is unlikely to drop out, people will still probably want to run the utility service locally for performance reasons. Or (like the good libre-minded nerds they are) host mirrors of the whole thing on independent forges to avoid giving Microsoft the keys to the kingdom.

I'm currently using the instructions here to clone the entire thing, and will report once it's done.

Last edited by plp (Yesterday 22:21:07)

cryptearth · Today 01:59:40

Lone_Wolf wrote:

That just means I have to add B to my local repo before I can build A.
...
Maybe you prefer a tool that can do many jobs ?

key - expand that to a 30-package multi-level tree - wanna deal with that by hand?
I don't seek for a tool that does multiple things - but if the one thing I miss out on is proper package management - na uh - not me
it's not about the inability to invest half an hour dealing putting my package together - it's about the 25 minutes wasted when not using a tool dealing with that in 5 minutes

plp wrote:

5TB on the other hand would be infeasible for many people

[main@main ~]$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
vault  21.8T  9.44T  12.4T        -         -    12%    43%  1.00x    ONLINE  -

say what now again?

Last edited by cryptearth (Today 02:01:01)

seth · Today 07:45:36

plp wrote:

for many people

I think you and the lonesome puppy are talking past each other.
The problem w/ most AUR helpers is that they in fact /do/ multiple things and more specifically that one of those things (the really really bad one) is that they blend between AUR and repos (and by inference also lure people into not scanning the PKGBUILDs *cough*)
So what you're looking for is an AUR helper that doesn't also wrap pacman - such exist, but are for some reason not the ones that frequently show up having gotten their users into a complete mess ¯\_(ツ)_/¯

plp · Today 09:26:16

The problem w/ most AUR helpers is that they in fact /do/ multiple things and more specifically that one of those things (the really really bad one) is that they blend between AUR and repos (and by inference also lure people into not scanning the PKGBUILDs *cough*)

Or perhaps AUR should have some sort of package verification system in place?

A simple way to do this without burdening the core developers would be to simply let users mark/sign packages as 'verified'. Packages with enough verifications can then be considered safe enough to be installed. This will of course depend on the verifiers' honesty, so perhaps there should also be in place some kind of mechanism for vetting them. Like a scoring system based on the user account's age, packages submitted, etc.

Sorry for digressing, but it's always nice to try and identify new problems to fix. Should I put this in another thread? Am I just being annoying?

cryptearth wrote:

say what now again?

21TB is 3 times what I have, yet 5TB would still waste over 40% of your free space.

Sorry, couldn't help talking past you again

BTW:

plp wrote:

I'm currently using the instructions here to clone the entire thing, and will report once it's done.

I went to bed and woke up, and git is still adding branches. It's on letter 'p'.

Last edited by plp (Today 09:28:02)

seth · Today 09:37:58

Have you checked how big the repo already is?
The 5GB were a complete estimate based on the branch count and reasonable average package size - but i've no idea how big the branches really are because of the backlog…

The AUR already has a voting system, but that doesn't verify anything - I can package an awesome tool, get plenty of votes (even from experienced users who read the PKGBUILD) and then change it to distribute malware.
Think of the AUR as structured wiki articles on how to package foobar - they're very convenient to apply, but you still need to read and understand them first.

plp · Today 10:36:31

seth wrote:

Have you checked how big the repo already is?

I already talked about it, git hasn't finished downloading/adding all the branches yet. Still on letter 'p'.

Succulent of your garden · Today 11:00:46

plp wrote:

A simple way to do this without burdening the core developers would be to simply let users mark/sign packages as 'verified'. Packages with enough verifications can then be considered safe enough to be installed. This will of course depend on the verifiers' honesty, so perhaps there should also be in place some kind of mechanism for vetting them. Like a scoring system based on the user account's age, packages submitted, etc.

Come On Jia Tan ! I'm ready for round two. Bring it on Shake it baby

seth wrote:

The AUR already has a voting system, but that doesn't verify anything - I can package an awesome tool, get plenty of votes (even from experienced users who read the PKGBUILD) and then change it to distribute malware.
Think of the AUR as structured wiki articles on how to package foobar - they're very convenient to apply, but you still need to read and understand them first.

This is the reason by the way friendly sea doggo puppy. You can have the verification and all that things but there is always some kind of trust in the hands of the developers in every piece of software, of course here we trust a lot in the package maintainers of core and extra repos because we know by years that they are nice people, but some times came Jian Tan and tries to steal your stuff, like the fox in Dora the explorer :C, but hey we have a lot of pretty neat Boots and Doras saying no to the Fox

sea doggo said wrote:

21TB is 3 times what I have, yet 5TB would still waste over 40% of your free space.

Maybe btrfs with deduplication ? But not sure to be honest.

plp · Today 11:21:36

seth wrote:

The AUR already has a voting system, but that doesn't verify anything - I can package an awesome tool, get plenty of votes (even from experienced users who read the PKGBUILD) and then change it to distribute malware.

That's the case for all software that's available online. Nobody can guarantee that a core package maintainer for Arch (or any other distro) won't suddenly go rogue, either.

The current voting system only indicates popularity. What I'm proposing is to allow veteran AUR users to examine the contents of new packages, satisfy themselves that they're not malicious, and then sign them as 'verified' for the rest of the community. Obviously, this process must be repeated with every new release.

But it need not be cumbersome. Here's how a hypothetical AUR helper that supports verification might work:

For newly released packages:

Inform the user that "this package hasn't been verified yet"
Display the contents of the PKGBUILD, helper files and local source files
Present the following choices: install, verify and install, report malicious
Proceed according to user's choice

For packages that have been verified:

Inform the user that "this package has been verified by: seth, plp"
Present the following options: install, examine
If the user chooses "install", install the package
If the user chooses "examine":
- Display the contents of the PKGBUILD, helper files and local source files
- Present choices: install, verify and install, report malicious
- Proceed according to user's choice

It should be noted that it's not sufficient to only examine the PKGBUILD. Helper files and local source files provided by the AUR package must also be reviewed, as must all links to online sources. But I think that having the tool show them to you, and step you through the process as above, will make things easy enough.

Last edited by plp (Today 11:25:43)

plp · Today 11:40:24

cactus in pot wrote:

sea doggo

I took that photo of him during a trip to Norway, a few years back. Had to crawl through the sand and under a fence, but it was worth it. His looks and attitude struck me as so similar to mine that I just had to use him as my avatar...

Succulent of your garden · Today 11:48:18

sea doggo wrote:

The current voting system only indicates popularity. What I'm proposing is to allow veteran AUR users to examine the contents of new packages, satisfy themselves that they're not malicious, and then sign them as 'verified' for the rest of the community. Obviously, this process must be repeated with every new release.

Mmm I guess in someway the "trusted" users can be people who are maintaining many packages from many years. I mean check seth:

https://aur.archlinux.org/packages?O=0& … &submit=Go

He is only maintaining one package, but Seth is from the beginning of Unix time, he's being around from many years here helping people fixing bugs. So Seth in some sense for many is a guy who you can trust in some sense, but it does have one package in the AUR, unless that seth is the trash hippopotamus version of the god of destruction and not the dog one, which probably is not the case here[ Also are you using sqriptor seth ? Why ? I'm curious ]

Also the AUR is being in examination constantly, it really takes off malware packages very quickly to be honest, I mean sometimes windows and mac takes weeks or months to patch a vulnerability in the OS, the community work around there is impressive from that point of view.

Also they have a clean up day ^^ https://wiki.archlinux.org/title/AUR_Cleanup_Day

So in some sense there are coordination. In some sense AUR is not the complete Wild West experience, there are some Clint Eastwood and Chuck Norris rangers checking the town over there

EDIT:

sea doggo wrote:

I took that photo of him during a trip to Norway, a few years back. Had to crawl through the sand and under a fence, but it was worth it. His looks and attitude struck me as so similar to mine that I just had to use him as my avatar...

Aww, that's pretty neat. if AMOC stops https://en.wikipedia.org/wiki/Atlantic_ … irculation and Europe went into ice age, then I would like to have a sea doggo adopted as a dog in my house ^^

Last edited by Succulent of your garden (Today 11:51:58)

plp · Today 12:04:40

cactus in pot wrote:

So in some sense there are coordination. In some sense AUR is not the complete Wild West experience, there are some Clint Eastwood and Chuck Norris rangers checking the town over there tongue

AUR is awesome, and that's especially because security violations are published as soon as they're caught (and not hidden behind walls of corporate bureaucracy). And because we're able to have discussions such as this, that actually lead to code being written and protocols changed.

Aww, that's pretty neat. if AMOC stops https://en.wikipedia.org/wiki/Atlantic_ … irculation and Europe went into ice age, then I would like to have a sea doggo adopted as a dog in my house ^^

I'm too far south for that.

Last edited by plp (Today 12:05:27)

Succulent of your garden · Today 12:22:21

sea doggo wrote:

and that's especially because security violations are published as soon as they're caught (and not hidden behind walls of corporate bureaucracy

funny-everything-open-source-read-assembly-sweatshirt-20240614150831-sqnouw2c-s1.jpg

sea doggo wrote:

I'm too far south for that.

me too. But the population of sea doggo is going to increase I guess, we need to do something about it meanwhile I design my igloo

Lone_Wolf · Today 13:38:46

For clarity :
Aur helpers that do not wrap pacman nor blend the difference between repo and aur can be very useful.

A few months ago I looked into aurutils and found it provides about 2/3 of what I achieve with my personal setup.
That's not enough to make me switch to it, but in my opinion it is the best aur helper available.

seth · Today 14:12:55

@plp
I meant in terms of gigabyte - do you mean git had not even started to download anything but was still busy collecting the branches?

That's the case for all software that's available online. Nobody can guarantee that a core package maintainer for Arch (or any other distro) won't suddenly go rogue, either.

Which is why you need to vet the sources before hitting make or have someone you trust (hopefully) do that (as good as possible)
As for your distro screwing its users: that would quickly be the end of that distro.

Obviously, this process must be repeated with every new release.

seth wrote:

The github mirror currently has 141620 branches

If every branch gets one update a year, that's 388 updates a day.
If you've a dozen trustworthy users and every PKGBUILD just gets one check and everybody checks 10 packages a day, it will take over 3 years to curate them all (and the aurweb w/ ~100000 packages doesn't look much better)

Also there's a process where trusted users check PKGBUILDs and build packages (and then even kindly share them): it's the arch repos.
So if you have a trusted user and they decide to curate a PKGBUILD, there's a more robust and maintainable process available that's even more enduser friendly.

Also I think we went OT w/ this (from the original problem being the AUR uptime - not with the entire seal thing )

Succulent of your garden · Today 14:16:08

Lone_Wolf wrote:

but in my opinion it is the best aur helper available.

Mmmhh I'm going to check it in summer ^^

plp · Today 14:28:43

@Lone_Wolf, what are you using, if I may ask? And what is the functionality you are missing?

My main use cases are:
- Software for my desktop and laptop, installed interactively so that I get to inspect the package contents
- Software for my collection of docker images, installed without any user interaction (--noconfirm option of yay)

So, I'm at risk of ending up with malware on the containers, unless I first install and test all packages manually on my desktop. And sometimes I can do that, but other times it's just not feasible. For instance, images used for CI have to be built unattended at scheduled intervals. Plus, manually checking packages can be a hassle.

This is why I like the package verification system idea.

I use yay but there's no reason I couldn't be using aurutils instead. I took a quick look at their manual pages, and they appear to offer all the functionality I need, including automatic resolution of pacman dependencies. (This actually means that they do blend pacman and aur after all. Unless there's something I didn't understand.)

Succulent of your garden · Today 14:32:36

Sea Doggo wrote:

So, I'm at risk of ending up with malware on the containers, unless I first install and test all packages manually on my desktop

Do you know this right ? https://archlinux.org/packages/extra/x86_64/trivy/

https://github.com/aquasecurity/trivy

Arch Linux

#26 Yesterday 10:16:59

Re: A more distributed AUR?

#27 Yesterday 12:07:58

Re: A more distributed AUR?

#28 Yesterday 12:39:15

Re: A more distributed AUR?

#29 Yesterday 13:08:50

Re: A more distributed AUR?

#30 Yesterday 20:19:07

Re: A more distributed AUR?

#31 Yesterday 20:23:55

Re: A more distributed AUR?

#32 Yesterday 20:33:04

Re: A more distributed AUR?

#33 Yesterday 20:48:54

Re: A more distributed AUR?

#34 Yesterday 22:05:18

Re: A more distributed AUR?

#35 Today 01:59:40

Re: A more distributed AUR?

#36 Today 07:45:36

Re: A more distributed AUR?

#37 Today 09:26:16

Re: A more distributed AUR?

#38 Today 09:37:58

Re: A more distributed AUR?

#39 Today 10:36:31

Re: A more distributed AUR?

#40 Today 11:00:46

Re: A more distributed AUR?

#41 Today 11:21:36

Re: A more distributed AUR?

#42 Today 11:40:24

Re: A more distributed AUR?

#43 Today 11:48:18

Re: A more distributed AUR?

#44 Today 12:04:40

Re: A more distributed AUR?

#45 Today 12:22:21

Re: A more distributed AUR?

#46 Today 13:38:46

Re: A more distributed AUR?

#47 Today 14:12:55

Re: A more distributed AUR?

#48 Today 14:16:08

Re: A more distributed AUR?

#49 Today 14:28:43

Re: A more distributed AUR?

#50 Today 14:32:36

Re: A more distributed AUR?

Board footer