A new AUR mirror: AUR3

keenerd · 2011-01-31 05:13:18

For the past year or so, I've been toying with mirroring the AUR. In the last week, I stepped it up a notch and started mirroring constantly. The result of all this:

http://aur3.org

Some stuff works great, some stuff is just... missing. Right now I am more curious if my little server can hold up to some real traffic. (If it does not, it would be a bigger show stopper than the 2% of packages that do not mirror properly.)

Please have fun with this!

Xyne · 2011-01-31 06:14:46

Interesting. Designing it with tools in mind is very promising.

*subscribes to topic*

karol · 2011-01-31 06:17:26

"Every pkgbuild from the AUR, in one convenient tarball" - is it retained or overwritten? Some people wanted previous versions of PKGBUILDs or ones that got removed some time ago.

keenerd · 2011-01-31 15:24:52

Seems there is already a project with the AUR3 name. I did not know this. Extofme, my apologies. I'll see if I can figure out I good way to rearrange this.

karol: For now, overwritten. Getting 100% coverage is more important than full versioning, but I am very passionate about tracking versions. All the meta data is fully tracked over time.

Xyne: This project would not have been possible without your work on parsing pkgbuilds. I owe you one on that; any crazy feature you'd like to see?

But running Python and Perl and Bash interpreters is killing my very limited ram. I'll probably port your code. From what I understand, you wrap the pkgbuild in a function (so nothing is executed) and pass it to bash. Bash parses the source and returns a pretty printed version. Then parse out the variable assignments and emulate a few things such as variable substitution. Is this mostly correct or did I skip a few steps?

karol · 2011-01-31 15:36:17

I actually thought you and extofme are working together and you haven't linked to his thread because not all the features are implemented.
I think he wants to create something like AUR300 not AUR3 ;P

Getting older versions of binaries (via ARM) or PKGBUILDs (from your AUR3) may not be enough. How do you configure foobar-3.03? You need to manually descent into wiki history (current version is foobar 5, actually, it's foobar-ng-5) or hope The Wayback Machine has the webpage of that project in its cache (as www.foobar.org gives you a 404).

keenerd · 2011-01-31 16:39:55

Eh, not really what I had in mind. I'd much rather add exciting new forwards looking features than clone archive.org. I want Arch development to be faster and more convenient.

It will be some time (if ever) before you can just build arbitrarily old versions of packages. The all_pkgbuilds only contains pkgbuilds. It is missing all the other files (patches and forgotten svn repos and whatnot) that come in the source tarball. I will probably start saving every edited pkgbuild later today, but saving every source tarball will not happen in the near future. Distributing/searching a historical pkgbuild collection is a whole ball of wax.

On the A.R.M. front, there is still low hanging fruit. Such as a command "unpacman --yesterday" that looks at your pacman.log, automatically downgrading recent changes. Which is something I would write, if I ever had a need for it.

karol · 2011-01-31 16:55:53

keenerd wrote:

On the A.R.M. front, there is still low hanging fruit. Such as a command "unpacman --yesterday" that looks at your pacman.log, automatically downgrading recent changes. Which is something I would write, if I ever had a need for it.

I think saving the timestamped packagelist and then installing every package from it would do.

aeosynth · 2011-02-01 02:40:34

What I really want are aur binaries, that's the only thing I miss from Ubuntu. Would you or the other aur3 project (link to their thread?) be willing to index the unofficial repos?

karol · 2011-02-01 02:44:13

aeosynth wrote:

What I really want are aur binaries, that's the only thing I miss from Ubuntu. Would you or the other aur3 project (link to their thread?) be willing to index the unofficial repos?

You mean to build all the packages from AUR? Unlikely, as it takes several people to take care of much smaller official repos.
I'm sure Arch devs would welcome some additional manpower - be it from Ubuntu or any other place.

If by indexing you mean to keep track which version of which package which repo is carrying, you can do it yourself, just "enable" those repos in your pacman.conf.
The Unofficial User Repositories already provide binaries.

Last edited by karol (2011-02-01 02:50:40)

aeosynth · 2011-02-01 07:40:37

I'm talking about just keeping track of which package is where.

Sure, I could refresh the wiki, copy / paste all the repos into my pacman.conf, search for / install packages, then remove the repos to not screw up my system - but I'd use the repos a lot more if I could search them without going through all this.

All I'm asking for is one additional piece of metadata - the repos the package is available in.

karol · 2011-02-01 07:48:28

So you say it's a problem to paste 30 lines into your pacman.conf and download the dbs? OK.

chris-kun · 2011-02-01 08:43:16

Honestly, it's just too subjective. Some people will prefer this, and others that. We can talk about it all day and go in circles. imo the developers should just pick something and go with it, and if they really wanted to they could have an option in your profile or something where you could change it to suit what you want.

keenerd · 2011-02-02 12:54:39

Extofme was okay with both of us using the AUR3 name, which was awesome of him.

Fixed a bunch of bugs, but I'll probably spend another day checking to root out any that were missed.

One new feature added - ABS is loaded into AUR3. What does this mean?

* Really fast searching. Similar to "pacman -Sii", but takes 0.3 seconds instead of 10 seconds (empty cache).
* More information. "Required by" stat for official packages includes AUR packages.
* Tarballs. Don't want the whole ABS tree around? Download a single source package instead.

hellnest · 2011-02-02 13:02:50

great stuff!, hmm try to add "auto exctract" feature when downloading the tarball would be nice

Last edited by hellnest (2011-02-02 14:43:34)

keenerd · 2011-02-03 10:45:25

hellnest: I'd rather work on the backend for now. The current aur3 app is meant to be as bare bones as possible. But if no one else writes a nice client I'll port Packer or something.

It seems like a lot of searches are coming in pairs, for both name and description. So, name+description searching has been added.

Use:
aur3 -nd query
echo 'nd query' | nc aur3.org 1819

keenerd · 2011-02-04 12:51:07

Comments are mirrored. You can read them with the reference client (aur3 -c pkgname) or download the json (aur3.org/mirror/pkgname/comments.gz)

keenerd · 2011-02-08 06:44:06

Cross posted to the aur-dev ML.

The new thing is uploads to AUR3. I was stuck on a few parts,
like how to split a .sig from a .gpg (for single file uploads) and
then after than how to PUT more than one file at a time. Turns out neither are
possible, so I hacked up something really ugly with CGI.

But except for the one little wart of a multifile POST, all the beauty
of signed packages is there.

If a package has a sig, its repo (in the rpc) is marked as aur3 and there will be a file at aur3.org/mirror/pkgname/pkgname.tar.gz.sig. But as I have not set up a keyserver yet, sigs are mostly for novelty value.

If you want to fool around with signing your own packages, download the AUR3 reference client (now in bash!) and email me your pubkey. Not the most secure means of establishing trust, but hey.

Arch finally has signed (source) packages.

More on http://aur3.org

muunleit · 2011-02-09 09:45:07

Is it possible to get more information from a query? Like package, "repo", description.
Example:

echo 'nd pacman'| nc aur3.org

pacman abs A library-based package manager with dependency support
pacman-color aur Command-line frontend for libalpm aka pacman with color patch
pacman-mirrorlist abs Arch Linux mirror list for use by pacman

This way a / the client could provide more information faster.

keenerd · 2011-02-09 12:11:50

Eventually. The idea is to have individual queries be simple, easy for a client to parse and fast. (Think unix philosophy.) Right now something on the server is hitting swap and that is killing response time. It'll get better.

Once the bigger problems are sorted out there will be more flexibility.

fauno_ · 2011-02-09 20:36:55

This is awesome! Any plans on releasing the code?

hellnest · 2011-02-09 21:17:57

@keenerd
on your PKGBUILD script, you make a depend on 'gpg' which is doesn't exist. I guess you try to point it to 'gnupg' package??

keenerd · 2011-02-09 23:14:20

fauno: Yes, when it gets a bit more done. I have a handful of core features still to do. Maybe not all of it though. I am a bit hesitant to release the AUR account crawler, for example. Stuff like the extensions to Xyne's pkgbuild parser, definitely.

hellnest: Fixed. Thanks.

muunleit · 2011-02-11 11:20:42

I've added some functions of your example to my ~/.zshrc

my_aur3 () {
    if [[ $1 == [nd] || $1 == "nd" ]]; then
        ztcp aur3.org 1819
        <<< "$1 $2" >&$REPLY
        <&$REPLY
        ztcp -c
    elif [[ $1 == "i" ]]; then
        curl -s "http://aur3.org/rpc/$2"
    elif [[ $1 == "g" ]]; then
        curl -sO "http://aur3.org/mirror/$2/${2}.tar.gz"
        tar xaf ${2}.tar.gz &> /dev/null
        rm ${2}.tar.gz
    else
        print "Unknown option"
    fi
}

Sadly I have no time to write a client.

fauno_ · 2011-02-11 13:09:47

keenerd wrote:

I am a bit hesitant to release the AUR account crawler, for example. Stuff like the extensions to Xyne's pkgbuild parser, definitely.

that means the code won't be released at all, or that it's too soon to show it?

steve___ · 2011-02-11 14:05:08

It's never too soon to show it... release often and early!

Arch Linux

#1 2011-01-31 05:13:18

A new AUR mirror: AUR3

#2 2011-01-31 06:14:46

Re: A new AUR mirror: AUR3

#3 2011-01-31 06:17:26

Re: A new AUR mirror: AUR3

#4 2011-01-31 15:24:52

Re: A new AUR mirror: AUR3

#5 2011-01-31 15:36:17

Re: A new AUR mirror: AUR3

#6 2011-01-31 16:39:55

Re: A new AUR mirror: AUR3

#7 2011-01-31 16:55:53

Re: A new AUR mirror: AUR3

#8 2011-02-01 02:40:34

Re: A new AUR mirror: AUR3

#9 2011-02-01 02:44:13

Re: A new AUR mirror: AUR3

#10 2011-02-01 07:40:37

Re: A new AUR mirror: AUR3

#11 2011-02-01 07:48:28

Re: A new AUR mirror: AUR3

#12 2011-02-01 08:43:16

Re: A new AUR mirror: AUR3

#13 2011-02-02 12:54:39

Re: A new AUR mirror: AUR3

#14 2011-02-02 13:02:50

Re: A new AUR mirror: AUR3

#15 2011-02-03 10:45:25

Re: A new AUR mirror: AUR3

#16 2011-02-04 12:51:07

Re: A new AUR mirror: AUR3

#17 2011-02-08 06:44:06

Re: A new AUR mirror: AUR3

#18 2011-02-09 09:45:07

Re: A new AUR mirror: AUR3

#19 2011-02-09 12:11:50

Re: A new AUR mirror: AUR3

#20 2011-02-09 20:36:55

Re: A new AUR mirror: AUR3

#21 2011-02-09 21:17:57

Re: A new AUR mirror: AUR3

#22 2011-02-09 23:14:20

Re: A new AUR mirror: AUR3

#23 2011-02-11 11:20:42

Re: A new AUR mirror: AUR3

#24 2011-02-11 13:09:47

Re: A new AUR mirror: AUR3

#25 2011-02-11 14:05:08

Re: A new AUR mirror: AUR3

Board footer