You are not logged in.

#1 2021-01-04 01:38:27

victrid
Member
Registered: 2021-01-04
Posts: 3

getting AUR database to perform local query

Hello,

I have observed a difference in the process of getting package information between pacman and AUR package helper. Pacman will download a .db file from the source and initiate a query locally. The AUR package helpers, on the other hand, is making a query request to the AUR page each time.

In my network situation, my school has a mirror of the main repo that downloads and updates at a satisfactory speed, whereas AUR can only request from the main server, which might be in Finland, and there is often a lot of interference with the network communication over the line.

Arch is a rolling update system and I see that people don't recommend partial updates. Many people also do the AUR package update at the same time as the system update. If a dynamic query is post every time, will it put a high load on AUR's servers? AUR and Arch's main repository have the same order of magnitude of the number of packages, and since AUR does not provide pre-compiled binary, the synchronisation pressure should be much less. I think it would be much more server-side and end-user friendly if it could be organised in the same way as Arch's main repository.

Offline

#2 2021-01-04 04:52:28

Awebb
Member
Registered: 2010-05-06
Posts: 5,675

Re: getting AUR database to perform local query

Sounds interesting.

Statistics
Packages	66768
...
Packages added in the past 7 days	283
Packages updated in the past 7 days	1384
Packages updated in the past year	23596
...

If adding a package and updating it both creates a new "state" of the AUR where an AUR mirror can be out-of-date, then there were 238 times per day over the past seven days, where a mirror would have to update itself in order to be not out of date. If we use the "past year" value, it's still 65 times a day. On top of that, we need security. As of now, we have one aurweb instance that stores all the information in an SQL database. You can access it over the website and the RPC interface. How am I going to make sure that my mirror isn't spreading malware?

Offline

#3 2021-01-04 09:11:13

victrid
Member
Registered: 2021-01-04
Posts: 3

Re: getting AUR database to perform local query

Awebb wrote:

If adding a package and updating it both creates a new "state" of the AUR where an AUR mirror can be out-of-date, then there were 238 times per day over the past seven days, where a mirror would have to update itself in order to be not out of date. If we use the "past year" value, it's still 65 times a day.

I don't think synchronisation of mirrors would be a problem. Even if we assume that a PKGBUILD takes up 5KB, with incremental updates, a mirror brings in no more than 5MB of requests a day.

Moreover, this update is at the same frequency as the main repository. 237 packages were updated in total on 3 January 2021. AUR only stores PKGBUILD and not the binary files, which is about the same size as even a package update.

Awebb wrote:

On top of that, we need security.

Malware can be prevented in the same way as the official repository, by signatures. If, as in the case of the official repository mirror only providing the profile database and PKGBUILD, while web service are hosted on the base server, users could easily verify the authenticity of the database by its signature.

Offline

#4 2021-01-04 09:36:28

Awebb
Member
Registered: 2010-05-06
Posts: 5,675

Re: getting AUR database to perform local query

Okay, start hacking. I'm really curious about your approach, maybe this picks up speed.

Offline

#5 2021-01-04 15:41:55

Awebb
Member
Registered: 2010-05-06
Posts: 5,675

Re: getting AUR database to perform local query

To get you started:

The aurweb sourcecode: https://gitlab.archlinux.org/archlinux/aurweb
the AUR web RPC: https://aur.archlinux.org/rpc.php
aurweb RPC Interface wiki article: https://wiki.archlinux.org/index.php/Au … _interface

Offline

#6 2021-01-04 15:58:19

victrid
Member
Registered: 2021-01-04
Posts: 3

Re: getting AUR database to perform local query

Awebb wrote:

To get you started:

The aurweb sourcecode: https://gitlab.archlinux.org/archlinux/aurweb
the AUR web RPC: https://aur.archlinux.org/rpc.php
aurweb RPC Interface wiki article: https://wiki.archlinux.org/index.php/Au … _interface

Thanks, I'll try to implement this once I've finished what I'm doing now.

Offline

Board footer

Powered by FluxBB