Attempting to write a script to compile each package in the aur.

theblazehen · 2013-03-23 04:39:21

How did I do?

cd /root;
git clone git://pkgbuild.com/aur-mirror.git;
cd aur-mirror

src="/root/aur-mirror"
IFS=$'\n'

for dir in `ls "$src"`
        do
                if [ -d "$src/$dir" ]; then
                        cd $dir;
                        echo "in $dir"
                        makepkg --asroot -d;
                        cd $src;
                        rm $dir/PKGBUILD; #dont retry failed build
                fi
        done

Xyne · 2013-03-23 05:44:15

Warning

Naïvely building packages from an untrusted source as root is a very bad idea.

General Questions

Do you understand that anyone can upload malicious code to the AUR?
Do you understand that running makepkg as root can break or infect the entire system? (even blindly running makepkg as a user can do significant damage and lead to data loss)
Are you doing this in a VM?
Why are you doing this?
Do you have any idea how much data you will need to download to build all of these packages?
Do you have the disk space to hold all of these packages?

theblazehen wrote:

How did I do?

It is not clear to me if you meant to ask this question or rather "How do I do it?"

For the former I would say "not so good".
For the latter I would say "you don't".
I may help once I am assured that you know what you are doing, but until then any technical help would likely make me an accomplice to your system's demise.

moetunes · 2013-03-23 06:30:59

Xyne wrote:

Lots of sane stuff

My first question is why would you...?

e.g. there's at least a half dozen urxvt's with different patches, why would you want them all?
As practise for coding there are better things to do.

Awebb · 2013-03-23 07:28:48

He's asking how he's doing [so far]. I once tried to do this, just to see if it was possible.

So far, OP, you have the problem, that some AUR packages depend on each other. You will need to parse the dependency array and install the deps from AUR first. You will also run intro trouble, when dependencies conflict with each other, so you probably want to either parse the deps first and let the script generate a hierarchy or, probably for every major package in a sub-hierarchy, decent into a chroot environment.

Or you could have it generate a source repository that works with ABS. This might be the most elegant way, as it reduces the lines of code you'd have to rewrite, because everything is there.

What you really should do, is grab one of the many AUR helpers and feed it its own search result, with a --bug-me-not switch turned on.

Ultimately, I stopped trying this, once I realized, that it is a waste of time and resources – especially the traffic you will create just for fun.

theblazehen · 2013-03-23 10:49:43

Xyne wrote:

Do you understand that anyone can upload malicious code to the AUR?

Yes. However I believe most people do not look at the pkgbuild before building a package.

Xyne wrote:

Do you understand that running makepkg as root can break or infect the entire system? (even blindly running makepkg as a user can do significant damage and lead to data loss)

Yes. I was doing this in a chroot on a debian VM. I didn't have anything to loose.

Xyne wrote:

Are you doing this in a VM?

To make it easier to install aur packages, as eventually I want to create a custom repository

Xyne wrote:

Why are you doing this?

To create a custom repository so I can install aur packages from pacman

Xyne wrote:

Do you have any idea how much data you will need to download to build all of these packages?

No, only that it is a large amount.

Xyne wrote:

Do you have the disk space to hold all of these packages?

Not yet

theblazehen · 2013-03-23 11:31:22

Awebb wrote:

He's asking how he's doing [so far]. I once tried to do this, just to see if it was possible.
So far, OP, you have the problem, that some AUR packages depend on each other. You will need to parse the dependency array and install the deps from AUR first. You will also run intro trouble, when dependencies conflict with each other, so you probably want to either parse the deps first and let the script generate a hierarchy or, probably for every major package in a sub-hierarchy, decent into a chroot environment.
Or you could have it generate a source repository that works with ABS. This might be the most elegant way, as it reduces the lines of code you'd have to rewrite, because everything is there.
What you really should do, is grab one of the many AUR helpers and feed it its own search result, with a --bug-me-not switch turned on.
Ultimately, I stopped trying this, once I realized, that it is a waste of time and resources – especially the traffic you will create just for fun.

Won't the packages build without problems? Then I create a custom repo with all the files and pacmsn can handle dependancy resolution

Trilby · 2013-03-23 11:36:09

theblazehen wrote:

Won't the packages build without problems? Then I create a custom repo with all the files and pacmsn can handle dependancy resolution

No.

Most dependencies will be required at built time. My X11 tools for example, require Xlib - they will not compile without it. The linker cannot link to libraries that don't exist on the system.

PKGBUILDs have dependencies and "build-deps". The difference is that build-deps are not needed after the package is built - they are both/all needed for the package to be built.

Awebb · 2013-03-24 00:13:32

To talk straight: You probably will not want more than 10% of the AUR in your custom repo, because you will not need them. It is practically impossible to keep track of all the changes and of packages that need a rebuild because of an update of a package in the main repos. You will cause a lot of traffic downloading source code and binary blobs for stuff you don't need. So if this is more than a practice exercise, you will waste not only your time and capacity, but also traffic, which is, opposed to the common image of flat rate internet plans, not free.

theblazehen · 2013-03-24 05:48:08

Awebb wrote:

To talk straight: You probably will not want more than 10% of the AUR in your custom repo, because you will not need them. It is practically impossible to keep track of all the changes and of packages that need a rebuild because of an update of a package in the main repos. You will cause a lot of traffic downloading source code and binary blobs for stuff you don't need. So if this is more than a practice exercise, you will waste not only your time and capacity, but also traffic, which is, opposed to the common image of flat rate internet plans, not free.

I plan on making the repository available to the public.

ewaller · 2013-03-24 15:36:33

theblazehen wrote:

I plan on making the repository available to the public.

How do you plan on handling key management?
Who is going to sign the packages? Elaborate on the chain of trust?
If you plan on signing the packages with your personal key, are you going to very each package is trustworthy?
What is your plan for complying with copyright on each and every package? A lot of stuff is in the AUR because of license problems that prohibit repackaging.

theblazehen · 2013-03-24 17:20:40

Would i need to sign the packages? And as for the legal, this would not be a official repo so arch wont have problems

Awebb · 2013-03-24 19:29:55

I would indeed limit my efforts to a selection of packages. Maintaining a repository is more than just running makepkg on a bunch of PKGBUILDs and upload the results to a server. Ask around a little, most maintainers will tell you, that providing a stable quality repository with a few packages is enough work to keep one guy busy.

Trilby · 2013-03-24 19:37:49

theblazehen wrote:

And as for the legal, this would not be a official repo so arch wont have problems

Oh, well in that case - as long as it's unofficial - then go right ahead and do something illegal. *sarcasm*

cfr · 2013-04-06 03:44:42

theblazehen wrote:

Would i need to sign the packages? And as for the legal, this would not be a official repo so arch wont have problems

Well you certainly should not sign them if you don't bother to verify them. But if you don't do that, you could be distributing code which could do literally anything. If you mean it to be of benefit to the public, you have to start by making sure that it will not be a positive menace.

For the record, I never do makepkg without looking at the PKGBUILD (and anything else in that comes with it) unless I've built an earlier version. Then I just read the diff and only look at the PKGBUILD if there are significant changes.

Unfortunately, there are certainly lots of things I would never spot but I figure I can at least see if it looks superficially "normal" and isn't doing say rm -rf /* in the install script.

I also only ever run makepkg as an unprivileged user who does not have access to the files in my own home directory. So makepkg can't run code which will wipe my work, for example.

But maybe I'm just paranoid.

cfr · 2013-04-06 03:46:15

Trilby wrote:

theblazehen wrote:
And as for the legal, this would not be a official repo so arch wont have problems
Oh, well in that case - as long as it's unofficial - then go right ahead and do something illegal. *sarcasm*

They don't prosecute people for unofficially breaking the law, surely?

Arch Linux

#1 2013-03-23 04:39:21