You are not logged in.

#1 2013-03-23 04:39:21

theblazehen
Member
Registered: 2012-12-05
Posts: 32

Attempting to write a script to compile each package in the aur.

How did I do?

cd /root;
git clone git://pkgbuild.com/aur-mirror.git;
cd aur-mirror

src="/root/aur-mirror"
IFS=$'\n'

for dir in `ls "$src"`
        do
                if [ -d "$src/$dir" ]; then
                        cd $dir;
                        echo "in $dir"
                        makepkg --asroot -d;
                        cd $src;
                        rm $dir/PKGBUILD; #dont retry failed build
                fi
        done

Offline

#2 2013-03-23 05:44:15

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: Attempting to write a script to compile each package in the aur.

Warning

Naïvely building packages from an untrusted source as root is a very bad idea.

General Questions

Do you understand that anyone can upload malicious code to the AUR?
Do you understand that running makepkg as root can break or infect the entire system? (even blindly running makepkg as a user can do significant damage and lead to data loss)
Are you doing this in a VM?
Why are you doing this?
Do you have any idea how much data you will need to download to build all of these packages?
Do you have the disk space to hold all of these packages?


theblazehen wrote:

How did I do?

It is not clear to me if you meant to ask this question or rather "How do I do it?"

For the former I would say "not so good".
For the latter I would say "you don't".
I may help once I am assured that you know what you are doing, but until then any technical help would likely make me an accomplice to your system's demise.

Offline

#3 2013-03-23 06:30:59

moetunes
Member
From: A comfortable couch
Registered: 2010-10-09
Posts: 1,019

Re: Attempting to write a script to compile each package in the aur.

Xyne wrote:

Lots of sane stuff

My first question is why would you...?

e.g. there's at least a half dozen urxvt's with different patches, why would you want them all?
As practise for coding there are better things to do.


You're just jealous because the voices only talk to me.

Offline

#4 2013-03-23 07:28:48

Awebb
Member
Registered: 2010-05-06
Posts: 4,319

Re: Attempting to write a script to compile each package in the aur.

He's asking how he's doing [so far]. I once tried to do this, just to see if it was possible.

So far, OP, you have the problem, that some AUR packages depend on each other. You will need to parse the dependency array and install the deps from AUR first. You will also run intro trouble, when dependencies conflict with each other, so you probably want to either parse the deps first and let the script generate a hierarchy or, probably for every major package in a sub-hierarchy, decent into a chroot environment.

Or you could have it generate a source repository that works with ABS. This might be the most elegant way, as it reduces the lines of code you'd have to rewrite, because everything is there.

What you really should do, is grab one of the many AUR helpers and feed it its own search result, with a --bug-me-not switch turned on.

Ultimately, I stopped trying this, once I realized, that it is a waste of time and resources – especially the traffic you will create just for fun.

Offline

#5 2013-03-23 10:49:43

theblazehen
Member
Registered: 2012-12-05
Posts: 32

Re: Attempting to write a script to compile each package in the aur.

Xyne wrote:

Do you understand that anyone can upload malicious code to the AUR?

Yes. However I believe most people do not look at the pkgbuild before building a package.

Xyne wrote:

Do you understand that running makepkg as root can break or infect the entire system? (even blindly running makepkg as a user can do significant damage and lead to data loss)

Yes. I was doing this in a chroot on a debian VM. I didn't have anything to loose.

Xyne wrote:

Are you doing this in a VM?

To make it easier to install aur packages, as eventually I want to create a custom repository

Xyne wrote:

Why are you doing this?

To create a custom repository so I can install aur packages from pacman

Xyne wrote:

Do you have any idea how much data you will need to download to build all of these packages?

No, only that it is a large amount.

Xyne wrote:

Do you have the disk space to hold all of these packages?

Not yet

Offline

#6 2013-03-23 11:31:22

theblazehen
Member
Registered: 2012-12-05
Posts: 32

Re: Attempting to write a script to compile each package in the aur.

Awebb wrote:

He's asking how he's doing [so far]. I once tried to do this, just to see if it was possible.

So far, OP, you have the problem, that some AUR packages depend on each other. You will need to parse the dependency array and install the deps from AUR first. You will also run intro trouble, when dependencies conflict with each other, so you probably want to either parse the deps first and let the script generate a hierarchy or, probably for every major package in a sub-hierarchy, decent into a chroot environment.

Or you could have it generate a source repository that works with ABS. This might be the most elegant way, as it reduces the lines of code you'd have to rewrite, because everything is there.

What you really should do, is grab one of the many AUR helpers and feed it its own search result, with a --bug-me-not switch turned on.

Ultimately, I stopped trying this, once I realized, that it is a waste of time and resources – especially the traffic you will create just for fun.

Won't the packages build without problems? Then I create a custom repo with all the files and pacmsn can handle dependancy resolution

Offline

#7 2013-03-23 11:36:09

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,810
Website

Re: Attempting to write a script to compile each package in the aur.

theblazehen wrote:

Won't the packages build without problems? Then I create a custom repo with all the files and pacmsn can handle dependancy resolution

No.

Most dependencies will be required at built time.  My X11 tools for example, require Xlib - they will not compile without it.  The linker cannot link to libraries that don't exist on the system.

PKGBUILDs have dependencies and "build-deps".  The difference is that build-deps are not needed after the package is built - they are both/all needed for the package to be built.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#8 2013-03-24 00:13:32

Awebb
Member
Registered: 2010-05-06
Posts: 4,319

Re: Attempting to write a script to compile each package in the aur.

To talk straight: You probably will not want more than 10% of the AUR in your custom repo, because you will not need them. It is practically impossible to keep track of all the changes and of packages that need a rebuild because of an update of a package in the main repos. You will cause a lot of traffic downloading source code and binary blobs for stuff you don't need. So if this is more than a practice exercise, you will waste not only your time and capacity, but also traffic, which is, opposed to the common image of flat rate internet plans, not free.

Offline

#9 2013-03-24 05:48:08

theblazehen
Member
Registered: 2012-12-05
Posts: 32

Re: Attempting to write a script to compile each package in the aur.

Awebb wrote:

To talk straight: You probably will not want more than 10% of the AUR in your custom repo, because you will not need them. It is practically impossible to keep track of all the changes and of packages that need a rebuild because of an update of a package in the main repos. You will cause a lot of traffic downloading source code and binary blobs for stuff you don't need. So if this is more than a practice exercise, you will waste not only your time and capacity, but also traffic, which is, opposed to the common image of flat rate internet plans, not free.

I plan on making the repository available to the public.

Offline

#10 2013-03-24 15:36:33

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 12,953

Re: Attempting to write a script to compile each package in the aur.

theblazehen wrote:

I plan on making the repository available to the public.

How do you plan on handling key management?
Who is going to sign the packages? Elaborate on the chain of trust?
If you plan on signing the packages with your personal key, are you going to very each package is trustworthy?
What is your plan for complying with copyright on each and every package?  A lot of stuff is in the AUR because of license problems that prohibit repackaging.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Online

#11 2013-03-24 17:20:40

theblazehen
Member
Registered: 2012-12-05
Posts: 32

Re: Attempting to write a script to compile each package in the aur.

Would i need to sign the packages? And as for the legal, this would not be a official repo so arch wont have problems

Offline

#12 2013-03-24 19:29:55

Awebb
Member
Registered: 2010-05-06
Posts: 4,319

Re: Attempting to write a script to compile each package in the aur.

I would indeed limit my efforts to a selection of packages. Maintaining a repository is more than just running makepkg on a bunch of PKGBUILDs and upload the results to a server. Ask around a little, most maintainers will tell you, that providing a stable quality repository with a few packages is enough work to keep one guy busy.

Offline

#13 2013-03-24 19:37:49

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,810
Website

Re: Attempting to write a script to compile each package in the aur.

theblazehen wrote:

And as for the legal, this would not be a official repo so arch wont have problems

Oh, well in that case - as long as it's unofficial - then go right ahead and do something illegal. *sarcasm*


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#14 2013-04-06 03:44:42

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,662

Re: Attempting to write a script to compile each package in the aur.

theblazehen wrote:

Would i need to sign the packages? And as for the legal, this would not be a official repo so arch wont have problems

Well you certainly should not sign them if you don't bother to verify them. But if you don't do that, you could be distributing code which could do literally anything. If you mean it to be of benefit to the public, you have to start by making sure that it will not be a positive menace.

For the record, I never do makepkg without looking at the PKGBUILD (and anything else in that comes with it) unless I've built an earlier version. Then I just read the diff and only look at the PKGBUILD if there are significant changes.

Unfortunately, there are certainly lots of things I would never spot but I figure I can at least see if it looks superficially "normal" and isn't doing say rm -rf /* in the install script.

I also only ever run makepkg as an unprivileged user who does not have access to the files in my own home directory. So makepkg can't run code which will wipe my work, for example.

But maybe I'm just paranoid.


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

#15 2013-04-06 03:46:15

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,662

Re: Attempting to write a script to compile each package in the aur.

Trilby wrote:
theblazehen wrote:

And as for the legal, this would not be a official repo so arch wont have problems

Oh, well in that case - as long as it's unofficial - then go right ahead and do something illegal. *sarcasm*

They don't prosecute people for unofficially breaking the law, surely?


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

Board footer

Powered by FluxBB