You are not logged in.

#1 2011-02-06 20:05:47

tomd123
Developer
Registered: 2008-08-12
Posts: 565

[Announcement] First public git repo of the complete AUR

Hey there Archers, just to reiterate from the aur-general ml (cause I know not everyone reads it)
I decided to work on this little project over the weekend.
It is a complete git clone of the AUR from the source and it will be
updated regularly (at least once a day) though probably as frequent as
30minutes (please see `Side Notes' for explanation).
What this means is that history for the AUR will now be possible!
Also, for those of you wanting to have an abs like tree for all of the
aur, it is now possible.
I hope this opens up new opportunities, and new possibilities for everyone.

To get the complete git repo, run:
git clone git://pkgbuild.com/aur-mirror.git

The web interface is at:
http://pkgbuild.com/git/aur-mirror.git/

Special Thanks to Ioni for enabling the git server & Bluewind for
setting up the web interface.

Cheers!
-Thomas Dziedzic

Side Note:
Although it will be updated at least once a day, it will most likely
be updated more frequently (around 30 minute intervals with a script),
I just can't guarantee that for now.
Work is ongoing to automate the sync with the aur tree, but at this
moment, there is no secure way of doing this.
Automating the sync will guarantee the intervals between syncs, I am
aiming for around 30 minute to 3 hour intervals between syncs.
A final goal might be to get continuous updates. Enjoy! smile

Last edited by tomd123 (2012-01-31 22:04:43)

Offline

#2 2011-02-06 20:48:58

Army
Member
Registered: 2007-12-07
Posts: 1,784

Re: [Announcement] First public git repo of the complete AUR

Just cloned it, pretty cool thing smile

Offline

#3 2011-02-06 21:06:08

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

Army wrote:

Just cloned it, pretty cool thing smile

Yup, what I really love to see though is the git diffs between different commits.
(we finally have history!)

For instance, this is what you get when you run the cleanup script on the aur server: (warning, may take a while to load)
http://pkgbuild.com/git/aur.git/commit/ … 2061db2c34

Last edited by tomd123 (2011-02-06 21:06:21)

Offline

#4 2011-02-06 21:08:24

archman-cro
Member
From: Croatia
Registered: 2010-04-04
Posts: 943
Website

Re: [Announcement] First public git repo of the complete AUR

What! Amazing stuff! Even my useless kernel PKGBUILD is up there! This will definitely add new possibilities to Arch. Thank you. smile

Offline

#5 2011-02-06 21:09:09

litemotiv
Forum Fellow
Registered: 2008-08-01
Posts: 5,026

Re: [Announcement] First public git repo of the complete AUR

Stickied. smile


ᶘ ᵒᴥᵒᶅ

Offline

#6 2011-02-07 14:12:13

skodabenz
Banned
From: Tamilnadu, India
Registered: 2010-04-11
Posts: 382

Re: [Announcement] First public git repo of the complete AUR

Thanks for implementing a version control system for AUR. But this doesn't fulfill the purpose of using a VCS for AUR in the first place. Checking out a 450 MB to be able to maintain custom PKGBUILDs of a few packages is useless IMO. Since git does not allow per directory checkouts like svn does, it would have been better if each package was given its own repo instead of one common repo for the whole AUR tree. The same problem exists with svntogit repo for official repos PKGBUILDs in http://projects.archlinux.org/ . But a single git repo is ok if this is only for viewing purpose.


My new forum user/nick name is "the.ridikulus.rat" .

Offline

#7 2011-02-07 15:54:10

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

skodabenz wrote:

Thanks for implementing a version control system for AUR. But this doesn't fulfill the purpose of using a VCS for AUR in the first place. Checking out a 450 MB to be able to maintain custom PKGBUILDs of a few packages is useless IMO. Since git does not allow per directory checkouts like svn does, it would have been better if each package was given its own repo instead of one common repo for the whole AUR tree. The same problem exists with svntogit repo for official repos PKGBUILDs in http://projects.archlinux.org/ . But a single git repo is ok if this is only for viewing purpose.

Thanks for your concern, I will take a look at possible options later this evening,
#git gave me the following link http://www.kernel.org/pub/software/scm/ … e_checkout

If anything I can always convert to svn tongue

Offline

#8 2011-02-07 16:51:42

skodabenz
Banned
From: Tamilnadu, India
Registered: 2010-04-11
Posts: 382

Re: [Announcement] First public git repo of the complete AUR

git sparse checkout is not exactly what i mentioned. For sparse checkout to work , still the entire 60 MB clone of the repo has to be performed. There is currently no feature in git to do like

git clone git://pkgbuild.com/aur.git/grub2-efi-bzr

where it downloads only part relevant to grub2-efi-bzr instead of the whole 60 MB (or is it 400 MB) like in case of svn. Converting to svn will solve the problem but will not provide the advantages of a distributed VCS. I don't know whether bzr can provide best of both.

What i meant is something like this http://pkgs.fedoraproject.org/gitweb/ (each package has its own repo). Although i have to perform a clone for every package i want, this can be performed easily with a shell script by the user.

Also regarding to the discussion in the ML, storing tarballs in git is not feasible as git is not know to be useful to store binaries. Storing tarballs instead of the extracted test files like PKGBUILDs and the post install scripts will cause the git repo to take up lots of server space in the long run since the diff between tarballs are stored by git instead of diff between text files.

See  http://stackoverflow.com/questions/5405 … s-with-git , http://stackoverflow.com/questions/9847 … git-limits , and http://stackoverflow.com/questions/2266 … nary-files . Thanks for considering my suggestions. I hope something similar is implemented for ABS so that svn usage can be removed altogether.

Last edited by skodabenz (2011-02-07 16:52:18)


My new forum user/nick name is "the.ridikulus.rat" .

Offline

#9 2011-02-08 06:06:03

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

skodabenz wrote:

git sparse checkout is not exactly what i mentioned. For sparse checkout to work , still the entire 60 MB clone of the repo has to be performed. There is currently no feature in git to do like

git clone git://pkgbuild.com/aur.git/grub2-efi-bzr

where it downloads only part relevant to grub2-efi-bzr instead of the whole 60 MB (or is it 400 MB) like in case of svn. Converting to svn will solve the problem but will not provide the advantages of a distributed VCS. I don't know whether bzr can provide best of both.

Well, as one guy put it in #git, it is only the initial 60MB download that is a small fee, but everything after that is free.
Currently I have no short term goals of implementing a per package repo, my task for now is getting per package commits.

After I deploy the per package system, I will look into the possibilities of adding support for per directory repos.
One thing that I haven't had a chance to explore that looks promising is, http://gitslave.sourceforge.net/

Also,
I would like to know what you guys prefer, the digested all in one commits, or would you rather prefer per package commits (stuff I'm working on right now)?

Last edited by tomd123 (2011-02-08 07:19:58)

Offline

#10 2011-02-08 19:55:17

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

Sorry, git repo is down for now.

Last edited by tomd123 (2011-02-08 20:01:32)

Offline

#11 2011-02-09 16:00:04

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

git repo back up

Offline

#12 2011-02-21 10:18:57

sironitomas
Member
From: Cordoba, Argentina
Registered: 2009-11-28
Posts: 174
Website

Re: [Announcement] First public git repo of the complete AUR

Great!

Offline

#13 2011-02-24 04:00:25

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

I have now set everything up in a cron job, so 30 minute intervals should now be guaranteed big_smile enjoy

Offline

#14 2011-02-24 11:10:54

geoisis
Member
From: greece
Registered: 2007-09-02
Posts: 100
Website

Re: [Announcement] First public git repo of the complete AUR

Great Work big_smile


Archlinux x86_64 | post-engineering | last.fm

Offline

#15 2011-02-24 19:24:03

milso
Member
Registered: 2010-11-16
Posts: 112
Website

Re: [Announcement] First public git repo of the complete AUR

This is beautiful, many thanks big_smile

Offline

#16 2011-02-25 06:41:01

mikesd
Member
From: Australia
Registered: 2008-02-01
Posts: 788
Website

Re: [Announcement] First public git repo of the complete AUR

Cloning...

This will make it possible to do all sorts of things locally. Thanks.

Offline

#17 2011-03-26 17:40:35

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

The aur git mirror hasn't been updated for 7 hours because of an error on the server which caused files to be lost. Luckily I pastebinned my update script a while back and managed to find it. I will try to get the aur git mirror updating soon enough.

Offline

#18 2011-03-26 18:21:30

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

aur mirror should be syncing regularly again!

Offline

#19 2011-03-26 19:01:44

linux-ka
Member
From: ADL
Registered: 2010-05-07
Posts: 232

Re: [Announcement] First public git repo of the complete AUR

whoa, great. it is a bit slow, but still !great!. This enhances my wrapper around abs  a lot. nice nice nice :-)

Offline

#20 2011-03-26 20:02:26

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: [Announcement] First public git repo of the complete AUR

tomd123 wrote:

I would like to know what you guys prefer, the digested all in one commits, or would you rather prefer per package commits (stuff I'm working on right now)?

per-package.
I wonder how much that would increase storage requirements, but I think probably not much.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#21 2011-06-22 11:31:10

Xilon
Member
Registered: 2007-01-01
Posts: 243

Re: [Announcement] First public git repo of the complete AUR

skodabenz wrote:

Since git does not allow per directory checkouts like svn does.

Actually it does since 1.7, they're called sparse checkouts; http://www.kernel.org/pub/software/scm/ … -1.7.0.txt. I think it will still grab the whole history, but you can also do a shallow checkout and grab only the first couple of commits (you likely won't need the history).

Offline

#22 2011-06-22 11:51:56

skodabenz
Banned
From: Tamilnadu, India
Registered: 2010-04-11
Posts: 382

Re: [Announcement] First public git repo of the complete AUR

Xilon wrote:
skodabenz wrote:

Since git does not allow per directory checkouts like svn does.

Actually it does since 1.7, they're called sparse checkouts; http://www.kernel.org/pub/software/scm/ … -1.7.0.txt. I think it will still grab the whole history, but you can also do a shallow checkout and grab only the first couple of commits (you likely won't need the history).

This will still take lots of bandwidth space in terms of download. I am not bothered about the amount of space the git repo mya take in my system. I am instead bothered about amount of data downloaded (initially) by "git clone" to create a local copy of the repo.


My new forum user/nick name is "the.ridikulus.rat" .

Offline

#23 2011-07-11 22:39:34

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: [Announcement] First public git repo of the complete AUR

released a quick abs-like tool which maintains a repo under /var/aur (as opposed to /var/abs for abs)

just run sudo aur and it should update/clone the aur git repo:

script at:
https://github.com/gostrc/aur

created a package for it:
aur-git
https://aur.archlinux.org/packages.php?ID=50660

Last edited by tomd123 (2011-07-13 16:17:57)

Offline

#24 2011-07-22 02:30:27

estevao
Member
From: Vitória, ES - Brazil
Registered: 2009-10-13
Posts: 100
Website

Re: [Announcement] First public git repo of the complete AUR

Very nice! Thanks smile

Offline

#25 2011-11-07 21:03:31

moonboots
Member
Registered: 2011-11-07
Posts: 2

Re: [Announcement] First public git repo of the complete AUR

Hey guys, what are your opinions about making each package its own repository and connecting them as submodules under an umbrella project? This would allow someone to clone individual projects, make changes, and submit pull requests to the maintainer if he or she also happens to be using git. This end result would be something like the  vim-scripts account on github.

I'm working on a simple script that separates each subdirectory in the aur repository into its own top level project with git filter-branch --subdirectory-filter. I created an 'aur' user on github and can mass import these into github with their deveoper api.

Last edited by moonboots (2011-11-07 23:22:53)

Offline

Board footer

Powered by FluxBB