makerepo: a tool to simplify repository maintenance

Xyne · 2009-01-29 13:31:55

Here's the main info page: http://xyne.archlinux.ca/info/makerepo

Makerepo can automatically update a local repository using local PKGBUILDs/ABS, the AUR, and even CPAN (the latter needs pacpan).

Potential uses:
set up a local repo, build new packages once and distribute them across your LAN or beyond
install abs and use it to compile packages such as the kernel from source
cron it so you don't have to wait for things to compile every time you -Syu
build a comprehensive Arch CPAN mirror that would impress even Larry

I hope most of the info is straight-forward. If not, just ask. Feel free to share feedback and feature requests too.

awesome_welles · 2009-02-01 00:26:18

"set up a local repo," does this mean that if I have a slow internet connection I can download a package only once on one machine and when it is required on others on the LAN it will not need to be downloaded again but instead transferred from the first computer to the second, if not could you please help me find some way of achieving this or point towards some reading material or something. I haven't found anything myself yet.

Xyne · 2009-02-01 13:54:10

You could do that with a local repo, but that would be somewhat of a pain to maintain and you would end up duplicating a lot of data by keeping a copy of all packages on one machine.

After I read your post in the other thread though, I started playing around with some code and ended up writing a daemon to do exactly what you're looking for. Take a look at the thread I've posted and the info page that's linked from it: http://bbs.archlinux.org/viewtopic.php?id=64391

fukawi2 · 2009-09-04 01:06:56

I get these errors when I run:

Use of uninitialized value $PACPAN in scalar chomp at /usr/bin/makerepo line 64, <GEN6> line 1.
Use of uninitialized value $PBGET in scalar chomp at /usr/bin/makerepo line 66, <GEN6> line 2.

Obviously because I haven't got pacpan and pbget installed, although after I installed pbget, it still borked. Both these lines are looking for pacpan - that that correct?

It also doesn't appear to be handling comments correctly:

root@platypus /srv/http/sites/repo.falconn.nl/any # makerepo
--> searching for # in AUR

Last edited by fukawi2 (2009-09-04 01:09:59)

Xyne · 2009-09-04 01:55:14

The pacpan/pbget check should work now. I'll double-check the way it handles comments.

It's probably worth mentioning that at some point I refactored makerepo twice. I wrote it in Perl originally, then rewrote it in bash to try to use my bash-xyne-common_functions but when I realized that bash is too slow, I rewrote it in Perl again. It seems some bugs snuck in.

*edit*
My local source file contained the lines in the parsing function to skip comments but for some reason they were absent in the release. I have no idea how that happened but it should work now.

Last edited by Xyne (2009-09-04 02:15:44)

fukawi2 · 2009-09-04 02:34:54

Thanks mate!

I'll update it and test as soon as I finish fiddling with my DNS and Virtual Hosts

Last edited by fukawi2 (2009-09-04 02:35:07)

fukawi2 · 2009-09-04 08:33:25

All working good now it seems

fukawi2 · 2009-09-04 08:54:55

I stand corrected:

fukawi2@kangaroo ~/i686  $ makerepo
--> checking nagios-nrpe... not found
...
fukawi2@kangaroo ~/i686  $ yaourt nagios-nrpe
1 aur/nagios-nrpe 2.12-6 (6)
    Daemon to answer plugin requests from Nagios on remote hosts.

Xyne · 2009-09-04 09:09:45

It would help if you posted the REPOBUILD. Also, does "pbget nagios-nrpe --aur-only" work for you?

fukawi2 · 2009-09-04 09:16:34

It seems to work as root, but not as a normal user - is that expected?

Xyne wrote:

Also, does "pbget nagios-nrpe --aur-only" work for you?

Yes.

REPOBUILD:

%DATABASE%
stable32.db.tar.gz

%BUILD_DIR%
/srv/repo/BUILD

%AUR_PKGS%
nagios-nrpe

%BINARY_PKGS%
cyrus-sasl-plugins

%LOCAL_PKGS%
/srv/repo/abs/courier-authlib
/srv/repo/abs/courier-imap
/srv/repo/abs/cyrus-sasl
/srv/repo/abs/iptables
/srv/repo/abs/pam_pgsql
/srv/repo/abs/postfix
/srv/repo/abs/postfix-relay
/srv/repo/abs/postgresql

%IGNORE_VERSION%

%MAKEPKG_ARGS%
--asroot

Last edited by fukawi2 (2009-09-04 09:17:02)

Xyne · 2009-09-04 09:27:32

It works when run as a normal user here. I have no problems updating nagios-nrpe with the same line in a REPOBUILD.

Does this work:

#!/usr/bin/perl

use strict;
use warnings;
use LWP::Simple qw/get/;

my $AUR_RPC='http://aur.archlinux.org/rpc.php';

sub get_aur_version
{
  my $json = get($AUR_RPC.'?type=info&arg='.$_[0]);
  return "failed to retrieve JSON data\n" if not defined($json);
  return ($json =~ m/"Version":"(.+?)"/) ? "detected version: $1\n" : "failed to parse version information in the following JSON data:\n$json\n";
}

print &get_aur_version(@ARGV);

Pass it "nagios-nrpe" as its first argument to see if it detects the version. That seems to be the part that's failing in the makerepo code. Maybe there was a transient problem connecting to rpc page when you ran it as a normal user.

fukawi2 · 2009-09-04 09:33:25

fukawi2@kangaroo /tmp  $ ./xyne-test.py nagios-nrpe
failed to retrieve JSON data

I have a proxy - will that matter? http_proxy is set correctly (yaourt and pacman respect it):

fukawi2@kangaroo /tmp  $ echo $http_proxy
proxy.fukawi2.local:3128

I don't think it's transient - I had it a few hours ago, and then just before, plus it works for root.

Does it matter that I've already built the repo once, then added nagios-nrpe as an after-thought?

Xyne · 2009-09-04 09:45:33

You should be able to add packages whenever you want. It could be an issue with the proxy but in that case it's due to something in LWP::Simple not respecting it. I don't actually know what do to about it.

If you can confirm that the miniscript works as root but not as a normal user then it's definitely an issue with LWP::Simple in which case I expect that this has come up before.

*edit*
Try this.

#!/usr/bin/perl

use strict;
use warnings;
use LWP::Simple qw/$ua get/;

if ( defined($ENV{'http_proxy'}) )
{
  print "detected proxy: ", $ENV{'http_proxy'}, "\n";
  $ua->proxy('http',$ENV{'http_proxy'});
}

my $AUR_RPC='http://aur.archlinux.org/rpc.php';

sub get_aur_version
{
  my $json = get($AUR_RPC.'?type=info&arg='.$_[0]);
  return "failed to retrieve JSON data\n" if not defined($json);
  return ($json =~ m/"Version":"(.+?)"/) ? "detected version: $1\n" : "failed to parse version information in the following JSON data:\n$json\n";
}

print &get_aur_version(@ARGV);

LWP::Simple should detect the proxy automatically it seems. Are you sure that you've properly exported the variable?

Last edited by Xyne (2009-09-04 09:51:41)

fukawi2 · 2009-09-04 09:52:15

Hmmm, fail

fukawi2@kangaroo /tmp  $ sudo ./xyne-test.py nagios-nrpe
detected version: 2.12-6

Oh well, I'll just have to do it as root, no biggie

Vrob · 2009-12-03 18:14:03

I have two questions/remarks.

I'm not sure if the first one fits into makerepos scope, but I find removing packages that I don't wan't to be built and served in the repo somewhat circumstantial.
Frirst of all, I delete the corresponding line in my REPOBUILD, afterwards I optionally delete the build dir/PKGBUILD-dir (depending on whether I plan to add it again later).
Then I have to care about consistency in my repository, so I will do a repo-remove and delete the .pkg.tar.gz - files by hand.
It would be really cool and timesaving if makerepo would do all repo-management stuff after removal of the corresponding line in the REPOBUILD. Maybe make it optional, so people who want to keep the binary packages but not the PKGBUILDs are not upset

The other issue is whether it would be possible to introduce some kind of "force rebuild" switch. Right now, I have a set of git, svn and so on packages that I want to build regularily on my desktop and then share them over LAN to other boxes. Since PKGBUILDs for Version Control System based packages don't change often, makerepo will not rebuild the package after it has done so once. So it would be great, if I could just put all my VCS packages into one seperate REPOBUILD and do something like

makerepo --force-build REPOBUILD-VCS

.

Greets,
Vrob

Xyne · 2009-12-03 18:34:53

@Vrob
I would need to double-check the code to be absolutely sure, but I believe the repo gets rebuilt from scratch each time something changes so you do not need to run "repo-remove" after removing something from the repobuild. I might add an option to remove binary packages that are not currently included in the database, but I'm reluctant to do so as it may lead to issues when hosting multiple repos in the same directory. I don't really think it's difficult for a user to (re)move packages manually so I'm in no hurry to do this.

As for the second issue, please read through the comments and sections of the REPOBUILD:

REPOBUILD wrote:

%IGNORE_VERSION%
# Uncommenting a line in this section will cause makerepo to skip version
# comparisons and update all packages. This can be desirable when building
# developmental versions.
#true

That option has always been there.

Last edited by Xyne (2009-12-03 18:37:54)

Xyne · 2009-12-22 20:11:06

Paths in the LOCAL_PKGS section of the REPOBUILD may now end in "/*". makerepo will then include all subdirectories in that directory which contain a PKGBUILD.

Last edited by Xyne (2009-12-22 22:19:40)

linopolus · 2009-12-31 09:55:51

I got

/bin/bash: Zeile 245: cd: /home/linopolus/repo.aur: Datei oder Verzeichnis nicht gefunden

(file or dir not found in english) after many ours of compiling aur packages so I created the directory and ran it again and now it worked. But so I had to compile all twice. It would be nice if makerepo would create the dir in this case or ask to do that.

Xyne · 2010-01-07 00:48:36

Which dir was it and at what stage of the build did it occur?

I'm too tired to code right now but I'll try to remember to add checks for directory existence. Sorry about the lost compilation time.

linopolus · 2010-01-30 11:12:20

It would be great if you can add an option to build only one package

Arch Linux

#1 2009-01-29 13:31:55

makerepo: a tool to simplify repository maintenance

#2 2009-02-01 00:26:18

Re: makerepo: a tool to simplify repository maintenance

#3 2009-02-01 13:54:10

Re: makerepo: a tool to simplify repository maintenance

#4 2009-09-04 01:06:56

Re: makerepo: a tool to simplify repository maintenance

#5 2009-09-04 01:55:14

Re: makerepo: a tool to simplify repository maintenance

#6 2009-09-04 02:34:54

Re: makerepo: a tool to simplify repository maintenance

#7 2009-09-04 08:33:25

Re: makerepo: a tool to simplify repository maintenance

#8 2009-09-04 08:54:55

Re: makerepo: a tool to simplify repository maintenance

#9 2009-09-04 09:09:45

Re: makerepo: a tool to simplify repository maintenance

#10 2009-09-04 09:16:34

Re: makerepo: a tool to simplify repository maintenance

#11 2009-09-04 09:27:32

Re: makerepo: a tool to simplify repository maintenance

#12 2009-09-04 09:33:25

Re: makerepo: a tool to simplify repository maintenance

#13 2009-09-04 09:45:33

Re: makerepo: a tool to simplify repository maintenance

#14 2009-09-04 09:52:15

Re: makerepo: a tool to simplify repository maintenance

#15 2009-12-03 18:14:03

Re: makerepo: a tool to simplify repository maintenance

#16 2009-12-03 18:34:53

Re: makerepo: a tool to simplify repository maintenance

#17 2009-12-22 20:11:06

Re: makerepo: a tool to simplify repository maintenance

#18 2009-12-31 09:55:51

Re: makerepo: a tool to simplify repository maintenance

#19 2010-01-07 00:48:36

Re: makerepo: a tool to simplify repository maintenance

#20 2010-01-30 11:12:20

Re: makerepo: a tool to simplify repository maintenance

Board footer