[resolved] Build local package repo from http/ftp mirror?

littleweseth · 2007-06-18 13:13:29

G'day;

I live at uni during the semester, with Phat Pipe(tm) - think 100mbps LAN - between me and the nearest archlinux mirror, http://mirror.pacific.net.au/linux/archlinux. However, I have an interesting problem :
a) I'm about to go home for quite some time. Home has dialup. 'nuff said.
b) Home-time is when i get all the ideas about spiffy cool stuff I want to do with my computer, and hence when i need random bits of software. (i don't get the time to play around during uni )
c) Internet here at uni costs $22.50AUD/gb download, excepting certain special servers on the PIPEnetworks peering exchange, which are $0.10/gb. Thus, I am restricted to mirror.pacific.net.au.

My previous solution, when I was an ubuntu user, was to use debmirror to pull down a local mirror of the entire package repository. However, it's quite obvious that Arch Is Not Ubuntu, and there don't seem to be any mirroring tools apart from rsync scripts on the wiki. (These won't work because the only server I can get to cheaply is mirror.pacific.net.au, http/ftp only.)

So, can anyone suggest a way to build a (preferably internally consistent) local package mirror given only http/ftp access? (not sure what happens if I get the mirror halfway through a sync...)

I've contemplated a simplistic python script or similar that recursively pulls down http directory listings and then wgets every.single.file, but this strikes me as immensely retarded and problably prone to some kind of b0rkage. Plus, I suck at programming, and the thing would problably break at 99% done

Any help appreciated, even 'you're a bad person for wanting your own mirror!!' posts.

cheers;
-lws :wq

Last edited by littleweseth (2007-06-20 18:50:52)

byte · 2007-06-18 13:25:11

Have a look at lftp ('mirror'). That will allow you to fetch directories recursively, to fetch only files newer than $foo, exclude stuff like x86_64 and so on.

littleweseth · 2007-06-18 13:41:31

Thanks; in process of checking out. (I knew this was a pre-solved problem - just needed to find $needle )

ty;
-lws :wq

littleweseth · 2007-06-18 14:53:17

I'm trying this;

lftp mirror.pacific.net.au:/linux/archlinux> mirror ./ /mnt/sata/archmirror -x /x86_64/|/iso/|/0\.8/ --dry-run

The intent is to skip over x86_64 packages (because I don't want them), iso's (because i already have them) and the directory /0.8/, which is a symlink to /current/. However, lftp is happily ignoring the regexp i've specified (or i've written it wrong.)

halp?

cheers;
-lws :wq

littleweseth · 2007-06-20 18:22:36

Resolved. I had an error in my patternmatching somewhere : I broke up the regular expression into multiple --exclude= options (because i don't trust | do do proper ORing here, since the RE expressions don't appear to act as egrep says they should. Or i suck at egrep.)

Final command;
lftp mirror.pacific.net.au:/linux/archlinux> mirror --dry-run -v --exclude=x86_64/ --exclude=iso/ --exclude=0.8/ --exclude=0.7.2/ ./ /mnt/sata/archmirror/0.8

(n.b. : obviously, remove the --dry-run. I was using it so i could verify how much i suck at mirror -x without burning precious internet quota. )

Thanks to //byte// for his suggestion of lftp as a tool. It's joining wget and vi in my Arsenal Of Infinite Awesome.

-lws
:wq

Arch Linux

#1 2007-06-18 13:13:29

[resolved] Build local package repo from http/ftp mirror?

#2 2007-06-18 13:25:11

Re: [resolved] Build local package repo from http/ftp mirror?

#3 2007-06-18 13:41:31

Re: [resolved] Build local package repo from http/ftp mirror?

#4 2007-06-18 14:53:17

Re: [resolved] Build local package repo from http/ftp mirror?

#5 2007-06-20 18:22:36

Re: [resolved] Build local package repo from http/ftp mirror?

Board footer