You are not logged in.

#1 2007-06-18 13:13:29

littleweseth
Member
From: James Cook University, Aust
Registered: 2007-06-18
Posts: 16
Website

[resolved] Build local package repo from http/ftp mirror?

G'day;

I live at uni during the semester, with Phat Pipe(tm) - think 100mbps LAN - between me and the nearest archlinux mirror, http://mirror.pacific.net.au/linux/archlinux. However, I have an interesting problem :
a) I'm about to go home for quite some time. Home has dialup. 'nuff said.
b) Home-time is when i get all the ideas about spiffy cool stuff I want to do with my computer, and hence when i need random bits of software. (i don't get the time to play around during uni sad )
c) Internet here at uni costs $22.50AUD/gb download, excepting certain special servers on the PIPEnetworks peering exchange, which are $0.10/gb. Thus, I am restricted to mirror.pacific.net.au.

My previous solution, when I was an ubuntu user, was to use debmirror to pull down a local mirror of the entire package repository. However, it's quite obvious that Arch Is Not Ubuntu, and there don't seem to be any mirroring tools apart from rsync scripts on the wiki. (These won't work because the only server I can get to cheaply is mirror.pacific.net.au, http/ftp only.)

So, can anyone suggest a way to build a (preferably internally consistent) local package mirror given only http/ftp access? (not sure what happens if I get the mirror halfway through a sync...)

I've contemplated a simplistic python script or similar that recursively pulls down http directory listings and then wgets every.single.file, but this strikes me as immensely retarded and problably prone to some kind of b0rkage. Plus, I suck at programming, and the thing would problably break at 99% done smile

Any help appreciated, even 'you're a bad person for wanting your own mirror!!' posts. smile

cheers;
-lws :wq

Last edited by littleweseth (2007-06-20 18:50:52)


"Tandis qu'ils dorment, nous gagnerons."

Offline

#2 2007-06-18 13:25:11

byte
Member
From: Düsseldorf (DE)
Registered: 2006-05-01
Posts: 2,045

Re: [resolved] Build local package repo from http/ftp mirror?

Have a look at lftp ('mirror'). That will allow you to fetch directories recursively, to fetch only files newer than $foo, exclude stuff like x86_64 and so on.


1000

Offline

#3 2007-06-18 13:41:31

littleweseth
Member
From: James Cook University, Aust
Registered: 2007-06-18
Posts: 16
Website

Re: [resolved] Build local package repo from http/ftp mirror?

Thanks; in process of checking out. (I knew this was a pre-solved problem - just needed to find $needle smile )

ty;
-lws :wq


"Tandis qu'ils dorment, nous gagnerons."

Offline

#4 2007-06-18 14:53:17

littleweseth
Member
From: James Cook University, Aust
Registered: 2007-06-18
Posts: 16
Website

Re: [resolved] Build local package repo from http/ftp mirror?

I'm trying this;

lftp mirror.pacific.net.au:/linux/archlinux> mirror ./ /mnt/sata/archmirror -x /x86_64/|/iso/|/0\.8/ --dry-run

The intent is to skip over x86_64 packages (because I don't want them), iso's (because i already have them) and the directory /0.8/, which is a symlink to /current/. However, lftp is happily ignoring the regexp i've specified (or i've written it wrong.)

halp?

cheers;
-lws :wq


"Tandis qu'ils dorment, nous gagnerons."

Offline

#5 2007-06-20 18:22:36

littleweseth
Member
From: James Cook University, Aust
Registered: 2007-06-18
Posts: 16
Website

Re: [resolved] Build local package repo from http/ftp mirror?

Resolved. I had an error in my patternmatching somewhere : I broke up the regular expression into multiple --exclude= options (because i don't trust | do do proper ORing here, since the RE expressions don't appear to act as egrep says they should. Or i suck at egrep.)

Final command;
lftp mirror.pacific.net.au:/linux/archlinux> mirror --dry-run -v --exclude=x86_64/ --exclude=iso/ --exclude=0.8/ --exclude=0.7.2/ ./ /mnt/sata/archmirror/0.8

(n.b. : obviously, remove the --dry-run. I was using it so i could verify how much i suck at mirror -x without burning precious internet quota. smile )

Thanks to //byte// for his suggestion of lftp as a tool. It's joining wget and vi in my Arsenal Of Infinite Awesome. smile

-lws
:wq


"Tandis qu'ils dorment, nous gagnerons."

Offline

Board footer

Powered by FluxBB