You are not logged in.

#1 2016-10-23 13:38:58

bboozzoo
Member
From: Poland
Registered: 2006-08-01
Posts: 125

viadown - on demand mirroring tool

I have a number of computers that run Arch and I want to share the upgrade download cache between those machines with minimal effort. I also have at least 2 other devices that are always on (or almost always on), namely a WiFi outer and a NAS (both ARM CPUs). There's a number of solutions one could come up with, like a shared NFS/CIFS mount, or a proxy (though there's some hassle to get pacman work with proxy), or just rsync packages from the PC that was upgraded most recently back to the cache and export everything over HTTP.

Unfortunately I'm too lazy to try any of these as each one seems a bit over the top effort wise. IMO the best thing I could get is some tool to mirror/cache package downloads in an on-demand fashion. Something I could easily deploy on one of the always-on devices. Something that I could just setup as a mirror in my mirrorlist and let pacman use it while the tool does the rest.

I'm announcing `viadown` - a tool for on demand mirroring. Grab it here: https://github.com/bboozzoo/viadown/

The tool is a kind of a proxy (but not implementing the actual PROXY protocol) that forwards GET requests to the upstream server. When a response with status 200 (OK) comes in, it will serve the body to the original requesting party (pacman in this case) and cache it locally. Next time someone asks for this resource, it will be served directly from the cache. Requests that set `If-Modified-Since` header, skip the cache and are directly forwarded to the upstream. This handles a case when the client believes that the resource is available under the same name, but it's contents may have changed. Since we're not a real mirror but a kind of a proxying one, it's best to just forward the request upstream. `Range` requests are not supported, but as pacman uses those for downloads that are already partially done, adding support is on a TODO list (maybe someone wants to jump in and contribute?).

Getting it is rather easy (although it assumes that one has a working $GOPATH setup already):

# pulls the code, pulls dependencies, builds and installs into $GOPATH/bin
go get -u github.com/bboozzoo/viadown

Rebuilding for ARM is rather trivial:

GOARCH=arm GOARM=<your-ARM-cpu-v> go build -v

I have added viadown as one of the mirrors to pacman's mirrorlist (/etc/pacman.d/mirrorlist). Seems to work fine so far and covers the case when I'm not connected to my home network.

Cached files are kept forever for now. So it's best if you setup a cron job to clean up older files manually.

Offline

Board footer

Powered by FluxBB