Arch Linux

bboozzoo

Member

From: Poland

Registered: 2006-08-01

Posts: 127

viadown - on demand mirroring tool

I have a number of computers that run Arch and I want to share the upgrade download cache between those machines with minimal effort. I also have at least 2 other devices that are always on (or almost always on), namely a WiFi outer and a NAS (both ARM CPUs). There's a number of solutions one could come up with, like a shared NFS/CIFS mount, or a proxy (though there's some hassle to get pacman work with proxy), or just rsync packages from the PC that was upgraded most recently back to the cache and export everything over HTTP.

Unfortunately I'm too lazy to try any of these as each one seems a bit over the top effort wise. IMO the best thing I could get is some tool to mirror/cache package downloads in an on-demand fashion. Something I could easily deploy on one of the always-on devices. Something that I could just setup as a mirror in my mirrorlist and let pacman use it while the tool does the rest.

I'm announcing `viadown` - a tool for on demand mirroring. Grab it here: https://github.com/bboozzoo/viadown/

The tool is a kind of a proxy (but not implementing the actual PROXY protocol) that forwards GET requests to the upstream server. When a response with status 200 (OK) comes in, it will serve the body to the original requesting party (pacman in this case) and cache it locally. Next time someone asks for this resource, it will be served directly from the cache. Requests that set `If-Modified-Since` header, skip the cache and are directly forwarded to the upstream. This handles a case when the client believes that the resource is available under the same name, but it's contents may have changed. Since we're not a real mirror but a kind of a proxying one, it's best to just forward the request upstream. `Range` requests are not supported, but as pacman uses those for downloads that are already partially done, adding support is on a TODO list (maybe someone wants to jump in and contribute?).

Getting it is rather easy (although it assumes that one has a working $GOPATH setup already):

# pulls the code, pulls dependencies, builds and installs into $GOPATH/bin
go get -u github.com/bboozzoo/viadown

Rebuilding for ARM is rather trivial:

GOARCH=arm GOARM=<your-ARM-cpu-v> go build -v

I have added viadown as one of the mirrors to pacman's mirrorlist (/etc/pacman.d/mirrorlist). Seems to work fine so far and covers the case when I'm not connected to my home network.

Cached files are kept forever for now. So it's best if you setup a cron job to clean up older files manually.

joseheitor

Member

Registered: 2025-02-18

Posts: 5

Re: viadown - on demand mirroring tool

Hi,

When I execute the command:
go generate github.com/bboozzoo/viadown/...

I get the following error and can go no further:
assets/generate.go:3: running "esc": exec: "esc": executable file not found in $PATH

I am not a Go developer, so do not know how to proceed - please help?

Thanks,
Jose

bboozzoo

Member

From: Poland

Registered: 2006-08-01

Posts: 127

Re: viadown - on demand mirroring tool

Hi,

When I execute the command:
go generate github.com/bboozzoo/viadown/...

I get the following error and can go no further:
assets/generate.go:3: running "esc": exec: "esc": executable file not found in $PATH

I am not a Go developer, so do not know how to proceed - please help?

oh wow, it's been years since I last used that. I believe you may need to install esc first. It's a helper library and tool to embed binary stuff in a binary. These days it's in Go standard library, but back in 2016 one had to use external packages for that. Try to install esc by running

go install github.com/maddyblue/esc@latest

and make sure it's in your $PATH.

joseheitor

Member

Registered: 2025-02-18

Posts: 5

Re: viadown - on demand mirroring tool

Hi,

Thanks for the prompt assistance - it worked!

I am now getting the following error when launching as follows:

./viadown -cache-root "./repo-cache" -listen ":9999" pidfile "viadown.pid" -syslog -debug -mirrors "mirrorlist"
ERRO[0000] no mirrors, cannot continue

What am I doing wrong?

Thanks,
Jose

bboozzoo

Member

From: Poland

Registered: 2006-08-01

Posts: 127

Re: viadown - on demand mirroring tool

Hi,

Thanks for the prompt assistance - it worked!

I am now getting the following error when launching as follows:

./viadown -cache-root "./repo-cache" -listen ":9999" pidfile "viadown.pid" -syslog -debug -mirrors "mirrorlist"
ERRO[0000] no mirrors, cannot continue

What am I doing wrong?

Thanks,
Jose

do you have any mirrors listed in mirrorlist file?

joseheitor

Member

Registered: 2025-02-18

Posts: 5

Re: viadown - on demand mirroring tool

Yes - a private mirror of the stable branch of the Manjaro repo.

joseheitor

Member

Registered: 2025-02-18

Posts: 5

Re: viadown - on demand mirroring tool

Hooray!

I have it working with the modified command:

./viadown -cache-root "./repo-cache" -listen ":9999" -mirrors "./mirrorlist" pidfile "./viadown.pid" -syslog -debug

and the following setup:

Viadown mirrorlist: http://10.0.0.200:8888/
/etc/pacmand.d/mirrorlist: http://localhost:9999/manjaro/stable/$repo/$arch

Last edited by joseheitor (2025-02-19 09:29:12)

joseheitor

Member

Registered: 2025-02-18

Posts: 5

Re: viadown - on demand mirroring tool

Can you please explain what the cache-purge does?

Last edited by joseheitor (2025-02-19 18:24:15)