rucksack - unify a local and remote pacman pkg cache

dtw · 2006-08-09 09:36:05

Anyone that runs Arch on more than one machine probably rues the time and bandwidth wasted by making the same basic updates several times.

Therefore I have created rucksack, a simple script that uses unionfs to mount a local and remote cache together on "client" machines. If, like me, you have one box that you update more often than others you can share your pacman pkg cache using NFS and access those pkgs across your network.

I'm sure some people already use a R/W NFS to share a single cache, but rucksack keeps the remote cache RO, while unionfs allows you to store pkgs used only by the local machine in a seamless cache.

For example, my girlfriend uses xfce-svn on her laptop. Using rucksack she can share my basic updates but I don't have to have all the xfce-svn pkgs cached on my machine. Also, because the remote cache is read-only, there are less security issues.

I hope you can all see the benefits.

I need to write a wiki page about how to set up an NFS server and clients, and then implement rucksack but after set-up it is as simple as listing daemons in rc.conf.

I have written and tested the script and it works perfectly.

However, I'm in about four minds about implementation. I'm undecided as to whether it should be focused entirely on an NFS mount, which narrows down error and sanity checking massively, or if it should provide options for other remote mounts (are there any?). I could also hardcode local paths for mounting caches, and even provide options in the config file for defining the NFS mount, rather than relying on fstab. Of these two the former is probably a good idea, the latter probably not, but I could use some thoughts!

stavrosg · 2006-08-09 13:32:59

Other possible mount you could use is smbfs / cifs. It is stable, reasonably fast and easy to set up.

Could ftp be used, too?

postlogic · 2006-08-09 16:33:17

Or how about even sshfs? It may not be as fast as other alternatives, but it's very, very nice.

wain · 2006-08-09 16:36:23

+1 for sshfs

dtw · 2006-08-09 20:51:57

Cool, well as it stands you can use any of them. Are they all configured in fstab?

dtw · 2006-08-15 14:02:32

I made a wiki page

Note: rucksack is not in [community] yet, please see the links in my signature for access to my repo where you can test rucksack. I have tested it extensively tonight and it seems to work perfectly

NitE · 2006-08-17 10:31:41

Wow, this is exactly what I was looking for. I was actually going to code something simular to this. As it stands now, I am currently manually uploading pacman's cache across several machine.

Will have to give this a go tonight.

dtw · 2006-08-17 10:52:36

Cool! It'll be great to have someone test it.

There are a few, somewhat obvious, caveats to usage:
· depending on your NFS setup bringing down the cache host can hang the clients
· if you do really big updates (50Mb+) on a slow network, e.g. 54Mbps or less, the package integrity check can be pretty slow. If you have a 100Mbps+ network though you should find it pretty speedy!Make sure you read the wiki and the NFS howtos that are linked. The initial setup is not something you should rush through. Let me know how it goes

Galdona · 2006-08-17 11:19:26

great! this is very useful. i will try it..

Romashka · 2006-08-18 08:09:46

There is Howto Upgrade via Home Network - a better approach (didn't have time to write it more nicely) which describes almost the same solution.
Using unionfs is nice idea. Hovewer I use shared package cache for all machines, and no local cache.
The script is nice, and wiki page is more clean then mine. Nice work, dtw!

dtw · 2006-08-18 08:12:30

Romashka wrote:

There is Howto Upgrade via Home Network - a better approach (didn't have time to write it more nicely) which describes almost the same solution.
Using unionfs is nice idea. Hovewer I use shared package cache for all machines, and no local cache.
The script is nice, and wiki page is more clean then mine. Nice work, dtw!

I imagine that solution is suitable to almost everyone to be honest, having a read-write cache is hardly a bad thing if you are behind a good firewall I still prefer not to clutter one machine with all sorts of packages from the others, I think.

Galdona · 2006-08-18 08:36:54

Ok, I have set up rucksack and it is now running as daemon. Um, then what? Do I just run pacman as usual?

Galdona · 2006-08-18 08:40:27

Oh, and what if the machine with the remote cache is not available? What happens?

Romashka · 2006-08-18 08:47:36

I updated and extended my howto. Feel free to comment and improve it.

Romashka · 2006-08-18 08:54:02

Galdona wrote:

Oh, and what if the machine with the remote cache is not available? What happens?

I didn't checked Rucksack but when setting up shared cache as described in my howto - it will work when the machine with the remote cache is not available.
If you want to install some package when remote cache is unavailable - just do pacman -Syu pkgname.

dtw · 2006-08-18 09:10:42

Galdona wrote:

Oh, and what if the machine with the remote cache is not available? What happens?

They way you have to look at it is that rucksack requires that remote cache. What should happen is that if the remote cache is not available at startup then rucksack should fail and /var/cache/pacman/pkg will be used as the cache as normal. What that means though is that any files downloaded during that time will not be available if/when rucksack works and mounts the combined cache over /var/cache/pacman/pkgs. You would need to copy them to /var/cache/pacman/local.

If you bring the remote cache host down while rucksack is running on a client then you have problems. This isn't a limitation of rucksack per se but of shared networrk FS in general. If you're connected it has to stay up. Does that help?

If you want to be sure then don't run rucksack as a daemon, just start it, update the system and stop it.

And to answer your other question, yes, you just run pacman -Syu on clients as normal.

Looking at what Romanshka has written rucksack should work just as well with sshfs as it does with NFS - you just need to have the mount properly setup on the host and listed in fstab correctly on the client.

Arch Linux

#1 2006-08-09 09:36:05

rucksack - unify a local and remote pacman pkg cache

#2 2006-08-09 13:32:59

Re: rucksack - unify a local and remote pacman pkg cache

#3 2006-08-09 16:33:17

Re: rucksack - unify a local and remote pacman pkg cache

#4 2006-08-09 16:36:23

Re: rucksack - unify a local and remote pacman pkg cache

#5 2006-08-09 20:51:57

Re: rucksack - unify a local and remote pacman pkg cache

#6 2006-08-15 14:02:32

Re: rucksack - unify a local and remote pacman pkg cache

#7 2006-08-17 10:31:41

Re: rucksack - unify a local and remote pacman pkg cache

#8 2006-08-17 10:52:36

Re: rucksack - unify a local and remote pacman pkg cache

#9 2006-08-17 11:19:26

Re: rucksack - unify a local and remote pacman pkg cache

#10 2006-08-18 08:09:46

Re: rucksack - unify a local and remote pacman pkg cache

#11 2006-08-18 08:12:30

Re: rucksack - unify a local and remote pacman pkg cache

#12 2006-08-18 08:36:54

Re: rucksack - unify a local and remote pacman pkg cache

#13 2006-08-18 08:40:27

Re: rucksack - unify a local and remote pacman pkg cache

#14 2006-08-18 08:47:36

Re: rucksack - unify a local and remote pacman pkg cache

#15 2006-08-18 08:54:02

Re: rucksack - unify a local and remote pacman pkg cache

#16 2006-08-18 09:10:42

Re: rucksack - unify a local and remote pacman pkg cache

Board footer