You are not logged in.

#1 2013-07-16 05:10:02

ftravers
Member
Registered: 2013-07-16
Posts: 5

Speed up install...use another computer's packages.

My goal is to speed up install on a computer.  Waiting for the 'pacstrap' command to download all the packages is too slow for this.

I'm trying to leverage the fact that I've got other arch computers already on my network and would like to (if possible) leverage their pacman cache/db.  So far I've NFS shared the them like so:

    # mkdir -p /mnt/var/lib/pacman/sync
    # mkdir -p /mnt/var/cache/pacman/pkg

    # mount -t nfs4 192.168.1.44:pacman-db /mnt/var/lib/pacman/sync
    # mount -t nfs4 192.168.1.44:pacman-cache /mnt/var/cache/pacman/pkg

which works fine.  However, running:

    # pacstrap -i /mnt base base-devel

just tries to download all the packages from the internet again.

I was following the instructions from:

https://wiki.archlinux.org/index.php/Pa … rite_cache

Any suggestions very much appreciated.

Offline

#2 2013-07-16 05:37:21

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: Speed up install...use another computer's packages.

Xyne has a tool that handles this.  I ma fairly certain there is a thread around here for it somewhere... here it is!  It is called pacserve.  I haven't actually tried it, but others seem to have had success with it.

Offline

#3 2013-07-16 12:58:04

ftravers
Member
Registered: 2013-07-16
Posts: 5

Re: Speed up install...use another computer's packages.

pacserve isn't a good option for installation, since you'd have to get pacserve onto the live cd usb in the first place, which requires yaourt, which requires binutils, which requires a c compiler, which requires..., well you get the point...

i wonder about doing an install, then rsync backing it up, then using that for the base of an install

btw: it took 10 minutes to do a base and base-devel install with an internet connection that speedtest.net's at 10M download from Vientienne, Laos, to Bangkok, Thailand.

Offline

#4 2013-07-16 12:59:59

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Speed up install...use another computer's packages.

Offline

#5 2013-07-16 13:13:39

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: Speed up install...use another computer's packages.

Export the pkg directory of the installed system with nfs, and mount it to /mnt/var/cache/pacman/pkg in the installer (after mounting the / partition to /mnt, etc). I've done this multiple times.

Edit: Facepalm! I completely missed that you're already doing this... well, I've never shared the sync dir, maybe that's the problem.

ftravers wrote:

acserve isn't a good option for installation, since you'd have to get pacserve onto the live cd usb in the first place, which requires yaourt, which requires binutils, which requires a c compiler, which requires..., well you get the point...

Last edited by alphaniner (2013-07-16 14:14:32)


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#6 2013-07-16 13:42:25

tomk
Forum Fellow
From: Ireland
Registered: 2004-07-21
Posts: 9,839

Re: Speed up install...use another computer's packages.

ftravers wrote:

However, running:

    # pacstrap -i /mnt base base-devel

just tries to download all the packages from the internet again.

Are you sure the versions in your shared cache are up to date? pacstrap uses 'pacman -Sy', which means it will look for the current versions of everything.

Offline

#7 2013-07-16 13:43:31

drcouzelis
Member
From: Connecticut, USA
Registered: 2009-11-09
Posts: 4,092
Website

Re: Speed up install...use another computer's packages.

Why does every response in this thread seem incredibly complicated? yikes

I think I did something like what you want to do... I have Arch Linux installed on /dev/sda1 and I wanted to install another copy on /dev/sda6. So I started installation like normal, and just before it started downloading packages I copied "/var/cache/pacman/pkg/*" from my original installation to the new installation. The only pacman needs to do then is verify that the checksums and look for updates.

Does that not work in this scenario?

Offline

#8 2013-07-16 14:05:12

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: Speed up install...use another computer's packages.

In a situation like your drcouzelis, you could also have simply mounted the partition holding /var/cache/pacman/pkg to somewhere else (like /media for example), and then bind mounted /media/var/cache/pacman/pkg.

I set my pacman cache up as its own subvolume in btrfs, so I can share it as I please amongst my system.

Offline

#9 2013-07-16 14:46:16

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Speed up install...use another computer's packages.

ftravers wrote:

pacserve isn't a good option for installation, since you'd have to get pacserve onto the live cd usb in the first place, which requires yaourt, which requires binutils, which requires a c compiler, which requires..., well you get the point...

A few points:

  • You do not need yaourt to install AUR packages.

  • All of my packages are available in my repo, so you don't even need to build them (and even if you did, most of them do not even need to be "built".

  • Pacman and makepkg should be available on the installation medium if you actually need to build some packages.

  • Most importantly, you don't even need to have pacserve on the local computer to use it. Just point pacman to one of the local pacserver's and it will do its magic during installation. After that, when the system is up, it can be installed locally to share the local computer's cache.

As for the other comment in this thread about complexity, installing and running Pacserve is very easy and does exactly what the OP wants.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#10 2013-07-17 08:14:17

ftravers
Member
Registered: 2013-07-16
Posts: 5

Re: Speed up install...use another computer's packages.

In the end I did the following:

1 - Ensure on my old computer my pacman cache only had the latest versions with:

pacman -Sc

2 - Copy this folder over to the new computer

This shaved my install time down from 10 minutes to 3 minutes.  Xyne's solution looks nice too, so eventually I may try the 'pacserve' option. 

BTW:

1 - I didn't know you could get away without using 'pacserve' on the client, that seems like a very good feature to have.
2 - It would be an error to share the pacman DB, as that is the source of truth of what is installed on a given computer, whereas the cache is just a dump of files.

Thanks all for your comments.

Offline

#11 2013-07-17 09:02:09

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,041

Re: Speed up install...use another computer's packages.

Truly without any snark, I'm curious: how much time did you spend trying to get this to work, compared to how much time you actually saved?

(edit: and by "without any snark", I mean, I understand that sometimes we do things like this just to see if we can, even if we obliterate any possible gains in efficiency by spending time on figuring out how to do it)

Last edited by jakobcreutzfeldt (2013-07-17 09:03:24)

Offline

#12 2013-07-17 09:37:15

ftravers
Member
Registered: 2013-07-16
Posts: 5

Re: Speed up install...use another computer's packages.

Time wasn't the real gain, understanding was.  I learned enough about pacman to figure out what the right thing to do was.  It took days.  It took frustration on IRC (for others as well as me).  But in the end, I'd say that this is (should be) the real goal of Arch...to learn better how the system works.  You can't imagine where this need came from!

1 - I was trying to get my desktop computer to work with 'Wake on Lan', so I didn't have to have it running all the time.
2 - It was working only for about 10 minutes after shutdown.  So I thought I should try to upgrade the BIOS.
3 - BIOS tools are mostly only available for windows!
4 - I thought if I could boot into FreeDOS then maybe I could get the BIOS updater to work.
5 - This led me to wanting to setup my partitions better...having one dedicated to GPartEd, one for FreeDOS.
6 - This led me to wanting to use GRUB and GPT over syslinux.
7 - This led to multiple screw ups and I wanted to reduce the time it took to do a full re-install, so I started to investigate how to reduce the 'pacstrap' time.

Well you know the rest of the story more or less.  Eventually I started a wiki page: "How Pacman works": https://wiki.archlinux.org/index.php/Ta … cman_works

There are comments like: this is just a subset of what is in 'man pacman', and you get the usual comments in IRC that are fairly disheartening.  Regardless, I think it's important to document at the right level of abstraction.  'man pacman' gives you how to use everything about pacman, which isn't really the same as 'what are the operating principles of pacman' such that I could figure out that only sharing the 'cache' folder was the right thing to do.  As opposed to the silly suggestion on the 'shared read/write' section in the Pacman Tips wiki page, which I believe is wrong and have made a note in the page about it...maybe i should just delete the section in the wiki...i don't know the protocol there.

I think there is a lot of comments around "how to do something" and not enough around, "how does this thing work".  Kind of "teach a man to fish" versus "give him a fish" idea.

Offline

#13 2013-07-17 09:44:31

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,041

Re: Speed up install...use another computer's packages.

OK, that's what I assumed you were doing: learning rather than aiming for significant gains.

For what it's worth, I think that will be a nice wiki page once it's finished. "man pacman" gives good instructions on how to use the program but it would be nice to have a quick overview of what it's actually doing without having to dig into things oneself.

Offline

#14 2013-07-17 10:55:29

tomk
Forum Fellow
From: Ireland
Registered: 2004-07-21
Posts: 9,839

Re: Speed up install...use another computer's packages.

ftravers wrote:

Ensure on my old computer my pacman cache only had the latest versions with:

pacman -Sc

FYI, this was unnecessary for this particular procedure - no harm, but not required.

Offline

#15 2013-07-18 01:48:39

ftravers
Member
Registered: 2013-07-16
Posts: 5

Re: Speed up install...use another computer's packages.

To expand on the purpose/effect of the

pacman -Sc

command, removes all old version of packages, so depending on the age of your system this could be multiples of what is necessary. 

second it removes packages that aren't installed on the computer executing the command, NOT JUST old versions.  It's important to consider this when you are sharing a cache between computers, because one computer would very likely remove packages that ARE actively being used by another computer...a no no.

you can restrict `pacman -Sc` to removing only old version by looking at: https://wiki.archlinux.org/index.php/Pa … che_purges.

Last edited by ftravers (2013-07-18 01:50:22)

Offline

#16 2013-07-18 04:01:46

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: Speed up install...use another computer's packages.

ftravers wrote:

<snip>
you can restrict `pacman -Sc` to removing only old version by looking at: https://wiki.archlinux.org/index.php/Pa … che_purges.

Even better, use paccache.  It is a script that is shipped in the pacman package (it used to be in the pacman-contrib package) that gives you much greater control of what goes and what stays.  Plus you can have it give you a verbose dry-run to see what is going to be removed from the cache.  Oh, and you can have it focus on old packages or uninstalled packages.

I have several packages from base that I have removed from my system.  If you think about what has to be included in base, you will realize that there are a lot of programs that are included for situations which may not apply to your system.  For example, I have no need for ppp and I just symlinked /usr/bin/vi to /usr/bin/vim (vi always frusturated me since I tended to forget I was not using vim).

@tomk, I think the reason the OP decided to -Sc the package cache was because he wanted to actually copy the contents of the cache over to another system.  If the directory was simply being shared over the LAN, then it would indeed have been unnecessary, but the way he did it, it most likely cut way down on transfer time.

Offline

Board footer

Powered by FluxBB