You are not logged in.

#1 2020-02-14 17:17:05

tony5429
Member
Registered: 2006-03-28
Posts: 978

[SOLVED] Custom Repository Based on Currently-Installed Packages

I'd like to create my own custom repository based on the packages I currently have installed on my system. This way I can include the database in the airootfs of a custom Arch installer and have a sort of backup ISO that can be installed offline. Below is what I'm thinking of doing. Two questions: (1) does this seem like it would work, and/or is there a better way to do it? (2) Will dependencies be automatically included? (I'd like them to be.) Finally, I'm aware AUR packages won't be included - that's ok.

#empty pacman package cache
rm /var/cache/pacman/pkg/*

#download all currently-installed packages
pacman -Qqn | pacman -w -

#build custom local repository
repo-add myrepo.db.tar $(find /var/cache/pacman/pkg -type f | xargs)

Last edited by tony5429 (2020-02-14 22:43:28)

Offline

#2 2020-02-14 18:06:35

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,015

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

tony5429 wrote:
#empty pacman package cache
rm /var/cache/pacman/pkg/*

You do realize this means you'll need to redownload every file every time you run this script? Once you downloaded all packages in the next step, you could just use pacman -Sp to print the filename for each installed package in the format:

file:///var/cache/pacman/pkg/..........

tony5429 wrote:
#download all currently-installed packages
pacman -Qqn | pacman -w -

pacman -w is not a valid option, unless combined with -S

tony5429 wrote:
#build custom local repository
repo-add myrepo.db.tar $(find /var/cache/pacman/pkg -type f | xargs)

Why are you using find here? Why are you piping the output of find to xargs, if the default behavior of find is to print results, and you are printing the results in a pipe to xargs, which is also printing the results? If you wanted to actually do something with them, you would use find -exec, not xargs. Since find is only finding files within a single directory, no subdirectories, you don't need to run a command to find them, just use the POSIX sh glob *, and if you did need to recurse into directories, you could use the bash "globstar" option and bash recursive glob **. You could also use `find /var/cache/pacman/pkg -type f -exec repo-add myrepo.db.tar {} +`.

See http://mywiki.wooledge.org/BashPitfalls … .2A.mp3.29 for why you should not be using `program $(find ...)` at all. Luckily, in this one cherry-picked case, you are almost certainly not going to find pacman packages with filenames other than ${pkgname}-${pkgver}-${pkgrel}-${arch}.pkg.tar.${COMPRESSION_EXT}, none of which can contain newlines, whitespace, or glob characters. But as a general shell-scripting rule you shouldn't be doing this anyway... *especially* since it's actually fully unnecessary.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#3 2020-02-14 19:09:15

tony5429
Member
Registered: 2006-03-28
Posts: 978

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

Thanks, eschwartz! Lots of great info here. I am aware that this will require re-downloading every time it's run (though I don't really anticipate needing to run it very often). The reason I'm planning to empty the cache is to ensure I don't end up loading multiple versions of the same package (due to previous updates of the system) into the repo database (valid concern?).

Thanks for the heads up about the -w pacman flag.

I didn't know * would work in this case; that definitely simplifies things, and I understand the concern with my first approach in terms of variability of the input data - I figured there was a better way!

So, my revised commands are as follows:

#empty pacman package cache
rm /var/cache/pacman/pkg/*

#download all currently-installed packages
pacman -Qqn | pacman -Sw -

#build custom local repository
repo-add myrepo.db.tar /var/cache/pacman/pkg/*

Finally, to re-address my question 2 in the first message, I think this process should ensure that dependencies are successfully included in the custom repo.

Offline

#4 2020-02-14 19:23:30

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,015

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

tony5429 wrote:

The reason I'm planning to empty the cache is to ensure I don't end up loading multiple versions of the same package (due to previous updates of the system) into the repo database (valid concern?).

This is why I suggested using pacman -Sp, since it will only print the cached filenames of the packages currently in the database. You can strip off the file:// from the beginning of each output line, and this (rather than nuking and redownloading only what you need) serves as your filter for "only current packages".

repo-add does also have a new option --prevent-downgrade which will ensure you only keep the latest version, if you try adding multiple versions of a package. This can still be in error if you have cached versions of a package from the [testing] repository...

tony5429 wrote:

Finally, to re-address my question 2 in the first message, I think this process should ensure that dependencies are successfully included in the custom repo.

Yes, since those packages are also returned by pacman -Qqn. wink


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#5 2020-02-14 20:20:10

tony5429
Member
Registered: 2006-03-28
Posts: 978

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

Ah, ok pacman -Sp makes sense now. Thanks.

One last question: If, rather than using my own current system to define the contents of the repo, I were to just run "pacman -Sw pkg1 pkg2 pkg3", would pacman also download the dependencies for those three packages (pkg1/2/3)? And, would it download packages/dependencies regardless of what I have installed currently on my system? I'd just try it out, but I'm not with my machine at the moment.

Offline

#6 2020-02-14 20:36:19

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,015

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

No, it would not. Or actually, the answer is "it depends".

Remember it is an option for the -S/--sync flag, which means that it will build up a list of packages to be installed, but instead of installing them, it will only download them.

So let's assume that "pkg1" depends on "dep1" and "pkg2" depends on "dep2", and "pkg3" depends on "dep3"... and you already have "dep2" and "pkg3" installed.

pacman -Sw pkg1 pkg2 pkg3 will:

  • make sure that pkg3 is downloaded, even though it is already installed, because you explicitly listed it

  • make sure that pkg2 is downloaded, and check the deptree that it can be installed. Since dep2 is already installed, even though it was rm'ed from the cache, it does not need to be downloaded because you don't need to reinstall it

  • make sure that pkg1 is downloaded, and check the deptree that it can be installed. Since dep1 is a dependency, and it must be installed, it is added to the transaction list; the transaction list

Now, even though for your goals you wanted to download all 6 to the cache, you have only downloaded 4: the 3 you listed explicitly, and the one (dep1) which in order to successfully install the first 3, you need to install as a dependency.

Since dep2 and dep3 were already installed, and you didn't explicitly say you wanted them, pacman -S would not bother *reinstalling* them, and -Sw did not bother downloading them.

This is why in my previous post, I highlighted the fact that pacman -Qqn lists them -- because otherwise they would not be downloaded. yikes

You can work around this by operating with a dummy --dbpath and initing a fresh system view which does not have any packages installed. (So -Sw always thinks all dependencies need to be downloaded.)

Last edited by eschwartz (2020-02-14 20:38:15)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#7 2020-02-14 20:59:47

tony5429
Member
Registered: 2006-03-28
Posts: 978

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

Ohh, interesting. So, assuming the same example packages/dependencies, would this be an acceptable way to force the download of all 6 packages?

pacman --dbpath /dev/null -Sw pkg1 pkg2 pkg3

Offline

#8 2020-02-14 22:25:51

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,015

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

dbpath is the parent directory of the directory that the sync databases will be downloaded to. You cannot use /dev/null where a directory is expected, and also, you cannot use /dev/null in a place where you need to persistently access data.

You could use --logfile /dev/null because the logfile option refers to a file which is simply appended to and doesn't store data which must be recalled, though.

For dbpath you'll need to setup a temporary workdir. e.g.

workdir=$(mktemp -d)
pacman --dbpath "${workdir}" -Sy # the new workdir has no sync databases yet
pacman --dbpath "${workdir}" -Sw pkg1 pkg2 pkg3

Of course, your use case doesn't need to worry about this, since you simply download all packages you currently have installed. But for the learning exercise of these 3 explicit packages plus all dependencies? Yes, that's how you would do it.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#9 2020-02-14 22:43:14

tony5429
Member
Registered: 2006-03-28
Posts: 978

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

Got it. Thanks for all your help. Actually, I realised at some point that there are a couple packages beyond what I currently have installed which would be nice to also have optionally available on the installation media for offline installation in the future, so the extra info (knowing how to download the packages and their dependencies) is beneficial.

Offline

#10 2020-02-16 01:53:01

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,015

Re: [SOLVED] Custom Repository Based on Currently-Installed Packages

Great. smile

Note that for packages which you do *not* have installed -- or when using a temporary --dbpath -- pacman -Sp will list the cached file:///var/cache/pacman/pkg/..... local file urls for the entire set of dependencies as well (because they would be part of a transaction list when installing them to a system without any packages, for example, using pacstrap).

So, the temporary --dbpath is not incompatible with using a persistent cachedir and having pacman list the files you want. In fact, using a temporary --dbpath actually make it easier. big_smile

workdir=$(mktemp -d)
wanted_packages=(base pkg1 pkg2 pkg3)

pacman --dbpath "${workdir}" -Sy "${wanted_packages[@]}"

mapfile -t allpkgfiles < <(pacman --dbpath "${workdir}" -Sp "${wanted_packages[@]}")
allpkgfiles=("${allpkgfiles[@]#file://}")

Automatically ensure the base metapackage and all its requirements are pulled in, add some extra packages, make sure they are downloaded, then grab their cache urls and remove the "file://" URL prefix to produce simple pathnames.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB