You are not logged in.

#1 2019-01-11 13:14:25

blochl
Member
Registered: 2018-08-31
Posts: 18

Maintaining a private sub-repository of Arch packages

Hi,

I'm using Arch in a closed-off environment, and I need to maintain a local repository. Obviously, it's unreasonable to maintain (and re-build) the entire Arch repos locally. But I need just a subset of packages, as I run a very minimal setup. I suppose it's around 500 packages total (inc. all the dependencies and makedepends), most of them found in the official repos, and few only in the AUR.

What is the best way to go about it? Is there some ready-made solution for this?

The packages which I install explicitly are just the base group, and about 15 more packages. All the rest are dependencies. But since I want to re-build all of the packages, I need the makedepends as well.
If it's something that needs to be designed from ground up, the first question is how can I get, recursively, all the needed packages + the makedepends? (There's pactree for the dependencies, but what about makedepends?)

Offline

#2 2019-01-11 16:17:45

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,700

Re: Maintaining a private sub-repository of Arch packages

Why do you need to rebuild them? Could you not simply use the official repository and mirror it yourself?

That being said, if you really needed to do this then you could either acquire the build files and repeatedly sort through them to read the makedepends info and pull in additional dependencies, or you could build something using the archweb json interface. For any package currently in the official repos, you can use e.g.

pacman: https://www.archlinux.org/packages/core/x86_64/pacman/
pacman's json description: https://www.archlinux.org/packages/core … cman/json/

This information is also in the *.pkg.tar.xz internal .PKGINFO as well as the sync db, but not in the localdb. pacman has no method for getting the makedepends of a package, the libalpm backend *does* in the development version... so you could write your own libalpm frontend to display this information. https://bugs.archlinux.org/task/60347


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#3 2019-01-11 16:36:49

progandy
Member
Registered: 2012-05-17
Posts: 3,226

Re: Maintaining a private sub-repository of Arch packages

For any package where you have the PKGBUILD you can extract the makedepends from "makepkg --printsrcinfo" if you prefer that over a web query.

Last edited by progandy (2019-01-11 16:38:36)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#4 2019-01-11 17:07:21

blochl
Member
Registered: 2018-08-31
Posts: 18

Re: Maintaining a private sub-repository of Arch packages

Thanks for the replies!
I see that getting all the dependencies (inc. makedepends) is not trivial, but is certainly doable.

My next question then, is if there is some tool for automation of the following procedure (or parts of it):

1) Update the PKGBUILDs (and the associated files) of a set of packages from the official repos and the AUR monthly.
2) Hold them in a Git repo, in corresponding subdirectories.
3) Download the package sources and hold them in a Git LFS in the same repository.
4) Rebuild the packages that have changed (first time rebuild all the packages) and hold the packages themselves in Git LFS as well.
5) On demand, export all the packages from the Git LFS, and the db.tar.gz, to a local directory, to produce a custom local repo in that directory.

What I need essentially, is a local repo of a subset of packages (about 500 total) which will all be locally compiled, and I need to have a local history of the sources and the previous versions of the packages themselves. These are hard requirements, which can not be changed. If there is a more efficient/standard/ready-made method to do it than I've described above, that'd be really helpful! smile

Offline

#5 2019-01-11 17:22:01

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,700

Re: Maintaining a private sub-repository of Arch packages

Every repository package is already tracked in version control: https://wiki.archlinux.org/index.php/Ar … to_use_ABS
And every AUR package is already a git repository.

All your other mysterious requirements are specific to your specific use case, so you'll need to script that yourself (acquire sources, cp them into your own repo?). Good luck.

Note: the AUR helper "aurutils" makes it convenient to do clean chroot builds of many packages in subdirectories and export them directly to a custom repository. This does not exactly meet your goals, but then, I have never met anyone before who required that everything be mediated through git LFS before reaching the custom repository, so who knows. ¯\_(ツ)_/¯

Last edited by eschwartz (2019-01-11 17:22:33)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#6 2019-01-11 17:29:55

progandy
Member
Registered: 2012-05-17
Posts: 3,226

Re: Maintaining a private sub-repository of Arch packages

Maybe using gentoo would be simpler? I think portage does match your requirements better than pacman and you can create your own binary repositories as well.

https://wiki.gentoo.org/wiki/Binary_package_guide


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#7 2019-01-12 08:52:08

blochl
Member
Registered: 2018-08-31
Posts: 18

Re: Maintaining a private sub-repository of Arch packages

Thanks!!
aurutils sounds like it could help me in my task. I've also found pacman-src, which could be helpful.


Why not Gentoo? I thought for some time to use Gentoo for my purpose, but I like the simplicity of PKGBUILDs over ebuilds, plus I have more experience with Arch.

Offline

#8 2019-01-13 11:07:02

blochl
Member
Registered: 2018-08-31
Posts: 18

Re: Maintaining a private sub-repository of Arch packages

eschwartz, I've noticed that you maintain your own repo. Do you use some automation to synchronize some of the packages in your repo to the versions that exist in the official repos/AUR, or do you do it manually?

Also, putting aside the compilation requirements for a moment, is there some "standard" way to make a private repo from a subset of packages in the official repos?

Offline

#9 2019-01-13 16:14:38

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,700

Re: Maintaining a private sub-repository of Arch packages

I build each package using extra-x86_64-build from the devtools package against the current repository state, then repo-add it directly.

I only have two package bases that need actual syncing, because I create forked gtk3 and glib2 packages which build with debug symbols. But those I get notifications when the repository package is updated (and it is part of my svn checkout of the repos) so I know when it is being updated and quickly bump those too.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#10 2019-01-13 16:25:50

ayekat
Member
Registered: 2011-01-17
Posts: 1,275
Website

Re: Maintaining a private sub-repository of Arch packages

eschwartz wrote:

But those I get notifications when the repository package is updated (and it is part of my svn checkout of the repos) so I know when it is being updated and quickly bump those too.

How do you get notifications for repo package updates? Is there something that normal users (non-devs, non-TUs) can subscribe to as well?

Offline

#11 2019-01-13 16:32:59

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,700

Re: Maintaining a private sub-repository of Arch packages

1) part of my daily update routine is to `svn up` the PKGBUILD sources from https://www.archlinux.org/svn/

2) tools like nvchecker (available in the official repositories) can check archweb for updated package information (it is easy as archweb exports a json interface when you append "json/" to a package details url or access the search via "search/json/?queryparams")


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB