You are not logged in.

#1 2013-12-04 23:48:03

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Setting up personal package dependencies

Hi all,

I'd like to bring some organization to my Arch system. From what I've been able to figure out, there is already a distinction between explicit packages (that I installed manually) and dependencies (that were pulled in). That's good. However, I seem to have a broader view of the term 'dependency' than pacman does. Right now I still have to manually install:

  1. optional dependencies (exist solely as reminders?),

  2. package groups (exist solely to save keystrokes?),

  3. "if you install X under certain circumstances, don't forget to also install Y, or X will break" kinds of dependencies (e.g., lib32-libpulse),

  4. "if you want X and Y to work together, also install Z" kinds of dependencies (e.g., skype4pidgin), and

  5. "to accomplish task T, I need to install W, X, Y and Z" kinds of dependencies (I came up with this one myself).

None of those are formally tracked as dependencies. They exist as explicit installs, indistinguishable from the rest, cluttering up my system even after they no longer serve a purpose.

Instead of installing them manually, I'd rather set up some personal dependency rules to pull them in. Unless there are more elegant solutions out there, I think I want to do the following:

  1. add personal dependency-relations between existing packages (to address 1 and 3),

  2. add personal 'dummy packages' to my database, only meant to pull in dependencies (to address 2 and 5), and

  3. make those relations smarter, such as: pull in 'skype4pidgin' if both 'skype' and 'pidgin' are pulled in (to address 4).

Are there existing tools or recommended practices to accomplish my goals? Oh, by the way, I'd like AUR to be included in this. I currently use aura.

More generally, where can I learn more about management of the pacman dependency tree?

Last edited by mhelvens (2013-12-04 23:58:09)

Offline

#2 2013-12-05 04:56:06

dgbaley27
Member
Registered: 2011-07-22
Posts: 47

Re: Setting up personal package dependencies

1) There is a patch set for much better optional dependency handling in pacman. I don't know when you can expect it to be released however.
2) Some package groups -- such as the KDE ones I believe -- have both groups and meta-packages with dependencies. Other than that, you have to create your own.
3) This seems like Y should be an optional dependency of X (if not a full dependency).
4) Similar to package groups, you need to create your own meta-package. Or you can argue with the pidgin/skype maintainers that skype4pidgin is an optional dependency of either.
5) This is pretty clearly a case where you'd want to create your own meta-package with the necessary dependencies.

a) This might be a patches-welcome type of thing, but...
b) Personal dummy packages are the only way to do it currently.
c) There is a design for 'hooks' support in pacman but I'm not sure how far along it is. With this you could potentially install a package based on a hook; how easily this can be done depends on whether the pacman db is locked when hooks are executed.

I find that

pacman -S --asdeps <pkg>

is enough for me to manage stuff I just need temporarly. With that I can manually insepct packages for cleanup with

pacman -Qdt

Also, if you create meta/dummy packages for yourself, don't feel the need to upload them all to the AUR. Chances nobody else will care smile.

Offline

#3 2013-12-05 11:17:47

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

dgbaley27 wrote:

1) There is a patch set for much better optional dependency handling in pacman. I don't know when you can expect it to be released however.

The thing is, "optional dependency" is an oxymoron. It's not really clear to me what it even means.

I come from Gentoo, which has a concept of USE-flags (i.e., features). Certain packages will be dependencies of other packages if specific USE-flags are activated. Will it be something like that?

3) This seems like Y should be an optional dependency of X (if not a full dependency).

Sure. But oversights happen. That's why we need a way to fix this manually.

b) Personal dummy packages are the only way to do it currently.

Well, that will get me a long way, at least. Can I assume that this page is the (best) full documentation for creating packages?

c) There is a design for 'hooks' support in pacman but I'm not sure how far along it is. With this you could potentially install a package based on a hook; how easily this can be done depends on whether the pacman db is locked when hooks are executed.

I don't think hooks are the answer here. Said packages would still not behave properly when cleaning out orphaned dependencies.

Also, if you create meta/dummy packages for yourself, don't feel the need to upload them all to the AUR. Chances nobody else will care smile.

I didn't think they would. ;-)

Thanks!

Offline

#4 2013-12-05 11:42:06

tomk
Forum Fellow
From: Ireland
Registered: 2004-07-21
Posts: 9,839

Re: Setting up personal package dependencies

mhelvens wrote:

The thing is, "optional dependency" is an oxymoron. It's not really clear to me what it even means.

Seems pretty clear to me - it's a dependency that is not essential, but provides additional functionality when installed.

Offline

#5 2013-12-05 12:15:43

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

tomk wrote:
mhelvens wrote:

The thing is, "optional dependency" is an oxymoron. It's not really clear to me what it even means.

Seems pretty clear to me - it's a dependency that is not essential

That's my point. The word 'dependency' (meaning 'need' or 'reliance') means pretty much the opposite of 'optional' (meaning 'not essential'). So: oxymoron.

tomk wrote:

not essential, but provides additional functionality when installed.

That seems to be true for every package that exists.

I guess it means that the two packages in question can work together somehow when installed at the same time. Perhaps a better term would be 'optional extension' or 'optional combination'?

Offline

#6 2013-12-05 13:04:42

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,784
Website

Re: Setting up personal package dependencies

It is an optional dependency because A will work without B, but B will provide more functionality if it is installed.

For example, firefox has gstreamer0.10-good-plugins as an optional dependency. If installed, the good-plugins package provides webm and mp4 demuxing. When not installed, firefox functions as usual, just without the additional functionality good-plugins would provide.

If you need webm and mp4 demuxing in firefox, then you need gstreamer0.10-good-plugins. If you don't need webm and mp4 demuxing in firefox, then you don't need gstreamer0.10-good-plugins. Hence it is an optional dependency.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#7 2013-12-05 13:20:30

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

Yeah, I understand the general idea. I'm sorry, I guess I'm just being pedantic. smile

Offline

#8 2013-12-06 18:47:49

sitquietly
Member
From: On the Wolf River
Registered: 2010-07-12
Posts: 219

Re: Setting up personal package dependencies

mhelvens wrote:
dgbaley27 wrote:

1) There is a patch set for much better optional dependency handling in pacman. I don't know when you can expect it to be released however.

The thing is, "optional dependency" is an oxymoron. It's not really clear to me what it even means.

I come from Gentoo, which has a concept of USE-flags (i.e., features).

Actually in Arch "optional dependency" is not always an oxymoron.  Arch pkgbuilds often use a trick to compile in support for a feature without requiring that feature to be present at runtime. The trick is to declare the feature to be a "makedepends" AND an "optdepends".  This often means that the binaries will link to "optional" libraries that may not be there at runtime.  This can lead to bizarre failures at runtime but some apps compiled this way will intelligently catch the missing library and issue an error message when you attempt to use some "optional" feature.

I build an Arch-like system (a system that started as Arch Linux but has been through several cycles of heavy modification of the pkgbuilds and complete rebuilds).  In this system I remove all of those broken dependencies and substitute a clean use of USE flags, the beautiful part of Gentoo!  (I also unsplit all split packages and add USE flags in the configure phase to build just what is needed). For example, the Arch Linux pkgbuild for vlc has many "optional dependencies"

depends=('a52dec' 'libdvbpsi' 'libxpm' 'libdca' 'qt4' 'libproxy' 
         'sdl_image' 'libdvdnav' 'libtiger' 'lua' 'libmatroska' 
         'zvbi' 'taglib' 'libmpcdec' 'ffmpeg' 'faad2' 'libupnp' 
         'libshout' 'libmad' 'libmpeg2' 'xcb-util-keysyms' 'libtar' 
         'libxinerama')
makedepends=('live-media' 'libnotify' 'libbluray' 'flac' 'kdelibs'
             'libdc1394' 'libavc1394' 'lirc-utils' 'libcaca' 
             'librsvg' 'portaudio' 'libgme' 'xosd' 'projectm' 
             'twolame' 'aalib' 'libmtp' 'libdvdcss' 'gnome-vfs' 
             'libgoom2' 'vcdimager' 'opus' 'libssh2' 'mesa')
optdepends=('avahi: for service discovery using bonjour protocol'
            'libnotify: for notification plugin'
            'ncurses: for ncurses interface support'
            'libdvdcss: for decoding encrypted DVDs'
            'lirc-utils: for lirc plugin'
            'libavc1394: for devices using the 1394ta AV/C'
            'libdc1394: for IEEE 1394 plugin'
            'kdelibs: KDE Solid hardware integration'
            'libva-vdpau-driver: vdpau back-end for nvidia'
            'libva-intel-driver: back-end for intel cards'
            'libbluray: for Blu-Ray support'
            'flac: for Free Lossless Audio Codec plugin'
            'portaudio: for portaudio support'
            'twolame: for TwoLAME mpeg2 encoder plugin'
            'projectm: for ProjectM visualisation plugin'
            'libcaca: for colored ASCII art video output'
            'libgme: for libgme plugin'
            'librsvg: for SVG plugin'
            'gnome-vfs: for GNOME Virtual File System support'
            'libgoom2: for libgoom plugin'
            'vcdimager: navigate VCD with libvcdinfo'
            'aalib: for ASCII art plugin'
            'libmtp: for MTP devices support'
            'smbclient: for SMB access plugin'
            'libcdio: for audio CD playback support'
            'ttf-freefont: for subtitle font '
            'ttf-dejavu: for subtitle font'
            'opus: for opus support'
            'libssh2: for sftp support'
            'lua-socket: for http interface')

Notice that those optional deps must be present at build time and many are actually linked to.  My own pkgbuild for vlc has

depends=(
  'a52dec' 
  'aalib' 
  'alsa-lib'
  'dbus'
  'faad2' 
  'flac'
  'libass'
  'libav' 
  'libcaca' 
  'libcddb'
  'libcdio'
  'libdca' 
  'libdvbpsi' 
  'libdvdcss' 
  'libdvdnav' 
  'libdvdread'
  'libgme' 
  'libmad' 
  'libmatroska' 
  'libmodplug'
  'libmpcdec' 
  'libmpeg2' 
  'libmtp' 
  'libnotify' 
  'libpng'
  'libproxy' 
  'librsvg'
  'libshout' 
  'libssh2' 
  'libtheora'
  'libtiger' 
  'libupnp' 
  'libva'
  'libvorbis'
  'libx11'
  'libxcb'
  'libxpm' 
  'lirc-utils' 
  'lua51' 
  'mesa'
  'ncurses'
  'opencv'
  'opus' 
  'portaudio' 
  'projectm' 
  'qt4' 
  'schroedinger'
  'sdl_image' 
  'speex'
  'taglib' 
  'ttf-dejavu'
  'ttf-freefont'
  'twolame' 
  'vcdimager'
  'x264'
  'xcb-util'
  'xcb-util-keysyms' 
  'xosd'
  'zlib'
  'zvbi' 
  $(use jack)
  $(use avahi) 
  $(use oss) 
  $(use pulseaudio)
  $(use samba smbclient) 
  )
makedepends=(
  'pkg-config'
  'xproto'
  $(use kde kdelibs)
  )

Of course in the configure command I have to configure for the optional features:

  ./configure --prefix=/usr \
    LUAC=luac5.1 \
    RCC=/usr/bin/rcc-qt4 \
    --sysconfdir=/etc \
    --disable-bluray \
    --disable-chromaprint \
    --disable-dc1394 \
    --disable-fluidsynth \
    --disable-goom \
    --disable-libfreerdp \
    --disable-libvnc \
    --disable-live555 \
    --disable-postproc \
    --disable-rpath \
    --disable-v4l2 \
    --enable-aa \
    --enable-faad \
    --enable-flac \
    --enable-lirc \
    --enable-mod \
    --enable-ncurses \
    --enable-nls \
    --enable-notify \
    --enable-opencv \
    --enable-opus \
    --enable-projectm \
    --enable-pvr \
    --enable-realrtsp \
    --enable-schroedinger \
    --enable-sftp \
    --enable-speex \
    --enable-upnp \
    --enable-vcdx \
    --enable-vorbis \
    --enable-xosd \
    $(use avahi '--enable-bonjour' '--disable-bonjour') \
    $(use kde '' '--without-kde-solid') \
    $(use oss '--enable-oss' '--disable-oss') \
    $(use pulseaudio '--enable-pulse' '--disable-pulse') \
    $(use samba '--enable-smb' '--disable-smb') \

In my pkgbuild there are no optdepends and a very small number of use flags.  I use "optdepends" for related packages that can actually be added afterward.  In the base layer I use optdepends in a few places to break ugly cyclic dependencies that  pull in higher level packages into a lower level (I call such dependencies "bad boys"). Since the system is built from source I can check for the presence of an optional package such as doxygen and enable it's use if it is present.

It seems to me that there are only two ways to get clean dependencies:
1. Compile from source and include only those dependencies actually wanted for the target system.
2. Build and distribute binaries that include a limited set of commonly needed features.

Arch Linux tries to include every feature that any use ever demands, although not in a very clean way (IMO).  The Arch way can lead to strange problems but it is probably one of the things that makes Arch popular.

Last edited by sitquietly (2013-12-06 18:55:56)

Offline

#9 2013-12-06 19:07:12

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

sitquietly wrote:

Actually in Arch "optional dependency" is not always an oxymoron.  Arch pkgbuilds often use a trick to compile in support for a feature without requiring that feature to be present at runtime. The trick is to declare the feature to be a "makedepends" AND an "optdepends".  This often means that the binaries will link to "optional" libraries that may not be there at runtime.  This can lead to bizarre failures at runtime but some apps compiled this way will intelligently catch the missing library and issue an error message when you attempt to use some "optional" feature.

Ah, that's interesting. :-)

It seems to me that there are only two ways to get clean dependencies:
1. Compile from source and include only those dependencies actually wanted for the target system.
2. Build and distribute binaries that include a limited set of commonly needed features.

I find that I no longer have the patience for option 1, which is part of the reason I'm switching to Arch. I can now set up an Arch netbook within half an hour. Installing the same software on a Gentoo box can take a couple of nights. wink

Option 2 is not really an option. The whole appeal of Gentoo and Arch is that they are fully customizable.

Arch Linux tries to include every feature that any use ever demands, although not in a very clean way (IMO).  The Arch way can lead to strange problems but it is probably one of the things that makes Arch popular.

Maybe. But I'll tell you this: I've been using my new Arch system for a couple of days now and there have been no problems. My Gentoo installation broke all the time. This is the other reason I'm switching.

Offline

#10 2013-12-06 21:10:18

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Setting up personal package dependencies

mhelvens wrote:
  1. add personal dependency-relations between existing packages (to address 1 and 3),

makedep can do this. It's basically a stopgap until Pacman implements better optdep management.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#11 2013-12-06 23:52:02

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

Xyne wrote:
mhelvens wrote:
  1. add personal dependency-relations between existing packages (to address 1 and 3),

makedep can do this. It's basically a stopgap until Pacman implements better optdep management.

Great! I didn't think I could be the only one who'd find this useful.

I've been playing with it for a bit. There are a few things you should fix in the --help text:

  • I tried to make pkg2 a dependency of pkg1. I followed the --help text and tried makedep -a pkg1 pkg2, but that didn't do anything (not even an error message). Then I realized that the -a argument takes a list of new dependencies, and pkg2 was swallowed into that list. This breaks the intuition of positional arguments a bit.

  • The --help text says about the -u option: "When this option is passed, ." ... what? smile

Other than that, it seems to work as advertised; nice!

I'll need to write a number of scripts around it to make it functional for myself: to (optionally) 'un-explicit' the package; to implement the meta-package idea (which you have listed under TODO); to run makedep -u after any mutational pacman runs; and to basically create batch-jobs around the most common operations I'll be performing.

Cheers!

Offline

#12 2013-12-07 18:13:50

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Setting up personal package dependencies

I've fixed the misplaced help text and added an epilog to explain the behavior of the -a and -r options. I'm used to it because it's standard for the argparse library and present in aurtomatic, but I remember how annoying it could be in the beginning.

I have also added two options for setting the install reason of the target dependencies: --add-as and --remove-as.

Although it isn't installed as such, the script is written as a module so you can use it in other scripts by linking or copying it into the working directory or Python path. That may be useful if you want to write your own scripts around makedep, especially if you want to use pyalpm directly instead of invoking the pacman executable.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#13 2013-12-07 20:24:30

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

Xyne wrote:

I've fixed the misplaced help text and added an epilog to explain the behavior of the -a and -r options.

Great. Don't forget to put it on AUR. That makes it so much easier for us Arch newbies. wink

I have also added two options for setting the install reason of the target dependencies: --add-as and --remove-as.

Is the install reason changed in the makedep database only, like the dependency status, and reset with --purge?

My first guess is no (otherwise, why would you need --remove-as to reverse the damage?). If I'm right, I'd rather do this through pacman itself, making it more obvious that the change is persistent.

Although it isn't installed as such, the script is written as a module so you can use it in other scripts by linking or copying it into the working directory or Python path. That may be useful if you want to write your own scripts around makedep, especially if you want to use pyalpm directly instead of invoking the pacman executable.

That's still a bit over my head. I was just thinking of writing a bash-script calling makedep and pacman (well, aura actually).

Last edited by mhelvens (2013-12-07 20:27:18)

Offline

#14 2013-12-07 21:16:06

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Setting up personal package dependencies

"if you want X and Y to work together, also install Z" kinds of dependencies (e.g., skype4pidgin), and

I would solve this like this: skype4pidgin is optdepends for pidgin and skype4pidgin depends on both skype and pidgin.

c.  make those relations smarter, such as: pull in 'skype4pidgin' if both 'skype' and 'pidgin' are pulled in (to address 4).

Never do that. Don't pretend to know what the user wants. What if one user needs skype and another one wants pidgin without skype? There is no need to install the bridge.

Last edited by progandy (2013-12-07 21:19:10)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#15 2013-12-07 21:27:08

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

progandy wrote:

c.  make those relations smarter, such as: pull in 'skype4pidgin' if both 'skype' and 'pidgin' are pulled in (to address 4).

Never do that. Don't pretend to know what the user wants. What if one user needs skype and another one wants pidgin without skype? There is no need to install the bridge.

You misunderstand:

  • I'm saying that if both Skype and Pidgin are installed anyway, then also install the package that makes them work together. Installing either Skype or Pidgin alone would not pull in anything else.

  • I want to set this up for myself, not impose it on others. I am the user. :-)

Last edited by mhelvens (2013-12-07 21:27:36)

Offline

#16 2013-12-07 23:24:30

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Setting up personal package dependencies

mhelvens wrote:

[*]I want to set this up for myself, not impose it on others. I am the user. :-)[/*]

There are multiuser-systems, not only single user installations. You could create a pacman wrapper that includes a list of recommendations with dependencies. If all dependencies for a recommendation are installed, then suggest the package, but don't install it automatically. I use skype and pidgin, but I don't want to integrate skype. Arch is not a handholding distro that guesses what the user wants, so this should not be part of the dependency system.

Last edited by progandy (2013-12-07 23:25:09)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#17 2013-12-07 23:33:57

mhelvens
Member
From: Leiden, the Netherlands
Registered: 2013-12-04
Posts: 69
Website

Re: Setting up personal package dependencies

progandy wrote:
mhelvens wrote:

[*]I want to set this up for myself, not impose it on others. I am the user. :-)[/*]

There are multiuser-systems, not only single user installations. You could create a pacman wrapper that includes a list of recommendations with dependencies. If all dependencies for a recommendation are installed, then suggest the package, but don't install it automatically. I use skype and pidgin, but I don't want to integrate skype. Arch is not a handholding distro that guesses what the user wants, so this should not be part of the dependency system.

I'm not sure which part of this you're not understanding. You even quoted the exact line. smile

I want to set this up for myself. Only for myself. I've decided that on whichever system I install both Pidgin and Skype, I want to also install pidgin4skype as a dependency (not an explicit). This will help me keep a clean system. And in the future it will give me insight into why certain packages are installed.

Last edited by mhelvens (2013-12-07 23:35:34)

Offline

#18 2013-12-07 23:39:44

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Setting up personal package dependencies

mhelvens wrote:
progandy wrote:
mhelvens wrote:

[*]I want to set this up for myself, not impose it on others. I am the user. :-)[/*]

There are multiuser-systems, not only single user installations. You could create a pacman wrapper that includes a list of recommendations with dependencies. If all dependencies for a recommendation are installed, then suggest the package, but don't install it automatically. I use skype and pidgin, but I don't want to integrate skype. Arch is not a handholding distro that guesses what the user wants, so this should not be part of the dependency system.

I'm not sure which part of this you're not understanding. You even quoted the exact line. smile

Sorry, I got the general idea but somehow I missed the term "personal dependency rule" and thought you wanted to integrate it into pacman. Of yourse its perfectly fine to do that on your own system with your own tools. I guess I got confused because at least better support for optional dependencies is planned for pacman.

You can easily create your own list of packages with requirements and then let a script run over them and install them if all requirements are fulfilled.

Last edited by progandy (2013-12-07 23:43:44)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

Board footer

Powered by FluxBB