A suggestion for handling optdepends.

Xyne · 2009-01-08 02:38:23

[UPDATE]
I've rewritten this post to present the idea more clearly.
[/UPDATE]

I've submitted a feature request: http://bugs.archlinux.org/task/12708
If you like this idea, please express your support there too.

The Current Situation
The pacman database contains a file named "depends" in each package's directory which specifies the package's depends in the following format:

%DEPENDS%
foo
bar
this
that

Pacman reads this file and creates an internal representation of this list for the package which it uses during the sync operation to handle dependencies. Each package may also list optional dependencies which provide further functionality for the package without being required to use the package. Let's take gimp as an example:

pacman -Si gimp
...
Depends On     : gtk2>=2.14.4  lcms>=1.17  libxpm>=3.5.7  libwmf>=0.2.8.4  libxmu>=1.0.4  librsvg>=2.22.3  libmng>=1.0.10  dbus-glib>=0.76  libexif>=0.6.16  pygtk>=2.13.0  desktop-file-utils  gegl>=0.0.22  curl  
Optional Deps  : gutenprint: for sophisticated printing only as gimp has built-in cups print support
                 libwebkit: for the help browser
                 poppler-glib: for pdf support
                 hal: for Linux input event controller module
                 alsa-lib: for MIDI event controller module

If you want to install libwebkit to use gimp's help browser, you have 2 choices:

pacman -S libwebkit

pacman -S --asdeps libwebkit

With the first choice, libwebkit will clutter the list of explicitly installed packages ("pacman -Qet"). With the second choice, libwebkit will be considered an orphan and will be listed in "pacman -Qdt", which not only means it clutters that list but it also means that you can no longer purge orphans with "pacman -Rs $(pacman -Qqdt)".
In both cases, when you uninstall gimp, you must remember to uninstall libwebkit too, because pacman doesn't know that you installed it as a dependency for gimp.
This may not be a problem for one package, but it will once the number or optdepends you have installed increases.

My Suggestion
Create an optdepends database in /var/lib/pacman/optdepends/ that follows the same format as the current depends files:

%OPTDEPENDS%
foo
bar
this
that

Add a function to pacman to check if a package has an entry in the optdepends database.
During a sync operation, treat any optdepends specified in the optdepends database as if they had been specified in the depends file.
Add a "--getoptdeps" flag to pacman to enable interactive installation of optdepends for a given package that follows the same pattern as the current group installation dialogue.
Store the results of this dialogue in the optdepends database.

Let's take gimp as an example again. You know that gimp has an optdepend that you want, so you do this:

pacman -S --getoptdeps gimp
gimp package found, checking optdepends
:: gimp has the following optdepends:
                 gutenprint: for sophisticated printing only as gimp has built-in cups print support
                 libwebkit: for the help browser
                 poppler-glib: for pdf support
                 hal: for Linux input event controller module
                 alsa-lib: for MIDI event controller module
:: Install whole content? [y/N] n
:: Install gutenprint as optdepend for gimp? [y/N] n
:: Install libwebkit as optdepend for gimp? [y/N] y
:: Install poppler-glib as optdepend for gimp? [y/N] n
:: Install hal as optdepend for gimp? [y/N] n
:: Install alsa-lib as optdepend for gimp? [y/N] n

Retrieving libwebkit...

Libwebkit will now be handled exactly as if it were a true dependency of gimp. It is neither explicitly installed nor an orphan. It will get removed with gimp unless it's a depend or optdepend for another package.
/var/lib/pacman/optdepends/gimp/optdepends now looks like this:

%OPTDEPENDS%
libwebkit

The Benefits of This Method
Default pacman behavior remains unchanged.
Most of the code is already in place (depends file parser, package selection dialogue, dependency handling during sync operation)
The existing databases (local, sync) would not require any changes.
The only extra overhead would be checking if a package has an entry in the optdepends database.
Users can define their own optional dependencies by adding them to the optdepends database (manually or with provided tools)
This opens the doors for metapackages to replace groups.

About Metapackages
A metapackage is a package that contains nothing itself but organizes other packages. For an example of how these work on Arch, take a look at metapax.

Every package group could be converted to a metapackage if this suggestion were implemented. To understand the benefits of using metapackages instead of groups, we need to consider how groups currently work.

When you install gnome, this is what happens:

pacman -S gnome
gnome package not found, searching for group...
:: group gnome (including ignored packages):
    epiphany  gnome-applets  gnome-backgrounds  gnome-control-center  gnome-desktop  gnome-icon-theme  gnome-media  gnome-mime-data  gnome-mount  gnome-panel  gnome-python  gnome-screensaver  gnome-session  gnome-settings-daemon  
    gnome-themes  gnome2-user-docs  libgail-gnome  metacity  nautilus  notification-daemon  yelp  
:: Install whole content? [Y/n] n
:: Install epiphany from group gnome? [Y/n] y
:: Install gnome-applets from group gnome? [Y/n] y
:: Install gnome-backgrounds from group gnome? [Y/n] y
:: Install gnome-control-center from group gnome? [Y/n] y
:: Install gnome-desktop from group gnome? [Y/n] y
...

Most users will install all of the packages, others won't. In either case, once the packages are on your system, pacman has no concept of the gnome "group". Each package is effectively independent of the gnome group. If a new package is added to the gnome group, for example "gnome-somenewpackage", pacman will not install it during your next update. It won't even ask you about it or tell you that there is a new package. There have been questions on this forum from users wondering why new gnome packages weren't installed automatically. This applies to all groups... kde, xorg, xfce, etc.

If we instead replaced groups with metapackages, each package in the group would become an optdepend of the metapackage. With my suggestion, this would lead to exactly the same dialogue as above. Each package in a metapackage would remain optional just as packages in groups currently are. The advantage would be that if "gnome-somenewpackage" is added to the gnome metapackage, it would be possible to inform the user during an update and prompt for installation.

Here's the discussion on flyspray about groups vs metapackages: http://bugs.archlinux.org/task/8242

Notes on Metapackages
The only complicated parts of handling metapackages are the following:
If a package is a metapackage, it should be detected during installation and automatically jump to the optdepends dialogue in order for it to behave exactly as groups do.
During a metapackage update, there should be a way to inform the user of new optdepends, but this might be as simple as including an upgrade message in the package install file.

Last edited by Xyne (2009-01-13 16:20:52)

ibendiben · 2009-01-11 15:11:58

I'm not sure if I understand what you are saying. With the first part, do you mean:

In case the file 'depends' lists deps that you actually don't use or need, you should be able to move those deps to another file called 'optdepends' and they won't be installed anymore, unless you later find them to be usefull after all and move them back to 'depends'?
So pacman needs to offer an option to 'not-install as optdeps' and pacman should keep track of both, by checking depends and optdepends separately each 'update' but ignoring the 'optdepends' if possible?

I suppose this would imply a lot of expertise from users wouldn't it? Since they should know every single task dependencies are used for by they parent packages. More user control sure is a good thing and in combination with a tool that can tell users which depends might be optional, by listing their tasks, I could see it's use. But to me it seems rather complicated too because pacman and many other packages might need a more modular approach (every task, single package) and different error handling because missing deps-checks errors could well be missing-'optdeps'.

What you say about the empty metapackages I didn't quite understand either, maybe you can try to clarify it more?

Xyne · 2009-01-11 17:53:40

No, this wouldn't affect a packages "true" dependencies in any way.

Packages now have 2 types of dependencies, "depends" and "optdepends". "depends" are installed with the package and are required for the package to run. "optdepends" just display message during installation to the effect of "optional dependencies for this package: foo - for foo support, bar - for bar support, baz - for web access and printing". "gimp" is an example of a package with optional dependencies.

As it is right now, optional dependencies are nothing more than installation messages. If you decide to install optional dependencies for a given package, they are completely independent of the target package. Let me give a concrete example:

pacman -Si gimp
...
Depends On     : gtk2>=2.14.4  lcms>=1.17  libxpm>=3.5.7  libwmf>=0.2.8.4  libxmu>=1.0.4  librsvg>=2.22.3  libmng>=1.0.10  dbus-glib>=0.76  libexif>=0.6.16  pygtk>=2.13.0  desktop-file-utils  gegl>=0.0.22  curl  
Optional Deps  : gutenprint: for sophisticated printing only as gimp has built-in cups print support
                 libwebkit: for the help browser
                 poppler-glib: for pdf support
                 hal: for Linux input event controller module

Ok, I want to install gimp and I want libwebkit to be able to use gimp's help browser. I have 2 options right now:
Option 1:

pacman -S gimp libwebkit

libwebkit is now installed as an explicit package.

Option 2:

pacman -S gimp
pacman -S --asdeps libwebkit

libwebkit is now installed as a dependency.

With option 1, libwebkit clutters my list of explicitly installed packages (pacman -Qet). With option 2, it is considered an orphan by pacman and would be removed with an orphan purge ("pacman -Rsn $(pacman -Qqdt)"). In both cases, if I remove gimp, libwebkit stays on my system even though I only want it for gimp. It will not be removed with "pacman -Rs gimp" because pacman has no idea that it has anything to do with gimp.

My suggestion therefore it to create a way for pacman to treat selected optdepends as depends. Given the gimp example, what this would mean for the user is that when the user runs "pacman -S gimp", it would present a dialogue as follows:

gimp has the following optional dependencies:
     gutenprint: for sophisticated printing only as gimp has built-in cups print support
     libwebkit: for the help browser
     poppler-glib: for pdf support
     hal: for Linux input event controller module
Would you like to install these optional dependencies? [y/N] y
Install all optional dependencies? [y/N] n
Install gutenprint? [y/N] n
Install libwebkit? [y/N] y
Install poppler-glib? [y/N] n
Install hal? [y/N] n

retrieving libwebkit...

libwebkit would now be treated as if it had been specified in gimp's depends array. When you uninstall gimp, it would be removed with gimp just as gimp's other dependencies.

There would also be tools to add optional dependencies to a package later (either with pacman or something else... I'll gladly contribute something to do this), so if you want to add gutenprint to gimp later, you could and then let pacman grab it as a dependency of gimp.

Again, this has nothing to do with "true" dependencies of packages. This is just a fix for the kludge now known as "optdepends".

First, let's look at what happens when you install the gnome

pacman -S gnome
gnome package not found, searching for group...
:: group gnome (including ignored packages):
    epiphany  gnome-applets  gnome-backgrounds  gnome-control-center  gnome-desktop  gnome-icon-theme  gnome-media  gnome-mime-data  gnome-mount  gnome-panel  gnome-python  gnome-screensaver  gnome-session  gnome-settings-daemon  
    gnome-themes  gnome2-user-docs  libgail-gnome  metacity  nautilus  notification-daemon  yelp  
:: Install whole content? [Y/n] n
:: Install epiphany from group gnome? [Y/n] n
:: Install gnome-applets from group gnome? [Y/n] y
:: Install gnome-backgrounds from group gnome? [Y/n] y
:: Install gnome-control-center from group gnome? [Y/n] y
:: Install gnome-desktop from group gnome? [Y/n] y
:: Install gnome-icon-theme from group gnome? [Y/n] y
:: Install gnome-media from group gnome? [Y/n] n
:: Install gnome-mime-data from group gnome? [Y/n]

After the installation, each of those packages is treated as an independently installed package. The "group" gnome only exists when you select packages for the initial installation. There have been threads on this forum posted by users who didn't understand why "pacman -Syu" failed to retrieve packages that had been added to the gnome "group". That's because pacman simply updates the existing packages and doesn't know about groups once their on the system. If they add "gnome-some-new-package", you have to either run "pacman -S gnome" again and either re-install all the packages or run through the dialogue until you get to the new package, or you have to explicitly install any new packages directly. You need to find out when a new package has been included in gnome too, because there is no way for pacman to know this (I posted a script somewhere to check if you have all packages in a group, forgot where though).

The idea of a metapackage is that it is an empty package that simply specifies other packages as dependencies (i.e it contains no files, just package information). That's what metapax creates (http://bbs.archlinux.org/viewtopic.php?id=53788). If a gnome metapackage is created with metapax, the user can install it and get all of the packages in gnome. If a new package is added to the gnome metapackage, this package will be retrieved on the next sync update. The user doesn't need to regularly check that he has everything in gnome because the metapackage handles all the packages in gnome.

The problem with this approach is that everything it specifies is a "depends", so you have to include everything. With "optdepends" though, you would get a similar dialogue to the one when installing a group (as my example above for gimp), but the installed metapackage would have all of the advantages of a package when syncing and uninstalling.

Users would also create their own metapackages. Lets say that you would like to create a custom DE from existing packages so that you can quickly install a simliar desktop on different systems. You could create a metapackage with your window manager, text editor, image viewer, video player, etc. You could then simply install that package on different machines and be presented with the choice of which components you'd like to install. You could distribute this over your network with a local user repository. If you later want to add another package to it, that package could be optionally included on the different machines during the next update.

Last edited by Xyne (2009-01-11 18:03:10)

Dieter@be · 2009-01-11 18:42:32

Things like these have been brought up several times on the mailing lists and bugtracker, but I'm afraid no one had enough time to implement a good solution, I think all efforts/discussions died because of lack of time/contributions.

But for your proposed solution.. if you install it as a "real" dependency, it will look as if it was an orphan and will run the risk of being cleaned up because it looks like you don't need it. I guess that's where your proposed database changes come in? to note that an "orphan" is listed as an optdep for some package, and hence, warn the user?
Installing it explicitly is also not an option, like you said.

Since my C skills are too rusty I didn't start hacking on pacman yet but I did create a shell script that lets you manage your optdeps. See http://bbs.archlinux.org/viewtopic.php?id=59385
I think this is sort of what you're talking about, although implemented in a workaround script instead of pacman core.
Also the discussion here (towards the end, at least) is in line with your ideas http://www.nabble.com/"explicit-dependencies",-a-compromise-between-explicit-and-deps-td19940889.html

I think that basically, the issue can be reduced to : "install optdep as real dep, but note that it was actually an optdep when doing things like orphan searching" and "pacman can sometimes never be sure when it's removing package foo which has dep bar, and bar is also an optdep for baz, but not a dep for anything else. in that case just tell the user this and ask him if he wants to remove bar or not"

Xyne · 2009-01-11 19:16:29

Bascially, but my proposal also involves a concrete way to do this with minimal changes the pacman code (I think).
(This is more an open response to anyone not aware of the way pacman tracks deps in the database) The "database" stores plaintext files named "depends'. As an example, here is gimp's depends file:

%DEPENDS%
gtk2>=2.14.4
lcms>=1.17
libxpm>=3.5.7
libwmf>=0.2.8.4
libxmu>=1.0.4
librsvg>=2.22.3
libmng>=1.0.10
dbus-glib>=0.76
libexif>=0.6.16
pygtk>=2.13.0
desktop-file-utils
gegl>=0.0.22
curl

%CONFLICTS%
gimp-devel

%OPTDEPENDS%
gutenprint: for sophisticated printing only as gimp has built-in cups print support
libwebkit: for the help browser
poppler-glib: for pdf support
hal: for Linux input event controller module
alsa-lib: for MIDI event controller module

What I suggest is to create a second file that follows this format, named "optdepends" and save it in /var/lib/pacman/optdepends/<pkgname>. This would keep it outside of the database to avoid persistence issues across updates. As an example, if /var/lib/pacman/optdepends/gimp/optdepends looks like this:

%DEPENDS%
libwebkit

then pacman would handle libwebkit as a true dependency for gimp. All it needs to do internally is append the %DEPENDS% section of this fle to it's internal array that it creates from the %DEPENDS% section in the standard depends file.

Maybe it would be easier to simply add a section to the "depends" file called "%USERDEPS% and just make sure that this can be preserved across database updates. The disadvantage of this is that it makes writing tools for changing/adding/removing optional dependencies more difficult because they need to resolve version numbers. Having the files in a separate directory with directory names corresponding to the package name makes this much easier.

I don't think this would take much new code to implement and I'm hoping that some of the core devs will support this. I think one of the major obstacles is that too many suggestions have been made and that has led to default negative impressions of new suggestions.

C is on my list of things to learn though and I don't think I'll be leaving Arch in the foreseeable future, so if I can't convince anyone that this is a good way to go, I'll probably end up trying to code it myself... it just might take ages for me to get around to it, though.

Last edited by Xyne (2009-01-11 19:22:28)

ibendiben · 2009-01-11 23:50:11

Now I understand what you mean with those metapackages, I find it a good idea.
metapax seems candidate for pacman-contrib imo. Like rankmirrors and stuff. As you pointed out, there's a lot of potential uses for it.

As for the optdepend 'orphan' problem. It seems a small hassle. But since they are already listed in the db, why not have them checked before calling them orphans... If pacman finds 'orphans' that are listed as optdepends for some package(s) a single notification would do the trick, won't it?

Xyne · 2009-01-12 00:12:07

That would just be another kludge imo. I think pacman should be able to really handle them instead of just passing around messages. I basically want a way to tell pacman "I want you to treat bar as a dependency of foo" and let pacman work his wacka wacka magic. I see several benefits in doing it this way (it opens the doors for user-defined metapackages, simplified package management and migration, system backups, etc).

alex_anthony · 2009-01-12 18:04:36

With the gnome idea, what would it ask you when you wanted to upgrade? For example, gnome-extra has a lot of desirable stuff, but I don't want the Assistive Technology stuff. Would I have to go through every time gnome has even a minor version bump saying which packages I want? Or only on major versions? Or only when the optdepends changes? I can see the advantages for when new packages are added, but I don't want to have to go through the whole list every time there's a gnome upgrade.

Xyne · 2009-01-12 18:11:10

You wouldn't need to go through the dialogue each time. I think it should be presented the first time you install it and they whenever a new optdepend is added, you would get prompted for the new one during an update.

Also take a look at Allan's comment about this on flyspray (linked in the first post) and my response.

Dieter@be · 2009-01-12 21:22:29

Xyne wrote:

That would just be another kludge imo. I think pacman should be able to really handle them instead of just passing around messages. I basically want a way to tell pacman "I want you to treat bar as a dependency of foo" and let pacman work his wacka wacka magic. I see several benefits in doing it this way (it opens the doors for user-defined metapackages, simplified package management and migration, system backups, etc).

The problem is that pacman cannot always be *sure* about what it should do.
See http://www.nabble.com/Re%3A-%22explicit … 71183.html
So in some cases, there is no good solution other then telling the user "this is the situation, you have this package for these reasons, what do you want to do?"

Xyne · 2009-01-13 09:20:30

With this suggestion it would be sure, because this provides the "bookkeeping" mentioned in that message.
Thanks for that link though, it made me realize the weakness in Allan's suggestion in the flyspray discussion.

Xyne · 2009-01-13 16:21:33

*bump*
I've rewritten the first post.

u_no_hu · 2009-01-13 16:41:38

Libwebkit will now be handled exactly as if it were a true dependency of gimp. It is neither explicitly installed nor an orphan. It will get removed with gimp unless it's a depend or optdepend for another package.

If that is the case why do you need a separate optdepends database? why not add it to the %DEPENDS% of /var/lib/pacman/local/gimp-2.6.2-2/depends if the user wants it to be installed as a dependency? then pacman will handle it as if it was a normal dependency of gimp while updating,chcking for orphans etc.........

Xyne · 2009-01-13 17:41:56

u_no_hu wrote:

If that is the case why do you need a separate optdepends database? why not add it to the %DEPENDS% of /var/lib/pacman/local/gimp-2.6.2-2/depends if the user wants it to be installed as a
dependency? then pacman will handle it as if it was a normal dependency of gimp while updating,chcking for orphans etc.........

Then you would need to edit the database every time gimp was updated.

u_no_hu · 2009-01-13 18:44:12

Xyne wrote:

u_no_hu wrote:
If that is the case why do you need a separate optdepends database? why not add it to the %DEPENDS% of /var/lib/pacman/local/gimp-2.6.2-2/depends if the user wants it to be installed as a
dependency? then pacman will handle it as if it was a normal dependency of gimp while updating,chcking for orphans etc.........
Then you would need to edit the database every time gimp was updated.

If my understanding is correct, the local database will be updated with every update. (not sure though, any body who knows how pacman works can correct me)

shining · 2009-01-13 19:47:17

u_no_hu wrote:

Xyne wrote:
u_no_hu wrote:
If that is the case why do you need a separate optdepends database? why not add it to the %DEPENDS% of /var/lib/pacman/local/gimp-2.6.2-2/depends if the user wants it to be installed as a
dependency? then pacman will handle it as if it was a normal dependency of gimp while updating,chcking for orphans etc.........
Then you would need to edit the database every time gimp was updated.
If my understanding is correct, the local database will be updated with every update. (not sure though, any body who knows how pacman works can correct me)

If you mix up dependencies with installed optdependencies, you will get in troubles when you want to update.
When you upgrade a package, you want to replace the old deps list by the new one. But you probably want to keep the installed_optdeps list intact.
Optionally, you could check that installed_optdeps is still a subset of optdeps, in case the new package lost some optdeps.
I hope everything is clear

By the way, the sync/ subdir is "read only" and contains the sync database.
The local/ subdir is not, it contains all the infos about the installed packages, that is where the installed optdepends belongs.
Also, the number of the files contained by pacman databases already hit us hard performance wise. We definitely don't want to add more files.
This information definitely belongs to /var/lib/pacman/local/gimp-2.6.2-2/depends.
However, this file could contain the same %OPTDEPENDS% field as in /var/lib/pacman/sync/gimp-2.6.2-2/depends. And in fact, it will be that way with the future pacman version.
The natural way would be to add %INSTALLED_OPTDEPENDS% to /var/lib/pacman/local/gimp-2.6.2-2/depends.

I didn't comment on this issue before because it is just a minor detail. I commented on the idea itself in the bug tracker.

Dieter@be · 2010-07-31 12:10:11

I've given this another read and thought. I'm not sure about the specific implementation, but I like the general ideas.
Maybe, if the pacman devs don't like this, it would be better to implement this in another libalpm frontend.
My first idea was to get my optdep solution (https://bbs.archlinux.org/viewtopic.php?id=59385, which is conceptually pretty similar to yours, although it is implemented in userspace which is - now i think about it - pretty bad) into clyde. But now I think it would be better to include your logic.

Allan · 2010-07-31 12:21:11

Seem I do not think this has been mentioned yet, here is the design document on how I would like optdepends handled: http://wiki.archlinux.org/index.php/Use … OptDepends . With pacman-3.4, we have added restrictions on the optdepends format into makepkg so we can actually start coding this. I am intending to get to this but it stands somewhere below implementing a tar based sync backend and package signing for the next releases aims...

Dieter@be · 2010-07-31 13:20:35

Allan wrote:

Seem I do not think this has been mentioned yet, here is the design document on how I would like optdepends handled: http://wiki.archlinux.org/index.php/Use … OptDepends . With pacman-3.4, we have added restrictions on the optdepends format into makepkg so we can actually start coding this. I am intending to get to this but it stands somewhere below implementing a tar based sync backend and package signing for the next releases aims...

Allan, on your page you don't really mention any specific way for pacman to track installed optdeps, so I assume there is none in your proposal.
You just install --asdep, which is not good IMHO. See for example your example "optdepends are not orphans unless a flag is specified"
Since nowhere any info is stored on whether tk is installed as dep or optdep, pacman cannot know for sure. Surely you can use your flag --optdep, but as far as I can see this just goes over all orphans and checks if they are an optdep for some package and displays them accordingly. But this is decoupled from what the user wants. The user still needs to think about "did I install tk as optdep for python or not?" and make manual choices accordingly.

The advantage of the approach mentioned by Xyne, is that if pacman tracks optdeps, it can know for sure what it can and can't do (clean up). Without needing thought from the user.

Allan · 2010-07-31 13:24:42

Why do we need an extra field? pacman does not track which package a dep is installed for. When that information is needed, it goes through the dependency list of all installed packages and creates the list. The same will happen for optdeps. Perhaps something to be added to that document is how these are displayed in the "Required by" field in "-Qi".

Edit: added.

Allan · 2010-07-31 13:29:36

Allan wrote:

pacman does not track which package a dep is installed for. When that information is needed, it goes through the dependency list of all installed packages and creates the list

Just to clarify this: pacman used to store "required by" information in the local database and add/remove entries as needed. This turned out to be very error prone and full of breakage so it was decided that it is better not to store that information but instead calculate it as needed. The handling of optdepends needs to be the same or it will not make it into the pacman codebase.

Dieter@be · 2010-07-31 13:34:02

Allan wrote:

Why do we need an extra field? pacman does not track which package a dep is installed for. When that information is needed, it goes through the dependency list of all installed packages and creates the list. The same will happen for optdeps. Perhaps something to be added to that document is how these are displayed in the "Required by" field in "-Qi".

Well, suppose you have tk installed as a dep on your system. And you have no package depending on it, but you do have a package which lists tk as an optdep (say, python).
In this case, you cannot do [a variant of] `pacman -Rs $(pacman -Qdtq)` to automatically clean up all orphans. because there is no way to know if tk was installed as a real dep for a (now removed) package, or as optdep for python. Say you have another package foo which also has tk as an optdep, then you have the same problem. You might want to use tk-based feature of python, but not the tk-based feature of foo. So after you remove python, there is again no way for pacman to automatically know what it can clean up and what it cannot clean up. The user needs to check the lists manually and try to remember for which reasons he might have installed some optdeps.

So, when dealing with real dependencies it's easy, because you always need them if you have one or more packages depending on them. period. With optdeps there is a lot of uncertainty whether you need them or not, unless you explicitly track for which packages they are installed.

Last edited by Dieter@be (2010-07-31 13:36:05)

Allan · 2010-07-31 13:45:01

So say I install python and then tk as an optdepends. Then I install another package which requires tk as an optdepend. If I want to use tk as an optdep for both packages, I need to run some command to manually add foo to the optdepends required by list for tk?

I suppose that could be another use for the -D flag, but it seems kinda crap to manually have to modify the database to keep track of these optdeps. Especially given only new optdeps are displayed on upgrade (and my proposal would extend that to only uninstalled optdeps) so adding these entries might be missed.

I much prefer the more limited manual intervention required for "pacman -Qtd --optdep" which would let you scan the list of packages only installed as optdeps and make decisions. But I guess I have a very good knowledge of my system and know what optdeps I would want.

Anyway, if this is to go forward, it needs to be discussed on the pacman-dev mailing list (and have patches submitted...). As a heads up, Dan likes optdepends to be informational only (as the were to replace stupid post_install messages) so I get the impression that any interface to them will need to be quite simple to get approved.

Dieter@be · 2010-07-31 15:09:33

Allan wrote:

So say I install python and then tk as an optdepends. Then I install another package which requires tk as an optdepend. If I want to use tk as an optdep for both packages, I need to run some command to manually add foo to the optdepends required by list for tk?

hmm, with my approach (and I think Xyne's also), that would indeed be the case.

Allan wrote:

I suppose that could be another use for the -D flag, but it seems kinda crap to manually have to modify the database to keep track of these optdeps. Especially given only new optdeps are displayed on upgrade (and my proposal would extend that to only uninstalled optdeps) so adding these entries might be missed.
I much prefer the more limited manual intervention required for "pacman -Qtd --optdep" which would let you scan the list of packages only installed as optdeps and make decisions. But I guess I have a very good knowledge of my system and know what optdeps I would want.

Well, it seems both approaches now have their pros and cons:
* track optdeps separately in db -> pacman can do more automatically (given some manual work to maintain the database), non-trivial implementation
* don't track optdeps separately -> cleaning up your system cannot be done automatically, you need to know why you installed certain stuff, simple implementation.

When I want to clean up orphans I will probably also know what can go and what can stay. However, I appreciate it when pacman can figure out things by itself. Especially, if you create backups by saving a list of installed packages (in a crontab or whatever), and restoring the system by reinstalling the packages. For this to work fluently, you need to explicitly track optdeps. But I guess the simple implementation is best for most users, and those folks who want something more advanced can still write their own scripts and wrappers (like i have already done anyway)

Xyne · 2010-08-01 03:06:42

Dieter@be wrote:

Allan wrote:
So say I install python and then tk as an optdepends. Then I install another package which requires tk as an optdepend. If I want to use tk as an optdep for both packages, I need to run some command to manually add foo to the optdepends required by list for tk?
hmm, with my approach (and I think Xyne's also), that would indeed be the case.

Fsck, I had written an extensive reply but my login timed out and I lost it when I hit the back button. Oh well, take 2...

Nope, that's not how my idea would work. Allan has inverted the tracking. My idea lets packages track their own optdepends. Here's how my idea would work using Allan's example. Note that all filepaths and syntax is just a conceptual example, so don't get hung up on names, flags and other trivialities:

Already installed: python, foo
Not installed: tk
Existing optdeps: none
Database directory: $dbDir

Install tk as an optdep for python (this could accept multiple optdepends):

pacman -S --asdepsfor python tk

What it would do:
* install tk as a dep (e.g. pacman -S --asdeps tk)
* create $dbDir/extra/python/optdepends:

%OPTDEPENDS%
tk

When pacman checks dependencies to determine which files are required by others and which files are orphans, it already checks $dbDir/local/$pkgname-$pkgver-$pkgrel/depends for every package in the local database. When checking "depends" for $pkgname, it would also check if $dbDir/extra/$pkgname/optdepends exists and, if it does, internally append all packages in it to %DEPENDS% (i.e. in memory, not in the file). This would allow any package listed in $dbDir/extra/$pkgname/optdepends to be treated as a first-class dependency. The additional overhead should be negligible because most local packages will not have any optdepends listed in $dbDir/extra/, so only the existence of the optdepends file needs to be checked.

To make tk an optdep of foo too, simply run

pacman -S --asdepsfor foo tk

What it would do:
* create $dbDir/extra/foo/optdepends:

%OPTDEPENDS%
tk

It would not re-install tk.

When removing $pkgname, pacman would check if $dbDir/extra/$pkgname/optdepends exists. If it does then it would apply the following logic to each package ($optdep) in that list:

if ($optdep is installed as a dep and ($optdep is neither a dep nor user-defined optdep))
{
  remove $optdep
}

where "user-defined optdep" is any dep specified in $dbDir/extra/*/optdepends

Removing foo would thus leave tk installed because it is still listed as a user-defined optdep for python (and vice versa when removing python first). Removing python would then remove tk.

Instead of removing python the user could remove tk as its optdep:

pacman -R --asdepsfor python tk

This would remove "tk" from $dbDir/extra/python/optdepends and, because the list is empty, delete it. Removing tk as an optdep would follow the same logic above as if python had been removed, i.e. remove tk if it is installed as a dep and no longer required as a dep or user-defined optdep.

Other notes:
* Any package could be made an optdep of any other package. The optdeps in the sync database would remain suggestions in that regard. They would nevertheless be used to create a dialogue similar to the group dialogue, as described in the OP. This dialogue could be invoke via e.g. "pacman -S --optdeps foo". Beyond that, pacman could also display a message when an optdep is removed from the sync database if it is detected as a user-defined optdep, e.g. "bar is no longer listed as an optdep for foo".

* Groups could be replaced by metapackages with optdeps. As mentioned in the previous note, it would be possible to bring up a dialogue identical to the current group dialogue so there would be no difference in installation. The benefit would be that groups could be updated (e.g. the KDE metapackage could change its optdepend list and the user would then be informed about it on the next upgrade, with the option to resync his optdeps to the current sync db optdeps). Users could also easily define their own metapackages for e.g. system backup and migration, organization (e.g. all games in one metapackage, all office apps in another).

* optdepends files could be handled in 1 of 2 ways when removing a package. They could be left in $dbDir/extra in case the user wants to re-install the package later with the same optdeps, or they could be removed with the package.

The first approach would thus ignore entries in $dbDir/extra when the corresponding package is not installed, but that requires no extra logic as the optdepends file would only be read via a hook when reading the depends file in $dbDir/local/, which itself only exists when the package is installed. Leftover files could be cleaned up with a hook when running "pacman -Sc", which already prompts for cleaning the cache and $dbDir/sync/.

Last edited by Xyne (2010-08-01 03:16:41)

Arch Linux

#1 2009-01-08 02:38:23

A suggestion for handling optdepends.

#2 2009-01-11 15:11:58

Re: A suggestion for handling optdepends.

#3 2009-01-11 17:53:40

Re: A suggestion for handling optdepends.

#4 2009-01-11 18:42:32

Re: A suggestion for handling optdepends.

#5 2009-01-11 19:16:29

Re: A suggestion for handling optdepends.

#6 2009-01-11 23:50:11

Re: A suggestion for handling optdepends.

#7 2009-01-12 00:12:07

Re: A suggestion for handling optdepends.

#8 2009-01-12 18:04:36

Re: A suggestion for handling optdepends.

#9 2009-01-12 18:11:10

Re: A suggestion for handling optdepends.

#10 2009-01-12 21:22:29

Re: A suggestion for handling optdepends.

#11 2009-01-13 09:20:30

Re: A suggestion for handling optdepends.

#12 2009-01-13 16:21:33

Re: A suggestion for handling optdepends.

#13 2009-01-13 16:41:38

Re: A suggestion for handling optdepends.

#14 2009-01-13 17:41:56

Re: A suggestion for handling optdepends.

#15 2009-01-13 18:44:12

Re: A suggestion for handling optdepends.

#16 2009-01-13 19:47:17

Re: A suggestion for handling optdepends.

#17 2010-07-31 12:10:11

Re: A suggestion for handling optdepends.

#18 2010-07-31 12:21:11

Re: A suggestion for handling optdepends.

#19 2010-07-31 13:20:35

Re: A suggestion for handling optdepends.

#20 2010-07-31 13:24:42

Re: A suggestion for handling optdepends.

#21 2010-07-31 13:29:36

Re: A suggestion for handling optdepends.

#22 2010-07-31 13:34:02

Re: A suggestion for handling optdepends.

#23 2010-07-31 13:45:01

Re: A suggestion for handling optdepends.

#24 2010-07-31 15:09:33

Re: A suggestion for handling optdepends.

#25 2010-08-01 03:06:42

Re: A suggestion for handling optdepends.

Board footer