Why does namcap say I don't need stuff I definitely do need?

colinkeenan · 2014-09-22 05:39:30

I have developed some bash scripts using ffmpeg, imagemagick's convert, yad, and lots of other tools that I had to explicitly install, but namcap claims I don't need to list any of those dependencies. It's called silentcast and makes it easy to create a variety of animated gif recordings of the screen. I haven't put it in the AUR yet because I'm still working on the README.md, but the package is basically ready for use and available on github: https://github.com/colinkeenan/silentcast/#silentcast. I created the animated gifs in that README.md with silentcast itself.

Anyway, I was listing all the dependencies in a table in the README.md and realized I had missed a lot of stuff so added more. Then, still worried I had missed something, I read the wiki to find out how to test for missing dependencies and discovered namcap. First, I tried running namcap on PKGBUILD, but that didn't do anything. Next, I did a makepkg -s and ran namcap on the package just created. Here's what namcap has to say:

namcap -i silentcast-1.0-1-any.pkg.tar.xz

    silentcast I: Script link detected (bash) in file ['usr/bin/genffcom', 'usr/bin/silentcast', 'usr/bin/temptoanim']
    silentcast I: Script link detected (python) in file ['usr/share/silentcast/transparent_window.py']
    silentcast W: Dependency included and not needed ('ffmpeg')
    silentcast W: Dependency included and not needed ('imagemagick')
    silentcast W: Dependency included and not needed ('yad')
    silentcast W: Dependency included and not needed ('xorg-xrandr')
    silentcast W: Dependency included and not needed ('wmctrl')
    silentcast W: Dependency included and not needed ('xdotool')
    silentcast W: Dependency included and not needed ('xorg-xwininfo')
    silentcast W: Dependency included and not needed ('python-gobject')
    silentcast W: Dependency included and not needed ('python-cairo')
    silentcast W: Dependency included and not needed ('xdg-utils')
    silentcast I: Dependency covered by dependencies from link dependence (bzip2)
    silentcast I: Dependency covered by dependencies from link dependence (readline)
    silentcast I: Dependency covered by dependencies from link dependence (linux-api-headers)
    silentcast I: Dependency covered by dependencies from link dependence (gdbm)
    silentcast I: Dependency covered by dependencies from link dependence (perl)
    silentcast I: Dependency covered by dependencies from link dependence (glibc)
    silentcast I: Dependency covered by dependencies from link dependence (iana-etc)
    silentcast I: Dependency covered by dependencies from link dependence (openssl)
    silentcast I: Dependency covered by dependencies from link dependence (ncurses)
    silentcast I: Dependency covered by dependencies from link dependence (gcc-libs)
    silentcast I: Dependency covered by dependencies from link dependence (db)
    silentcast I: Dependency covered by dependencies from link dependence (expat)
    silentcast I: Dependency covered by dependencies from link dependence (tzdata)
    silentcast I: Dependency covered by dependencies from link dependence (filesystem)
    silentcast I: Dependency covered by dependencies from link dependence (libffi)
    silentcast I: Dependency covered by dependencies from link dependence (sh)
    silentcast I: Dependency covered by dependencies from link dependence (zlib)
    silentcast I: Depends as namcap sees them: depends=(bash python)

So, namcap thinks I only need bash and python! This is crazy. All the stuff I listed as dependencies are explicitly called in the bash scripts or imported in the python script because they were needed in that script. Many of these I had to install myself and were certainly not included with bash and python! Why is namcap saying I don't need this stuff?

It occurs to me that maybe I haven't properly created the PKGBUILD so that namcap thinks the dependencies are covered by installing the various included scripts. Do I have to somehow make it clear all those scripts ARE the package? Here is my PKGBUILD.

# Maintainer: Colin Keenan <colinnkeenan at gmail dot com>

pkgname=silentcast
pkgver=1.0
pkgrel=1
pkgdesc="Silent Screencast: video record your screen and make it into an animated gif"
arch=('any')
url="https://github.com/colinkeenan/silentcast"
license=('GPL')
depends=('ffmpeg' 'imagemagick' 'yad' 'xorg-xrandr' 'wmctrl' 'xdotool' 'xorg-xwininfo' 'python-gobject' 'python-cairo' 'xdg-utils')
install=${pkgname}.install

source=('silentcast' 'genffcom' 'temptoanim' 'transparent_window.py' 'silentcast.desktop' 'README.md')
md5sums=('de05e0140e4640bf0b52077f4ade5a26'
         '3b7311751662d8bd3034e4720239b070'
         '6ce4638b3358ca388838cf650083a528'
         '6bf9e695750a3f65e4421544c53bdb96'
         'f1641d3959f479a97b6bcd26f103f900'
         '539141058e7f8bcb99218e632a817e1e')

package() {
  install -D -m755 silentcast "$pkgdir/usr/bin/silentcast"
  install -D -m755 genffcom "$pkgdir/usr/bin/genffcom"
  install -D -m755 temptoanim "$pkgdir/usr/bin/temptoanim"
  install -D -m755 transparent_window.py "$pkgdir/usr/share/silentcast/transparent_window.py"
  install -D -m755 silentcast.desktop "$pkgdir/usr/share/applications/silentcast.desktop"
  install -D -m755 README.md "$pkgdir/usr/share/doc/silentcast/README.md"
}

Last edited by colinkeenan (2014-09-22 05:56:37)

Scimmia · 2014-09-22 06:23:50

namcap isn't good with scripts.

colinkeenan · 2014-09-22 06:25:40

If namcap is no good for scripts, is there any way other than me just being really careful to verify I've covered all the dependencies?

colinkeenan · 2014-09-22 06:40:50

I see that there's a feature request to get namcap to stop saying dependencies aren't needed when it actually has not checked anything: https://bugs.archlinux.org/task/28694

I think anyone reading this should vote for that feature at that above link.

Last edited by colinkeenan (2014-09-22 06:41:36)

tomk · 2014-09-22 06:56:04

I'm a bit puzzled here. You wrote these scripts, how can you not know what the dependencies are?

Apart from that, I'm guessing you would like users of all distros to benefit from your work, so don't rely on a tool that only checks one package format.

colinkeenan · 2014-09-22 15:11:11

tomk wrote:

I'm a bit puzzled here. You wrote these scripts, how can you not know what the dependencies are?
...

The specific answer I was looking for and hoping namcap would give me had to do with xdg-open from the xdg-utils package. I use it to open the desktop's file-browser. On the Packages page for xdg-utils, optional dependencies are listed according to what desktop it's being run on. So, for example, I use Xfce and it lists 'exo' and 'xorg-xprop' as optional dependencies. I have no idea if those optional dependencies are actually required in order for my script to open the file-browser in Xfce. If so, it would be very difficult to actually spell out the dependencies of my script because it would depend on what desktop the user has installed.

Just now I checked

pacman -Qi exo

and see that I did explicitly install it but that it's also a dependency for Thunar. So, I'm going to assume that whatever xdg-utils needs to open the file-browser will have been installed as a dependency for the file-browser in question. So, I'm not going to worry about that anymore.

A more general concern I had was that each time I read through my scripts, I realized I had missed another dependency. It's not as if I was keeping a separate list as I went along. So, I wanted namcap to double check if I had overlooked something. I'll just have to go through my scripts over and over until I don't see or remember anything else.

Another concern: all my scripts are bash and python but I haven't specified bash or python as dependencies yet. I didn't specify python because it's a dependency of other python modules that I did specify. I also didn't specify bash because it seems like other shells could work as well. But, I googled how to do stuff and used a couple of snippets of code and built on them from there. I often solved the same problem in different ways according to whatever snippet of code I happened to find from google. Sometimes, when I found a method I preferred, I'd go back and change my other code to match, but not always. But, I did end up liking and using 'let $x=$x+$y' types of arithmetic, for example, and that requires a certain version of bash. Should I be listing bash > whatever version as a dependency?

In general, do each of my dependencies need to be a certain version or greater for my scripts to work?

These are the types of questions I wanted namcap to answer for me.

Last edited by colinkeenan (2014-09-22 15:18:48)

owain · 2014-09-22 15:13:08

colinkeenan wrote:

If namcap is no good for scripts, is there any way other than me just being really careful to verify I've covered all the dependencies?

Install a base system on a VM, install your package, check for errors when used?

colinkeenan · 2014-09-22 15:24:52

owain wrote:

Install a base system on a VM, install your package, check for errors when used?

I may do that, but then again, I'm not sure I care quite *that* much. I'm not even sure anyone else will use this. So, I'm just going to hope for feedback from those that are having issues if anyone else actually uses silentcast. But, I do realize that a base system in a VM would be the best way for me to verify I've listed all the dependencies. It probably won't help me to decide on versions for those dependencies though.

Scimmia · 2014-09-22 15:30:17

Versions of the dependencies don't matter in Arch. Users are expected to be on the current version as partial updates are not supported. The only time versions are useful are when you need something OTHER than what's in the repos (ie, weston needs libinput 0.4, but the repos are stuck at 0.2, or when the current version doesn't work and you need an older one.).

Last edited by Scimmia (2014-09-22 15:31:03)

Trilby · 2014-09-22 15:37:03

colinkeenan wrote:

The specific answer I was looking for and hoping namcap would give me had to do with xdg-open from the xdg-utils package. I use it to open the desktop's file-browser. On the Packages page for xdg-utils, optional dependencies are listed according to what desktop it's being run on. So, for example, I use Xfce and it lists 'exo' and 'xorg-xprop' as optional dependencies. I have no idea if those optional dependencies are actually required in order for my script to open the file-browser in Xfce.

How can you not know? Do you use exo or xprop in the script or not? If you only use xdg-open, then only specify that as a dependency. Users can then figure out which specific tools they wish to use to satisfy that dependency. I use my own replacement for xdg-open, and if I wanted I could package it listing "xdg-open" in the provides array so your tool would then call my xdg-open replacement tool. It's not up to you to worry about how other users chose to implement xdg-open unless your scripts depend on a specific implementation.

Much the same could be bash. If your script uses a bash hashbang, then bash is a dependency. If you use a "sh" hashbang than any shell will do - but then you must use only posix shell commands and conditionals - there are a lot of "bashisms" that will not work in other shells (e.g. the [[ bash builtin for conditionals). But even if bash is a dependency it does not need to be listed: you should not list dependencies that are in base or base-devel.

On a related question that you touched on in this thread, you do not need to list recursive dependencies. If you list python modules that have python as a hard dependency, then you do not need to list python. But just like the 'bashisms' example above, if you use a specific version of python (2.7 or 3) these are generally not compatible. The modules might not care which python is installed to meet it's needs, but your script may require a specific implementation of python. In such a case, you'd have to specify which python you used in addition to the python module(s).

EDIT:
Bear in mind I just whipped this up on the spot, and I have no idea how reliable it is - but in principle the following line should find dependencies for any script or anything else for that matter - provided that simply running the script/command will use everything that is required:

pacman -Qqo $(strace ./script  2>&1 | awk '/^(open|stat|access)/ {sub(/[a-z]*\("/,"",$1); sub(/",/,"",$1); print $1}') 2>/dev/null | sort -u

This is assuming the script is in the current working directory and named "script". I'll put together a more flexible and useful version momentarily.

Scimmia · 2014-09-22 15:43:36

Trilby wrote:

... But even if bash is a dependency it does not need to be listed: you should not list dependencies that are in base or base-devel.

Hold on there. You *can* omit dependencies in the base group, but I certainly wouldn't say *should*. That's completely up to the packager.

Base-devel can only be assumed at build time, not at run time.

Trilby · 2014-09-22 15:54:07

I see the wiki page on PKGBUILDs does not actually specify anything about base, but it does say that base-devel packages *should* be omitted not only from the makedepends, but also from the depends. If we're considering cases where a package is built on one machine and installed on others, then leaving out base-devel members from dependencies would be worse than leaving out base members - yet the wiki said they should be left out.

Scimmia · 2014-09-22 15:56:37

Trilby wrote:

I see the wiki page on PKGBUILDs does not actually specify anything about base, but it does say that base-devel packages *should* be omitted not only from the makedepends, but also from the depends. If we're considering cases where a package is built on one machine and installed on others, then leaving out base-devel members from dependencies would be worse than leaving out base members - yet the wiki said they should be left out.

Then the wiki is wrong.

colinkeenan · 2014-09-22 16:30:12

Trilby wrote:

...
Bear in mind I just whipped this up on the spot, and I have no idea how reliable it is - but in principle the following line should find dependencies for any script or anything else for that matter - provided that simply running the script/command will use everything that is required:
pacman -Qqo $(strace ./script  2>&1 | awk '/^(open|stat|access)/ {sub(/[a-z]*\("/,"",$1); sub(/",/,"",$1); print $1}') 2>/dev/null | sort -u
This is assuming the script is in the current working directory and named "script". I'll put together a more flexible and useful version momentarily.

Thanks, I gave that a try, but it only listed the following:

bash
coreutils
filesystem
glibc
ncurses
readline

Which would seem to be the dependencies of the actual packages I'm using in the script, and not sufficient for my script at all. I think I'll go ahead and list bash as a dependency though. Since my script requires a systray notification area, I think it can be safely assumed the rest is already installed.

By the way, is there some common base package that all systray notification areas use across different DEs? I'm thinking no, which is why I'm not trying to list that as an explicit dependency. But, anyone taking a glance at my README.md will realize they will need it for reasonable operation.

colinkeenan · 2014-09-22 16:35:00

Trilby wrote:

... if you use a specific version of python (2.7 or 3) these are generally not compatible. The modules might not care which python is installed to meet it's needs, but your script may require a specific implementation of python. In such a case, you'd have to specify which python you used in addition to the python module(s).

Thanks for pointing that out, especially for others that may read this thread. I was already aware and had tested my script against both python2 and python3 and it works for both. At first, I was using a snip of code that imported gtk which is only available for python2, but I changed it to another snip of code that does 'from gi.repository import Gtk' which works in both versions.

tomk · 2014-09-22 17:11:41

colinkeenan wrote:

A more general concern I had was that each time I read through my scripts, I realized I had missed another dependency. It's not as if I was keeping a separate list as I went along.

Sounds like a good idea for your next project.

Trilby · 2014-09-22 17:43:21

I'd be curious which dependencies my ugly line missed. If the script only uses certain components when specific flags are passed, that could explain it - so the command line could be replicated with various invocations of the script. But every single file that is touched in the running of the script should be reported - so I'm curious what a dependency could be that it misses.

Arch Linux

#1 2014-09-22 05:39:30

Why does namcap say I don't need stuff I definitely do need?

#2 2014-09-22 06:23:50

Re: Why does namcap say I don't need stuff I definitely do need?

#3 2014-09-22 06:25:40

Re: Why does namcap say I don't need stuff I definitely do need?

#4 2014-09-22 06:40:50

Re: Why does namcap say I don't need stuff I definitely do need?

#5 2014-09-22 06:56:04

Re: Why does namcap say I don't need stuff I definitely do need?

#6 2014-09-22 15:11:11

Re: Why does namcap say I don't need stuff I definitely do need?

#7 2014-09-22 15:13:08

Re: Why does namcap say I don't need stuff I definitely do need?

#8 2014-09-22 15:24:52

Re: Why does namcap say I don't need stuff I definitely do need?

#9 2014-09-22 15:30:17

Re: Why does namcap say I don't need stuff I definitely do need?

#10 2014-09-22 15:37:03

Re: Why does namcap say I don't need stuff I definitely do need?

#11 2014-09-22 15:43:36

Re: Why does namcap say I don't need stuff I definitely do need?

#12 2014-09-22 15:54:07

Re: Why does namcap say I don't need stuff I definitely do need?

#13 2014-09-22 15:56:37

Re: Why does namcap say I don't need stuff I definitely do need?

#14 2014-09-22 16:30:12

Re: Why does namcap say I don't need stuff I definitely do need?

#15 2014-09-22 16:35:00

Re: Why does namcap say I don't need stuff I definitely do need?

#16 2014-09-22 17:11:41

Re: Why does namcap say I don't need stuff I definitely do need?

#17 2014-09-22 17:43:21

Re: Why does namcap say I don't need stuff I definitely do need?

Board footer