namcap: outside of a valid path

solskogen · 2017-12-19 09:48:29

Hi!

I'm about to create a PKGBUILD for a cross compiler, and before I submit it I wanted to check it with namcap.
namcap does not complain about the PKGBUILD but there are some errors and warnings on the built package.

m68k-elf-toolchain E: ELF file ('usr/m68k-elf/lib/mfidoa/fido-ram-crt0.o') outside of a valid path.
m68k-elf-toolchain E: Cross-directory hardlink in package (usr/bin/m68k-elf-as, usr/m68k-elf/bin/as)

Is this okay?

Also some warnings like this:

m68k-elf-toolchain W: File (usr/m68k-elf/include/libgen.h) exists in a non-standard directory.

eschwartz · 2017-12-19 16:38:30

This is totally okay, it is a warning because namcap isn't smart enough to figure out that for cross-compilers these are normal directories.
See for example the arm-none-eabi-* packages in [community].

As for the hardlink, those are conventionally symbolic links, I don't know why m68k is using hardlinks instead.

solskogen · 2017-12-19 22:04:39

Ok, here it is!

https://aur.archlinux.org/packages/m68k-elf-toolchain

Please review if you have the time! :-)

eschwartz · 2017-12-20 15:11:44

1) You need to quote "$srcdir" and "$pkgdir", as they can contain spaces depending on the directory the PKGBUILD is in.

2) Don't use find | xargs, use find -print0 | xargs -0. Or really, since this is a more involved pipeline, I would use something like:

local regex='ELF ().*(executable|shared object).*'
while read -r -d '' filename; do
    if [[ $(file -b "$filename") =~ $regex ]]; then
        # do stuff on executable files
    fi
done < <(find ... -print0)

But I don't really understand why you don't simply use the standard strip routines and are doing it manually instead. Also it does a perfectly good case statement against file -bi which works quite nicely I think...

3) What is the benefit of a unified package when split packages already exist -- note that it seems to make the pkgver rather awkward.

4) Why are you doing this?

#hack to speed up compilation a bit by replacing debug info with -pipe.
sed -i 's/\-g /-pipe /' Makefile

-pipe is already in the default makepkg.conf and you shouldn't be overriding the user choice to remove that, nor should you do so by unilaterally disabling debug flags that the user set by enabling debug in their makepkg.conf

5) Depending on the order of globs in `ln -sv ../binutils-*/$i` is as a general rule of thumb fragile, as it depends on LC_COLLATE for ordering and breaks if it ends up matching more than two file paths. Although in this case it is likely safe.

6) You do not need to use backslashes to escape newlines when declaring the source=() array. Notice how on the next line, the sha512sums array is not backslash-escaped.

Last edited by eschwartz (2017-12-20 15:16:39)

solskogen · 2017-12-20 19:03:20

1) Fixed!

2) Almost fixed. I found that little snipplet of your a bit hard to understand, but I've added print0 and 0 to find and xargs.

I need to strip manually because makepkg doesn't handle cross compiled packages very well. In order to strip a binary or library for another architecture you must use $target-strip.

3) It doesn't exist, but it might seem like so. The packages in question only support one cpu, and not all of them.
IMHO the three packages are so dependent of each other that it makes almost no sense to have them seperated. Also this way, you don't need the seperate bootstrap-gcc package, since all of that magic is already in gcc/binutils/newlib if you combine them.

4) Remenants of some old script I had. Removed it now.

5) Fixed!

6) And fixed.

solskogen · 2017-12-20 19:03:53

And I forgot to add: Thanks for taking your time to review it!

eschwartz · 2017-12-20 20:33:31

Sure.

As for the separately implemented stripping using the package's own strip binaries, interesting point. I know you can specify STRIP_SHARED etc. in makepkg.conf, sadly it seems that using your own strip binaries will not work. You might be able to get around that by manipulating the PATH but I guess at that point it is not worth it.

And find -print0 | xargs -0 doesn't quite work when you pass the output to file/grep/cut, but all those programs also allow you to specify -z or -0 to operate on null-terminated data as well. Unfortunately, I just realized file messes this up due to having multiple fields that must also be separated by nulls, meaning for grep you must use grep -zZ -B1 and then grab every other null-terminated line instead of using cut.

Basically, that snippet *was* the easy option. I can explain what it does if you like.

# local variable is scoped to the function, for general tidiness. 
local regex='ELF ().*(executable|shared object).*'
# read null-terminated filenames from stdin, and use a while loop to operate on each one
# for each run of the loop, the filename is stored in the intuitive variable "filename". :) 
while read -r -d '' filename; do
    # test if the output of `file` matches the regular expression defined earlier
    if [[ $(file -b "$filename") =~ $regex ]]; then
        # awesome, it matches! So, do the standard strip routine since this isn't an m68k executable
        strip --strip-unneeded "$filename"
    fi
# this find command uses process substitution to pass the output of find into the `while read` loop
done < <(find ... -print0)

# finish off by stripping m68k executables
# - find accepts multiple starting points, so combine both find commands into one
# - find also has the -exec option which can replace a single use of xargs
find "${pkgdir}/usr/lib/gcc/${_target}" "${pkgdir}/usr/${_target}/lib" -type f -name '*.o' -o -name '*.a' -exec "${pkgdir}"/usr/bin/${_target}-strip -g {} +

This makes use of Process Substitution (more on this later), and bash regular expressions which is faster than using external commands like grep.

You can also use `find ... print0 | while read -rd '' filename; do` instead of process substitution, either one passes the stdout for `find` into the stdin for `read`. It's mostly a stylistic choice to use < <() versus pipes, in the same way `cat somefile | othercommand` is a stylistic choice over `othercommand < somefile` -- although granted in the specific case of cat, using the shell < is vastly superior to a Useless Use Of Cat.
You can also replace [[ $(file -b "$filename") =~ $regex ]] with the `case ... esac` loop from makepkg's own strip implementation, as referenced here: https://git.archlinux.org/pacman.git/tr … e6a56#n115
Then you'd just run `strip $strip_flags $binary`.

But that might be overkill just to get --strip-all only for binary executables while still using strip-unneeded for shared libraries and -g for .o files. Instead you could just use the following additional arguments to find, in order to make it ignore .o files and pass shared libraries and executables to "the standard strip routine":

-name \*.o -prune -o -type f

What this does is first tell `find` to locate .o files, and exclude them from the search path. Then, afterwards, to look for anything that is a file and not a directory. You will *need* to use -print or -print0 for this.
See https://stackoverflow.com/questions/148 … 05#1489405 for more details about using find.

Last edited by eschwartz (2017-12-20 20:40:04)

solskogen · 2017-12-21 10:21:29

Again, I really appreciate the time you take here. It's been a while since I wrote my first PKGBUILD (2005!) and I always have something to learn when it comes to bash scripts.

I've updated the pckages now. I still want to use xargs, as it doesn't print out all findings that find does when using exec.
Also, I found out that the PKGBUID is very dynamic. Just change _target to for instance arm-none-eabi, and it will compile that. powerpc-none-eabi works as well.

eschwartz · 2017-12-25 20:28:46

I don't see find printing anything out, what are you seeing???

Arch Linux

#1 2017-12-19 09:48:29

namcap: outside of a valid path

#2 2017-12-19 16:38:30

Re: namcap: outside of a valid path

#3 2017-12-19 22:04:39

Re: namcap: outside of a valid path

#4 2017-12-20 15:11:44

Re: namcap: outside of a valid path

#5 2017-12-20 19:03:20

Re: namcap: outside of a valid path

#6 2017-12-20 19:03:53

Re: namcap: outside of a valid path

#7 2017-12-20 20:33:31

Re: namcap: outside of a valid path

#8 2017-12-21 10:21:29

Re: namcap: outside of a valid path

#9 2017-12-25 20:28:46

Re: namcap: outside of a valid path

Board footer