You are not logged in.

#1 2020-02-20 14:12:39

GFdevelop
Member
Registered: 2017-09-18
Posts: 24

Node.js packaging wiki review

(I am not a veteran of nodejs packaging)
Assume as example the package that I am co-maintainer angular-cli;
I want to propose the following changes to the Node.js package guidelines

  • I want to add a tip to easly find the sha1sum of a package:

    curl -s "https://registry.npmjs.org/NPMID" | jq '.versions."PKGVER".dist.shasum'
  • Why xargs?

    find "$pkgdir" -name package.json -print0 | xargs -r -0 sed -i '/_where/d'

    Is this a better alternative?

    find "${pkgdir}" -type f -name package.json -exec sed -i -e '/_where/d' {} \;
  • Instead of this long block:

    local tmppackage="$(mktemp)"
    local pkgjson="$pkgdir/usr/lib/node_modules/$pkgname/package.json"
    jq '.|=with_entries(select(.key|test("_.+")|not))' "$pkgjson" > "$tmppackage"
    mv "$tmppackage" "$pkgjson"
    chmod 644 "$pkgjson"

    we can use this oneline?

    sed -i -n -e '1p;/  "[^_].*": {$/,$p' "$pkgdir"/usr/lib/node_modules/NPMID/package.json

    This print the first line (there is only the symbol '{'), then skip all keys that start with an underscore and print all remaining lines.
    (and you don't need jq as makedepends)

See the angular-cli package for understand better.

What do you think? We can do these changes?

Offline

#2 2020-02-20 18:11:04

Awebb
Member
Registered: 2010-05-06
Posts: 5,354

Re: Node.js packaging wiki review

Offline

#3 2020-02-20 18:34:12

Slithery
Forum Moderator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 3,697

Re: Node.js packaging wiki review

Try adding your ideas to the relevant talk page of the wiki as well - that's where changes are usually discussed.
https://wiki.archlinux.org/index.php?ti … &redlink=1


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#4 2020-02-20 19:24:57

GFdevelop
Member
Registered: 2017-09-18
Posts: 24

Re: Node.js packaging wiki review

I am see this https://www.everythingcli.org/find-exec-vs-find-xargs/

time find "${pkgdir}" -type f -name package.json -exec sed -i -e '/_where/d' {} \;
real	0m0,592s
user	0m0,569s
sys	0m0,060s
time find "${pkgdir}" -type f -name package.json -exec sed -i -e '/_where/d' {} \+
real	0m0,050s
user	0m0,037s
sys	0m0,011s
time find "${pkgdir}" -type f -name package.json -print0 | xargs -r -0 sed -i '/_where/d'
real	0m0,048s
user	0m0,040s
sys	0m0,005s
Slithery wrote:

Try adding your ideas...

Oh great, thanks.

Offline

#5 2020-02-21 02:47:46

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,159

Re: Node.js packaging wiki review

The author of this blog (who is so confident in his knowledge that he's going around trying to advise other people on how to use find) is a fool that doesn't realize:

find . -name "*.jpg" -exec ls {} +

(Alternatively, said author is being disingenuous by not mentioning that his comparison only applies to -exec {} \; which the common advice is to not use if you can get away with -exec {} + instead.)

This completely nullifies the given argument for xargs on every possible level, and you are left with find -exec being much, much safer for passing arguments around, plus being faster by one pipeline and one subshell fork to `xargs`.

(I will acknowledge that GNU findutils xargs has many useless options to do things you don't need, which if you do need anyway, find -exec cannot replace. And for POSIX xargs, there is the -L, -n, -s arguments, or the -t/--verbose option which is implied by -p/--interactive, which in turn is equivalent to find -ok. find does not have a direct analogue to xargs -t.)

@GFdevelop,

Using xargs here seems silly indeed, but should be replaced by the "-exec command {} +" variant.

Finding the sha1sum as a tip seems fairly reasonable. It would be something the packager does by hand before updating the PKGBUILD...

As for replacing jq with sed... you're relying on the fact that sed natively supports editing a temporary copy of a file and then updating the original, to avoid needing to construct a tempfile yourself (jq doesn't have an --inplace option to write its output back to the input file). This would seem fairly reasonable, except for one thing: how sure are you that package.json will have such a regular structure? A proper json parsing tool has the advantage over sed, of being able to handle arbitrary formatting.

Last edited by eschwartz (2020-02-21 02:53:27)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#6 2020-02-21 13:26:55

Awebb
Member
Registered: 2010-05-06
Posts: 5,354

Re: Node.js packaging wiki review

which the common advice is to not use if you can get away with -exec {} + instead

Didn't know that, thank you. The whole -exec vs. xargs discussion has no definitive answer and boils down to " use what's most efficient in your scenario". Now OP can try the + and use the fastest.

Offline

#7 2020-02-21 14:31:06

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,159

Re: Node.js packaging wiki review

More or less, yeah, with the caveat that as you have seen, people are often wrong on the internet due to not truly understanding the tools they have -- so they hear of a feature one program has, but don't hear about the same feature another program has, and start blogging about how program #1 is better because only it has the feature. big_smile

GNU findutils xargs does have the -P/--max-procs option which is a simple implementation of `parallel`, and that's about the only practical use I've ever had for xargs specifically, I think. So if your program greatly benefits from parallelism and it doesn't have a mode to operate on many files at once (thus allowing use of -exec {} +) then that may come in handy. The short form -P is also available in busybox xargs and freebsd xargs, so I'd call it fairly reliably there.

POSIX xargs -I {} can replace multiple occurrences of {} anywhere in the invoked command, including arguments where {} is a substring, such as "dir2/{}" -- this behavior in POSIX find is "implementation-defined", but implementations where `find dir1/ -exec cp {} dir2/{} \;` will work include busybox find, GNU find, and (if I read its online manpage correctly) freebsd find as well. So you don't really need this xargs feature in practice, though it would be nice if POSIX did not permit less common find implementations to potentially implement non-useful behavior.

Last edited by eschwartz (2020-02-21 14:33:47)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#8 2020-02-21 15:15:46

GFdevelop
Member
Registered: 2017-09-18
Posts: 24

Re: Node.js packaging wiki review

Thanks for your replies boys.

eschwartz wrote:

As for replacing jq with sed... you're relying on the fact that sed natively supports editing a temporary copy of a file and then updating the original, to avoid needing to construct a tempfile yourself (jq doesn't have an --inplace option to write its output back to the input file). This would seem fairly reasonable, except for one thing: how sure are you that package.json will have such a regular structure? A proper json parsing tool has the advantage over sed, of being able to handle arbitrary formatting.

GFdevelop wrote:

(I am not a veteran of nodejs packaging)

For this I ask to the community...with the latest npm version I see that in the package.json the keys are sorted in the alphabetical order, where the keys that begins with an underscore are before the keys that begins with a symbol in the [a-z] set;
Is this related with LC_COLLATE or others system variable/setting?!?
Is this for all npm version?!?
The results of my sed version and that of the wiki jq version is the same....I ask to the community because I am not sure of all my questions...

Offline

#9 2020-02-21 15:21:42

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,159

Re: Node.js packaging wiki review

I'm no veteran either. big_smile

I would not worry about the sort order of the keys myself, but I would ask whether all npm versions always pretty-print the json with one key per line such that it can be interpreted via a line-based regular expression parser and not a full-fledged json parser. Maybe someone more familiar with npm could tell us the answer.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB