parallelization in bash

Cody Learner · 2018-04-05 01:34:39

A quick search turned up a few options for parallelizing with bash. GNU parallel, xargs, and backgrounding, which seemed by far the most straight forward method to implement and test.

I parallelized the series of long running stuff within a function in my script. After limited testing, it seems to work ok, and went from taking ~5 min to ~1.5 min, to process ~100 AUR packages.

My questions, would this be considered a fragile hack? How could I control overloading the cpu on a more limited system? Should I move on, to focus on a better solution and stop wasting time with backgrounding?

somefunct ()
	... some commands
for loop...
do

{ 	cd "${builddir}" 2>/dev/null || install

	makepkg -so --needed --noconfirm &>/dev/null

	C2=$(makepkg --printsrcinfo | awk '/pkg[vr]/ {print $3}' | awk '{printf $0 "-"; getline; print $0}')
	epoch=$(makepkg --printsrcinfo | awk '/epoch/ {print $3}')
if	[[ -n $epoch ]]; then
	C2="$epoch:$C2"
fi							# Check if installed version different from available version
							# If different, add package to update list and message user.
if	[[ $C1 != "$C2" ]]; then

	. aurt-info --syncprint3
	. aurt-info --syncprint4
	echo "${P}" >> /tmp/updates
fi; } &
	... more commands
done
	wait < <(jobs -p)
}

Complete function / script: https://github.com/Cody-Learner/aurt.au … /aurt#L123

Trilby · 2018-04-05 02:58:10

I'd recommend first focusing on streamlining the code to make it more efficient. Minimize subshells first (i.e. get rid of needless pipelines). There is no reason to pipe from one awk to another to get pkgver-pkgrel and then use yet another to prepend an epoch number. If you want to parse the output of printsrcinfo, just do it in one shot:

makepkg --printsrcinfo | sed -n '/pkgver/{s/.*= //;x} /pkgrel/{s/.*= /-/;H} /epoch/{s/.*= //;s/$/:/;x;H} ${x;s/\n//g;p}'

Alternatively you could just trim whitespace then source the result of printsrcinfo and use the variables:

 makepkg --printsrcinfo |sed 's/\s//g' > tmp
. ./tmp
printf $epoch:$pkgver-$pkgrel

Also, you seem to get the installed version number of each package individually, but certianly you already obtained a list of packages. Get the list once, then just keep using it - there is no need to call pacman again for every package.

But for your main question, backgrounding should be fine. EDIT: no, scratch that, do not background paralleled jobs calling makepkg -s. You'll end up with multiple pacman processes trying to lock the database. All but one will fail.

(edit: updated sed command to deal with presence/absence of epoch numbers)

Last edited by Trilby (2018-04-05 11:04:48)

graysky · 2018-04-05 08:56:50

Checkout GNU parallel for this... many for loops in bash can bnifit from it via parrallazation.

Last edited by graysky (2018-04-05 08:57:34)

N_BaH · 2018-04-05 18:36:40

hi,

i'm wondering about this syntax:

wait < <(jobs -p)

shouldn't it be

wait $(jobs -p)

however, if you want to wait for the last backgrounded task initiated, then just wait, no need to give it an argument, right ?

Cody Learner · 2018-04-05 19:50:08

Thanks Trilby,

If you want to parse the output of printsrcinfo, just do it in one shot:
makepkg --printsrcinfo | sed -n '/pkgver/{s/.*= //;x} /pkgrel/{s/.*= /-/;H} /epoch/{s/.*= //;s/$/:/;x;H} ${x;s/\n//g;p}'

Getting an error on my system for some reason:

 $ makepkg --printsrcinfo | sed -n '/pkgver/{s/.*= //;x} /pkgrel/{s/.*= /-/;H} /epoch/{s/.*= //;s/$/:/;x;H} ${x;s/\n//g;p}'
sed: -e expression #1, char 22: extra characters after command

Unfortunately, I don't know enough (any) extended regex to be able to make sense of the sed command. I did think of the ":" for epoch in the original sed command, and see it's updated for it now.

I did not think of removing whitespace on the scrinfo and then sourcing it. Very cool! So I played around with some of what I know, and came up with the following, although not sure if it qualifies as better than what I had.

I try to use code I can understand and read. I've said this before, I need to learn e regex. Takes much time and effort on my part to learn, use this stuff, even with reference notes. I'm learning as I progress with this script, and hopefully don't move backwards!

. <(makepkg --printsrcinfo|tr -d "/[:blank:]/")
	if [[ -n $epoch ]]; then pkgver="$epoch:$pkgver"; fi
	echo "$pkgver-$pkgrel"

As for pacman lockout while using makepkg -s, I have all the makedeps, so I got lucky and believe I avoided any errors, at least that I could detect. Good point though. I have a function that displays makedeps for all installed AUR packages, and checks for any missing and displays them as well. I will have to look into using it to install any missing prior to running makepkg without -s. Still need to straighten out pkg -vs- library or executable in it though.

$ aurt -lm
  
:: List of all AUR pkgs makedeps: 
asciidoc
automake
boost
cmake
dep
docbook-xsl
doxygen
gcc-multilib
git
gnome-doc-utils
go
gobject-introspection
gtk-doc
intltool
libcurl-openssl-1.0
libpcap
libx11
mercurial
ncurses
perl
python
python-distutils-extra
python-docutils
python-setuptools
subversion
svn
systemd
xmlto
xorg-server-xvfb
xz
yarn
yelp-tools
  
:: Missing makedeps: 
gcc-multilib
svn

GNU parallel adds another dependency, and so far, I've had no luck getting it to work. The backgrounding method was a no brainer, and seemed to work well at reducing the wait for the downloads initiated by makepkg, and reduced the function runtime to less than half.

I just didn't want to waste a bunch of time on it to learn later it was a bad direction.

Last edited by Cody Learner (2018-04-05 20:08:23)

Cody Learner · 2018-04-05 19:53:19

graysky wrote:

Checkout GNU parallel for this... many for loops in bash can bnifit from it via parrallazation.

Thanks graysky,

No luck with it yet, but still in the works. I'd like to compare results of backgrounding and using parallel.

Cody Learner · 2018-04-05 19:59:35

N_BaH wrote:

i'm wondering about this syntax:
wait < <(jobs -p)
shouldn't it be
wait $(jobs -p)
however, if you want to wait for the last backgrounded task initiated, then just wait, no need to give it an argument, right ?

Thanks N_BaH,

I tried using just wait, but didn't seem to work. I had something like 50 jobs running in background when I was testing with wait < <(jobs -p). Need to do more testing if I end up going this way though.

Last edited by Cody Learner (2018-04-05 20:03:09)

graysky · 2018-04-05 20:02:02

Cody Learner wrote:

GNU parallel adds another dependency, and so far, I've had no luck getting it to work.

Bah, minimal dep. Learn how to use it... it's powerful.

Example converting images with guetzli:

parallel 'guetzli --quality 85 {} {.}.jpg' ::: *.PNG

Example using it to cycle through a bunch ping tests (like a for loop but in parallel):

export RED="\e[01;31m" GRN="\e[01;32m" YLW="\e[01;33m" NRM="\e[00m" BLD="\e[01m"

doit() {
  ping -W1 -c1 "$1" &> /dev/null
  if [[ $? -eq 0 ]]; then
    status="${BLD}${GRN}$1 ${NRM}"
    echo -e -n "$status"
  else
    status="${BLD}${RED}$1 ${NRM}"
    echo -e -n "$status"
  fi
}

HOSTS="mercury venus earth mars phobos deimos jupiter saturn uranus neptune"
export -f doit
SHELL=/bin/bash parallel --gnu -k doit ::: $HOSTS

It can also take input from a file... here "$HOME/parallel_build_list" is something I use to pass tokens separated by a comma to the function:

...

build() {
 # arbitrary function that requires 2 tokens separated by a comma
}

SHELL=/bin/bash parallel --delay=0.5 -P 11 -a $HOME/parallel_build_list --colsep ',' build

Last edited by graysky (2018-04-05 20:10:01)

Trilby · 2018-04-05 22:31:02

Sorry about the sed command - it seems gnu's core utils sed is different from mine ... stand by.

EDIT: here's the gnu version:

makepkg --printsrcinfo | /bin/sed -n '/pkgver/{s/.*= //;x}; /pkgrel/{s/.*= /-/;H}; /epoch/{s/.*= //;s/$/:/;x;H}; ${x;s/\n//g;p}'

Last edited by Trilby (2018-04-05 22:36:58)

ewaller · 2018-04-05 23:37:39

Oh no! Trilby is falling victim to artificial selection

eschwartz · 2018-04-08 18:59:47

graysky wrote:

Cody Learner wrote:
GNU parallel adds another dependency, and so far, I've had no luck getting it to work.
Bah, minimal dep. Learn how to use it... it's powerful.
Example converting images with guetzli:
parallel 'guetzli --quality 85 {} {.}.jpg' ::: *.PNG
Example using it to cycle through a bunch ping tests (like a for loop but in parallel):
export RED="\e[01;31m" GRN="\e[01;32m" YLW="\e[01;33m" NRM="\e[00m" BLD="\e[01m"

doit() {
  ping -W1 -c1 "$1" &> /dev/null
  if [[ $? -eq 0 ]]; then
    status="${BLD}${GRN}$1 ${NRM}"
    echo -e -n "$status"
  else
    status="${BLD}${RED}$1 ${NRM}"
    echo -e -n "$status"
  fi
}

HOSTS="mercury venus earth mars phobos deimos jupiter saturn uranus neptune"
export -f doit
SHELL=/bin/bash parallel --gnu -k doit ::: $HOSTS
It can also take input from a file... here "$HOME/parallel_build_list" is something I use to pass tokens separated by a comma to the function:
...

build() {
 # arbitrary function that requires 2 tokens separated by a comma
}

SHELL=/bin/bash parallel --delay=0.5 -P 11 -a $HOME/parallel_build_list --colsep ',' build

It's also completely unnecessary when the kernel is already pretty good at parallelizing tasks.

parallel 'guetzli --quality 85 {} {.}.jpg' ::: *.PNG

becomes

for img in *.PNG; do 
    guetzli --quality 85 $img ${img%.PNG}.jpg &
done

graysky · 2018-04-08 19:41:05

Eschwartz wrote:

It's also completely unnecessary when the kernel is already pretty good at parallelizing tasks.
parallel 'guetzli --quality 85 {} {.}.jpg' ::: *.PNG
becomes
for img in *.PNG; do 
    guetzli --quality 85 $img ${img%.PNG}.jpg &
done

Agreed, but if you have a directory of say 200 PNG files and you use the code you propose, I believe all 200 of them will kick off at a time, no? GNU parallel knows how many cores your system has and will use them or you can pick any number of tasks to run in parallel as the minimum batch size (ie groups of 8 or 12, etc).

Trilby · 2018-04-08 20:17:29

So what if they all start at the same time?

graysky · 2018-04-08 20:22:23

Trilby wrote:

So what if they all start at the same time?

I guess you could nice it -19 or something... just not very elegant.

Trilby · 2018-04-08 20:27:08

Why would you need to nice it? Are you worried about other processes lagging? You already said parallels will use as many cores as are available - so what would be the difference?

graysky · 2018-04-08 21:36:07

Have to run the experiment.

frostschutz · 2018-04-08 23:44:59

& method is fine if you know it will only ever be a handful of tasks.

Using different nice levels works to even it out further. I use this method in initramfs, early bootup, to open a few LUKS containers in parallel.

With nice, a cryptsetup process of lower priority won't fight for CPU time as long as a process of higher priority exists (for each core).

# Unlock HDD
for i in $(seq 1 32)
do
    # nice is the poor man's parallel
    nice -n $((($i % 40) - 20)) /sbin/cryptsetup luksOpen ... luksHDD"$i" &
done

wait # for cryptsetup

Note you shouldn't use higher-than-normal priority nice levels outside of early boot, unless you wish to starve your regular processes completely. There pretty much are no other processes in initramfs so it is fine here.

parallel is for big jobs. Hundreds, thousands, millions of files? The simple & method would hit all sorts of limits and the CPU would spend 99% of its time handling context switch overhead, instead of doing any useful work.

Last edited by frostschutz (2018-04-08 23:48:23)

NoSuck · 2018-04-09 00:00:28

■ for p in *.jpg ; do time convert "$p" "bash/$p.png" & done

real    0m0.768s
user    0m0.216s
sys     0m0.005s

real    0m0.327s
user    0m0.051s
sys     0m0.005s

real    0m0.258s
user    0m0.057s
sys     0m0.005s

real    0m0.902s
user    0m0.221s
sys     0m0.014s

real    0m0.807s
user    0m0.185s
sys     0m0.006s

real    0m1.157s
user    0m0.365s
sys     0m0.010s

real    0m1.238s
user    0m0.346s
sys     0m0.016s

real    0m0.919s
user    0m0.385s
sys     0m0.011s

real    0m1.247s
user    0m0.249s
sys     0m0.011s

real    0m0.879s
user    0m0.410s
sys     0m0.013s

real    0m1.346s
user    0m0.516s
sys     0m0.010s

real    0m1.294s
user    0m0.484s
sys     0m0.007s

real    0m1.516s
user    0m0.453s
sys     0m0.012s

real    0m1.199s
user    0m0.358s
sys     0m0.010s

real    0m1.560s
user    0m0.738s
sys     0m0.018s

real    0m2.183s
user    0m1.064s
sys     0m0.022s

real    0m3.556s
user    0m2.322s
sys     0m0.038s

real    0m3.527s
user    0m2.656s
sys     0m0.104s

real    0m4.535s
user    0m3.458s
sys     0m0.051s

real    0m8.000s
user    0m6.969s
sys     0m0.094s

real    0m10.598s
user    0m9.179s
sys     0m0.132s

real    0m13.674s
user    0m12.864s
sys     0m0.175s

■ time parallel 'convert {} par/{}.png' ::: *.jpg

real    0m14.470s
user    0m42.351s
sys     0m0.992s

61.49 vs 14.47 seconds. GNU parallel wins. Thanks for teaching me about it, graysky.

Trilby · 2018-04-09 00:23:41

NoSuck, that's completely apples and oranges. In the first you had the "time" inside the loop which doubles the number of processes, with parallel you had it outside the loop.

EDIT: I just did a test with the "time" outside the loop:

$ time ./script
real    0m 24.09s
user    0m 43.97s
sys     0m 1.15s

$ time parallel 'convert {} par/{}.png' ::: *.jpg
real    0m 24.55s
user    0m 43.51s
sys     0m 1.38s

Where "script" is:

#!/bin/sh

for p in *.jpg; do
        convert "$p" "bash/$p.png" &
done

wait

Last edited by Trilby (2018-04-09 00:34:02)

NoSuck · 2018-04-09 00:42:25

Bah, edits.

Last edited by NoSuck (2018-04-09 00:47:59)

Trilby · 2018-04-09 00:47:07

Actually your test was doubly unfair as you sum up the total of the individual "time" outputs. I just did an additional test by adding another "time" inside the loop. The sum of the individual "time" commands within the script was quite a bit longer than the total time reported outside the script. And of course this would be the case as the individual "times" were overlapping. You didn't time how long it took for the processes to run in parallel, you summed the total running time of all processes ignoring the fact that they were running in parallel: when 4 images were converted in 3 seconds, each one individually reported 3 seconds used, but it was the same 3 seconds.

It turns out though I was wrong too - the additional time processes have a trivial if any effect. The real discrepancy was multiple countings of overlapping time intervals.

Last edited by Trilby (2018-04-09 00:50:48)

NoSuck · 2018-04-09 00:48:04

Excellent example. GNU parallel is bloat.

Trilby · 2018-04-09 00:49:04

NoSuck wrote:

GNU ~~parallel~~ is bloat.

FTFY

NoSuck · 2018-04-09 00:50:22

Blasphemy! GCC made the world upon which we stand today.

Cody Learner · 2018-04-09 05:45:56

Here are some preliminary test results. I'm still scratching my head on implementing GNU parallel in a similar way to the current function for comparison.

Function WITHOUT PARALLELIZATION

real	4m23.176s
user	1m56.779s
sys	1m9.898s

$ time aurt -Vc
  
:: Creating AUR package check list .... 
:: Checking for updates ....
 
:: Checking : astyle-svn r655-1
:: Checking : aur-out-of-date 1.1.0-1
:: Checking : aurget 4.7.2-1
:: Checking : aurora 20160907.16_f78867f-1
:: Checking : aurutils 1.5.3-10
:: Checking : aurvote-git r27.fd413f1-2
:: Checking : burgaur-git 2.2.5.gb282099-1
:: Checking : cloc-svn 1.62-1
:: Checking : cower 17-2
:: Checking : dbus-x11 1.10.18-1
:: Checking : downgrade 6.0.0-1
:: Checking : downgrader 2.0.0-3
:: Checking : duff 0.5.2-1
:: Checking : enchant1.6 1.6.1-8
:: Checking : f 0.5.4-1
:: Checking : fetchpkg 0.3-7
:: Checking : firefox-nightly 61.0a1.20180408-1
:: Checking : gconf-gtk2 3.2.6-5
:: Checking : gedit2 2.30.4-9
:: Checking : git-remote-hg 0.3-1
:: Checking : github-desktop-git 1.1.2.beta1.r104.g2dfc68ad0-1
:: Checking : grive 0.5.0-9
:: Checking : grub-legacy 0.97-25
:: Checking : hgsubversion-hg 1563.f44543ffab9b-1
:: Checking : idutils 4.6-1
:: Checking : inxi 2.9.12-1
:: Checking : ld-lsb 3-7
:: Checking : libcurl-openssl-1.0 7.57.0-1
:: Checking : menulibre 2.2.0-1
:: Checking : mime-editor 0.6-6
:: Checking : mimeo 2017.6.6-1
:: Checking : modprobed-db 2.37-1
:: Checking : package-query-git 1.9.r1.gdd242f4-1
:: Checking : pacnews-neovim 0.0.2-1
:: Checking : pacvcs 1:1.0.0-1
:: Checking : pacvim-git v1.1.1.r10.gb0934ca-1
:: Checking : pkgbrowser 0.21-1
:: Checking : pkgbuilder 4.2.17-1
:: Checking : plymouth 0.9.3-3
:: Checking : polkit-explorer 10.84983b1-1
:: Checking : pvim v0.3.1-1
:: Checking : python3-aur 2018-1
:: Checking : python3-memoizedb 2017.3.30-1
:: Checking : python3-xcgf 2017.3-1
:: Checking : python3-xcpf 2017.12-1
:: Checking : raur-git 49.89f019b-1
:: Checking : repoctl 0.18-1
:: Checking : ruby-neovim 0.7.0-2
:: Checking : ruby-redcarpet 3.4.0-1
:: Checking : surfraw-git 2018.02.03.g9b1b780-2
:: Checking : terminfo-neovim-tmux 1.0-1
:: Checking : trizen 1:1.46-1
:: Checking : trizen .................... UPDATE AVAILABLE 
:: Local git HEAD : ca52ffad2ae585ed46149aab9b9ba46ffb5d8d91
:: Remote git HEAD: a0e4ddbd52b6690be46045e5b0d314d696356e23
:: Checking : tstat-svn 775-1
:: Checking : ttf-ms-win10 10.0.16299.192-1
:: Checking : typescript-vim-git 7843f89-1
:: Checking : urxvt-vim-insert 14.07a4afb-1
:: Checking : vi-vim-symlink 1-2
:: Checking : vim-ack 1.0.9-1
:: Checking : vim-apprentice 1.9-1
:: Checking : vim-assistant 1.5.8-1
:: Checking : vim-bash-support 4.2.1-1
:: Checking : vim-bitbake-git r67.e1c3218-1
:: Checking : vim-brainfuck 1.0.3-1
:: Checking : vim-buftabline-git r64.14d208b-1
:: Checking : vim-buftabs 0.18-2
:: Checking : vim-codepad 1.4-1
:: Checking : vim-coffeescript-git v003.23.g9e3b4de-1
:: Checking : vim-command-t 4.0-1
:: Checking : vim-comments 2.12-1
:: Checking : vim-csv-git 20150115-1
:: Checking : vim-dein-git 1.0.r29.gec5a35c-1
:: Checking : vim-diffchar 7.50-1
:: Checking : vim-easytags 3.11-1
:: Checking : vim-endwise 1.2-3
:: Checking : vim-fluxkeys 20091101-5
:: Checking : vim-json-git 1:r102.3727f08-1
:: Checking : vim-markdown 2.0.0.r250.8ace663-1
:: Checking : vim-misc-xolox 1.17.6-1
:: Checking : vim-openscad r14.2ac407d-1
:: Checking : vim-pep8 1.1-1
:: Checking : vim-pkgbuild 0.1.20141125.ca8e436-1
:: Checking : vim-rest 0.1-1
:: Checking : vim-session 2.13.1-2
:: Checking : vim-tern 94.3cffc28-1
:: Checking : vim-vim-support 2.3-1
:: Checking : vim-vimproc-git ver.9.3.r6.2300224-1
:: Checking : virtualbox-ext-oracle 5.2.8-1
:: Checking : wrapaur 2.0.10-1
:: Checking : yarn-nightly 1.6.0.20180405.1011-1
:: Checking : you-get-git 0.4.1040.20180329.1905-1

real	4m23.176s
user	1m56.779s
sys	1m9.898s

#==========================================================================================
#                  SCRIPT WITHOUT PARALLELIZATION
#==========================================================================================

sync () {

# set -x
	rm /tmp/pid   2>/dev/null
	rm /tmp/jobs1 2>/dev/null
	. aurt-info --syncprint1

if 	[[ -e /tmp/updates ]]; then
	rm /tmp/updates || exit
fi
							# Create list AUR packages to check.
	dlist=$(pacman -Qq|aursift 2>/dev/null)

	. aurt-info --syncprint2

for P in $dlist
do							# Prepare to check installed version -vs- available version
							# Avl vers via 'makepkg -so' parse ver from 'makepkg --printsrcinfo'
	builddir="${HOME}"/z-AUR-Aurt.git/"${P}"
	C1=$(pacman -Q "${P}" | awk '{print $2}')

	. aurt-info --syncprint33
#{
	cd "${builddir}" 2>/dev/null || install

	makepkg -o --needed --noconfirm &>/dev/null

	C2=$(makepkg --printsrcinfo | /bin/sed -n '/pkgver/{s/.*= //;x}; /pkgrel/{s/.*= /-/;H}; /epoch/{s/.*= //;s/$/:/;x;H}; ${x;s/\n//g;p}')

#	C2=$(makepkg --printsrcinfo | awk '/pkg[vr]/ {print $3}' | awk '{printf $0 "-"; getline; print $0}')
#	epoch=$(makepkg --printsrcinfo | awk '/epoch/ {print $3}')
#	if [[ -n $epoch ]]; then C2="$epoch:$C2" ; fi
#
#	makepkg -so --needed --noconfirm &>/dev/null
#	. <(makepkg --printsrcinfo|tr -d [:blank:])
#	if [[ -n $epoch ]]; then pkgver="$epoch:$pkgver" ; fi
#	echo "$pkgver-$pkgrel"
							# Check if installed version different from available version
							# If different, add package to update list and message user.
if	[[ $C1 != "$C2" ]]; then

	. aurt-info --syncprint3
	. aurt-info --syncprint4
	echo "${P}" >> /tmp/updates
fi
#} &
	echo "$!" >>/tmp/pid
							# Prepare for git HEAD local -vs- remote backup check

	C3=$(git ls-remote https://aur.archlinux.org/"${P}".git HEAD|awk '{print $1}')
	C4=$(git -C "${builddir}" rev-parse HEAD)
							# For pkgs only included as part of others & w/o git repo. 
							# ie: aursec-tui is part of aursec
							# Eliminate false positive update status.
	if [[ -z ${C3} ]]; then C3=$C4 ; fi
							# Check if local git HEAD different from remote as backup check
							# If different, add package to update list and message user.
if	[[ ${C3} != "${C4}" ]]; then

	. aurt-info --syncprint5
	echo "${P}" >> /tmp/updates
fi

done							# If no updates available, create file with message and current date.
							# Notify user with message.
if	[[ ! -e /tmp/updates ]]; then

	. aurt-info --syncprint6
	echo ":: No AUR updates available as of last sync on ${pdate}" >/tmp/noupdates
	exit
fi

	jobs -p >>/tmp/jobs
	wait < <(jobs -p)

	md5sum /tmp/updates >/tmp/md5sum 2>/dev/null
}

#==========================================================================================

/tmp/pid: empty 91 line file

/tmp/jobs: empty 1 line file

Function WITH PARALLELIZATION

real	1m42.655s
user	1m53.492s
sys	1m1.513s

$ time aurt -Vc
  
:: Creating AUR package check list .... 
:: Checking for updates ....
 
:: Checking : astyle-svn r655-1
:: Checking : aur-out-of-date 1.1.0-1
:: Checking : aurget 4.7.2-1
:: Checking : aurora 20160907.16_f78867f-1
:: Checking : aurutils 1.5.3-10
:: Checking : aurvote-git r27.fd413f1-2
:: Checking : burgaur-git 2.2.5.gb282099-1
:: Checking : cloc-svn 1.62-1
:: Checking : cower 17-2
:: Checking : dbus-x11 1.10.18-1
:: Checking : downgrade 6.0.0-1
:: Checking : downgrader 2.0.0-3
:: Checking : duff 0.5.2-1
:: Checking : enchant1.6 1.6.1-8
:: Checking : f 0.5.4-1
:: Checking : fetchpkg 0.3-7
:: Checking : firefox-nightly 61.0a1.20180408-1
:: Checking : gconf-gtk2 3.2.6-5
:: Checking : gedit2 2.30.4-9
:: Checking : git-remote-hg 0.3-1
:: Checking : github-desktop-git 1.1.2.beta1.r104.g2dfc68ad0-1
:: Checking : grive 0.5.0-9
:: Checking : grub-legacy 0.97-25
:: Checking : hgsubversion-hg 1563.f44543ffab9b-1
:: Checking : idutils 4.6-1
:: Checking : inxi 2.9.12-1
:: Checking : ld-lsb 3-7
:: Checking : libcurl-openssl-1.0 7.57.0-1
:: Checking : menulibre 2.2.0-1
:: Checking : mime-editor 0.6-6
:: Checking : mimeo 2017.6.6-1
:: Checking : modprobed-db 2.37-1
:: Checking : package-query-git 1.9.r1.gdd242f4-1
:: Checking : pacnews-neovim 0.0.2-1
:: Checking : pacvcs 1:1.0.0-1
:: Checking : pacvim-git v1.1.1.r10.gb0934ca-1
:: Checking : pkgbrowser 0.21-1
:: Checking : pkgbuilder 4.2.17-1
:: Checking : plymouth 0.9.3-3
:: Checking : polkit-explorer 10.84983b1-1
:: Checking : pvim v0.3.1-1
:: Checking : python3-aur 2018-1
:: Checking : python3-memoizedb 2017.3.30-1
:: Checking : python3-xcgf 2017.3-1
:: Checking : python3-xcpf 2017.12-1
:: Checking : raur-git 49.89f019b-1
:: Checking : repoctl 0.18-1
:: Checking : ruby-neovim 0.7.0-2
:: Checking : ruby-redcarpet 3.4.0-1
:: Checking : surfraw-git 2018.02.03.g9b1b780-2
:: Checking : terminfo-neovim-tmux 1.0-1
:: Checking : trizen 1:1.46-1
:: Checking : trizen .................... UPDATE AVAILABLE 
:: Local git HEAD : ca52ffad2ae585ed46149aab9b9ba46ffb5d8d91
:: Remote git HEAD: a0e4ddbd52b6690be46045e5b0d314d696356e23
:: Checking : tstat-svn 775-1
:: Checking : ttf-ms-win10 10.0.16299.192-1
:: Checking : typescript-vim-git 7843f89-1
:: Checking : urxvt-vim-insert 14.07a4afb-1
:: Checking : vi-vim-symlink 1-2
:: Checking : vim-ack 1.0.9-1
:: Checking : vim-apprentice 1.9-1
:: Checking : vim-assistant 1.5.8-1
:: Checking : vim-bash-support 4.2.1-1
:: Checking : vim-bitbake-git r67.e1c3218-1
:: Checking : vim-brainfuck 1.0.3-1
:: Checking : vim-buftabline-git r64.14d208b-1
:: Checking : vim-buftabs 0.18-2
:: Checking : vim-codepad 1.4-1
:: Checking : vim-coffeescript-git v003.23.g9e3b4de-1
:: Checking : vim-command-t 4.0-1
:: Checking : vim-comments 2.12-1
:: Checking : vim-csv-git 20150115-1
:: Checking : vim-dein-git 1.0.r29.gec5a35c-1
:: Checking : vim-diffchar 7.50-1
:: Checking : vim-easytags 3.11-1
:: Checking : vim-endwise 1.2-3
:: Checking : vim-fluxkeys 20091101-5
:: Checking : vim-json-git 1:r102.3727f08-1
:: Checking : vim-markdown 2.0.0.r250.8ace663-1
:: Checking : vim-misc-xolox 1.17.6-1
:: Checking : vim-openscad r14.2ac407d-1
:: Checking : vim-pep8 1.1-1
:: Checking : vim-pkgbuild 0.1.20141125.ca8e436-1
:: Checking : vim-rest 0.1-1
:: Checking : vim-session 2.13.1-2
:: Checking : vim-tern 94.3cffc28-1
:: Checking : vim-vim-support 2.3-1
:: Checking : vim-vimproc-git ver.9.3.r6.2300224-1
:: Checking : virtualbox-ext-oracle 5.2.8-1
:: Checking : wrapaur 2.0.10-1
:: Checking : yarn-nightly 1.6.0.20180405.1011-1
:: Checking : you-get-git 0.4.1040.20180329.1905-1

real	1m42.655s
user	1m53.492s
sys	1m1.513s


#==========================================================================================
#                  SCRIPT WITH PARALLELIZATION
#==========================================================================================
sync () {

# set -x
	rm /tmp/pid   2>/dev/null
	rm /tmp/jobs1 2>/dev/null
	. aurt-info --syncprint1

if 	[[ -e /tmp/updates ]]; then
	rm /tmp/updates || exit
fi
							# Create list AUR packages to check.
	dlist=$(pacman -Qq|aursift 2>/dev/null)

	. aurt-info --syncprint2

for P in $dlist
do							# Prepare to check installed version -vs- available version
							# Avl vers via 'makepkg -so' parse ver from 'makepkg --printsrcinfo'
	builddir="${HOME}"/z-AUR-Aurt.git/"${P}"
	C1=$(pacman -Q "${P}" | awk '{print $2}')

	. aurt-info --syncprint33
{	cd "${builddir}" 2>/dev/null || install

	makepkg -o --needed --noconfirm &>/dev/null

	C2=$(makepkg --printsrcinfo | /bin/sed -n '/pkgver/{s/.*= //;x}; /pkgrel/{s/.*= /-/;H}; /epoch/{s/.*= //;s/$/:/;x;H}; ${x;s/\n//g;p}')

#	C2=$(makepkg --printsrcinfo | awk '/pkg[vr]/ {print $3}' | awk '{printf $0 "-"; getline; print $0}')
#	epoch=$(makepkg --printsrcinfo | awk '/epoch/ {print $3}')
#	if [[ -n $epoch ]]; then C2="$epoch:$C2" ; fi
#
#	makepkg -so --needed --noconfirm &>/dev/null
#	. <(makepkg --printsrcinfo|tr -d [:blank:])
#	if [[ -n $epoch ]]; then pkgver="$epoch:$pkgver" ; fi
#	echo "$pkgver-$pkgrel"
							# Check if installed version different from available version
							# If different, add package to update list and message user.
if	[[ $C1 != "$C2" ]]; then

	. aurt-info --syncprint3
	. aurt-info --syncprint4
	echo "${P}" >> /tmp/updates
fi
} &
	echo "$!" >>/tmp/pid
							# Prepare for git HEAD local -vs- remote backup check

	C3=$(git ls-remote https://aur.archlinux.org/"${P}".git HEAD|awk '{print $1}')
	C4=$(git -C "${builddir}" rev-parse HEAD)
							# For pkgs only included as part of others & w/o git repo. 
							# ie: aursec-tui is part of aursec
							# Eliminate false positive update status.
	if [[ -z ${C3} ]]; then C3=$C4 ; fi
							# Check if local git HEAD different from remote as backup check
							# If different, add package to update list and message user.
if	[[ ${C3} != "${C4}" ]]; then

	. aurt-info --syncprint5
	echo "${P}" >> /tmp/updates
fi

done							# If no updates available, create file with message and current date.
							# Notify user with message.
if	[[ ! -e /tmp/updates ]]; then

	. aurt-info --syncprint6
	echo ":: No AUR updates available as of last sync on ${pdate}" >/tmp/noupdates
	exit
fi

	jobs -p >>/tmp/jobs
	wait < <(jobs -p)

	md5sum /tmp/updates >/tmp/md5sum 2>/dev/null
}

#==========================================================================================
$ cat /tmp/pid
19026
19263
19502
19716
20212
20404
20601
21148
21961
22287
22583
22806
23399
23710
25344
26309
26952
27383
27996
28650
29469
30030
30325
31082
31597
31951
32666
1046
1646
2263
3138
3609
4293
5125
5964
6788
7452
8338
8943
9384
9947
10623
11261
11724
12174
12625
13000
13432
14097
14603
14827
15405
15749
16039
16393
16966
17986
18915
19711
20108
20688
21241
21462
21856
22413
22947
23375
23765
24194
25089
25367
25572
26122
26786
27309
27755
27988
28386
29069
29777
30590
31232
32064
331
986
2116
2604
3201
3535
4061

#==========================================================================================
$ cat /tmp/jobs
19026
19263
19502
19716
20212
20404
20601
21148
21961
22287
22583
22806
23399
23710
25344
26309
26952
27383
27996
28650
29469
30030
30325
31082
31597
31951
32666
1046
1646
2263
3138
3609
4293
5125
5964
6788
7452
8338
8943
9384
9947
10623
11261
11724
12174
12625
13000
13432
14097
14603
14827
15405
15749
16039
16393
16966
17986
18915
19711
20108
20688
21241
21462
21856
22413
22947
23375
23765
24194
25089
25367
25572
26122
26786
27309
27755
27988
28386
29069
29777
30590
31232
32064
331
986
2116
2604
3201
3535
4061

Last edited by Cody Learner (2018-04-09 06:06:18)

Arch Linux

#1 2018-04-05 01:34:39

parallelization in bash

#2 2018-04-05 02:58:10

Re: parallelization in bash

#3 2018-04-05 08:56:50

Re: parallelization in bash

#4 2018-04-05 18:36:40

Re: parallelization in bash

#5 2018-04-05 19:50:08

Re: parallelization in bash

#6 2018-04-05 19:53:19

Re: parallelization in bash

#7 2018-04-05 19:59:35

Re: parallelization in bash

#8 2018-04-05 20:02:02

Re: parallelization in bash

#9 2018-04-05 22:31:02

Re: parallelization in bash

#10 2018-04-05 23:37:39

Re: parallelization in bash

#11 2018-04-08 18:59:47

Re: parallelization in bash

#12 2018-04-08 19:41:05

Re: parallelization in bash

#13 2018-04-08 20:17:29

Re: parallelization in bash

#14 2018-04-08 20:22:23

Re: parallelization in bash

#15 2018-04-08 20:27:08

Re: parallelization in bash

#16 2018-04-08 21:36:07

Re: parallelization in bash

#17 2018-04-08 23:44:59

Re: parallelization in bash

#18 2018-04-09 00:00:28

Re: parallelization in bash

#19 2018-04-09 00:23:41

Re: parallelization in bash

#20 2018-04-09 00:42:25

Re: parallelization in bash

#21 2018-04-09 00:47:07

Re: parallelization in bash

#22 2018-04-09 00:48:04

Re: parallelization in bash

#23 2018-04-09 00:49:04

Re: parallelization in bash

#24 2018-04-09 00:50:22

Re: parallelization in bash

#25 2018-04-09 05:45:56

Re: parallelization in bash

Board footer