Help with analyzing a bash script [SOLVED]

NuSkool · 2019-05-11 18:06:21

I'm looking at the following lines in a script.

mapfile -t updates < <(pacman -Qu --dbpath "$CHECKUPDATES_DB" 2> /dev/null | grep -v '\[.*\]')

sudo pacman -Sw --noconfirm "${updates[@]%% *}" --dbpath "$CHECKUPDATES_DB" --logfile /dev/null

First line, can't figure out why it doesn't use "pacman -Quq" rather than "pacman -Qu", and what the piped grep inverted match is looking for?
Second line, is that pattern matching "%% *" in the array, and what's it doing?

Knowing who wrote the code, I feel confident there's a good reason it's the way it is. I'd really like to know why though.
Also, I really don't want to bother the author with this question.

Last edited by NuSkool (2019-05-12 17:28:16)

Trilby · 2019-05-11 18:56:48

If you can't figure out what it's doing, don't use it. I can say that your questions are good, and the answers seem to be that the author of those lines doesn't really know what they're doing (hence the advice not to use it).

The first grep at the end of the first line seems to be for filtering out packages that are already [installed]. The string manipulation in the second line is likely to get rid of the version numbers to just leave package names.

But all this is basically irrelevant as it seems the sole purpose is to just download available updates using the CHECKUPDATES_DB. This can be done much simpler as one non-convoluted line:

pacman -Suw --noconfirm --dbpath "$CHECKUPDATES_DB" --logfile /dev/null

NuSkool wrote:

Knowing who wrote the code, I feel confident there's a good reason it's the way it is.

Who did write it? If you want to know their reason, ask them. But from just looking at the code I'm not inclined to agree.

Last edited by Trilby (2019-05-11 18:58:15)

NuSkool · 2019-05-11 19:58:50

The patch was written by one of the smarter guys here on the forums. I may have posted a few lines that are possibly out of context or not provided enough info.

I got the script from: https://aur.archlinux.org/packages/pacman-contrib-git/

I already wrote a script that checks for, download updates without installing, and sends a log. However, I feel "I" don't know what I'm doing, so I like to look at scripts written by others, doing similar things to learn.

Last edited by NuSkool (2019-05-11 20:00:59)

Trilby · 2019-05-11 20:07:53

What do you want your code to do. Those lines in pacman contrib have other relevant lines between them. If you just want to use these two lines, they are far more complex than they need to be. Much of that complexity is to do the other stuff in between that you left out of your initial post.

So do you want to generate the list of packages that have updates waiting, or do you just want to fill your cache with any available updates? These are two distinct goals.

NuSkool · 2019-05-12 00:47:54

Hey thanks Trilby, but I really don't want to use the code or need to create anything new.

I've already made a script, around 6 months ago that checks, downloads any available updates without installing, and sends me a log. It's on a daily timer and works as expected. My script initially used the checkupdates, but I added the check update ability to my script so I didn't need it after that.

I downloaded the development checkupdates script just to take a look at how the new -d option had been implemented, and hoped to learn something from it. After spending some time, I still had/have difficulty deciphering the lines I posted and thought I could get some clarification from posting them here.

I'll dig into it more in depth tonight and try to figure it out.

Trilby · 2019-05-12 01:12:51

What remains unclear?

NuSkool · 2019-05-12 02:41:59

Specifically: from the first line:

'\[.*\]'

And from the second line:

%% *

Although you gave a brief explanation, I want to understand it more in depth.

I'd like to understand the details. What look like escapes '\' proceeding '[]' don't make sense to me, but this is on me to learn more about.

I've seen '[installed]' when running pacman -Sl, for example. I have no experience running pacman -Qu on my systems and watching the output.

I just edited (pacman -Quq to -Qu) in my script (sets up separate database) and ran it in bash -x. Found '[ignored]' output from pacman -Qu. I just need to figure out any other possibilities inside '[]' when running -Qu, so mostly solved on this one.

You also gave me this '%% *' is "string manipulation" in the array. That's plenty enough for me to follow up on and learn about it.

I'd like to understand the decisions made in the checkupdates script. I feel if I had more details/knowledge I'm lacking, it would (should) be clear to me why it was written the way it is. Also, I won't be satisfied with myself until I fully understand and test what I've learned here.

Thanks for being helpful.

Trilby · 2019-05-12 02:54:40

For the second line portion is, as noted, to get rid of version numbers:
http://www.tldp.org/LDP/abs/html/string … ation.html

NuSkool · 2019-05-12 17:27:55

The reason for using 'pacman -Qu' rather than 'pacman -Quq' is finally adding up. I may be slow, but tenacity paid off in this case. I thought sharing what I learned may be worth while.

'pacman -Qu' output includes "ignored" packages (at least and possibly more) set in /etc/pacman.conf, within brackets like; [ignored].

Example:

gtk3 3.24.4-1 -> 1:3.24.8-1 [ignored]

Piping 'pacman -Qu' output to grep invert-match '-v' with regex '\[.*\]' eliminates the ignored packages from the updates array.

Regex breakdown for: grep -v '\[.*\]'

Escapes '\' give brackets [] literal meaning, and also prevents character classing the contents.
dot '.' = any character
asterisk '*' = quantifier to proceeding character, zero or more times

grep examples:

Grep without invert-match '-v' option returns line.

$ echo "gtk3 3.24.4-1 -> 1:3.24.8-1 [ignored]" | grep '\[.*\]'
gtk3 3.24.4-1 -> 1:3.24.8-1 [ignored]

Grep with invert-match '-v' option does not return line.

$ echo "gtk3 3.24.4-1 -> 1:3.24.8-1 [ignored]" | grep -v '\[.*\]'
$

String manipulation examples:

Create array of a package formatted as output of pacman -Qu

$ readarray -t test1 <<< "gtk3 3.24.4-1 -> 1:3.24.8-1"

Array expansion without the string manipulation:

$ echo ${test1[@]}
gtk3 3.24.4-1 -> 1:3.24.8-1

Array expansion with string manipulation to eliminate everything except the package name:

$ echo ${test1[@]%% *}
gtk3

TLDR:
Based on this, I'd guess the script uses 'pacman -Qu' rather than 'pacman -Quq' to eliminate downloading updates of ignored packages.

Also, I have MUCH to learn about regex and string manipulation just to name a few. This exersize introduced the subject of string
manipulation to me, and I can see advantages of using it in many cases over using awk, sed, grep, etc. It also reemphasized the importance of learning regex.

I think it may be appropriate to go ahead and post the script I was trying to figure out due to it's relevance:

#!/usr/bin/bash
#
#   checkupdates: Safely print a list of pending updates.
#
#   Copyright (c) 2013 Kyle Keen <keenerd@gmail.com>
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program.  If not, see <http://www.gnu.org/licenses/>.
#

declare -r myname='checkupdates'
declare -r myver='1.1.0-24-g4e22'

DOWNLOAD_CACHE=0

plain() {
	(( QUIET )) && return
	local mesg=$1; shift
	printf "${BOLD}    ${mesg}${ALL_OFF}\n" "$@" >&1
}

msg() {
	(( QUIET )) && return
	local mesg=$1; shift
	printf "${GREEN}==>${ALL_OFF}${BOLD} ${mesg}${ALL_OFF}\n" "$@" >&1
}

msg2() {
	(( QUIET )) && return
	local mesg=$1; shift
	printf "${BLUE}  ->${ALL_OFF}${BOLD} ${mesg}${ALL_OFF}\n" "$@" >&1
}

ask() {
	local mesg=$1; shift
	printf "${BLUE}::${ALL_OFF}${BOLD} ${mesg}${ALL_OFF}" "$@" >&1
}

warning() {
	local mesg=$1; shift
	printf "${YELLOW}==> $(gettext "WARNING:")${ALL_OFF}${BOLD} ${mesg}${ALL_OFF}\n" "$@" >&2
}

error() {
	local mesg=$1; shift
	printf "${RED}==> $(gettext "ERROR:")${ALL_OFF}${BOLD} ${mesg}${ALL_OFF}\n" "$@" >&2
}

# check if messages are to be printed using color
unset ALL_OFF BOLD BLUE GREEN RED YELLOW
if [[ -t 2 && ! $USE_COLOR = "n" && ! $TERM = "dumb" ]]; then
	# prefer terminal safe colored and bold text when tput is supported
	if tput setaf 0 &>/dev/null; then
		ALL_OFF="$(tput sgr0)"
		BOLD="$(tput bold)"
		BLUE="${BOLD}$(tput setaf 4)"
		GREEN="${BOLD}$(tput setaf 2)"
		RED="${BOLD}$(tput setaf 1)"
		YELLOW="${BOLD}$(tput setaf 3)"
	else
		ALL_OFF="\e[1;0m"
		BOLD="\e[1;1m"
		BLUE="${BOLD}\e[1;34m"
		GREEN="${BOLD}\e[1;32m"
		RED="${BOLD}\e[1;31m"
		YELLOW="${BOLD}\e[1;33m"
	fi
fi
readonly ALL_OFF BOLD BLUE GREEN RED YELLOW

# getopt-like parser
parseopts() {
	local opt= optarg= i= shortopts=$1
	local -a longopts=() unused_argv=()

	shift
	while [[ $1 && $1 != '--' ]]; do
		longopts+=("$1")
		shift
	done
	shift

	longoptmatch() {
		local o longmatch=()
		for o in "${longopts[@]}"; do
			if [[ ${o%:} = "$1" ]]; then
				longmatch=("$o")
				break
			fi
			[[ ${o%:} = "$1"* ]] && longmatch+=("$o")
		done

		case ${#longmatch[*]} in
			1)
				# success, override with opt and return arg req (0 == none, 1 == required)
				opt=${longmatch%:}
				if [[ $longmatch = *: ]]; then
					return 1
				else
					return 0
				fi ;;
			0)
				# fail, no match found
				return 255 ;;
			*)
				# fail, ambiguous match
				printf "checkupdates: $(gettext "option '%s' is ambiguous; possibilities:")" "--$1"
				printf " '%s'" "${longmatch[@]%:}"
				printf '\n'
				return 254 ;;
		esac >&2
	}

	while (( $# )); do
		case $1 in
			--) # explicit end of options
				shift
				break
				;;
			-[!-]*) # short option
				for (( i = 1; i < ${#1}; i++ )); do
					opt=${1:i:1}

					# option doesn't exist
					if [[ $shortopts != *$opt* ]]; then
						printf "checkupdates: $(gettext "invalid option") -- '%s'\n" "$opt" >&2
						OPTRET=(--)
						return 1
					fi

					OPTRET+=("-$opt")
					# option requires optarg
					if [[ $shortopts = *$opt:* ]]; then
						# if we're not at the end of the option chunk, the rest is the optarg
						if (( i < ${#1} - 1 )); then
							OPTRET+=("${1:i+1}")
							break
						# if we're at the end, grab the the next positional, if it exists
						elif (( i == ${#1} - 1 )) && [[ $2 ]]; then
							OPTRET+=("$2")
							shift
							break
						# parse failure
						else
							printf "checkupdates: $(gettext "option requires an argument") -- '%s'\n" "$opt" >&2
							OPTRET=(--)
							return 1
						fi
					fi
				done
				;;
			--?*=*|--?*) # long option
				IFS='=' read -r opt optarg <<< "${1#--}"
				longoptmatch "$opt"
				case $? in
					0)
						# parse failure
						if [[ $optarg ]]; then
							printf "checkupdates: $(gettext "option '%s' does not allow an argument")\n" "--$opt" >&2
							OPTRET=(--)
							return 1
						# --longopt
						else
							OPTRET+=("--$opt")
						fi
						;;
					1)
						# --longopt=optarg
						if [[ $optarg ]]; then
							OPTRET+=("--$opt" "$optarg")
						# --longopt optarg
						elif [[ $2 ]]; then
							OPTRET+=("--$opt" "$2" )
							shift
						# parse failure
						else
							printf "checkupdates: $(gettext "option '%s' requires an argument")\n" "--$opt" >&2
							OPTRET=(--)
							return 1
						fi
						;;
					254)
						# ambiguous option -- error was reported for us by longoptmatch()
						OPTRET=(--)
						return 1
						;;
					255)
						# parse failure
						printf "checkupdates: $(gettext "invalid option") '--%s'\n" "$opt" >&2
						OPTRET=(--)
						return 1
						;;
				esac
				;;
			*) # non-option arg encountered, add it as a parameter
				unused_argv+=("$1")
				;;
		esac
		shift
	done

	# add end-of-opt terminator and any leftover positional parameters
	OPTRET+=('--' "${unused_argv[@]}" "$@")
	unset longoptmatch

	return 0
}


usage() {
	cat << __EOF__
${myname} v${myver}

Safely print a list of pending updates

Usage: ${myname} [options]

  Options:
    -d, --download        download pending updates to the pacman cache.
    -h, --help            display this help message and exit.

Note: Export the "CHECKUPDATES_DB" variable to change the path of the temporary database.

__EOF__
}

OPT_SHORT='dh'
OPT_LONG=('download' 'help')

if ! parseopts "$OPT_SHORT" "${OPT_LONG[@]}" -- "$@"; then
	exit 1
fi
set -- "${OPTRET[@]}"
unset OPT_SHORT OPT_LONG OPTRET

while :; do
	case $1 in
		-d|--download)
			DOWNLOAD_CACHE=1 ;;
		-h|--help)
			usage
			exit 0 ;;
		--)
			shift
			break ;;
	esac
	shift
done

if ! type -P fakeroot >/dev/null; then
	error 'Cannot find the fakeroot binary.'
	exit 1
fi

if [[ -z $CHECKUPDATES_DB ]]; then
	CHECKUPDATES_DB="${TMPDIR:-/tmp}/checkup-db-${USER}/"
fi

trap 'rm -f $CHECKUPDATES_DB/db.lck' INT TERM EXIT

DBPath="$(pacman-conf DBPath)"
if [[ -z "$DBPath" ]] || [[ ! -d "$DBPath" ]]; then
	DBPath="/var/lib/pacman/"
fi

mkdir -p "$CHECKUPDATES_DB"
ln -s "${DBPath}/local" "$CHECKUPDATES_DB" &> /dev/null
if ! fakeroot -- pacman -Sy --dbpath "$CHECKUPDATES_DB" --logfile /dev/null &> /dev/null; then
	error 'Cannot fetch updates'
	exit 1
fi
mapfile -t updates < <(pacman -Qu --dbpath "$CHECKUPDATES_DB" 2> /dev/null | grep -v '\[.*\]')

if (( ${#updates[@]} )); then
	printf '%s\n' "${updates[@]}"
	if (( DOWNLOAD_CACHE )); then
		sudo pacman -Sw --noconfirm "${updates[@]%% *}" --dbpath "$CHECKUPDATES_DB" --logfile /dev/null
	fi
else
	exit 1
fi

# vim: set noet:

Last edited by NuSkool (2019-05-12 18:07:32)

Arch Linux

#1 2019-05-11 18:06:21

Help with analyzing a bash script [SOLVED]

#2 2019-05-11 18:56:48

Re: Help with analyzing a bash script [SOLVED]

#3 2019-05-11 19:58:50

Re: Help with analyzing a bash script [SOLVED]

#4 2019-05-11 20:07:53

Re: Help with analyzing a bash script [SOLVED]

#5 2019-05-12 00:47:54

Re: Help with analyzing a bash script [SOLVED]

#6 2019-05-12 01:12:51

Re: Help with analyzing a bash script [SOLVED]

#7 2019-05-12 02:41:59

Re: Help with analyzing a bash script [SOLVED]

#8 2019-05-12 02:54:40

Re: Help with analyzing a bash script [SOLVED]

#9 2019-05-12 17:27:55

Re: Help with analyzing a bash script [SOLVED]

Board footer