Post your handy self made command line utilities

karol · 2014-09-09 20:07:43

I'd simply do

$ for i in "world" "asynchronous processing"; do echo "hello," $i; done
hello, world
hello, asynchronous processing

;P

Trilby · 2014-09-09 21:38:28

Nothing parallel there (except maybe the tick marks in the quotes - but that all depends on the font)

stevenhoneyman · 2014-09-10 18:23:06

Trilby wrote:

Nothing parallel there (except maybe the tick marks in the quotes - but that all depends on the font)

No, but it's probably faster (guessing!)

Xaero252 · 2014-09-12 01:49:41

I just mostly finished a script (I'm sure it can be improved, but it gets the job done) for downloading a large list of large urls using WGET, while showing both current and overall progress.
I had a list of many urls (~27,000) of varying sizes (~1M-1G) that change regularly (It's data blobs for projects) rsync isn't available remotely, and I don't have permission for SSH access etc, but I can still grab them from the web server.
Anyways, it was frustrating to let wget -i run for hours and have no clue how far through the process I was. So I wrote this, it's pretty crude but functional.

#!/bin/bash
total=$(wc -l < $1)
current=1
while read line; do
	wgetpid=$(wget --progress=bar:force -b -o /tmp/wget.tmp $line |grep pid|cut -d"," -f2|cut -d" " -f3|sed -e 's/\.//')
	while test -d /proc/$wgetpid; do
		progress=$(tac /tmp/wget.tmp | sed -n 1p | tr '\n' ' ')
		destination=$(grep "Saving to:" /tmp/wget.tmp)
		size=$(grep "Length:" /tmp/wget.tmp)
		loga=$(tac /tmp/wget.tmp | sed -n 6p | tr '\n' ' ')
		logb=$(tac /tmp/wget.tmp | sed -n 5p | tr '\n' ' ')
		logc=$(tac /tmp/wget.tmp | sed -n 4p | tr '\n' ' ')
		overall=$(printf "%.2f%%" $(echo "scale=2; $current*100/$total" | bc))
		echo "Downloading: $current of $total... [$overall]"
		echo "$destination $size"
		echo "$progress"
		echo $loga
		echo $logb
		echo $logc
		sleep 0.2
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
	done
	rm /tmp/wget.tmp
        ((++current))
done < $1

I didn't write a usage output, but it's pretty straightforward:
script listofurls.txt

Last edited by Xaero252 (2014-09-12 02:12:20)

snakeroot · 2014-09-15 16:27:06

Wikmatize Text Files

In the Arch wiki, text boxes showing file contents are created by putting a space at the beginning of each line in the relevant file. This is easy to do with sed but annoying to type every time.

The below script takes a text file as a parameter and outputs a file of the same name with a space prepended to each line and with a filename identical to the input filename with a ".wiki" extension.

#!/bin/bash
filename=$1
sed -e 's/^/ /g' < "${filename}" > "${filename}".wiki

karol · 2014-09-15 16:31:58

Err, why not use wiki templates, as outlined by the styleguide?
https://wiki.archlinux.org/index.php/He … formatting
https://wiki.archlinux.org/index.php/Template:Bc
https://wiki.archlinux.org/index.php/Template:Hc

Last edited by karol (2014-09-15 16:33:48)

snakeroot · 2014-09-15 16:41:45

Thanks. I've had trouble getting the format to work right with those.

karol · 2014-09-15 18:51:32

snakeroot wrote:

Thanks. I've had trouble getting the format to work right with those.

Please report such issues. Post a sample of the wrong output on https://wiki.archlinux.org/index.php/Help_talk:Style

karkhaz · 2014-09-16 01:40:18

snakeroot wrote:

Wikmatize

This is my new favourite verb. Well done! A google search for 'wikmatize' turns out this thread as the only result.

karol · 2014-09-16 18:26:25

Ordinarily you say 'wikify' when you want to transform a piece of text to one using the wiki style and markup.

dummyano · 2014-09-20 01:43:10

Not exactly rocket science but... I'm a big fan of gTranslate for firefox,
I wanted something similar that works across all desktop, like
select some text from whatever app, press a combo, and get a notification with translated text

Using

xclip
notify-send
xbindkeys
aur/google-translate-cli-git

Create a script

dummyano@caos ~ % cat bin/triclip 

text_orig=$(xclip -o)
text_tra=$(translate \{en=it\} "$text_orig") # <- change language here
notify-send "$text_tra"

I assigned it to Alt+L via xbindkeys_config

qinohe · 2014-09-30 17:28:09

Hi Archers, in addition to jasonwryan's tool for mounting encrypted volumes with tcplay, I made one to create the volumes.
See http://jasonwryan.com/blog/2013/01/10/truecrypt/
Some parts are taken from that script, see link.

The tool can create an encrypted container with a defined size& name and mount it.

Hash & block-cipher are optional, but, if you input the hash you aslo need to give the cipher, otherwise it defaults to: whirlpool AES-256-XTS
It can create different file systems (ext3,4, vfat – these I tested!),
and mount them. (optional)

For future mounting use Jason's script!
Don't forget to change the user name ;-)

If it deserves it's own thread, we can work it out..

edit:
The tool now has the ability to create a hidden volume. No key files, yet, working on that one.
Updated with checks/tests for user input.
Readability.
What do you think of this script?

edit:
Updated tests for cipher&hash.

edit:
lots of updates

#!/bin/bash
## create encrypted containers, for use with tcplay
## create hidden volumes; plausible deniability
## use of keyfiles; working on it.
## see man tcplay
## https://wiki.archlinux.org/index.php/Tcplay
## mount volumes after creation with this script, see:
## http://jasonwryan.com/blog/2013/01/10/truecrypt/


user=mark
runuser -l  "$user" -c 'losetup -f'
loopdev=$(losetup -f)

if [[ $EUID -ne 0 ]]; then
  printf "%s\n" "$0 must be run as root user."
  exit 1
fi

retvalins=$?
   read -p "full path, where volume will be created?  : " tcplaypath
   read -p "name for the volume to create?  : " tcplayname
   read -p "size?, e.g: 10M, 10G  : " tcplaysize
   read -p "hash function? valid options: tcplay -a help , default: whirlpool [ENTER] : " encrypthash
   read -p "block cipher? valid options: tcplay -b help , default: AES-256-XTS [ENTER] : " encryptcipher
   
[[ $retvalins -ne 0 ]]
   mkdir -p "$tcplaypath"
   cd "$tcplaypath"

   fallocate -l "$tcplaysize" "$tcplayname"
   losetup "$loopdev" "$tcplayname"

## prompt user for hidden folder creation, only Y or N are accepted.
while true
do
  read -p "create hidden volume? Y(es) or N(o) : " hide
  case $hide in
   [Y]* )
  break;;
   [N]* )
  break;;		
   * ) printf "What will it be, Y or N ??\n"
  esac
done

    if [[ -z "$encrypthash" ]] || [[ -z "$encryptcipher" ]] && [[ "$hide" == Y ]]; then
      printf "Default cipher & hash & hidden\n"
      tcplay -c -d "$loopdev" -a whirlpool -b AES-256-XTS -g
      elif [[ "$encrypthash" ]] && [[ "$encryptcipher" ]] && [[ "$hide" == Y ]]; then
        printf "You're choice %s\n" "$encrypthash $encryptcipher & hidden."
        tcplay -c -d "$loopdev" -a "$encrypthash" -b "$encryptcipher" -g
        elif [[ -z "$encrypthash" ]] || [[ ! "$encrypthash" ]] || [[ -z "$encryptcipher" ]] || [[ ! "$encryptcipher" ]]  && [[ "$hide" == N ]]; then
          printf "Cipher or hash not found, using default, no hidden.\n"
          tcplay -c -d "$loopdev" -a whirlpool -b AES-256-XTS
          elif [[ "$encrypthash" ]] && [[ "$encryptcipher" ]] && [[ "$hide" == N ]]; then
            printf "You're choice %s\n" "$encrypthash $encryptcipher, no hidden."
            tcplay -c -d "$loopdev" -a "$encrypthash" -b "$encryptcipher"
    fi

   ## map container to loop device
   tcplay -m "$tcplayname" -d "$loopdev"


retvalfsmount=$?
   read -p "file system?, default: ext4 [ENTER] : " tcplayfs
   read -p "mount point?, or exit script? [ENTER] : " tcplaymount
  
[[ $retvalfsmount -ne 0 ]] && exit 0
    if [[ ! "$tcplayfs" ]]; then
      printf "File system not found, using default\n."
      mkfs.ext4 /dev/mapper/"$tcplayname"
    else
      printf "You're choice, formating to %s\n" "$tcplayfs"
      mkfs."$tcplayfs" /dev/mapper/"$tcplayname"
    fi
      if [[ ! "$tcplaymount" ]]; then
        dmsetup remove "$tcplayname"
        losetup -d "$loopdev"
        exit 1
        ## mount options
        ##  ext3, -4, vfat
        else
        if [[ "$tcplayfs" == ext* ]]; then
          mount -o nodev,nosuid /dev/mapper/"$tcplayname" "$tcplaymount"
            elif [[ "$tcplayfs" == vfat ]]; then          
              mount -o nodev,nosuid,uid=1000,gid=100 /dev/mapper/"$tcplayname" "$tcplaymount"
        fi
      fi

Last edited by qinohe (2014-10-05 18:41:43)

Saint0fCloud · 2014-10-04 12:15:25

^Just a thought, your style for case statements, if/else conditionals, and the manner which you test return values makes your script pretty hard to read. Also, avoid all the messy compound conditionals and use variable defaults, i.g. "${encrypthash:-whirlpool}".

qinohe · 2014-10-04 12:33:39

Hi Saint0fCloud, thank you very much for your answer/suggestions. I'm still learning bash, as you can see ;-)
Would you explain a little bit more, how to change the style for if/else conditionals? How should I test these return values in a nice way?
I will try and understand the way you use the variable, instead of compound conditionals, thanks for the suggestion.

Trilby · 2014-10-04 13:13:26

Indent according to the depth of the conditional:

if [[ some condition ]]; then
  do_one_thing
else
if [[ other condition ]]; then
  do_other_thing
fi;fi

#Change to

if [[ one conditon ]]; then
  do_one_thing
  if [[ other condition]]; then
    do_other_thing
  fi
fi

In the second version it is more clear that the second if condition is nested within the first. The first version makes it look like the two ifs are of equal depth/importance when they are not.

You also use a lot of case statements when there is only one case to check if the input matches a pattern. Just use an if statment or in some cases just a binary operator:

case $retval in
  0)
    do_lots_of_stuff
    ...
esac
exit 0

#Change to
[[ $retval -ne 0 ]] && exit 0

do_lots_of_stuff

Also, there is no point is using a string format specifier for printf when you just pass a static string all three of the below will do the same thing - either of the last two are good, the first is rather odd:

printf "%s\n" "some message"
printf "some message\n"
echo "some message"

EDIT: actually is seems *many* of your nested if statements should be replaced by elifs.

qinohe · 2014-10-04 13:42:30

Depth of an if statement: clear
printf: okay, I'll use the second option.
case statement: I don't really get what you mean, should I remove all or most cases, and bind them together into one?
p.s. as always, thanks for your answer Trilby ;-)

Last edited by qinohe (2014-10-04 13:45:01)

Trilby · 2014-10-04 13:45:34

A case statement with only one case doesn't serve any purpose. Case statements are only useful when there are multiple cases.

qinohe · 2014-10-04 14:00:22

Hmm, so, I see you back when I solved all suggestions given, I have to dive deeper into case statements, anyway. Thanks

Trilby. is this better, or am I still missing some points, thanks.
I have upgraded the script, it's 50 lines shorter, and can probably be made smaller still.
Haven't changed the compound part yet.

Last edited by qinohe (2014-10-05 18:38:37)

progandy · 2014-10-06 01:27:13

Hi, I have a neat script I use to inspect files hidden below a mountpoint. It creates a bindmount in a temporary directory and waits until you close the shell before unmounting it.

#!/bin/sh
# Author: ProgAndy

# print error message and exit.
# [ $1 - optional: integer as exit code ]
#   $* - message
error() {
    _errno=1
    if test 0 -ne "$1" 2>/dev/null; then
        _errno=$1
        shift
    fi
    [ $# -gt 0 ] && echo "$*" >&2
    exit $_errno
}

mountpoint -q "$1" || error 1 "'$1' is not a mountpoint."

dir=$(dirname "$1")
base=$(basename "$1")
[ "$dir"x = "$base"x ] && error 2 "Cannot unmask root mount."

SUDO="sudo"
SUDOV="sudo -v"
[ x"$(id -u)" = x"0" ] && unset SUDO && SUDOV=":"

$SUDOV || error 3 "Could not aquire sudo privileges."

tmp=$(mktemp -d)
[ x"" = x"$tmp" ] && error 4 "Could not create mountpoint."

echo "Unmasking mountpoint ..."
if $SUDO mount --bind "$dir" "$tmp"; then
    cd "$tmp/$base"
    echo "Opening shell on unmasked mountpoint. Exit to unmount."
    "$SHELL"
    echo "Removing unmasked mountpoint ..."
    cd "$tmp/.."
    $SUDO umount "$tmp"
else
  error 6 "Could not mount '$dir' to '$tmp'."
fi
mountpoint -q "$tmp" && sleep 1
mountpoint -q "$tmp" && error 5 "Could not unmount '$tmp'. Remove manually!"
rmdir "$tmp"

Last edited by progandy (2014-10-06 01:29:32)

alphaniner · 2014-10-10 21:37:33

After discovering that which can search aliases and functions, I wrote a function to combine everything:

which() {

    local tmp00 err00 verbose args tmplen maxlen match showall showhelp cont

    tmp00=$(mktemp)
    err00=$(mktemp)
    verbose=0
    maxlen=10
    match=0
    showall=0
    showhelp=0
    cont=1

    trap 'rm -f "$tmp00" "$err00"' EXIT

    [[ ! "$1" ]] && showhelp=1

    while (( $# > 0 )); do

        if [[ $1 == '-v' ]]; then
            verbose=1
            shift

        elif [[ $1 == '-m' ]]; then
            maxlen="$2"
            shift 2

        elif [[ $1 == '-q' ]]; then
            maxlen=0
            shift

        elif [[ $1 == '-a' ]]; then
            showall=1
            shift

        elif [[ $1 = '-h' ]]; then
            showhelp=1
            shift

        else
            args+=("$1")
            shift
        fi

    done

    if (( showhelp == 1 )); then
        cat <<HELPEOF

 Usage:
  which [opts] pattern

  -h         - this help
  -v         - show entire function regardless of length
  -m <num>   - show functions up to length <num> (default $maxlen)
  -q         - never print function body
  -a         - show all matches (alias,function,PATH)

HELPEOF
        return 0
    fi

    # Matching alias
    #  PATH is unset so only aliases will be found.
    #  The >| redirector overrides possible noclobber setting.
    if alias | PATH='' /usr/bin/which -i "${args[@]}" >|"$tmp00" 2>/dev/null; then

        if (( showall == 1 )); then
            echo "Aliases:"
            match=1
        else
            cont=0
        fi
        
        grep "^alias" "$tmp00"
        (( showall == 0 )) && match=1
    fi

    # Matching function
    #  PATH is unset so only aliases will be found.
    #   Could possibly avoid this, but that might make parsing more difficult.
    if (( cont == 1 )) && declare -f | PATH='' /usr/bin/which --read-functions \
                            "${args[@]}" >|"$tmp00" 2>|"$err00"; then

        if (( showall == 1 )); then
            if (( match == 1 )); then
                echo
            else
                match=1
            fi
            echo "Functions:"
        else
            cont=0
        fi

        tmplen="$(awk 'END { print NR }' $tmp00)"
        if (( verbose == 1 )) || (( tmplen < maxlen )); then
            cat "$tmp00"
        else
            grep "^[[:alnum:]]\+ ()$" "$tmp00"
        fi

    fi

    # Normal PATH search
    if (( cont == 1 )) && /usr/bin/which "${args[@]}" >|"$tmp00" 2>/dev/null; then
        if (( showall == 1 )); then
            if (( match == 1 )); then
                echo
            fi
            echo "In PATH:"
        fi
        cat "$tmp00"

    # No match, no output.
    else

        rm -f "$tmp00" "$err00"
        return 1

    fi

    rm -f "$tmp00" "$err00"
    return 0
}

It may have, umm, gotten a bit out of hand.

parchd · 2014-10-21 11:19:42

This probably isn't a generic solution, but to deal with a couple of pdfs that were white text on black background I threw this together:

Usage:
pdfcolinv input.pdf output.pdf

Requires:
pdftk (in AUR as pdftk-bin - I can't understand why something so useful isn't in extra!)

#!/usr/bin/env bash

pdftk $1 output /tmp/pdfinvtmp uncompress
sed 's/1 1 1 sc/x x x sc/g' </tmp/pdfinvtmp|sed 's/0 0 0 sc/1 1 1 sc/g'|sed 's/x x x sc/0 0 0 sc/g' > /tmp/pdfinvtmp2
pdftk /tmp/pdfinvtmp2 output $2 compress
rm /tmp/pdfinvtmp*

There are probably better ways to do it, I rarely write shell scripts these days.

How it works:
Magic. pdftk decompresses the pdf (and later on compresses it again). sed replaces three ones (white) with three noughts (black) and vice-versa. Trial and error, and lots of it, told me what these did.

Last edited by parchd (2014-10-21 11:54:52)

drcouzelis · 2014-10-21 11:45:49

Looks like it could be very handy!

Could you please edit your post and explain what it does? It just looks like magic to me.

EDIT: Thanks, parchd!

Last edited by drcouzelis (2014-10-21 12:15:23)

skanky · 2014-10-21 12:11:28

EDIT: sorry, I think I made a big mistake and rather than confuse people, it's better to delete what I wrote.

Last edited by skanky (2014-10-21 12:18:56)

parchd · 2014-10-21 12:17:10

Thanks skanky, I didn't realise that I could combine them and get rid of the intermediate bit

skanky · 2014-10-21 12:20:11

parchd wrote:

Thanks skanky, I didn't realise that I could combine them and get rid of the intermediate bit

Did it work? I thought I'd messed up.
Don't have time to try it now, as I really should be working.

Arch Linux

#2451 2014-09-09 20:07:43

Re: Post your handy self made command line utilities

#2452 2014-09-09 21:38:28

Re: Post your handy self made command line utilities

#2453 2014-09-10 18:23:06

Re: Post your handy self made command line utilities

#2454 2014-09-12 01:49:41

Re: Post your handy self made command line utilities

#2455 2014-09-15 16:27:06

Re: Post your handy self made command line utilities

Wikmatize Text Files

#2456 2014-09-15 16:31:58

Re: Post your handy self made command line utilities

#2457 2014-09-15 16:41:45

Re: Post your handy self made command line utilities

#2458 2014-09-15 18:51:32

Re: Post your handy self made command line utilities

#2459 2014-09-16 01:40:18

Re: Post your handy self made command line utilities

Wikmatize

#2460 2014-09-16 18:26:25

Re: Post your handy self made command line utilities

#2461 2014-09-20 01:43:10

Re: Post your handy self made command line utilities

#2462 2014-09-30 17:28:09

Re: Post your handy self made command line utilities

#2463 2014-10-04 12:15:25

Re: Post your handy self made command line utilities

#2464 2014-10-04 12:33:39

Re: Post your handy self made command line utilities

#2465 2014-10-04 13:13:26

Re: Post your handy self made command line utilities

#2466 2014-10-04 13:42:30

Re: Post your handy self made command line utilities

#2467 2014-10-04 13:45:34

Re: Post your handy self made command line utilities

#2468 2014-10-04 14:00:22

Re: Post your handy self made command line utilities

#2469 2014-10-06 01:27:13

Re: Post your handy self made command line utilities

#2470 2014-10-10 21:37:33

Re: Post your handy self made command line utilities

#2471 2014-10-21 11:19:42

Re: Post your handy self made command line utilities

#2472 2014-10-21 11:45:49

Re: Post your handy self made command line utilities

#2473 2014-10-21 12:11:28

Re: Post your handy self made command line utilities

#2474 2014-10-21 12:17:10

Re: Post your handy self made command line utilities

#2475 2014-10-21 12:20:11

Re: Post your handy self made command line utilities

Board footer