You are not logged in.

#2451 2014-09-09 20:07:43

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Post your handy self made command line utilities

I'd simply do

$ for i in "world" "asynchronous processing"; do echo "hello," $i; done
hello, world
hello, asynchronous processing

;P

Offline

#2452 2014-09-09 21:38:28

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Post your handy self made command line utilities

Nothing parallel there (except maybe the tick marks in the quotes - but that all depends on the font)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2453 2014-09-10 18:23:06

stevenhoneyman
Member
From: England
Registered: 2014-05-25
Posts: 241

Re: Post your handy self made command line utilities

Trilby wrote:

Nothing parallel there (except maybe the tick marks in the quotes - but that all depends on the font)

No, but it's probably faster (guessing!)

Offline

#2454 2014-09-12 01:49:41

Xaero252
Member
Registered: 2011-11-28
Posts: 107

Re: Post your handy self made command line utilities

I just mostly finished a script (I'm sure it can be improved, but it gets the job done) for downloading a large list of large urls using WGET, while showing both current and overall progress.
I had a list of many urls (~27,000) of varying sizes (~1M-1G) that change regularly (It's data blobs for projects) rsync isn't available remotely, and I don't have permission for SSH access etc, but I can still grab them from the web server.
Anyways, it was frustrating to let wget -i run for hours and have no clue how far through the process I was. So I wrote this, it's pretty crude but functional.

#!/bin/bash
total=$(wc -l < $1)
current=1
while read line; do
	wgetpid=$(wget --progress=bar:force -b -o /tmp/wget.tmp $line |grep pid|cut -d"," -f2|cut -d" " -f3|sed -e 's/\.//')
	while test -d /proc/$wgetpid; do
		progress=$(tac /tmp/wget.tmp | sed -n 1p | tr '\n' ' ')
		destination=$(grep "Saving to:" /tmp/wget.tmp)
		size=$(grep "Length:" /tmp/wget.tmp)
		loga=$(tac /tmp/wget.tmp | sed -n 6p | tr '\n' ' ')
		logb=$(tac /tmp/wget.tmp | sed -n 5p | tr '\n' ' ')
		logc=$(tac /tmp/wget.tmp | sed -n 4p | tr '\n' ' ')
		overall=$(printf "%.2f%%" $(echo "scale=2; $current*100/$total" | bc))
		echo "Downloading: $current of $total... [$overall]"
		echo "$destination $size"
		echo "$progress"
		echo $loga
		echo $logb
		echo $logc
		sleep 0.2
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
		tput cuu1
		tput el
	done
	rm /tmp/wget.tmp
        ((++current))
done < $1

I didn't write a usage output, but it's pretty straightforward:
script listofurls.txt

Last edited by Xaero252 (2014-09-12 02:12:20)

Offline

#2455 2014-09-15 16:27:06

snakeroot
Member
Registered: 2012-10-06
Posts: 164

Re: Post your handy self made command line utilities

Wikmatize Text Files

In the Arch wiki, text boxes showing file contents are created by putting a space at the beginning of each line in the relevant file. This is easy to do with sed but annoying to type every time.

The below script takes a text file as a parameter and outputs a file of the same name with a space prepended to each line and with a filename identical to the input filename with a ".wiki" extension.

#!/bin/bash
filename=$1
sed -e 's/^/ /g' < "${filename}" > "${filename}".wiki

Offline

#2456 2014-09-15 16:31:58

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Post your handy self made command line utilities

Err, why not use wiki templates, as outlined by the styleguide?
https://wiki.archlinux.org/index.php/He … formatting
https://wiki.archlinux.org/index.php/Template:Bc
https://wiki.archlinux.org/index.php/Template:Hc

Last edited by karol (2014-09-15 16:33:48)

Offline

#2457 2014-09-15 16:41:45

snakeroot
Member
Registered: 2012-10-06
Posts: 164

Re: Post your handy self made command line utilities

Thanks. I've had trouble getting the format to work right with those.

Offline

#2458 2014-09-15 18:51:32

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Post your handy self made command line utilities

snakeroot wrote:

Thanks. I've had trouble getting the format to work right with those.

Please report such issues. Post a sample of the wrong output on https://wiki.archlinux.org/index.php/Help_talk:Style

Offline

#2459 2014-09-16 01:40:18

karkhaz
Member
Registered: 2014-01-25
Posts: 79

Re: Post your handy self made command line utilities

snakeroot wrote:
Wikmatize

This is my new favourite verb. Well done! A google search for 'wikmatize' turns out this thread as the only result.

Offline

#2460 2014-09-16 18:26:25

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Post your handy self made command line utilities

Ordinarily you say 'wikify' when you want to transform a piece of text to one using the wiki style and markup.

Offline

#2461 2014-09-20 01:43:10

dummyano
Member
Registered: 2014-02-14
Posts: 10

Re: Post your handy self made command line utilities

Not exactly rocket science but... I'm a big fan of gTranslate for firefox,
I wanted something similar that works across all desktop, like
select some text from whatever app, press a combo, and get a notification with translated text

Using

xclip
notify-send
xbindkeys
aur/google-translate-cli-git

Create a script

dummyano@caos ~ % cat bin/triclip 

text_orig=$(xclip -o)
text_tra=$(translate \{en=it\} "$text_orig") # <- change language here
notify-send "$text_tra"

I assigned it to Alt+L via xbindkeys_config

Offline

#2462 2014-09-30 17:28:09

qinohe
Member
From: Netherlands
Registered: 2012-06-20
Posts: 1,494

Re: Post your handy self made command line utilities

Hi Archers, in addition to jasonwryan's tool for mounting encrypted volumes with tcplay, I made one to create the volumes.
See  http://jasonwryan.com/blog/2013/01/10/truecrypt/
Some parts are taken from that script, see link.

The tool can create an encrypted container with a defined size& name and mount it.

Hash & block-cipher are optional, but, if you input the hash you aslo need to give the cipher, otherwise it defaults to: whirlpool AES-256-XTS
It can create different file systems (ext3,4, vfat – these I tested!),
and mount them. (optional)

For future mounting use Jason's script!
Don't forget to change the user name ;-)

If it deserves it's own thread, we can work it out..

edit:
The tool now has the ability to create a hidden volume. No key files, yet, working on that one.
Updated with checks/tests for user input.
Readability.
What do you think of this script?

edit:
Updated tests for cipher&hash.

edit:
lots of updates

#!/bin/bash
## create encrypted containers, for use with tcplay
## create hidden volumes; plausible deniability
## use of keyfiles; working on it.
## see man tcplay
## https://wiki.archlinux.org/index.php/Tcplay
## mount volumes after creation with this script, see:
## http://jasonwryan.com/blog/2013/01/10/truecrypt/


user=mark
runuser -l  "$user" -c 'losetup -f'
loopdev=$(losetup -f)

if [[ $EUID -ne 0 ]]; then
  printf "%s\n" "$0 must be run as root user."
  exit 1
fi

retvalins=$?
   read -p "full path, where volume will be created?  : " tcplaypath
   read -p "name for the volume to create?  : " tcplayname
   read -p "size?, e.g: 10M, 10G  : " tcplaysize
   read -p "hash function? valid options: tcplay -a help , default: whirlpool [ENTER] : " encrypthash
   read -p "block cipher? valid options: tcplay -b help , default: AES-256-XTS [ENTER] : " encryptcipher
   
[[ $retvalins -ne 0 ]]
   mkdir -p "$tcplaypath"
   cd "$tcplaypath"

   fallocate -l "$tcplaysize" "$tcplayname"
   losetup "$loopdev" "$tcplayname"

## prompt user for hidden folder creation, only Y or N are accepted.
while true
do
  read -p "create hidden volume? Y(es) or N(o) : " hide
  case $hide in
   [Y]* )
  break;;
   [N]* )
  break;;		
   * ) printf "What will it be, Y or N ??\n"
  esac
done

    if [[ -z "$encrypthash" ]] || [[ -z "$encryptcipher" ]] && [[ "$hide" == Y ]]; then
      printf "Default cipher & hash & hidden\n"
      tcplay -c -d "$loopdev" -a whirlpool -b AES-256-XTS -g
      elif [[ "$encrypthash" ]] && [[ "$encryptcipher" ]] && [[ "$hide" == Y ]]; then
        printf "You're choice %s\n" "$encrypthash $encryptcipher & hidden."
        tcplay -c -d "$loopdev" -a "$encrypthash" -b "$encryptcipher" -g
        elif [[ -z "$encrypthash" ]] || [[ ! "$encrypthash" ]] || [[ -z "$encryptcipher" ]] || [[ ! "$encryptcipher" ]]  && [[ "$hide" == N ]]; then
          printf "Cipher or hash not found, using default, no hidden.\n"
          tcplay -c -d "$loopdev" -a whirlpool -b AES-256-XTS
          elif [[ "$encrypthash" ]] && [[ "$encryptcipher" ]] && [[ "$hide" == N ]]; then
            printf "You're choice %s\n" "$encrypthash $encryptcipher, no hidden."
            tcplay -c -d "$loopdev" -a "$encrypthash" -b "$encryptcipher"
    fi

   ## map container to loop device
   tcplay -m "$tcplayname" -d "$loopdev"


retvalfsmount=$?
   read -p "file system?, default: ext4 [ENTER] : " tcplayfs
   read -p "mount point?, or exit script? [ENTER] : " tcplaymount
  
[[ $retvalfsmount -ne 0 ]] && exit 0
    if [[ ! "$tcplayfs" ]]; then
      printf "File system not found, using default\n."
      mkfs.ext4 /dev/mapper/"$tcplayname"
    else
      printf "You're choice, formating to %s\n" "$tcplayfs"
      mkfs."$tcplayfs" /dev/mapper/"$tcplayname"
    fi
      if [[ ! "$tcplaymount" ]]; then
        dmsetup remove "$tcplayname"
        losetup -d "$loopdev"
        exit 1
        ## mount options
        ##  ext3, -4, vfat
        else
        if [[ "$tcplayfs" == ext* ]]; then
          mount -o nodev,nosuid /dev/mapper/"$tcplayname" "$tcplaymount"
            elif [[ "$tcplayfs" == vfat ]]; then          
              mount -o nodev,nosuid,uid=1000,gid=100 /dev/mapper/"$tcplayname" "$tcplaymount"
        fi
      fi

Last edited by qinohe (2014-10-05 18:41:43)

Offline

#2463 2014-10-04 12:15:25

Saint0fCloud
Member
Registered: 2009-03-31
Posts: 137

Re: Post your handy self made command line utilities

^Just a thought, your style for case statements, if/else conditionals, and the manner which you test return values makes your script pretty hard to read. Also, avoid all the messy compound conditionals and use variable defaults, i.g. "${encrypthash:-whirlpool}".

Offline

#2464 2014-10-04 12:33:39

qinohe
Member
From: Netherlands
Registered: 2012-06-20
Posts: 1,494

Re: Post your handy self made command line utilities

Hi Saint0fCloud, thank you very much for your answer/suggestions. I'm still learning bash, as you can see ;-)
Would you explain a little bit more, how to change the style for if/else conditionals? How should I test these return values in a nice way?
I will try and understand the way you use the variable, instead of compound conditionals, thanks for the suggestion.

Offline

#2465 2014-10-04 13:13:26

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Post your handy self made command line utilities

Indent according to the depth of the conditional:

if [[ some condition ]]; then
  do_one_thing
else
if [[ other condition ]]; then
  do_other_thing
fi;fi

#Change to

if [[ one conditon ]]; then
  do_one_thing
  if [[ other condition]]; then
    do_other_thing
  fi
fi

In the second version it is more clear that the second if condition is nested within the first.  The first version makes it look like the two ifs are of equal depth/importance when they are not.

You also use a lot of case statements when there is only one case to check if the input matches a pattern.  Just use an if statment or in some cases just a binary operator:

case $retval in
  0)
    do_lots_of_stuff
    ...
esac
exit 0

#Change to
[[ $retval -ne 0 ]] && exit 0

do_lots_of_stuff

Also, there is no point is using a string format specifier for printf when you just pass a static string all three of the below will do the same thing - either of the last two are good, the first is rather odd:

printf "%s\n" "some message"
printf "some message\n"
echo "some message"

EDIT: actually is seems *many* of your nested if statements should be replaced by elifs.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2466 2014-10-04 13:42:30

qinohe
Member
From: Netherlands
Registered: 2012-06-20
Posts: 1,494

Re: Post your handy self made command line utilities

Depth of an if statement: clear
printf: okay, I'll use the second option.
case statement: I don't really get what you mean, should I remove all or most cases, and bind them together into one?
p.s. as always, thanks for your answer Trilby ;-)

Last edited by qinohe (2014-10-04 13:45:01)

Offline

#2467 2014-10-04 13:45:34

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Post your handy self made command line utilities

A case statement with only one case doesn't serve any purpose.  Case statements are only useful when there are multiple cases.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2468 2014-10-04 14:00:22

qinohe
Member
From: Netherlands
Registered: 2012-06-20
Posts: 1,494

Re: Post your handy self made command line utilities

Hmm, so, I see you back when I solved all suggestions given, I have to dive deeper into case statements, anyway. Thanks wink

Trilby. is this better, or am I still missing some points, thanks.
I have upgraded the script, it's 50 lines shorter, and can probably be made smaller still.
Haven't changed the compound part yet.

Last edited by qinohe (2014-10-05 18:38:37)

Offline

#2469 2014-10-06 01:27:13

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Post your handy self made command line utilities

Hi, I have a neat script I use to inspect files hidden below a mountpoint. It creates a bindmount in a temporary directory and waits until you close the shell before unmounting it.

#!/bin/sh
# Author: ProgAndy

# print error message and exit.
# [ $1 - optional: integer as exit code ]
#   $* - message
error() {
    _errno=1
    if test 0 -ne "$1" 2>/dev/null; then
        _errno=$1
        shift
    fi
    [ $# -gt 0 ] && echo "$*" >&2
    exit $_errno
}

mountpoint -q "$1" || error 1 "'$1' is not a mountpoint."

dir=$(dirname "$1")
base=$(basename "$1")
[ "$dir"x = "$base"x ] && error 2 "Cannot unmask root mount."

SUDO="sudo"
SUDOV="sudo -v"
[ x"$(id -u)" = x"0" ] && unset SUDO && SUDOV=":"

$SUDOV || error 3 "Could not aquire sudo privileges."

tmp=$(mktemp -d)
[ x"" = x"$tmp" ] && error 4 "Could not create mountpoint."

echo "Unmasking mountpoint ..."
if $SUDO mount --bind "$dir" "$tmp"; then
    cd "$tmp/$base"
    echo "Opening shell on unmasked mountpoint. Exit to unmount."
    "$SHELL"
    echo "Removing unmasked mountpoint ..."
    cd "$tmp/.."
    $SUDO umount "$tmp"
else
  error 6 "Could not mount '$dir' to '$tmp'."
fi
mountpoint -q "$tmp" && sleep 1
mountpoint -q "$tmp" && error 5 "Could not unmount '$tmp'. Remove manually!"
rmdir "$tmp"

Last edited by progandy (2014-10-06 01:29:32)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#2470 2014-10-10 21:37:33

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: Post your handy self made command line utilities

After discovering that which can search aliases and functions, I wrote a function to combine everything:

which() {

    local tmp00 err00 verbose args tmplen maxlen match showall showhelp cont

    tmp00=$(mktemp)
    err00=$(mktemp)
    verbose=0
    maxlen=10
    match=0
    showall=0
    showhelp=0
    cont=1

    trap 'rm -f "$tmp00" "$err00"' EXIT

    [[ ! "$1" ]] && showhelp=1

    while (( $# > 0 )); do

        if [[ $1 == '-v' ]]; then
            verbose=1
            shift

        elif [[ $1 == '-m' ]]; then
            maxlen="$2"
            shift 2

        elif [[ $1 == '-q' ]]; then
            maxlen=0
            shift

        elif [[ $1 == '-a' ]]; then
            showall=1
            shift

        elif [[ $1 = '-h' ]]; then
            showhelp=1
            shift

        else
            args+=("$1")
            shift
        fi

    done

    if (( showhelp == 1 )); then
        cat <<HELPEOF

 Usage:
  which [opts] pattern

  -h         - this help
  -v         - show entire function regardless of length
  -m <num>   - show functions up to length <num> (default $maxlen)
  -q         - never print function body
  -a         - show all matches (alias,function,PATH)

HELPEOF
        return 0
    fi

    # Matching alias
    #  PATH is unset so only aliases will be found.
    #  The >| redirector overrides possible noclobber setting.
    if alias | PATH='' /usr/bin/which -i "${args[@]}" >|"$tmp00" 2>/dev/null; then

        if (( showall == 1 )); then
            echo "Aliases:"
            match=1
        else
            cont=0
        fi
        
        grep "^alias" "$tmp00"
        (( showall == 0 )) && match=1
    fi

    # Matching function
    #  PATH is unset so only aliases will be found.
    #   Could possibly avoid this, but that might make parsing more difficult.
    if (( cont == 1 )) && declare -f | PATH='' /usr/bin/which --read-functions \
                            "${args[@]}" >|"$tmp00" 2>|"$err00"; then

        if (( showall == 1 )); then
            if (( match == 1 )); then
                echo
            else
                match=1
            fi
            echo "Functions:"
        else
            cont=0
        fi

        tmplen="$(awk 'END { print NR }' $tmp00)"
        if (( verbose == 1 )) || (( tmplen < maxlen )); then
            cat "$tmp00"
        else
            grep "^[[:alnum:]]\+ ()$" "$tmp00"
        fi

    fi

    # Normal PATH search
    if (( cont == 1 )) && /usr/bin/which "${args[@]}" >|"$tmp00" 2>/dev/null; then
        if (( showall == 1 )); then
            if (( match == 1 )); then
                echo
            fi
            echo "In PATH:"
        fi
        cat "$tmp00"

    # No match, no output.
    else

        rm -f "$tmp00" "$err00"
        return 1

    fi

    rm -f "$tmp00" "$err00"
    return 0
}

It may have, umm, gotten a bit out of hand.


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2471 2014-10-21 11:19:42

parchd
Member
Registered: 2014-03-08
Posts: 421

Re: Post your handy self made command line utilities

This probably isn't a generic solution, but to deal with a couple of pdfs that were white text on black background I threw this together:

Usage:
pdfcolinv input.pdf output.pdf

Requires:
pdftk (in AUR as pdftk-bin - I can't understand why something so useful isn't in extra!)

#!/usr/bin/env bash

pdftk $1 output /tmp/pdfinvtmp uncompress
sed 's/1 1 1 sc/x x x sc/g' </tmp/pdfinvtmp|sed 's/0 0 0 sc/1 1 1 sc/g'|sed 's/x x x sc/0 0 0 sc/g' > /tmp/pdfinvtmp2
pdftk /tmp/pdfinvtmp2 output $2 compress
rm /tmp/pdfinvtmp*

There are probably better ways to do it, I rarely write shell scripts these days.

How it works:
Magic. pdftk decompresses the pdf (and later on compresses it again). sed replaces three ones (white) with three noughts (black) and vice-versa. Trial and error, and lots of it, told me what these did.

Last edited by parchd (2014-10-21 11:54:52)

Offline

#2472 2014-10-21 11:45:49

drcouzelis
Member
From: Connecticut, USA
Registered: 2009-11-09
Posts: 4,092
Website

Re: Post your handy self made command line utilities

Looks like it could be very handy!

Could you please edit your post and explain what it does? It just looks like magic to me. tongue

EDIT: Thanks, parchd!

Last edited by drcouzelis (2014-10-21 12:15:23)

Offline

#2473 2014-10-21 12:11:28

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: Post your handy self made command line utilities

EDIT: sorry, I think I made a big mistake and rather than confuse people, it's better to delete what I wrote.

Last edited by skanky (2014-10-21 12:18:56)


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#2474 2014-10-21 12:17:10

parchd
Member
Registered: 2014-03-08
Posts: 421

Re: Post your handy self made command line utilities

Thanks skanky, I didn't realise that I could combine them and get rid of the intermediate bit smile

Offline

#2475 2014-10-21 12:20:11

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: Post your handy self made command line utilities

parchd wrote:

Thanks skanky, I didn't realise that I could combine them and get rid of the intermediate bit smile

Did it work? I thought I'd messed up.
Don't have time to try it now, as I really should be working. neutral


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

Board footer

Powered by FluxBB