Post your handy self made command line utilities

nick@archpi2 · 2016-06-15 22:28:23

I tend to write little scripts that run certain parts of utilities and then create an alias so I can run them with a single command.

I'm still learning so nothing like the level of some of the scripts in this thread but someone may find it useful.

This is one that I use to run clamav scans on my home directory and external drives.

#!/bin/bash

# This script will update the clamscan virus definitions,
# scan the following locations recursively: /mnt/ and  ~/
# move any infected files to /home/alarm/clamscan/quarantine/
# and write a log to /home/alarm/clamscan/log

# update clamscan virus definitions:
echo -e "\e[30;48;5;10m***** UPDATING CLAMSCAN DATABASE *****\e[0m"
sudo freshclam
echo ""

# scan home directory recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING HOME DIRECTORY RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ ~/
echo ""

# scan /mnt/ recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING MNT RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ /mnt/
echo ""
echo -e "\e[1;10mDone.\e[0m"

nick@archpi2 · 2016-06-19 11:51:00

Another little script that I wrote to carry out some maintenance jobs as outlined in the Arch Wiki.

Fairly simple chain of tests and commands but handy for updating and checking my bare bones install of Arch that I use as a media server.

#!/bin/bash

# Script to update all packages, prune the pacman cache, remove
# unused (orphan) packages, write list of installed packages to file,
# check for broken symlinks, check for failed systemd services and
# write error logs to display.

# Updating packages:
echo -e "\e[30;48;5;10m***** UPDATING PACMAN REPOSITORY AND PACKAGES *****\e[0m"
sudo pacman -Syyu
echo ""

# Pruning the cache:
echo -e "\e[30;48;5;10m***** PRUNING THE PACKAGE CACHE *****\e[0m"
sudo paccache -r
echo ""

# Removing unused packages (orphans)
echo -e "\e[30;48;5;10m***** REMOVING UNUSED PACKAGES (ORPHANS) *****\e[0m"
sudo pacman -Rns $(pacman -Qtdq)
echo ""

# Writing installed package lists to file: ~/Documents/Sysinfo/
echo -e "\e[30;48;5;10m***** WRITING INSTALLED PACKAGE LISTS TO FILE *****\e[0m"
sudo pacman -Qe > ~/Documents/Sysinfo/packagesQe.txt
sudo pacman -Qn > ~/Documents/Sysinfo/packagesQn.txt
echo -e "\e[1;10mDone.\e[0m"
echo ""

# Checking for broken symlinks:
echo -e "\e[30;48;5;10m***** CHECKING FOR BROKEN SYMLINKS *****\e[0m"
find . -type l -! -exec test -e {} \; -print
echo -e "\e[1;10mDone.\e[0m"
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** FAILED ACTIVE SYSTEMD SERVICES *****\e[0m"
systemctl --failed
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** ALL FAILED SYSTEMD SERVICES *****\e[0m"
systemctl --failed --all
echo ""

# -Logfiles:
echo -e "\e[30;48;5;10m***** LOGFILES *****\e[0m"
journalctl -p 3 -xb
echo ""
echo -e "\e[1;10mDone.\e[0m"

Last edited by nick@archpi2 (2016-06-19 11:54:39)

whitelight999 · 2016-06-23 14:06:16

edit

Last edited by whitelight999 (2016-10-14 23:25:00)

Slithery · 2016-06-23 14:22:43

The shebang needs to be on the first line without a space between #! and /bin/bash otherwise it won't be recognised by the system as a bash script.

Running it through https://www.shellcheck.net/ throws up a couple of pointers as well.

Last edited by Slithery (2016-06-23 14:26:09)

Docbroke · 2016-06-23 15:53:11

my backup script, (it is quite basic one)

#!/bin/bash
# use with "su"

rsync -a --delete /home/sharad/ /media/BackUP/home
rsync -a --delete /etc/ /media/BackUP/etc
rsync -a --delete /var/ /media/BackUP/var

## backup list of installed packages ##
## can be reinstalled with pacman -S $(< pkglist.txt) ##

pacman -Qqen > /media/BackUP/pkglist.txt

alphaniner · 2016-08-09 16:51:09

Often I want to know how long a command takes for simple benchmarking. I know about time but it's a bit of a pain for compound commands and such. Plus TMI. So I would often do something like

start=$(date +%s); <command(s)>; echo $(( $(date +%s) - start ))

But that's a handful, hence timeit:

timeit() {
    if [[ "$1" ]] || [[ ! "$__timeit__" ]]; then
        __timeit__="$(date +%s)"
    else
        echo $(( $(date +%s) - __timeit__ ))
        unset __timeit__
    fi
}

Usage:

$ timeit <any argument>; <command(s)>; timeit

Trilby · 2016-08-09 18:52:49

I know you said you know about `time`, but have you used it with a subshell? It can handle everything timeit can (and possibly more), and seems simpler to me:

$ time (<command1>; <command2>; ... <commandN>)

alphaniner · 2016-08-09 20:02:26

I never considered using a subshell, always used eg.

$ time { command; command; command; }

It's probably the case that sometimes forgetting the unusual requirements of {} (; after final command, ' ' before closing }) influenced things.

Plus I like getting the results in seconds without any extra baggage, and that if I was so inclined I could effortlessly integrate speed calculation by passing a size as an argument to the first call. Sure you could do the same by parsing the output of 'time -p' but at that point I think timeit wins for simplicity.

nick@archpi2 · 2016-08-24 11:38:20

Handy little loop to batch remux mpeg2.ts files to a more mobile friendly format (mp4) using ffmpeg.

#!/bin/bash
# convert mpeg2-ts files to mp4
cd /mnt/2tb_hd/con/
for a in *.ts ; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done

Alad · 2016-08-25 01:12:08

You should use ./* instead of * to avoid issues with names starting in a hyphen (can be interpreted as options to ffmpeg).

Or just drop the cd call, and put the full path in the for loop. Mind you, using positional parameters is more flexible than hardcoding the path.

#!/bin/bash
# convert mpeg2-ts files to mp4
for a; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done

fooconvert /mnt/2tb_hd/con/*.ts

Last edited by Alad (2016-08-25 01:29:34)

dimich · 2016-08-25 01:57:26

nick@archpi2 wrote:

sleep 20

By the way, what is the sense of sleep() here?

Last edited by dimich (2016-08-25 02:02:25)

dimich · 2016-08-25 02:01:48

Alad wrote:

using positional parameters is more flexible than hardcoding the path.

"find … -print0 | xargs -0 …" should be better for this purpose

Trilby · 2016-08-25 11:48:31

The challenge with using find and xargs - or find with it's exec flag - is that you need to then invoke a separate subshell for every single file just to do the string manipulation to get the new name. If ffmpeg just acted on each file it'd be ridiculously easy:

find /path/to/files -name '*.ts' -exec ffmpeg '{}' \;

But you need an input and output file name and you can use the bash string manipulation without calling bash - so that becomes

find /path/to/files -name '*.ts' -exec bash -c "f='{}'; ffmpeg -i \"$f\" -acodec copy -vcodec copy \"${f/%ts/mp4}\"" \;

This is ugly, and error prone due to it being easy to get all the nesting and quoting wrong (I'm not even sure if this is quite right) - but more importantly, it - again - invokes a separate shell for every single file, once this is needed, a for loop is more efficient: one shell.

parchd · 2016-08-25 12:00:45

What about GNU parallel?

parallel ffmpeg -i {} -acodec copy -vcodec copy {.}.mp4 ::: *.ts

Depending on hardware maybe it will speed up the whole thing?

Last edited by parchd (2016-08-25 12:05:53)

Alad · 2016-08-25 12:52:30

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Last edited by Alad (2016-08-25 12:52:36)

parchd · 2016-08-25 13:04:14

Alad wrote:

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?
cf. https://wiki.archlinux.org/index.php/Co … el_version

Erm, no. I mean the tool which fairly reasonably suggests that if it is a core piece of your scientific breakthrough, you give them the credit due.

The "nagging screen" you mention can be disabled by running:

parallel --citation

and then typing

will cite

or simply

touch ~/.parallel/will-cite

Sachiko · 2016-08-29 08:36:36

This is a non-repeating wallpaper changing script I've been trying to get working for the last 3 days. Originally, I wanted it to echo the file name to a file on a new line every time it was set and if the file had lines equal to the number of files in said directory, it would remove the list file and retouch it so it could be echo'd to again, but that ran into a few snags...so I went with method 2. Move the file after it was set and when the file count got to 1, move the last file and then move all the files back to the first directory.

#!/bin/bash
until [ 1 -eq 2 ]; do

DIR="$HOME/.wallpaper/"
PIC=$(find $DIR -type f | shuf -n1)
WPC=$(find $DIR -type f | wc -l)

if [ "$WPC" = "1" ]; then
	mv $HOME/.wallpaper-used/*.{jpg,png,jpeg} $HOME/.wallpaper/ && nitrogen --set-scaled "$PIC" && mv "$PIC" $HOME/.wallpaper-used && sleep 20s
else
	nitrogen --set-scaled "$PIC" && mv "$PIC" $HOME/.wallpaper-used && sleep 20s
fi
done

Of course, the prerequisites are:

- Nitrogen(I prefer this over feh, but you can switch it to feh if you wish)
- folders in your users home directory named .wallpaper and .wallpaper-used
- files to populate ~/.wallpaper with(jpg, png, or jpeg formats only unless you add the extension you wish to use to both line with {jpg,png,jpeg} in them.

user-edit: Changed the method of getting the files and the number of files at the urging of many who have pointed me to the BashGuide which says to use find rather than ls so I complied. But of course credit for the switch goes to Alad who first pointed such an egregious error on my part.

Last edited by Sachiko (2016-09-01 14:40:28)

Alad · 2016-08-29 13:31:30

http://mywiki.wooledge.org/BashPitfalls … .2A.mp3.29
http://mywiki.wooledge.org/ParsingLs

As to preferring nitrogen, feh can do all this in a single line:

https://wiki.archlinux.org/index.php/fe … ound_image

whitelight999 · 2016-08-31 00:45:37

edit

Last edited by whitelight999 (2016-10-14 23:23:27)

Trilby · 2016-08-31 01:00:00

whitelight, from what I skimmed that all looks well done, but for code mainetenance and readability I think it would benefit quite a bit from pulling some of those bits out into separate functions.

Alad · 2016-08-31 06:49:13

All that echo! Use cat:

cat <<-EOF
line1
line2
line3
EOF

Use printf for the remaining echo invocations:

http://unix.stackexchange.com/a/65819

In particular, put the color escapes into functions like msg().

Also the classic mistakes of missing quotes, using backticks instead of $(), parsing ls, etc. Shellcheck is your friend.

Last edited by Alad (2016-08-31 10:48:48)

whitelight999 · 2016-08-31 20:02:02

.

Last edited by whitelight999 (2016-10-14 23:22:37)

Trilby · 2016-08-31 20:04:40

It seems I didn't skim that well.

skanky · 2016-08-31 20:36:20

whitelight999 wrote:

Can you expand on "parsing ls" Whats the mistake with the way I use ls ?

See http://mywiki.wooledge.org/BashGuide/Pr … r_Do_These

The whole BashGuide is must read, though.

http://mywiki.wooledge.org/BashGuide

kokoko3k · 2016-09-01 08:54:16

Probably there are a bunch of similar scripts, but here is mine.
Check hosts state by pinging them, and notify user (notify-send and kdialog) when they change state:

HOSTS=(testhost1 testhost2 testhost3)
CHECKTIME=60


TMPDIR="/tmp/check_hosts12345"

CheckHost(){
    #Init Host:
        if ! test -e $TMPDIR/$HOST ; then
            echo Init $HOST:
            echo UP>$TMPDIR/$HOST
        fi

    read state < $TMPDIR/$HOST

    if ! ping -c 1 -W 1 $HOST &>/dev/null ; then

        #Host down:
        if [ "$state" = "UP" ] ; then
            state="DOWN"
            NOW=$(date +"%d-%m-%Y %r")
            notify-send -a "Check Host" -t 0 "Check Host:" "$NOW:\n $HOST Down"
            kdialog --sorry "$NOW:\n$HOST Down" --title "Check Host:" &
        fi

            else

        #Host up
        if [ "$state" = "DOWN" ] ; then
            state="UP"
            NOW=$(date +"%d-%m-%Y %r")
            notify-send -a "Check Host" -t 0 "Check Host:" "$NOW:\n $HOST Up"
            kdialog --sorry "$NOW:\n$HOST Up" --title "Check Host:" &
        fi

    fi
    echo $state > $TMPDIR/$HOST
}


#
rm -R $TMPDIR &>/dev/null
mkdir $TMPDIR

while true ; do
    for HOST in "${HOSTS[@]}" ; do
        CheckHost
    done
    sleep $CHECKTIME
done

Last edited by kokoko3k (2016-09-01 08:55:07)

Arch Linux

#2776 2016-06-15 22:28:23

Re: Post your handy self made command line utilities

#2777 2016-06-19 11:51:00

Re: Post your handy self made command line utilities

#2778 2016-06-23 14:06:16

Re: Post your handy self made command line utilities

#2779 2016-06-23 14:22:43

Re: Post your handy self made command line utilities

#2780 2016-06-23 15:53:11

Re: Post your handy self made command line utilities

#2781 2016-08-09 16:51:09

Re: Post your handy self made command line utilities

#2782 2016-08-09 18:52:49

Re: Post your handy self made command line utilities

#2783 2016-08-09 20:02:26

Re: Post your handy self made command line utilities

#2784 2016-08-24 11:38:20

Re: Post your handy self made command line utilities

#2785 2016-08-25 01:12:08

Re: Post your handy self made command line utilities

#2786 2016-08-25 01:57:26

Re: Post your handy self made command line utilities

#2787 2016-08-25 02:01:48

Re: Post your handy self made command line utilities

#2788 2016-08-25 11:48:31

Re: Post your handy self made command line utilities

#2789 2016-08-25 12:00:45

Re: Post your handy self made command line utilities

#2790 2016-08-25 12:52:30

Re: Post your handy self made command line utilities

#2791 2016-08-25 13:04:14

Re: Post your handy self made command line utilities

#2792 2016-08-29 08:36:36

Re: Post your handy self made command line utilities

#2793 2016-08-29 13:31:30

Re: Post your handy self made command line utilities

#2794 2016-08-31 00:45:37

Re: Post your handy self made command line utilities

#2795 2016-08-31 01:00:00

Re: Post your handy self made command line utilities

#2796 2016-08-31 06:49:13

Re: Post your handy self made command line utilities

#2797 2016-08-31 20:02:02

Re: Post your handy self made command line utilities

#2798 2016-08-31 20:04:40

Re: Post your handy self made command line utilities

#2799 2016-08-31 20:36:20

Re: Post your handy self made command line utilities

#2800 2016-09-01 08:54:16

Re: Post your handy self made command line utilities

Board footer