You are not logged in.

#2776 2016-06-15 22:28:23

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

I tend to write little scripts that run certain parts of utilities and then create an alias so I can run them with a single command.

I'm still learning so nothing like the level of some of the scripts in this thread but someone may find it useful.

This is one that I use to run clamav scans on my home directory and external drives.

#!/bin/bash

# This script will update the clamscan virus definitions,
# scan the following locations recursively: /mnt/ and  ~/
# move any infected files to /home/alarm/clamscan/quarantine/
# and write a log to /home/alarm/clamscan/log

# update clamscan virus definitions:
echo -e "\e[30;48;5;10m***** UPDATING CLAMSCAN DATABASE *****\e[0m"
sudo freshclam
echo ""

# scan home directory recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING HOME DIRECTORY RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ ~/
echo ""

# scan /mnt/ recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING MNT RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ /mnt/
echo ""
echo -e "\e[1;10mDone.\e[0m"

Offline

#2777 2016-06-19 11:51:00

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

Another little script that I wrote to carry out some maintenance jobs as outlined in the Arch Wiki.

Fairly simple chain of tests and commands but handy for updating and checking my bare bones install of Arch that I use as a media server.


#!/bin/bash

# Script to update all packages, prune the pacman cache, remove
# unused (orphan) packages, write list of installed packages to file,
# check for broken symlinks, check for failed systemd services and
# write error logs to display.

# Updating packages:
echo -e "\e[30;48;5;10m***** UPDATING PACMAN REPOSITORY AND PACKAGES *****\e[0m"
sudo pacman -Syyu
echo ""

# Pruning the cache:
echo -e "\e[30;48;5;10m***** PRUNING THE PACKAGE CACHE *****\e[0m"
sudo paccache -r
echo ""

# Removing unused packages (orphans)
echo -e "\e[30;48;5;10m***** REMOVING UNUSED PACKAGES (ORPHANS) *****\e[0m"
sudo pacman -Rns $(pacman -Qtdq)
echo ""

# Writing installed package lists to file: ~/Documents/Sysinfo/
echo -e "\e[30;48;5;10m***** WRITING INSTALLED PACKAGE LISTS TO FILE *****\e[0m"
sudo pacman -Qe > ~/Documents/Sysinfo/packagesQe.txt
sudo pacman -Qn > ~/Documents/Sysinfo/packagesQn.txt
echo -e "\e[1;10mDone.\e[0m"
echo ""

# Checking for broken symlinks:
echo -e "\e[30;48;5;10m***** CHECKING FOR BROKEN SYMLINKS *****\e[0m"
find . -type l -! -exec test -e {} \; -print
echo -e "\e[1;10mDone.\e[0m"
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** FAILED ACTIVE SYSTEMD SERVICES *****\e[0m"
systemctl --failed
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** ALL FAILED SYSTEMD SERVICES *****\e[0m"
systemctl --failed --all
echo ""

# -Logfiles:
echo -e "\e[30;48;5;10m***** LOGFILES *****\e[0m"
journalctl -p 3 -xb
echo ""
echo -e "\e[1;10mDone.\e[0m"

Last edited by nick@archpi2 (2016-06-19 11:54:39)

Offline

#2778 2016-06-23 14:06:16

whitelight999
Member
Registered: 2016-05-17
Posts: 12

Re: Post your handy self made command line utilities

My First Arch Linux script( 4th ever) , a backup utility. might add some options to it later. Criticisms and  advice welcome!

#!/bin/bash

#GWB 19-05-16 
#Backup script backs up files to remote directory
#1: Make sure destination  is mounted
#2: Run as root



echo "******************************"
echo "*Written By GWB      *"
echo "*Version 1 21-06-16         *"
echo "*Backup script               *"
echo "******************************"
echo " "

#Check that user ran as sudo
if (( $EUID != 0 )); then
    echo "Please run as root"
    exit
fi

#get user input for which path for backup
Dest1="/run/media/gwb/Linux_backup"
Dest2="/run/media/gwb/iomeaga_320"
echo "(1)   "  $Dest1
echo "(2)   "  $Dest2
echo ""
echo "Pick destination directory for backup :-"
echo "NOTE : The Hard drives  internal and external must be mounted"
echo ""
echo "Press 1 or 2 , 3 to exit,  followed by [ENTER]"
echo ""
read choice
echo "You picked option " $choice 
echo "" 

#check that paths exist and change path to dest
case "$choice" in

1)  echo $Dest1
    if  [ -d $Dest1 ] 
    then
      cd $Dest1
    else
	  echo "Path not found to destination directory"
	  echo "NOTE : The Hard drives  internal and external must be mounted"
          echo ""
          exit
    fi
    ;;

2)  echo  $Dest2
    if  [ -d $Dest2 ] 
    then
      cd $Dest2
    else
	  echo "Path not found to destination directory"
	  echo "NOTE : The Hard drives  internal and external must be mounted"
          echo ""
          exit
    fi
    
    ;;
3)  echo  "exit"
    exit
    ;;

*) echo "exit"
    exit
   ;;
esac

#make the backup directory
export TODAYSBACKUPDATE=`date +%d-%b-%Y`
mkdir $TODAYSBACKUPDATE
cd $TODAYSBACKUPDATE
echo "directory made at " "$TODAYSBACKUPDATE" at "backup path"
echo ""

#begin the backup
echo "1 Make copy of first 512 bytes MBR with dd"
dd if=/dev/sdb1 of=hda-mbr.bin bs=512 count=1
echo "1 Done "
echo ""

echo "2 Make a copy of etc dir"
cp -a  -u /etc .
echo "2 Done "
echo ""

echo "3 Make a copy of home dir"
cp -a  -u /home .
echo "3 Done "
echo ""
sync


echo "4 Make tarball of all except tmp dev proc sys run"
tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/run/* -pzcf RootFS_backup.tar.gz /
echo " 4 done"
echo ""
sync

echo "5 Make copy of package lists"
pacman -Qqen > pkglist.txt
pacman -Qm > pkglistAUR.txt
echo "5 done"
echo ""
echo "All done"

edit:
fixed shebang typo

Last edited by whitelight999 (2016-06-23 18:18:24)


cylon (AUR) Maintenance , backups and system checks in a menu driven Bash script
https://aur.archlinux.org/packages/cylon/

Offline

#2779 2016-06-23 14:22:43

slithery
Member
Registered: 2013-12-01
Posts: 800

Re: Post your handy self made command line utilities

The shebang needs to be on the first line without a space between #! and /bin/bash otherwise it won't be recognised by the system as a bash script.

Running it through https://www.shellcheck.net/ throws up a couple of pointers as well.

Last edited by slithery (2016-06-23 14:26:09)

Offline

#2780 2016-06-23 15:53:11

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 529

Re: Post your handy self made command line utilities

my backup script, (it is quite basic one)

#!/bin/bash
# use with "su"

rsync -a --delete /home/sharad/ /media/BackUP/home
rsync -a --delete /etc/ /media/BackUP/etc
rsync -a --delete /var/ /media/BackUP/var

## backup list of installed packages ##
## can be reinstalled with pacman -S $(< pkglist.txt) ##

pacman -Qqen > /media/BackUP/pkglist.txt

Arch is home!
i3 rofi w3m mc vim mutt pass
w3m-dmenu

Offline

#2781 2016-08-09 16:51:09

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,754

Re: Post your handy self made command line utilities

Often I want to know how long a command takes for simple benchmarking. I know about time but it's a bit of a pain for compound commands and such. Plus TMI. So I would often do something like

start=$(date +%s); <command(s)>; echo $(( $(date +%s) - start ))

But that's a handful, hence timeit:

timeit() {
    if [[ "$1" ]] || [[ ! "$__timeit__" ]]; then
        __timeit__="$(date +%s)"
    else
        echo $(( $(date +%s) - __timeit__ ))
        unset __timeit__
    fi
}

Usage:

$ timeit <any argument>; <command(s)>; timeit

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2782 2016-08-09 18:52:49

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 14,220
Website

Re: Post your handy self made command line utilities

I know you said you know about `time`, but have you used it with a subshell?  It can handle everything timeit can (and possibly more), and seems simpler to me:

$ time (<command1>; <command2>; ... <commandN>)

InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#2783 2016-08-09 20:02:26

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,754

Re: Post your handy self made command line utilities

I never considered using a subshell, always used eg.

$ time { command; command; command; }

It's probably the case that sometimes forgetting the unusual requirements of {} (; after final command, ' ' before closing }) influenced things.

Plus I like getting the results in seconds without any extra baggage, and that if I was so inclined I could effortlessly integrate speed calculation by passing a size as an argument to the first call. Sure you could do the same by parsing the output of 'time -p' but at that point I think timeit wins for simplicity.


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2784 2016-08-24 11:38:20

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

Handy little loop to batch remux mpeg2.ts files to a more mobile friendly format (mp4) using ffmpeg.

#!/bin/bash
# convert mpeg2-ts files to mp4
cd /mnt/2tb_hd/con/
for a in *.ts ; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done

Offline

#2785 2016-08-25 01:12:08

Alad
Wiki Admin
Registered: 2014-05-04
Posts: 1,186

Re: Post your handy self made command line utilities

You should use ./* instead of * to avoid issues with names starting in a hyphen (can be interpreted as options to ffmpeg).

Or just drop the cd call, and put the full path in the for loop. Mind you, using positional parameters is more flexible than hardcoding the path.

#!/bin/bash
# convert mpeg2-ts files to mp4
for a; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done
fooconvert /mnt/2tb_hd/con/*.ts

Last edited by Alad (2016-08-25 01:29:34)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
User:Alad

Offline

#2786 2016-08-25 01:57:26

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 35

Re: Post your handy self made command line utilities

nick@archpi2 wrote:
sleep 20

By the way, what is the sense of sleep() here?

Last edited by dimich (2016-08-25 02:02:25)

Offline

#2787 2016-08-25 02:01:48

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 35

Re: Post your handy self made command line utilities

Alad wrote:

using positional parameters is more flexible than hardcoding the path.

"find … -print0 | xargs -0 …" should be better for this purpose

Offline

#2788 2016-08-25 11:48:31

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 14,220
Website

Re: Post your handy self made command line utilities

The challenge with using find and xargs - or find with it's exec flag - is that you need to then invoke a separate subshell for every single file just to do the string manipulation to get the new name.  If ffmpeg just acted on each file it'd be ridiculously easy:

find /path/to/files -name '*.ts' -exec ffmpeg '{}' \;

But you need an input and output file name and you can use the bash string manipulation without calling bash - so that becomes

find /path/to/files -name '*.ts' -exec bash -c "f='{}'; ffmpeg -i \"$f\" -acodec copy -vcodec copy \"${f/%ts/mp4}\"" \;

This is ugly, and error prone due to it being easy to get all the nesting and quoting wrong (I'm not even sure if this is quite right) - but more importantly, it - again - invokes a separate shell for every single file, once this is needed, a for loop is more efficient: one shell.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#2789 2016-08-25 12:00:45

parchd
Member
Registered: 2014-03-08
Posts: 288

Re: Post your handy self made command line utilities

What about GNU parallel?

parallel ffmpeg -i {} -acodec copy -vcodec copy {.}.mp4 ::: *.ts

Depending on hardware maybe it will speed up the whole thing?

Last edited by parchd (2016-08-25 12:05:53)

Offline

#2790 2016-08-25 12:52:30

Alad
Wiki Admin
Registered: 2014-05-04
Posts: 1,186

Re: Post your handy self made command line utilities

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Last edited by Alad (2016-08-25 12:52:36)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
User:Alad

Offline

#2791 2016-08-25 13:04:14

parchd
Member
Registered: 2014-03-08
Posts: 288

Re: Post your handy self made command line utilities

Alad wrote:

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Erm, no. I mean the tool which fairly reasonably suggests that if it is a core piece of your scientific breakthrough, you give them the credit due.

The "nagging screen" you mention can be disabled by running:

parallel --citation

and then typing

will cite

or simply

touch ~/.parallel/will-cite

Offline

#2792 2016-08-29 08:36:36

Sachiko
Member
Registered: 2016-07-01
Posts: 8

Re: Post your handy self made command line utilities

This is a non-repeating wallpaper changing script I've been trying to get working for the last 3 days. Originally, I wanted it to echo the file name to a file on a new line every time it was set and if the file had lines equal to the number of files in said directory, it would remove the list file and retouch it so it could be echo'd to again, but that ran into a few snags...so I went with method 2. Move the file after it was set and when the file count got to 1, move the last file and then move all the files back to the first directory.

#!/bin/bash
until [ 1 -eq 2 ]; do

DIR="$HOME/.wallpaper/"
PIC=$(find $DIR -type f | shuf -n1)
WPC=$(find $DIR -type f | wc -l)

if [ "$WPC" = "1" ]; then
	mv $HOME/.wallpaper-used/*.{jpg,png,jpeg} $HOME/.wallpaper/ && nitrogen --set-scaled "$PIC" && mv "$PIC" $HOME/.wallpaper-used && sleep 20s
else
	nitrogen --set-scaled "$PIC" && mv "$PIC" $HOME/.wallpaper-used && sleep 20s
fi
done

Of course, the prerequisites are:

- Nitrogen(I prefer this over feh, but you can switch it to feh if you wish)
- folders in  your users home directory named .wallpaper and .wallpaper-used
- files to populate ~/.wallpaper with(jpg, png, or jpeg formats only unless you add the extension you wish to use to both line with {jpg,png,jpeg} in them.

user-edit: Changed the method of getting the files and the number of files at the urging of many who have pointed me to the BashGuide which says to use find rather than ls so I complied. But of course credit for the switch goes to Alad who first pointed such an egregious error on my part.

Last edited by Sachiko (2016-09-01 14:40:28)

Offline

#2793 2016-08-29 13:31:30

Alad
Wiki Admin
Registered: 2014-05-04
Posts: 1,186

Re: Post your handy self made command line utilities


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
User:Alad

Offline

#2794 2016-08-31 00:45:37

whitelight999
Member
Registered: 2016-05-17
Posts: 12

Re: Post your handy self made command line utilities

EDITED for AUR link with updated code

Hi

This is my first contribution to the AUR and the Linux world.

What is Cylon?
A  Bash shell script. The goal was to create a script to do  maintenance,
backups and system checks in a single menu driven optional
Command line utility  for  Arch.
This script provides numerous tools
to an Arch Linux user  for maintenance, system checks and backups.
Its quite handy if you use cower as an AUR helper,

Criticism highly welcome as are suggestions for future additions, options.

AUR page:
https://aur.archlinux.org/packages/cylon/
Github project page: 
https://github.com/whitelight999/cylon

Last edited by whitelight999 (2016-09-13 10:41:17)


cylon (AUR) Maintenance , backups and system checks in a menu driven Bash script
https://aur.archlinux.org/packages/cylon/

Offline

#2795 2016-08-31 01:00:00

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 14,220
Website

Re: Post your handy self made command line utilities

whitelight, from what I skimmed that all looks well done, but for code mainetenance and readability I think it would benefit quite a bit from pulling some of those bits out into separate functions.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#2796 2016-08-31 06:49:13

Alad
Wiki Admin
Registered: 2014-05-04
Posts: 1,186

Re: Post your handy self made command line utilities

All that echo! Use cat:

cat <<-EOF
line1
line2
line3
EOF

Use printf for the remaining echo invocations:

http://unix.stackexchange.com/a/65819

In particular, put the color escapes into functions like msg().

Also the classic mistakes of missing quotes, using backticks instead of $(), parsing ls, etc. Shellcheck is your friend.

Last edited by Alad (2016-08-31 10:48:48)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
User:Alad

Offline

#2797 2016-08-31 20:02:02

whitelight999
Member
Registered: 2016-05-17
Posts: 12

Re: Post your handy self made command line utilities

Alad wrote:

All that echo! Use cat:

cat <<-EOF
line1
line2
line3
EOF

Use printf for the remaining echo invocations:

http://unix.stackexchange.com/a/65819

In particular, put the color escapes into functions like msg().

Also the classic mistakes of missing quotes, using backticks instead of $(), parsing ls, etc. Shellcheck is your friend.

Thanks for the feedback and I am optimizing the code,
Can you expand on "parsing ls" Whats the mistake with the way I use ls ?
Whats wrong with using  ls command with -A option to check  whether a directory is empty?
DIR="/tmp"
# init
# look for empty dir
if [ "$(ls -A $DIR)" ]; then
     echo "Take action $DIR is not Empty"
else
    echo "$DIR is Empty"
fi
http://www.cyberciti.biz/faq/linux-unix … ory-empty/
I should point out this is my fifth ever script.

Last edited by whitelight999 (2016-09-01 12:04:26)


cylon (AUR) Maintenance , backups and system checks in a menu driven Bash script
https://aur.archlinux.org/packages/cylon/

Offline

#2798 2016-08-31 20:04:40

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 14,220
Website

Re: Post your handy self made command line utilities

It seems I didn't skim that well.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#2799 2016-08-31 20:36:20

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,841

Re: Post your handy self made command line utilities

whitelight999 wrote:

Can you expand on "parsing ls" Whats the mistake with the way I use ls ?

See http://mywiki.wooledge.org/BashGuide/Pr … r_Do_These

The whole BashGuide is must read, though. smile

http://mywiki.wooledge.org/BashGuide


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#2800 2016-09-01 08:54:16

kokoko3k
Member
Registered: 2008-11-14
Posts: 1,526

Re: Post your handy self made command line utilities

Probably there are a bunch of similar scripts, but here is mine.
Check hosts state by pinging them, and notify user (notify-send and kdialog) when they change state:

HOSTS=(testhost1 testhost2 testhost3)
CHECKTIME=60


TMPDIR="/tmp/check_hosts12345"

CheckHost(){
    #Init Host:
        if ! test -e $TMPDIR/$HOST ; then
            echo Init $HOST:
            echo UP>$TMPDIR/$HOST
        fi

    read state < $TMPDIR/$HOST

    if ! ping -c 1 -W 1 $HOST &>/dev/null ; then

        #Host down:
        if [ "$state" = "UP" ] ; then
            state="DOWN"
            NOW=$(date +"%d-%m-%Y %r")
            notify-send -a "Check Host" -t 0 "Check Host:" "$NOW:\n $HOST Down"
            kdialog --sorry "$NOW:\n$HOST Down" --title "Check Host:" &
        fi

            else

        #Host up
        if [ "$state" = "DOWN" ] ; then
            state="UP"
            NOW=$(date +"%d-%m-%Y %r")
            notify-send -a "Check Host" -t 0 "Check Host:" "$NOW:\n $HOST Up"
            kdialog --sorry "$NOW:\n$HOST Up" --title "Check Host:" &
        fi

    fi
    echo $state > $TMPDIR/$HOST
}


#
rm -R $TMPDIR &>/dev/null
mkdir $TMPDIR

while true ; do
    for HOST in "${HOSTS[@]}" ; do
        CheckHost
    done
    sleep $CHECKTIME
done

Last edited by kokoko3k (2016-09-01 08:55:07)

Offline

Board footer

Powered by FluxBB