You are not logged in.

#2776 2016-06-15 22:28:23

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

I tend to write little scripts that run certain parts of utilities and then create an alias so I can run them with a single command.

I'm still learning so nothing like the level of some of the scripts in this thread but someone may find it useful.

This is one that I use to run clamav scans on my home directory and external drives.

#!/bin/bash

# This script will update the clamscan virus definitions,
# scan the following locations recursively: /mnt/ and  ~/
# move any infected files to /home/alarm/clamscan/quarantine/
# and write a log to /home/alarm/clamscan/log

# update clamscan virus definitions:
echo -e "\e[30;48;5;10m***** UPDATING CLAMSCAN DATABASE *****\e[0m"
sudo freshclam
echo ""

# scan home directory recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING HOME DIRECTORY RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ ~/
echo ""

# scan /mnt/ recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING MNT RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ /mnt/
echo ""
echo -e "\e[1;10mDone.\e[0m"

Offline

#2777 2016-06-19 11:51:00

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

Another little script that I wrote to carry out some maintenance jobs as outlined in the Arch Wiki.

Fairly simple chain of tests and commands but handy for updating and checking my bare bones install of Arch that I use as a media server.


#!/bin/bash

# Script to update all packages, prune the pacman cache, remove
# unused (orphan) packages, write list of installed packages to file,
# check for broken symlinks, check for failed systemd services and
# write error logs to display.

# Updating packages:
echo -e "\e[30;48;5;10m***** UPDATING PACMAN REPOSITORY AND PACKAGES *****\e[0m"
sudo pacman -Syyu
echo ""

# Pruning the cache:
echo -e "\e[30;48;5;10m***** PRUNING THE PACKAGE CACHE *****\e[0m"
sudo paccache -r
echo ""

# Removing unused packages (orphans)
echo -e "\e[30;48;5;10m***** REMOVING UNUSED PACKAGES (ORPHANS) *****\e[0m"
sudo pacman -Rns $(pacman -Qtdq)
echo ""

# Writing installed package lists to file: ~/Documents/Sysinfo/
echo -e "\e[30;48;5;10m***** WRITING INSTALLED PACKAGE LISTS TO FILE *****\e[0m"
sudo pacman -Qe > ~/Documents/Sysinfo/packagesQe.txt
sudo pacman -Qn > ~/Documents/Sysinfo/packagesQn.txt
echo -e "\e[1;10mDone.\e[0m"
echo ""

# Checking for broken symlinks:
echo -e "\e[30;48;5;10m***** CHECKING FOR BROKEN SYMLINKS *****\e[0m"
find . -type l -! -exec test -e {} \; -print
echo -e "\e[1;10mDone.\e[0m"
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** FAILED ACTIVE SYSTEMD SERVICES *****\e[0m"
systemctl --failed
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** ALL FAILED SYSTEMD SERVICES *****\e[0m"
systemctl --failed --all
echo ""

# -Logfiles:
echo -e "\e[30;48;5;10m***** LOGFILES *****\e[0m"
journalctl -p 3 -xb
echo ""
echo -e "\e[1;10mDone.\e[0m"

Last edited by nick@archpi2 (2016-06-19 11:54:39)

Offline

#2778 2016-06-23 14:06:16

whitelight999
Member
Registered: 2016-05-17
Posts: 4

Re: Post your handy self made command line utilities

My First Arch Linux script( 4th ever) , a backup utility. might add some options to it later. Criticisms and  advice welcome!

#!/bin/bash

#GWB 19-05-16 
#Backup script backs up files to remote directory
#1: Make sure destination  is mounted
#2: Run as root



echo "******************************"
echo "*Written By GWB      *"
echo "*Version 1 21-06-16         *"
echo "*Backup script               *"
echo "******************************"
echo " "

#Check that user ran as sudo
if (( $EUID != 0 )); then
    echo "Please run as root"
    exit
fi

#get user input for which path for backup
Dest1="/run/media/gwb/Linux_backup"
Dest2="/run/media/gwb/iomeaga_320"
echo "(1)   "  $Dest1
echo "(2)   "  $Dest2
echo ""
echo "Pick destination directory for backup :-"
echo "NOTE : The Hard drives  internal and external must be mounted"
echo ""
echo "Press 1 or 2 , 3 to exit,  followed by [ENTER]"
echo ""
read choice
echo "You picked option " $choice 
echo "" 

#check that paths exist and change path to dest
case "$choice" in

1)  echo $Dest1
    if  [ -d $Dest1 ] 
    then
      cd $Dest1
    else
	  echo "Path not found to destination directory"
	  echo "NOTE : The Hard drives  internal and external must be mounted"
          echo ""
          exit
    fi
    ;;

2)  echo  $Dest2
    if  [ -d $Dest2 ] 
    then
      cd $Dest2
    else
	  echo "Path not found to destination directory"
	  echo "NOTE : The Hard drives  internal and external must be mounted"
          echo ""
          exit
    fi
    
    ;;
3)  echo  "exit"
    exit
    ;;

*) echo "exit"
    exit
   ;;
esac

#make the backup directory
export TODAYSBACKUPDATE=`date +%d-%b-%Y`
mkdir $TODAYSBACKUPDATE
cd $TODAYSBACKUPDATE
echo "directory made at " "$TODAYSBACKUPDATE" at "backup path"
echo ""

#begin the backup
echo "1 Make copy of first 512 bytes MBR with dd"
dd if=/dev/sdb1 of=hda-mbr.bin bs=512 count=1
echo "1 Done "
echo ""

echo "2 Make a copy of etc dir"
cp -a  -u /etc .
echo "2 Done "
echo ""

echo "3 Make a copy of home dir"
cp -a  -u /home .
echo "3 Done "
echo ""
sync


echo "4 Make tarball of all except tmp dev proc sys run"
tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/run/* -pzcf RootFS_backup.tar.gz /
echo " 4 done"
echo ""
sync

echo "5 Make copy of package lists"
pacman -Qqen > pkglist.txt
pacman -Qm > pkglistAUR.txt
echo "5 done"
echo ""
echo "All done"

edit:
fixed shebang typo

Last edited by whitelight999 (2016-06-23 18:18:24)

Offline

#2779 2016-06-23 14:22:43

slithery
Member
Registered: 2013-12-01
Posts: 790

Re: Post your handy self made command line utilities

The shebang needs to be on the first line without a space between #! and /bin/bash otherwise it won't be recognised by the system as a bash script.

Running it through https://www.shellcheck.net/ throws up a couple of pointers as well.

Last edited by slithery (2016-06-23 14:26:09)

Offline

#2780 2016-06-23 15:53:11

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 529

Re: Post your handy self made command line utilities

my backup script, (it is quite basic one)

#!/bin/bash
# use with "su"

rsync -a --delete /home/sharad/ /media/BackUP/home
rsync -a --delete /etc/ /media/BackUP/etc
rsync -a --delete /var/ /media/BackUP/var

## backup list of installed packages ##
## can be reinstalled with pacman -S $(< pkglist.txt) ##

pacman -Qqen > /media/BackUP/pkglist.txt

Arch is home!
i3 rofi w3m mc vim mutt pass
w3m-dmenu

Offline

#2781 2016-08-09 16:51:09

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,725

Re: Post your handy self made command line utilities

Often I want to know how long a command takes for simple benchmarking. I know about time but it's a bit of a pain for compound commands and such. Plus TMI. So I would often do something like

start=$(date +%s); <command(s)>; echo $(( $(date +%s) - start ))

But that's a handful, hence timeit:

timeit() {
    if [[ "$1" ]] || [[ ! "$__timeit__" ]]; then
        __timeit__="$(date +%s)"
    else
        echo $(( $(date +%s) - __timeit__ ))
        unset __timeit__
    fi
}

Usage:

$ timeit <any argument>; <command(s)>; timeit

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2782 2016-08-09 18:52:49

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 14,001
Website

Re: Post your handy self made command line utilities

I know you said you know about `time`, but have you used it with a subshell?  It can handle everything timeit can (and possibly more), and seems simpler to me:

$ time (<command1>; <command2>; ... <commandN>)

InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#2783 2016-08-09 20:02:26

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,725

Re: Post your handy self made command line utilities

I never considered using a subshell, always used eg.

$ time { command; command; command; }

It's probably the case that sometimes forgetting the unusual requirements of {} (; after final command, ' ' before closing }) influenced things.

Plus I like getting the results in seconds without any extra baggage, and that if I was so inclined I could effortlessly integrate speed calculation by passing a size as an argument to the first call. Sure you could do the same by parsing the output of 'time -p' but at that point I think timeit wins for simplicity.


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2784 2016-08-24 11:38:20

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

Handy little loop to batch remux mpeg2.ts files to a more mobile friendly format (mp4) using ffmpeg.

#!/bin/bash
# convert mpeg2-ts files to mp4
cd /mnt/2tb_hd/con/
for a in *.ts ; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done

Offline

#2785 2016-08-25 01:12:08

Alad
Wiki Admin
Registered: 2014-05-04
Posts: 1,099

Re: Post your handy self made command line utilities

You should use ./* instead of * to avoid issues with names starting in a hyphen (can be interpreted as options to ffmpeg).

Or just drop the cd call, and put the full path in the for loop. Mind you, using positional parameters is more flexible than hardcoding the path.

#!/bin/bash
# convert mpeg2-ts files to mp4
for a; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done
fooconvert /mnt/2tb_hd/con/*.ts

Last edited by Alad (2016-08-25 01:29:34)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
User:Alad

Offline

#2786 2016-08-25 01:57:26

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 35

Re: Post your handy self made command line utilities

nick@archpi2 wrote:
sleep 20

By the way, what is the sense of sleep() here?

Last edited by dimich (2016-08-25 02:02:25)

Offline

#2787 2016-08-25 02:01:48

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 35

Re: Post your handy self made command line utilities

Alad wrote:

using positional parameters is more flexible than hardcoding the path.

"find … -print0 | xargs -0 …" should be better for this purpose

Offline

#2788 2016-08-25 11:48:31

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 14,001
Website

Re: Post your handy self made command line utilities

The challenge with using find and xargs - or find with it's exec flag - is that you need to then invoke a separate subshell for every single file just to do the string manipulation to get the new name.  If ffmpeg just acted on each file it'd be ridiculously easy:

find /path/to/files -name '*.ts' -exec ffmpeg '{}' \;

But you need an input and output file name and you can use the bash string manipulation without calling bash - so that becomes

find /path/to/files -name '*.ts' -exec bash -c "f='{}'; ffmpeg -i \"$f\" -acodec copy -vcodec copy \"${f/%ts/mp4}\"" \;

This is ugly, and error prone due to it being easy to get all the nesting and quoting wrong (I'm not even sure if this is quite right) - but more importantly, it - again - invokes a separate shell for every single file, once this is needed, a for loop is more efficient: one shell.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#2789 2016-08-25 12:00:45

parchd
Member
Registered: 2014-03-08
Posts: 284

Re: Post your handy self made command line utilities

What about GNU parallel?

parallel ffmpeg -i {} -acodec copy -vcodec copy {.}.mp4 ::: *.ts

Depending on hardware maybe it will speed up the whole thing?

Last edited by parchd (2016-08-25 12:05:53)

Offline

#2790 2016-08-25 12:52:30

Alad
Wiki Admin
Registered: 2014-05-04
Posts: 1,099

Re: Post your handy self made command line utilities

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Last edited by Alad (2016-08-25 12:52:36)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
User:Alad

Offline

#2791 2016-08-25 13:04:14

parchd
Member
Registered: 2014-03-08
Posts: 284

Re: Post your handy self made command line utilities

Alad wrote:

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Erm, no. I mean the tool which fairly reasonably suggests that if it is a core piece of your scientific breakthrough, you give them the credit due.

The "nagging screen" you mention can be disabled by running:

parallel --citation

and then typing

will cite

or simply

touch ~/.parallel/will-cite

Offline

Board footer

Powered by FluxBB