You are not logged in.

#2776 2016-06-15 22:28:23

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

I tend to write little scripts that run certain parts of utilities and then create an alias so I can run them with a single command.

I'm still learning so nothing like the level of some of the scripts in this thread but someone may find it useful.

This is one that I use to run clamav scans on my home directory and external drives.

#!/bin/bash

# This script will update the clamscan virus definitions,
# scan the following locations recursively: /mnt/ and  ~/
# move any infected files to /home/alarm/clamscan/quarantine/
# and write a log to /home/alarm/clamscan/log

# update clamscan virus definitions:
echo -e "\e[30;48;5;10m***** UPDATING CLAMSCAN DATABASE *****\e[0m"
sudo freshclam
echo ""

# scan home directory recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING HOME DIRECTORY RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ ~/
echo ""

# scan /mnt/ recursively and move files to quarantine:
echo -e "\e[30;48;5;10m***** SCANNING MNT RECURSIVELY *****\e[0m"
sudo clamscan -r -i --log=/home/alarm/clamscan/log --move=/home/alarm/clamscan/quarantine/ /mnt/
echo ""
echo -e "\e[1;10mDone.\e[0m"

Offline

#2777 2016-06-19 11:51:00

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

Another little script that I wrote to carry out some maintenance jobs as outlined in the Arch Wiki.

Fairly simple chain of tests and commands but handy for updating and checking my bare bones install of Arch that I use as a media server.


#!/bin/bash

# Script to update all packages, prune the pacman cache, remove
# unused (orphan) packages, write list of installed packages to file,
# check for broken symlinks, check for failed systemd services and
# write error logs to display.

# Updating packages:
echo -e "\e[30;48;5;10m***** UPDATING PACMAN REPOSITORY AND PACKAGES *****\e[0m"
sudo pacman -Syyu
echo ""

# Pruning the cache:
echo -e "\e[30;48;5;10m***** PRUNING THE PACKAGE CACHE *****\e[0m"
sudo paccache -r
echo ""

# Removing unused packages (orphans)
echo -e "\e[30;48;5;10m***** REMOVING UNUSED PACKAGES (ORPHANS) *****\e[0m"
sudo pacman -Rns $(pacman -Qtdq)
echo ""

# Writing installed package lists to file: ~/Documents/Sysinfo/
echo -e "\e[30;48;5;10m***** WRITING INSTALLED PACKAGE LISTS TO FILE *****\e[0m"
sudo pacman -Qe > ~/Documents/Sysinfo/packagesQe.txt
sudo pacman -Qn > ~/Documents/Sysinfo/packagesQn.txt
echo -e "\e[1;10mDone.\e[0m"
echo ""

# Checking for broken symlinks:
echo -e "\e[30;48;5;10m***** CHECKING FOR BROKEN SYMLINKS *****\e[0m"
find . -type l -! -exec test -e {} \; -print
echo -e "\e[1;10mDone.\e[0m"
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** FAILED ACTIVE SYSTEMD SERVICES *****\e[0m"
systemctl --failed
echo ""

# -systemd --failed:
echo -e "\e[30;48;5;10m***** ALL FAILED SYSTEMD SERVICES *****\e[0m"
systemctl --failed --all
echo ""

# -Logfiles:
echo -e "\e[30;48;5;10m***** LOGFILES *****\e[0m"
journalctl -p 3 -xb
echo ""
echo -e "\e[1;10mDone.\e[0m"

Last edited by nick@archpi2 (2016-06-19 11:54:39)

Offline

#2778 2016-06-23 14:06:16

whitelight999
Banned
Registered: 2016-05-17
Posts: 16

Re: Post your handy self made command line utilities

edit

Last edited by whitelight999 (2016-10-14 23:25:00)

Offline

#2779 2016-06-23 14:22:43

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: Post your handy self made command line utilities

The shebang needs to be on the first line without a space between #! and /bin/bash otherwise it won't be recognised by the system as a bash script.

Running it through https://www.shellcheck.net/ throws up a couple of pointers as well.

Last edited by Slithery (2016-06-23 14:26:09)


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#2780 2016-06-23 15:53:11

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Post your handy self made command line utilities

my backup script, (it is quite basic one)

#!/bin/bash
# use with "su"

rsync -a --delete /home/sharad/ /media/BackUP/home
rsync -a --delete /etc/ /media/BackUP/etc
rsync -a --delete /var/ /media/BackUP/var

## backup list of installed packages ##
## can be reinstalled with pacman -S $(< pkglist.txt) ##

pacman -Qqen > /media/BackUP/pkglist.txt

Offline

#2781 2016-08-09 16:51:09

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: Post your handy self made command line utilities

Often I want to know how long a command takes for simple benchmarking. I know about time but it's a bit of a pain for compound commands and such. Plus TMI. So I would often do something like

start=$(date +%s); <command(s)>; echo $(( $(date +%s) - start ))

But that's a handful, hence timeit:

timeit() {
    if [[ "$1" ]] || [[ ! "$__timeit__" ]]; then
        __timeit__="$(date +%s)"
    else
        echo $(( $(date +%s) - __timeit__ ))
        unset __timeit__
    fi
}

Usage:

$ timeit <any argument>; <command(s)>; timeit

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2782 2016-08-09 18:52:49

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: Post your handy self made command line utilities

I know you said you know about `time`, but have you used it with a subshell?  It can handle everything timeit can (and possibly more), and seems simpler to me:

$ time (<command1>; <command2>; ... <commandN>)

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2783 2016-08-09 20:02:26

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: Post your handy self made command line utilities

I never considered using a subshell, always used eg.

$ time { command; command; command; }

It's probably the case that sometimes forgetting the unusual requirements of {} (; after final command, ' ' before closing }) influenced things.

Plus I like getting the results in seconds without any extra baggage, and that if I was so inclined I could effortlessly integrate speed calculation by passing a size as an argument to the first call. Sure you could do the same by parsing the output of 'time -p' but at that point I think timeit wins for simplicity.


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#2784 2016-08-24 11:38:20

nick@archpi2
Member
Registered: 2016-03-22
Posts: 9

Re: Post your handy self made command line utilities

Handy little loop to batch remux mpeg2.ts files to a more mobile friendly format (mp4) using ffmpeg.

#!/bin/bash
# convert mpeg2-ts files to mp4
cd /mnt/2tb_hd/con/
for a in *.ts ; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done

Offline

#2785 2016-08-25 01:12:08

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Post your handy self made command line utilities

You should use ./* instead of * to avoid issues with names starting in a hyphen (can be interpreted as options to ffmpeg).

Or just drop the cd call, and put the full path in the for loop. Mind you, using positional parameters is more flexible than hardcoding the path.

#!/bin/bash
# convert mpeg2-ts files to mp4
for a; do
	f="${a[@]/%ts/mp4}"
	ffmpeg -i "$a" -acodec copy -vcodec copy "$f"
	sleep 20
done
fooconvert /mnt/2tb_hd/con/*.ts

Last edited by Alad (2016-08-25 01:29:34)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#2786 2016-08-25 01:57:26

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 237

Re: Post your handy self made command line utilities

nick@archpi2 wrote:
sleep 20

By the way, what is the sense of sleep() here?

Last edited by dimich (2016-08-25 02:02:25)

Offline

#2787 2016-08-25 02:01:48

dimich
Member
From: Kharkiv, Ukraine
Registered: 2009-11-03
Posts: 237

Re: Post your handy self made command line utilities

Alad wrote:

using positional parameters is more flexible than hardcoding the path.

"find … -print0 | xargs -0 …" should be better for this purpose

Offline

#2788 2016-08-25 11:48:31

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: Post your handy self made command line utilities

The challenge with using find and xargs - or find with it's exec flag - is that you need to then invoke a separate subshell for every single file just to do the string manipulation to get the new name.  If ffmpeg just acted on each file it'd be ridiculously easy:

find /path/to/files -name '*.ts' -exec ffmpeg '{}' \;

But you need an input and output file name and you can use the bash string manipulation without calling bash - so that becomes

find /path/to/files -name '*.ts' -exec bash -c "f='{}'; ffmpeg -i \"$f\" -acodec copy -vcodec copy \"${f/%ts/mp4}\"" \;

This is ugly, and error prone due to it being easy to get all the nesting and quoting wrong (I'm not even sure if this is quite right) - but more importantly, it - again - invokes a separate shell for every single file, once this is needed, a for loop is more efficient: one shell.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2789 2016-08-25 12:00:45

parchd
Member
Registered: 2014-03-08
Posts: 421

Re: Post your handy self made command line utilities

What about GNU parallel?

parallel ffmpeg -i {} -acodec copy -vcodec copy {.}.mp4 ::: *.ts

Depending on hardware maybe it will speed up the whole thing?

Last edited by parchd (2016-08-25 12:05:53)

Offline

#2790 2016-08-25 12:52:30

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Post your handy self made command line utilities

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Last edited by Alad (2016-08-25 12:52:36)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#2791 2016-08-25 13:04:14

parchd
Member
Registered: 2014-03-08
Posts: 421

Re: Post your handy self made command line utilities

Alad wrote:

You mean the tool which asks you to pay 10.000 EUR to disable it's nagging screen?

cf. https://wiki.archlinux.org/index.php/Co … el_version

Erm, no. I mean the tool which fairly reasonably suggests that if it is a core piece of your scientific breakthrough, you give them the credit due.

The "nagging screen" you mention can be disabled by running:

parallel --citation

and then typing

will cite

or simply

touch ~/.parallel/will-cite

Offline

#2792 2016-08-29 08:36:36

Sachiko
Member
Registered: 2016-07-01
Posts: 17

Re: Post your handy self made command line utilities

This is a non-repeating wallpaper changing script I've been trying to get working for the last 3 days. Originally, I wanted it to echo the file name to a file on a new line every time it was set and if the file had lines equal to the number of files in said directory, it would remove the list file and retouch it so it could be echo'd to again, but that ran into a few snags...so I went with method 2. Move the file after it was set and when the file count got to 1, move the last file and then move all the files back to the first directory.

#!/bin/bash
until [ 1 -eq 2 ]; do

DIR="$HOME/.wallpaper/"
PIC=$(find $DIR -type f | shuf -n1)
WPC=$(find $DIR -type f | wc -l)

if [ "$WPC" = "1" ]; then
	mv $HOME/.wallpaper-used/*.{jpg,png,jpeg} $HOME/.wallpaper/ && nitrogen --set-scaled "$PIC" && mv "$PIC" $HOME/.wallpaper-used && sleep 20s
else
	nitrogen --set-scaled "$PIC" && mv "$PIC" $HOME/.wallpaper-used && sleep 20s
fi
done

Of course, the prerequisites are:

- Nitrogen(I prefer this over feh, but you can switch it to feh if you wish)
- folders in  your users home directory named .wallpaper and .wallpaper-used
- files to populate ~/.wallpaper with(jpg, png, or jpeg formats only unless you add the extension you wish to use to both line with {jpg,png,jpeg} in them.

user-edit: Changed the method of getting the files and the number of files at the urging of many who have pointed me to the BashGuide which says to use find rather than ls so I complied. But of course credit for the switch goes to Alad who first pointed such an egregious error on my part.

Last edited by Sachiko (2016-09-01 14:40:28)

Offline

#2793 2016-08-29 13:31:30

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Post your handy self made command line utilities


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#2794 2016-08-31 00:45:37

whitelight999
Banned
Registered: 2016-05-17
Posts: 16

Re: Post your handy self made command line utilities

edit

Last edited by whitelight999 (2016-10-14 23:23:27)

Offline

#2795 2016-08-31 01:00:00

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: Post your handy self made command line utilities

whitelight, from what I skimmed that all looks well done, but for code mainetenance and readability I think it would benefit quite a bit from pulling some of those bits out into separate functions.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2796 2016-08-31 06:49:13

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Post your handy self made command line utilities

All that echo! Use cat:

cat <<-EOF
line1
line2
line3
EOF

Use printf for the remaining echo invocations:

http://unix.stackexchange.com/a/65819

In particular, put the color escapes into functions like msg().

Also the classic mistakes of missing quotes, using backticks instead of $(), parsing ls, etc. Shellcheck is your friend.

Last edited by Alad (2016-08-31 10:48:48)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#2797 2016-08-31 20:02:02

whitelight999
Banned
Registered: 2016-05-17
Posts: 16

Re: Post your handy self made command line utilities

.

Last edited by whitelight999 (2016-10-14 23:22:37)

Offline

#2798 2016-08-31 20:04:40

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: Post your handy self made command line utilities

It seems I didn't skim that well.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2799 2016-08-31 20:36:20

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: Post your handy self made command line utilities

whitelight999 wrote:

Can you expand on "parsing ls" Whats the mistake with the way I use ls ?

See http://mywiki.wooledge.org/BashGuide/Pr … r_Do_These

The whole BashGuide is must read, though. smile

http://mywiki.wooledge.org/BashGuide


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#2800 2016-09-01 08:54:16

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,389

Re: Post your handy self made command line utilities

Probably there are a bunch of similar scripts, but here is mine.
Check hosts state by pinging them, and notify user (notify-send and kdialog) when they change state:

HOSTS=(testhost1 testhost2 testhost3)
CHECKTIME=60


TMPDIR="/tmp/check_hosts12345"

CheckHost(){
    #Init Host:
        if ! test -e $TMPDIR/$HOST ; then
            echo Init $HOST:
            echo UP>$TMPDIR/$HOST
        fi

    read state < $TMPDIR/$HOST

    if ! ping -c 1 -W 1 $HOST &>/dev/null ; then

        #Host down:
        if [ "$state" = "UP" ] ; then
            state="DOWN"
            NOW=$(date +"%d-%m-%Y %r")
            notify-send -a "Check Host" -t 0 "Check Host:" "$NOW:\n $HOST Down"
            kdialog --sorry "$NOW:\n$HOST Down" --title "Check Host:" &
        fi

            else

        #Host up
        if [ "$state" = "DOWN" ] ; then
            state="UP"
            NOW=$(date +"%d-%m-%Y %r")
            notify-send -a "Check Host" -t 0 "Check Host:" "$NOW:\n $HOST Up"
            kdialog --sorry "$NOW:\n$HOST Up" --title "Check Host:" &
        fi

    fi
    echo $state > $TMPDIR/$HOST
}


#
rm -R $TMPDIR &>/dev/null
mkdir $TMPDIR

while true ; do
    for HOST in "${HOSTS[@]}" ; do
        CheckHost
    done
    sleep $CHECKTIME
done

Last edited by kokoko3k (2016-09-01 08:55:07)


Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

Offline

Board footer

Powered by FluxBB