Post your handy self made command line utilities

klixon · 2008-11-14 23:49:14

brisbin33 wrote:

i'd like to add a piece to prevent from operating on dot-files, but i haven't gotten around to it.

Find man-page wrote:

-name pattern
Base of file name (the path with the leading directories removed) matches shell pattern pattern. The metacharacters (`*', `?', and `[]') do not match a `.' at the start of the base name. To ignore a directory and the files under it, use -prune; see an example in the description of -path.

If I read this correct, you already did

Mashi · 2008-11-15 16:05:05

Here's linkgrep, pass it an URL and it will print all the links contained therein.

eg:
$ linkgrep http://bbs.archlinux.org/viewtopic.php?id=56646&p=4
http://bbs.archlinux.org/style/Archer.css
http://www.archlinux.org
http://www.archlinux.org/download/
etc etc etc

There are probably some edge cases where it falls down, so feel free to fix it up and PM me if you come across any.

#!/usr/bin/env python

import os, re, urllib, urlparse, sys

url = urlparse.urlparse(sys.argv[1])

sitebase = "%s://%s" % (url.scheme, url.hostname)
try:
    path = url.path[:url.path.rindex('/')] + '/'
except IndexError:
    path = '/'

data = urllib.urlopen(url.geturl()).read()
re_url = re.compile('(src|href)=(?P<qo>[\'\"]?)(.+?)((?P=qo)|>|\s)')
matches = re.findall(re_url, data)

urls = []

if not matches:
    sys.exit()

re_scheme = re.compile('\w+?://')
for u in (m[2] for m in matches):
    if not re_scheme.match(u):
        if u.startswith('/'):
            u = sitebase + u
        else:
            u = sitebase + path + u
    if u not in urls:
        urls.append(u.strip("""'"> \t"""))

print "\n".join(urls)

Now somebody will come along and say "wget can do that"

Last edited by Mashi (2008-11-15 16:07:53)

Dieter@be · 2008-11-18 20:13:27

I just wrote this oneliner in 2 minutes or so. Can be improved but worked for me
Use-case: you have several packages who are "too new", eg you've been running testing, then you disabled testing and updated your package cache. You want to get the stable packages back:

while read package version;  do echo "**** Downgrading $package to version $version"; sudo pacman -U /var/cache/pacman/pkg/$package-$version-i686.pkg.tar.gz; done < <(sudo pacman -Su | awk '/local.* is newer than.*/ {print $2,$9}' | sed 's/://' | sed 's/(//' | sed 's/)//')

Daenyth · 2008-11-18 21:29:28

I cleaned up the code a bit:

while read package version; do 
   echo "**** Downgrading $package to version $version" 
   sudo pacman -U /var/cache/pacman/pkg/$package-$version-i686.pkg.tar.gz; done < \
   <(sudo pacman -Su | awk '/local.* is newer than.*/ {print $2,$9}' | sed 's/[:()]//g' )

Note that this version will also remove all :, not sure if that's what you want.

Dieter@be · 2008-11-18 21:31:43

Daenyth wrote:

Note that this version will also remove all :, not sure if that's what you want.

Yep, all :,( and ) can be removed. I really wrote that in a hurry

omgwtfbyobbq · 2008-11-18 22:05:15

Here's a kludge I use at startup and 0800 to ensure the computer always wakes up at a given time.

declare -i final final1

echo 0 > /sys/class/rtc/rtc0/wakealarm
time=28800 # time in seconds after 0000 that the system will wake at
ctime=`date +%H%M`

if [ $ctime -gt 0800 ] ; then
timeg0=`date +%Y-%m-%d -d tomorrow` #get tomorrow's date
timeg1=`date --date=$timeg0 +%s` #tomorrow's date in time since epoch
final=$time+$timeg1 #tomorrow's date in time since epoch at 8am
echo $final > /sys/class/rtc/rtc0/wakealarm #set wakealarm to tomorrow at 8am

elif [ $ctime -lt 0800 ] ; then
timel0=`date +%Y-%m-%d` #today's date
timel1=`date --date=$timel0 +%s` #in seconds since epoch
final1=$time+$timel1 #8am today
echo $final1 > /sys/class/rtc/rtc0/wakealarm #set wakealarm to today at 8am

fi

Last edited by omgwtfbyobbq (2008-11-18 22:11:21)

chilebiker · 2008-11-19 00:39:32

A script to rename/sort pics from a digital camera:

#!/bin/bash

# global variables
Tempfile=/tmp/$$.$RANDOM

# argument check
if [ $# -ne 3 ] ; then
  echo "Usage: `basename $0` [directory] [prefix] [file_extension]"
  echo "       for example: `basename $0` /home/pics/ pic jpg"
  exit 65
fi

# generate file time-sorted file list
# -printf arguments: %T+ : file date and time
#                    %p  : file name
#                    \n  : new line
# sed removes the date/time from the filename, which was needed for sorting
find $1 -maxdepth 1 -type f -printf "%T+%p\n" | grep -i $3 | sort | sed -r 's:(.+)/(.+):\2:' > $Tempfile

# replace . with absolute directory
if [ "$1" == . ] ; then
  dir=`pwd`/
else
  dir=$1
fi

# rename files {prefix}{counter}.{extension} with counter starting from 0001
i=1
while read filename
do
#  echo $filename
  if [ "$i" -lt "10" ] ; then
    mv -v "$dir$filename" "$dir${2}000$i.$3"
#    mv -v "$1$filename" "${1}${2}000${i}.${3}"
  elif [ "$i" -lt "100" ] ; then
    mv -v "$dir$filename" "$dir${2}00$i.$3"
  elif [ "$i" -lt "1000" ] ; then
    mv -v "$dir$filename" "$dir${2}0$i.$3"
  fi
  (( i++ ))
done < $Tempfile

# done
rm $Tempfile
exit 0

Daenyth · 2008-11-19 03:05:26

I meant to make a new post but hit edit instead of quote. In any case, my updated np spam is here:

http://bbs.archlinux.org/viewtopic.php? … 44#p431744

.Maleficus. · 2008-11-19 11:53:26

Hello all. This is my first post here, and I have to say, you guys have a very nice forum! Very nice OS too . Anyways, to contribute, here's one I started messing with last night. Just playing around with the last.fm API:

#!/usr/bin/env python

# last-fm-recs - A simple Python script to grab recommended
# artists from last.fm, put them in a dictionary based on
# number of times recommended and the average match of each
# artist.  You know, to see what might be relevant to my
# interests.

from pylast import *
import os, sys

# Some necessary vars, such as API key and secret --! DO
# NOT EDIT !--
API_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
# SESSION_KEY is unique to each last.fm user, and thus this
# key only works for my account.
SESSION_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxx'

# List of artists that should be searched, separate by comma
# and include single quotes.
ARTISTS = ['DragonForce', 'Dream Theater', 'Pain of Salvation']

# Loop through each artist specified, find similar artists
# and store them.
for x in ARTISTS:
    print 'Recommendations based on ' + '\x1b[1;31m' + x + '\x1b[0m'
    # Create new instance of artist with 'x' as artist
    # name
    artist = Artist(x, API_KEY, SECRET, SESSION_KEY)
    # New list to hold recommendations
    recs = artist.getSimilar(limit = 10)
    # Print names of recommendations
    for rec in recs:
        print rec.getName()

Basically it just searches for artists based on those that I specify and prints them off. It doesn't have a whole lot of functionality yet, but I've got some updates in the works. You'll notice that the script doesn't do what the first comment says it does; it's still in it's infant stages .

Last edited by .Maleficus. (2008-11-19 12:09:39)

Daenyth · 2008-11-19 12:05:27

Not totally sure on this, but maybe you shouldn't post your "secret" stuff here?

.Maleficus. · 2008-11-19 12:11:41

Daenyth wrote:

Not totally sure on this, but maybe you shouldn't post your "secret" stuff here?

I guess that's what happens when you copy/paste... .

Last edited by .Maleficus. (2008-11-19 12:13:49)

Daenyth · 2008-11-20 13:24:56

find . -type l -printf '%h/%f %l\n' | while read file link; do if [ ! -e "$link" ]; then ls -ld $file; fi;done

This will find broken symlinks in the current directory. Replace '.' with any path you like (/usr/lib is handy). This will almost certainly break for files with spaces in the name.

finferflu · 2008-11-20 13:42:38

I used to have this script running when listening to music in bed, before getting asleep:

#!/bin/bash

while true; do

        lines=`mpc | wc -l`

        sleep 120;
        if [ "$lines" = "1" ]; then
                sudo halt
        fi
done

So when the playlist is through, the machine shuts down (I have passwordless sudo).

Ah! and let's not forget my Jamendo CLI browser

_AA_ · 2008-11-22 09:57:32

Another backup script for nightly backups. Creates a single compressed archive, encrypts (pgp) and uploads to remote FTP server.
It's not all mine, I found part of it online somewhere and modified it to suite my needs.

#!/bin/bash
# Nightly backup script.

# MySQL backup script http://sourceforge.net/projects/automysqlbackup/
`/root/scripts/backups/mysql-backup.sh`

PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
MAILTO=aa@uplinkzero.nospam.com

# Includes
DATE=`date +%F` ;
TIME=`date +%H-%M` ;
TEMPDIR="/tmp/bakup-$DATE-$TIME" ;
BACKUPDIR="/backup" ;
FTPSVR='4.2.2.2'
FTPUSER='username'
FTPPASS='password'
FTPPATH='/'
PGPSTRING='a_64_char_random_string_in_here'


# Change Dir
cd $BACKUPDIR ;

mkdir $TEMPDIR

# Directories to backup. Separate with a space. Exclude trailing slash!
SOURCES="/export /etc /root /backup/mysql/latest"

# Directory to backup to. This is where your backup(s) will be stored.
# Exclude trailing slash!
TARGET="$TEMPDIR"

# Your EXCLUDE_FILE tells rsync what NOT to backup. Leave it unchanged if you want
# to backup all files in your SOURCES. If performing a FULL SYSTEM BACKUP, ie.
# Your SOURCES is set to "/", you will need to make use of EXCLUDE_FILE.
# The file should contain directories and filenames, one per line.
# An example of a EXCLUDE_FILE would be:
# /proc/
# /tmp/
# /mnt/
# *.SOME_KIND_OF_FILE

# EXCLUDE_FILE="~/scripts/cronjobs/backup/exclude.txt"
EXCLUDE_FILE=""

# Comment out the following line to disable verbose output
# VERBOSE="-v"
###########################

# If $TEMPDIR is not executable then quit
if [ ! -x $TEMPDIR ]; then
  echo "Backup target does not exist or you don't have permission!"
  echo "Exiting..."
  exit 2
fi

# If source dirs are not executable then quit
echo "Verifying Sources..." 
for source in $SOURCES; do
        echo "Checking $source..."
        if [ ! -x $source ]; then
     echo "Error with $source!"
     echo "Directory either does not exist, or you do not have proper permissions."
     exit 2
   fi
done

# Paths to exclude (1 per line)
if [ -f $EXCLUDE_FILE ]; then
  EXCLUDE="--exclude-from=$EXCLUDE_FILE"
fi

echo "Sources verified. Running rsync..."
for source in $SOURCES; do

  # Create directories in $TEMPDIR to mimick source directory hiearchy 
  if [ ! -d $TEMPDIR/$source ]; then
    mkdir -p $TEMPDIR/$source
  fi

  # Rsync the source paths to our mimick path
  rsync $VERBOSE --exclude=$TEMPDIR/ $EXCLUDE -a --delete $source/ $TEMPDIR/$source/

done


### At this point we have duplicated all the files we want to back up.
### all files to be backed up are in $TEMPDIR


# Create single compressed archive in our nightly backup dir
cd $TEMPDIR ;
tar czf $BACKUPDIR/backup.$DATE.$TIME.tar.gz $TEMPDIR ;

# Remove temp directory
rm -Rf $TEMPDIR ;

# Change permissions on single backup archive
chmod -f 600 $BACKUPDIR/backup.$DATE.$TIME.tar.gz ;

# Encrypt backup archive for offsite storage
gpg -c --passphrase $PGPSTRING $BACKUPDIR/backup.$DATE.$TIME.tar.gz ;

# Delete unencrypted archive
rm -f $BACKUPDIR/backup.$DATE.$TIME.tar.gz ;

#ncftpput -u gleason -p my.password Elwood.probe.net /home/gleason stuff.txt
ncftpput -u $FTPUSER -p $FTPPASS $FTPSVR $FTPPATH $BACKUPDIR/backup.$DATE.$TIME.tar.gz.gpg ;

_AA_ · 2008-11-22 10:03:38

And another one, used to check uptime on servers and to then email me if uptime is less than 1 hour. Runs from a 30 min cronjob. requires mailx.

#!/bin/bash
#
# Check uptime and mail if uptime is less than 1 hour. 
#

UPTIME=`gawk -F . '{ print $1 }' /proc/uptime`

MINS=$(($UPTIME/60))
HOURS=$(($MINS/60))
DAYS=$(($HOURS/24))

# Unhash below line for testing.
#HOURS='0'

if [ "$HOURS" -lt '1' ] ;
then
    uptime | mail -s "`hostname` has been rebooted" email@uplinkzero.nospam.com ;
fi

Astrodomine · 2008-11-23 22:56:33

I use it with dwm, binded it to the alt+- and alt+= keys. I use it to control my main volume with the keyboard. There is surely a better and simple way to do it but I wanted to learn Python

#! /usr/bin/env python
# Volume manager

import os.path
import os
import sys

# Print the usage and exits
def printUsage():
    print "Usage: vol-manager [-o] [<operation>] [-i] [-u]"
    print "\tOperation:\tincrease\t(Increases the volume by 10)"
    print "\t\t\tdecrease\t(Decreases the volume by 10)"
    print "\ti: Print the volume and exit"
    print "\tu: Update the volume and exit"
    sys.exit()
    return

# Set the system volume
def setVolume(volume):
    command = "aumix -v" + str(volume)
    os.popen(command)
    return

# Check if the storage file exists, if not creates it with default value 70
# Returns the filename
def checkFile(desiredFilename):
    filename = os.environ['HOME'] + "/" + desiredFilename

    if not os.path.exists(filename):
        file = open(filename, 'w')
        file.write('70')
        file.close()
    return filename

# Get the stored volume
# Returns the stored volume
def getStoredVolume(filename):
    file = open(filename, "r")
    volume = int(file.read())
    file.close()
    return volume

# Updates the system volume
def updateVolume(filename):
    file = open(filename, "r")
    volume = int(file.read())
    file.close()
    setVolume(volume)
    return


#
#Main program
#

#Global variables
incrementor = 5
minimum = 0
maximum = 100

if len(sys.argv) < 2:
    printUsage()

filename = checkFile(".vol-manager")

if sys.argv[1] == "-i":
        print getStoredVolume(filename)
    sys.exit()

if sys.argv[1] == "-u":
    updateVolume(filename)
    sys.exit()

if sys.argv[1] != "-o":
    printUsage()


operation = sys.argv[2]

volume = getStoredVolume(filename)

if operation == "increase" and volume + incrementor <= maximum:
    volume += incrementor
elif operation == "decrease" and volume - incrementor >= minimum:
    volume -= incrementor

setVolume(volume)

file = open(filename, "w")
file.write(str(volume));
file.close()

Dieter@be · 2008-11-26 10:45:46

Dieter@be wrote:

I just wrote this oneliner in 2 minutes or so. Can be improved but worked for me
Use-case: you have several packages who are "too new", eg you've been running testing, then you disabled testing and updated your package cache. You want to get the stable packages back:
while read package version;  do echo "**** Downgrading $package to version $version"; sudo pacman -U /var/cache/pacman/pkg/$package-$version-i686.pkg.tar.gz; done < <(sudo pacman -Su | awk '/local.* is newer than.*/ {print $2,$9}' | sed 's/://' | sed 's/(//' | sed 's/)//')

Errm... scratch this

I just found this in man yaourt:

DOWNGRADE OPTIONS
-Su --downgrade
reinstall all packages which are marked as "newer than extra or core" in -Su output (this is specially for users who experience problems with [testing] and want to revert back to current)

Heller_Barde · 2008-11-30 12:02:11

I have a script for changing the wallpaper randomly.. its a little more sophisticated than the one already posted, and it requires you to be a little more disciplined with the names for your wallpaper. it uses nitrogen or feh, just change the "bgset" variable.

my naming scheme works like this:

format.style.name.of.wallpaper.suffix

the format part can be "center" or "scale"
the style part is not yet used in this script but i plan to add support (and i think it's useful to categorize your wallpapers somehow)

here goes nothing:

#!/bin/sh
picture="`ls /path/to/your/wallpapers | sort -R | head -1`"
format="`echo $picture | awk -F \. '{ print $1 }'`"
bgset="nitrogen"               #nitrogen or feh
if [ $format = 'center' ] ; 
then 
    if [ $bgset = 'nitrogen' ]
    then
        format='--set-centered'
    else 
        format='--bg-center'
    fi
else 
    if [ $format = 'scale' ]
    then
        if [ $bgset = 'nitrogen' ]
        then
            format='--set-scaled'
        else 
            format='--bg-scale'
        fi
    else 
        if [ $bgset = 'nitrogen' ]
        then
            format='--set-best'
        else
            format='--bg-seamless'
        fi
    fi
fi
$bgset $format /path/to/your/wallpapers/$picture

cheers Barde

Last edited by Heller_Barde (2008-11-30 12:02:49)

rson451 · 2008-12-02 16:23:25

I often find myself running an older kernel version than what I have installed on my server, simply because I cannot reboot my server on a whim. I whipped this up this morning. It tests whether you have upgraded the kernel since the last reboot.

#!/bin/bash
#
# Check whether a kernel update has been done since the last reboot
# if so, exit with a message and exit status of 1
# else exit with exit status 0

KERNELPKG="kernel26"

last_reboot=$(date -d "$(cat /proc/uptime | awk '{ print $1 }') seconds ago" +"%s")
kernel_installed=$(date +"%s" --date "$(tac /var/log/pacman.log | sed -r "s/\[(.*)\].*$(echo $KERNELPKG).*/\1/;tx;d;:x;q")")

diff=$(( $kernel_installed - $last_reboot ))

if [[ $diff > 0 ]]
then
  echo "A new kernel has been installed since the last reboot.  Please schedule a reboot."
  exit 1
else
  exit 0
fi

Last edited by rson451 (2008-12-02 16:26:47)

Daenyth · 2008-12-02 17:23:39

Heller_Barde wrote:

picture="`ls /path/to/your/wallpapers | sort -R | head -1`"

You need to use LC_ALL=C sort -R to get random sorting. Read the manpage.

Procyon · 2008-12-02 17:34:15

Daenyth wrote:

You need to use LC_ALL=C sort -R to get random sorting. Read the manpage.

I don't think that influences the -R switch.

brisbin33 · 2008-12-02 20:39:04

@ Procyon i just executed ls /path/to/wallpapers | sort -R | head -1 to test; i get a random image just as promised

here's a backup script i use weekly.

i figure if anything happens i can use the pacman log to script some auto reinstall of everything i have now, then restore /etc and /home and i'm pretty close to my working system

#!/bin/bash
#
# script for backing up home/etc and list installed packages as of today
# run as root (fcron) to capture all files

time=`date +%m.%d.%y`
dir="/path/to/backup/dir/"

# remove/replace old package list
rm ""$dir"paclog_"*
pacman -Q > ""$dir"paclog_"$time""

# backup home/etc
tar cvpzf "$dir"home.tgz /home/USER/
tar cvpzf "$dir"etc.tgz /etc

echo "back up completed successfully - ["$time"]"

rson451 · 2008-12-02 20:50:21

You guys really should start contributing to the script wiki. It could be a great resource in the long run. http://scriptwiki.twilightlair.net/index.php/Main_Page

Daenyth · 2008-12-02 23:48:38

Procyon wrote:

Daenyth wrote:
You need to use LC_ALL=C sort -R to get random sorting. Read the manpage.
I don't think that influences the -R switch.

Huh... I know that at one point it was true. I guess they fixed the related bug. Sweet!

haxit · 2008-12-02 23:52:35

Whiped this up quickly, to connect to archlinux irc server with ease:

#!/bin/bash
# Quick n' Dirty IRSSI Connect Script Script
clear
echo -n "Would you like to load defualt settings or new settings? [D/n]: "
read ANSWER

if [ "$ANSWER" = "n" ]; then
    echo -n "Please enter the server you would like to connect to $USER: "
    read SERVER
    echo -n "Do you need a password?: [y/n]: " 
    read AUTH

    if [ "$AUTH" = "y" ]; then
        irssi -c $SERVER -w $PASS
    else
        irssi -c $SERVER
    fi

else
    irssi -c irc.freenode.net -w enteryourpassword
fi

Arch Linux

#76 2008-11-14 23:49:14

Re: Post your handy self made command line utilities

#77 2008-11-15 16:05:05

Re: Post your handy self made command line utilities

#78 2008-11-18 20:13:27

Re: Post your handy self made command line utilities

#79 2008-11-18 21:29:28

Re: Post your handy self made command line utilities

#80 2008-11-18 21:31:43

Re: Post your handy self made command line utilities

#81 2008-11-18 22:05:15

Re: Post your handy self made command line utilities

#82 2008-11-19 00:39:32

Re: Post your handy self made command line utilities

#83 2008-11-19 03:05:26

Re: Post your handy self made command line utilities

#84 2008-11-19 11:53:26

Re: Post your handy self made command line utilities

#85 2008-11-19 12:05:27

Re: Post your handy self made command line utilities

#86 2008-11-19 12:11:41

Re: Post your handy self made command line utilities

#87 2008-11-20 13:24:56

Re: Post your handy self made command line utilities

#88 2008-11-20 13:42:38

Re: Post your handy self made command line utilities

#89 2008-11-22 09:57:32

Re: Post your handy self made command line utilities

#90 2008-11-22 10:03:38

Re: Post your handy self made command line utilities

#91 2008-11-23 22:56:33

Re: Post your handy self made command line utilities

#92 2008-11-26 10:45:46

Re: Post your handy self made command line utilities

#93 2008-11-30 12:02:11

Re: Post your handy self made command line utilities

#94 2008-12-02 16:23:25

Re: Post your handy self made command line utilities

#95 2008-12-02 17:23:39

Re: Post your handy self made command line utilities

#96 2008-12-02 17:34:15

Re: Post your handy self made command line utilities

#97 2008-12-02 20:39:04

Re: Post your handy self made command line utilities

#98 2008-12-02 20:50:21

Re: Post your handy self made command line utilities

#99 2008-12-02 23:48:38

Re: Post your handy self made command line utilities

#100 2008-12-02 23:52:35

Re: Post your handy self made command line utilities

Board footer