Post your handy self made command line utilities

lolilolicon · 2009-11-05 10:48:20

@mikesd
nice one.
i'd recommend using X SECONDARY selection:

alias pud='pwd | xsel -s'
alias pod='cd "$(xsel -so)"'

since it is almost never used.
also note the double quotes in case path contains white spaces.

Last edited by lolilolicon (2009-11-05 10:51:21)

mikesd · 2009-11-05 10:53:08

lolilolicon wrote:

@mikesd
nice one.
i'd recommend using X SECONDARY selection:
alias pud='pwd | xsel -s'
alias pod='cd "$(xsel -so)"'
since it is almost never used.
also note the double quotes in case path contains white spaces.

Ah, nice. No file needed. Thanks.

Daenyth · 2009-11-05 13:31:12

Or you could use, you know, pushd and popd.

"cd -" also changes to $OLDPWD. It's useful if you didn't push first.

Last edited by Daenyth (2009-11-05 13:32:13)

lolilolicon · 2009-11-05 17:02:05

So I've just came up with a function, which tests if a string contains unicode characters.

has_unicode () {
    local string="$1" num_chars num_bytes
    num_chars=$(echo -n "${string}" | wc -m)
    num_bytes=$(echo -n "${string}" | wc -c)
    [ "${num_chars}" != "${num_bytes}" ]
}
has_unicode "$1" && echo 'String contains UNICODE characters' || echo 'ANSI only string'

it works by far (and it came to my mind only 52.6 seconds ago, so i'm asking for correction/confirmation from you guys)

Daenyth · 2009-11-05 17:09:29

How does it work when you have things like \t or \n in the string?

lolilolicon · 2009-11-05 17:13:08

\t and \n are both counted as '1' both by `wc -c` and `wc -m`, i think it's no problem.

Daenyth · 2009-11-05 17:19:46

Cool! I think a great extension would be to test the bytestring using several encodings, and then try to figure out the intended encoding. You could look for common constructs that mis-encoding gives. There's no foolproof way to do it, and there's not a *huge* need for it, but I could imagine it being useful.

lolilolicon · 2009-11-05 17:34:47

Sounds like something enca does (and sometimes it does poorly...)
i've always wanted to find something that will automatically figure out the intended encoding of any string, and optionally converts it to utf8.
seems iconv is missing a brother called iguess, huh?

*...has no clue how to do such thing...*

Procyon · 2009-11-05 17:59:52

By the way, wc could use a wcswidth() wrapper for testing for multi-column characters too. And you're right lolilolicon, I remember when I had to find the encoding of some id3tags and I had it run through everything in iconv -l (and firefox is better at decoding than iconv by the way) so you still had to go through several pages of garbage. enca is completely useless for this.

mikesd · 2009-11-05 20:27:03

Daenyth wrote:

Or you could use, you know, pushd and popd.

pushd and popd only seem to work in the terminal you are using. I wanted something that would work globally, ie, push in one term and pop in another.

Daenyth · 2009-11-05 21:25:38

Ah, I suppose.

lolilolicon · 2009-11-06 04:53:39

Procyon wrote:

By the way, wc could use a wcswidth() wrapper for testing for multi-column characters too.

i'm not sure I understand what you were talking about...
i searched for wcswidth(), and this is what i've achieved by far:

#include <stdio.h>
#include <wchar.h>

void main(int argc, char *argv[])
{
  wchar_t *ws = L"test string";
  int count;
  count = wcswidth(ws,100);
  printf("%d\n",count);
}

i'd like to change L"test string" to argv[1]. how do I do it?

Also, Procyon, could you post an example presenting what you were saying? i don't think i get it

Last edited by lolilolicon (2009-11-06 05:41:19)

Procyon · 2009-11-06 08:52:14

There's a man page. The function tells you that for instance ア takes 2 columns, while it's 3 bytes and 1 character. It's useful for ncurses, but for ncurses you can use large pads with only a portion of it displayed.

For the iconv -l thing here is an example:

string=$(curl -s 'http://www.itit.co.jp/' | grep 'ebisuya.gif' | sed 's/.*alt=//'); for ENCODING in $(iconv -l | sed 's/..$//'); do echo -n $ENCODING :: ; echo "$string" | iconv -f $ENCODING -t utf8 2>/dev/null; done | less

(that wasn't easy getting that string... just took the first site I saw, but grep/sed won't work at all with these sjis strings.)
The kind of output you have to go through:

RK1048 ::"ѓGѓrѓXѓ„"
ROMAN8 ::"<U+0083>G<U+0083>r<U+0083>X<U+0083><U+0084>"
ROMAN9 ::"<U+0083>G<U+0083>r<U+0083>X<U+0083><U+0084>"
RUSCII ::"ГGГrГXГД"
SE ::"SE2 ::"SEN_850200_B ::"SEN_850200_C ::"SHIFT-JIS ::"エビスヤ"
SHIFT_JIS ::"エビスヤ"
SHIFT_JISX0213 ::"エビスヤ"
SJIS-OPEN ::"エビスヤ"
SJIS-WIN ::"エビスヤ"
SJIS ::"エビスヤ"
SS636127 ::"STRK1048-2002 ::"ѓGѓrѓXѓ„"
ST_SEV_358-88 ::"<U+0083>G<U+0083>r<U+0083>X<U+0083><U+0084>"
T.61-8BIT ::"<U+0083>G<U+0083>r<U+0083>X<U+0083><U+0084>"
T.61 ::"<U+0083>G<U+0083>r<U+0083>X<U+0083><U+0084>"
T.618BIT ::"<U+0083>G<U+0083>r<U+0083>X<U+0083><U+0084>"

nh · 2009-11-07 22:50:39

This is in my .bashrc to give a sorted summary of directory sizes within a directory:

du1 () { du -h --max-depth=1 "$@" | sort -k 1,1hr -k 2,2f; }

And this uses an ed script to do a global search and replace in place (i.e., retaining the inode and permissions) across many files. If environment variable REPLBACKUP exists, the function backs up all changed files and tells you which ones it has updated. The grep is there to filter out "No match" and "Newline appended" ed "errors", while reporting any unexpected (and possibly genuine) errors.

die () {
    declare -i r=${2:-9}
    echo "$1" 1>&2
    return $r
}

repl () {
    (( $# >= 3 )) || return $(die "Usage: repl from-string to-string file...")
    declare a="$1" b="$2" t e
    while (( $# >= 3 )); do
        [[ -f "$3" ]] || return $(die "$0: $FUNCNAME: Regular file $3 not found")
        [[ -r "$3" && -w "$3" ]] || return $(die "$0: $FUNCNAME: Need read and write access to file $3")
        if [[ -n "$REPLBACKUP" ]]; then
            { t="$REPLBACKUP/$3_$(date +'%F_%T')_$RANDOM" && cp --preserve=all -i -- "$3" "$t"; } || return $(die "$0: $FUNCNAME: Cannot create backup file $t")
        fi
        e=$({ ed -s -- "$3" <<-EOF
            H
            ,s/$a/$b/g
            w
        EOF
        } 2>&1 | gawk '!(/^\?$/ || /No match$/ || /Newline appended$/)')
        [[ -n "$e" ]] && return $(die "$0: $FUNCNAME: Error in ed while updating $3: $e")
        if [[ -n "$REPLBACKUP" ]]; then
            if cmp -s -- "$3" "$t"; then
                rm "$t"
            else
                echo "$3 updated"
            fi
        fi
        shift
    done
}

crouse · 2009-11-08 02:40:04

steve___ wrote:

crouse wrote:
I normally use RCS to edit most files, but sometimes I just want a quick backup of a file.
This is in my .bashrc file.
bu () { cp $1 ~/Backups/${1}-`date +%Y%m%d%H%M`.backup ; }
Called with "bu somefile" it puts a timestamped copy of whatever file in my Backups directory.
I think it is best to quote your variable here. Or else it will break if the filename has a space.
bu () { cp "$1" ~/Backups/"${1}"-`date +%Y%m%d%H%M`.backup ; }

True..... BUT I never need to deal with files with spaces in them.... I live in a UNIX world
IE: It's my system, and NONE of my files have spaces in them....

but yes.....if you MUST <yuck>use filenames with spaces</yuck>, then quoting the $1 will allow you to backup files with spaces lol.

bu () { cp "$1" ~/Backups/"$1"-`date +%Y%m%d%H%M`.backup ; }

lolilolicon · 2009-11-10 04:53:07

First cd into a clean directory. Then run this:

#!/bin/bash
# Downloads "Magic Eye Image of The Week"
day=$(date -d'20091029' +%u) #4
today=$(date +%u)
back=$(( ($today - $day + 7) % 7 ))

# Seems it started July 11th, 2002
until [ "$date" == "071102" ] ; do
    date=$(date -d"-$back day" +%m%d%y) ; [ -f "${date}s.jpg" ] && break
    wget "http://www.magiceye.com/gallery/images/${date}s.jpg"
    ((back+=7))
done

I got 368 Magic Eye images. And they are a total of 24MB. Next time I run it, it just looks for updates.

I hope someday I'll come up with a imagemagick command that actually creates Magic Eye images.

Last edited by lolilolicon (2009-11-10 05:06:03)

mjunx · 2009-11-12 07:24:58

Going back to the whole YouTube audio extractor thing, if the .flv already uses MP3 for audio (which it usually does), you can simply use `mplayer -dumpaudio -dumpfile foo.mp3 foo.flv`.

And to the guy who wrote the sh function: bad name (sh is the shell normally), and you could simply use git or bzr or anything like that instead.

steve___ · 2009-11-12 15:20:45

youtube's High Quality videos do not use mp3. It's aac and I use faad to convert.

Procyon · 2009-11-12 15:39:54

Procyon wrote:

I recently broke the default playlist size limit of mpd (16000?), and thought something was wrong with python-mpd or my netcat script, so thinking of the other mpd.db scripts in this thread I wrote a script to sort the database to an m3u playlist.
awk 'BEGIN {RS="key:"; FS="\n"} { i=0; while (i++<=NF) { if ( $i ~ /^file:/ ) { file=substr($i,7); }; if ($i ~ /^Album:/) {album=substr($i,8) }; if ( $i ~ /^Artist:/) {artist=substr($i,9) }; if ( $i ~ /^Track:/) { track=substr($i,8) }; if ( $i ~ /^Title:/) { title=substr($i,8) }; }; if ( file != "" ) { print track"\t"album"\t"file"\t"title }; };' ~/.mpd/mpd.db | sort -f -k3,3 | sort -f -s -k4,4 | sort -n -f -s -k1,1 | sort -f -s -k2,2 | cut -f 3 > $(date '+%Y-%M-%d').m3u
I don't sort by artist, but it should be easy to add. The sort pipes go from least significant to most (this works thanks to sort -s). In this case: [filename, title], track, album. The first field must be numeric (track) for sort to work numerically (i.e. you can't sort numerically on fields above 1 due to a coreutils bug). If you add an artist field note that the filename field repeats in cut -f 3.

I just noticed that sort is not working very well with unicode. For instance UTF8 album names that differ in an ASCII number will be brought together and for some reason my tracks were unsorted near the end of the playlist. I have a python one in pyampc, but with a really large database it will complain about lists/dicts being too large (did not look at the error carefully though), so I may rewrite this in another language.

mjunx · 2009-11-12 16:02:04

steve___ wrote:

youtube's High Quality videos do not use mp3. It's aac and I use faad to convert.

Well, either way, mplayer's -dumpaudio option will dump the audio stream in a separate file. Many MP3 players can do AAC as well, so no need for a loss of quality.

Daenyth · 2009-11-12 16:27:46

Maybe dump and then use "file" to figure out if it should be converted?

colbert · 2009-11-12 20:19:30

I've been using this for ages in my .bash_functions to make mp3's of YT vids:

# Convert Flash video to MP3
flvmp3() {
  ffmpeg -i $1 -ar 44100 -ab 192k -ac 2 $2
}

simongmzlj · 2009-11-12 20:55:27

Command line power management:

if [[ -n $1 ]]
then
    case $1 in
        shutdown)
            dbus-send --system --print-reply --dest=org.freedesktop.Hal \
                /org/freedesktop/Hal/devices/computer \
                org.freedesktop.Hal.Device.SystemPowerManagement.Shutdown > /dev/null
            ;;
        restart)
            dbus-send --system --print-reply --dest=org.freedesktop.Hal \
                /org/freedesktop/Hal/devices/computer \
                org.freedesktop.Hal.Device.SystemPowerManagement.Reboot > /dev/null
            ;;
        suspend)
            dbus-send --system --print-reply --dest=org.freedesktop.Hal \
                /org/freedesktop/Hal/devices/computer \
                org.freedesktop.Hal.Device.SystemPowerManagement.Suspend \
                int32:0 > /dev/null
            ;;
        hibernate)
            dbus-send --system --print-reply --dest=org.freedesktop.Hal \
                /org/freedesktop/Hal/devices/computer \
                org.freedesktop.Hal.Device.SystemPowerManagement.Hibernate > /dev/null
            ;;
        *)
            echo "Operation not supported"
    esac
else
    echo "Operation not supported"
fi

nh · 2009-11-14 01:46:44

This is handy for viewing information about all disk partitions:

alias partitions='blkid /dev/sd?? /dev/sd??? | sort -V'

Could also be /dev/hd?? /dev/hd?? , of course.
blkid /dev/sd* also works fine; blkid seems to silently ignore disks, such as /dev/sda.
blkid doesn't list USB stick partitions, at least on my system.

Procyon · 2009-11-14 11:43:15

One more try for the mpd playlist sorting. I wrote a python replacement of sort, called colsort.py

#! /usr/bin/python
#colsort.py COL [-n] [-r]
#COL is which col to sort on
#-n is numeric. Takes the first positive integer found
#-r is reverse
import sys, re
from operator import itemgetter

def firstint(data):
    data=re.sub("^[^0-9]*","",data)
    data=re.sub("[^0-9].*$","",data)
    try:
        data=int(data)
    except:
        data=0
    return data
        
numeric=False
reverse=False
try:
    col=int(sys.argv[1])-1
except:
    sys.stderr.write("Specify what column to sort on.\n")
    sys.exit(1)
if "-n" in sys.argv:
    numeric=True
if "-r" in sys.argv:
    reverse=True

input=sys.stdin.readlines()
for nr in xrange(len(input)):
    input[nr]=input[nr][:-1].split('\t')
    if nr == 0:
        nrcols=len(input[nr])
    else:
        if len(input[nr]) != nrcols:
            sys.stderr.write("Unmatched amount of columns on line "+str(nr+1)+". Data: "+repr(input[nr])+"\n")
            sys.exit(1)
if col+1 > nrcols:
    sys.stderr.write("Can't sort on column "+str(col+1)+". Not enough columns in input. Data: "+repr(input[0])+"\n")
    sys.exit(1)
if numeric:
    input.sort(lambda a,b: cmp(firstint(a.split("/")[0]), firstint(b.split("/")[0])),key=itemgetter(col),reverse=reverse)
else:
    input.sort(key=itemgetter(col),reverse=reverse)

for line in input:
    print "\t".join(line)

The mpd.db script:

awk 'BEGIN {RS="key:"; FS="\n"}
{ i=0
  file="";album="";artist="";track="";title="";disc=""
  while (i++<=NF) { 
   if ( $i ~ /^file:/ ) { file=substr($i,7) }
   if ( $i ~ /^Album:/ ) { album=substr($i,8) }
   if ( $i ~ /^Artist:/ ) { artist=substr($i,9) }
   if ( $i ~ /^Track:/ ) { track=substr($i,8) }
   if ( $i ~ /^Title:/ ) { title=substr($i,8) }
   if ( $i ~ /^Disc:/ ) { disc=substr($i,8) }
 }
 if (file != "") { print file"\t"title"\t"track"\t"disc"\t"album }
}' ~/.mpd/mpd.db | 
colsort.py 1 | 
colsort.py 2 | 
colsort.py 3 -n | 
colsort.py 4 | 
colsort.py 5 |
cut -f 1 > $(date '+%Y-%m-%d').m3u

EDIT: added disc sorting, before album.
EDIT: fixed disc's substr argument.
EDIT: reset variables so if something is missing it isn't influenced by the previous track
EDIT: made numeric colsort more lenient, it takes the first int or else 0.

Last edited by Procyon (2009-11-23 08:54:27)

Arch Linux

#626 2009-11-05 10:48:20

Re: Post your handy self made command line utilities

#627 2009-11-05 10:53:08

Re: Post your handy self made command line utilities

#628 2009-11-05 13:31:12

Re: Post your handy self made command line utilities

#629 2009-11-05 17:02:05

Re: Post your handy self made command line utilities

#630 2009-11-05 17:09:29

Re: Post your handy self made command line utilities

#631 2009-11-05 17:13:08

Re: Post your handy self made command line utilities

#632 2009-11-05 17:19:46

Re: Post your handy self made command line utilities

#633 2009-11-05 17:34:47

Re: Post your handy self made command line utilities

#634 2009-11-05 17:59:52

Re: Post your handy self made command line utilities

#635 2009-11-05 20:27:03

Re: Post your handy self made command line utilities

#636 2009-11-05 21:25:38

Re: Post your handy self made command line utilities

#637 2009-11-06 04:53:39

Re: Post your handy self made command line utilities

#638 2009-11-06 08:52:14

Re: Post your handy self made command line utilities

#639 2009-11-07 22:50:39

Re: Post your handy self made command line utilities

#640 2009-11-08 02:40:04

Re: Post your handy self made command line utilities

#641 2009-11-10 04:53:07

Re: Post your handy self made command line utilities

#642 2009-11-12 07:24:58

Re: Post your handy self made command line utilities

#643 2009-11-12 15:20:45

Re: Post your handy self made command line utilities

#644 2009-11-12 15:39:54

Re: Post your handy self made command line utilities

#645 2009-11-12 16:02:04

Re: Post your handy self made command line utilities

#646 2009-11-12 16:27:46

Re: Post your handy self made command line utilities

#647 2009-11-12 20:19:30

Re: Post your handy self made command line utilities

#648 2009-11-12 20:55:27

Re: Post your handy self made command line utilities

#649 2009-11-14 01:46:44

Re: Post your handy self made command line utilities

#650 2009-11-14 11:43:15

Re: Post your handy self made command line utilities

Board footer