You are not logged in.

#3251 2019-05-19 05:24:33

pellertson
Member
Registered: 2018-03-16
Posts: 5

Re: Post your handy self made command line utilities

I'm a bit of an odd duck and I use Rob Pike's acme as my text editor/general programming environment.  Recently, I wrote an RSS feed for it by converting all the RSS feeds to JSON feeds and using the following to display a summary of my subscriptions:

#!/bin/sh
# RSS/Atom/JSON Feed reader for acme
# This code is in the public domain
for url in $(cat $URLS)
do
	temp=$(mktemp)
	curl -s $url > $temp
	jq -r '.title' $temp
	echo
	jq -r ' .items | .[] | [.title, .url] | join("\t") ' $temp | head -n 5
	echo
	echo =====
	echo
	rm $temp
done | 9p write acme/new/body

writetoacme JSON-Feeds

"writetoacme" is also a script I wrote to help with writing acme screens.

#!/bin/sh
# Shortcut for common writing acme applications
# USAGE: writetoacme $1
# $1: name of window

last=$(9p ls acme | sort -g | tail -n 1)
echo "name $1" | 9p write acme/$last/ctl
echo -n "clean" | 9p write acme/$last/ctl
echo -n "0,0" | 9p write acme/$last/addr
echo -n "dot=addr" | 9p write acme/$last/ctl
echo -n "show" | 9p write acme/$last/ctl

EDIT: before anyone asks, I simply use a web-service called feed2json to do all the converting for me.  I'll probably write my own program to do that later on.  But this works for now.

Last edited by pellertson (2019-05-19 05:31:47)

Offline

#3252 2019-05-19 12:21:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 21,708
Website

Re: Post your handy self made command line utilities

for url in $(cat $URLS)
do
# ...
done

Save the cats!

while read url
do
# ...
done < $URLS

You also may be interested in python's feedparser which would drastically simplify the whole process.

Last edited by Trilby (2019-05-19 12:21:35)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3253 2019-05-19 18:37:20

escondida
Trusted User (TU)
Registered: 2008-04-03
Posts: 157

Re: Post your handy self made command line utilities

Props for the acme rss script, pellertson! You've inspired me to finish putting together my own version (sans the Web service), something I've been meaning to do for a while.

Offline

#3254 2019-05-20 01:01:14

pellertson
Member
Registered: 2018-03-16
Posts: 5

Re: Post your handy self made command line utilities

Trilby wrote:

You also may be interested in python's feedparser which would drastically simplify the whole process.

That could be useful, but I've found through experience that shell scripts are truly the best way to write to acme screens.  As readable as python is, I cannot go without pipes.  Plus I wanted to learn how to use "jq".  But thanks for the tip on removing cat!

Offline

#3255 2019-05-21 18:36:26

pellertson
Member
Registered: 2018-03-16
Posts: 5

Re: Post your handy self made command line utilities

escondida wrote:

Props for the acme rss script, pellertson! You've inspired me to finish putting together my own version (sans the Web service), something I've been meaning to do for a while.

Thanks!  Can I see your version?

Offline

#3256 2019-06-05 00:39:02

Alad
Wiki Admin/IRC Op/TU
From: Bagelstan
Registered: 2014-05-04
Posts: 1,958
Website

Re: Post your handy self made command line utilities

I was looking to watch websites for changes, but all I found was a paid service. So I decided to host my own solution.

The script is run from cron, and supports three methods of comparison:

1. direct HTML comparison (using curl). This may result in false positives, especially with javascript-heavy sites.
2. text rendering (using w3m). This does not necessarily render all content (again, javascript).
3. full rendering (using cutycapt/qtwebengine). Heavy-handed but reliable approach for most websites. An image overlay is sent as email attachment.

#!/bin/bash
# requires: diff mail file nice imagemagick curl w3m cutycapt xvfb-run
argv0=webwatcher
XDG_CACHE_HOME=${XDG_CACHE_HOME:-$HOME/.cache}

# default arguments
destination=your@face.com
mode=exact

while getopts :d:m: opt; do
    case $opt in
	d) destination=$OPTARG
	   ;;
	m) mode=$OPTARG
	   ;;
	*) echo "usage: ${0##*/} [+-mi} [--] ARGS..."
	   exit 2
    esac
done
shift $(( OPTIND - 1 ))
OPTIND=1

if [[ ! $1 ]]; then
    printf >&2 '%s\n' "$argv0: error: website not specified"
    exit 2
fi

uri=$1
uri_id=$(sha256sum <<< "$1" | cut -d' ' -f1)

case $mode in
    exact)
        tmp_extension=.htm
	# "small changes", may have false positives	
	ww_fetch() { curl -fsS "$1" --output "$2"; }
	;;
    text)
	tmp_extension=.txt
	# text-browser mode, may not render all contents
	ww_fetch() { w3m -dump "$1" > "$2"; }
	;;
    image)
	tmp_extension=.png
	# browser mode, will render full page
	# xvfb-run -a is used to allow multiple processes
	ww_fetch() { nice -n 19 xvfb-run -a --server-args "-screen 0, 1024x768x24" \
			  cutycapt --url="$1" --out="$2"; }
	;;
esac

# xdg base directory
known=$XDG_CACHE_HOME/$argv0/$uri_id$tmp_extension
mkdir -p "$XDG_CACHE_HOME/$argv0"

if [[ -s $known ]]; then
    tmp=$(mktemp --suffix "$tmp_extension")
    ww_fetch "$uri" "$tmp"

    case $mode in
	exact|text)
	    if ! diff -q "$known" "$tmp" >/dev/null; then
		diff -u "$known" "$tmp" | mail -s "$argv0: changes for $1" "$destination"
		mv "$tmp" "$known"
	    else
		rm "$tmp"
	    fi ;;
	image)
	    # when comparing screenshots, metadata has to be taken into account. See:
	    # http://www.libpng.org/pub/png/spec/1.2/PNG-Chunks.html#C.Anc-text (PNG meta-data)
	    # http://www.imagemagick.org/Usage/compare/ ("IM Image Signatures", imagemagick)
	    image_signatures=()
	    while read -r sig; do
		image_signatures+=("$REPLY")
	    done < <(identify -quiet -format '%#\n' "$tmp" "$known")

	    if [[ ${image_signatures[0]} != "${image_signatures[1]}" ]]; then
		compare "$tmp" "$known" -compose src diff.png
		mail -s "$argv0: changes for $1" "$destination" -A diff.png
		mv "$tmp" "$known"
	    else
		rm "$tmp"
	    fi ;;
    esac
else
    printf >&2 '%s\n' "$argv0: retrieving website for the first time"
    ww_fetch "$uri" "$known"
fi

Last edited by Alad (2019-06-05 00:39:41)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
Honest Alad's Package Emporium—Now with added bugs! (Grand reopening: December 1st 2018)

Offline

#3257 2019-06-05 01:41:29

bulletmark
Member
From: Brisbane, Australia
Registered: 2013-10-22
Posts: 435

Re: Post your handy self made command line utilities

Reading Alad's post above and seeing how he uses email for his notification system from his script has prompted me to recommend pushover.net which I have been using for a few years. E.g I have a personal script "notify-phone <message>" which I can call from scripts like the above (PS, I long ago wrote a similar "site-monitor" script myself). Pushover sends notifications direct to your phone, is extremely fast and reliable, and can be configured with different messages and priorities, i.e. with/without sound/pop up etc. You can create groups, so e.g my wife and I are in a "home alarm" group so our phones get instantly notified by my home alarm system (running on a Raspberry Pi using Arch ARM of course). That one is a high priority notification with a distinct sound and popup, which we must instantly deal with.

I normally avoid recommending any paid service but pushover only costs a single one-off "cup of coffee" for lifetime use of the client app. So it is a cheap, easy, and modern but reliable way to add phone notifications to your scripts and programs.

Offline

#3258 2019-06-05 07:00:30

boban_dj
Member
Registered: 2015-03-17
Posts: 127

Re: Post your handy self made command line utilities

@Alad,
CutyCapt is executable name instead cutycapt, found in AUR: https://aur.archlinux.org/packages/cutycapt-qt5-git/
also the website url argument must be full eg: https://example.com

thanks for the script!

Offline

#3259 2019-06-25 04:45:29

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,164

Re: Post your handy self made command line utilities

At the end of my .bash_profile

clear
wm() {
echo ""
echo ""
echo "                                        MY DISPLAY MANAGER"
echo ""
echo ""
echo "Choose your window manager (defaults to i3)?"
echo "s) sway i) i3 k) kodi c) back to console or type command name (e.g., firefox)"
echo ""
echo "You can run this script again with command 'wm'"
echo ""
echo "Make your choice.:"
read wm
case $wm in
    s|sway) sway ;;
    i|i3) sx /usr/bin/i3 ;;
    k|kodi) sx /usr/bin/kodi ;;
    c) ;;
    *) [ $wm ] && sx /usr/bin/$wm || sx /usr/bin/i3 ;;
esac
}
wm

Arch is home!
sway rofi qutebrowser vifm vim mutt lizzy pass

Offline

#3260 2019-06-29 20:27:18

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,327

Re: Post your handy self made command line utilities

A little go script to play my favourite radio stations resiliently without buffer latency (uses MPV for playback)

package main

import (
	"flag"
	"fmt"
	"io"
	"log"
	"net"
	"net/http"
	"os"
	"os/exec"
	"time"
)

func main() {
	noskip := flag.Bool("noskip", false, "do not skip buffer")
	flag.Usage = func() {
		fmt.Println("help:")
		fmt.Println("\tavailable stations")
		fmt.Println("\t\t1, 2, 5, swh, heart")
		fmt.Println("\t-noskip")
		fmt.Println("\t\tdisables buffer skiping")
		os.Exit(0)
	}
	flag.Parse()

	stations := map[string]struct {
		stream  string
		buffer  int
		bitrate int
		codec   string
	}{
		//Latvijas Radio 1,2,5
		"1": {"http://lr1mp1.latvijasradio.lv:8010/", 15, 128, "aac"},
		"2": {"http://lr2mp1.latvijasradio.lv:8000/", 15, 128, "aac"},
		"5": {"http://live.pieci.lv/live19-hq.mp3", 3, 128, "mp3"},
		//Radio SWH
		"swh": {"http://87.110.219.34:8000/swhaac", 3, 128, "aac"},
		//Heart London
		"heart": {"http://media-ice.musicradio.com/HeartLondonMP3", 12, 128, "mp3"},
	}

	args := flag.Args()
	// If no args, play LR2
	station := stations["2"]
	if len(args) > 0 {
		if v, ok := stations[args[0]]; ok {
			station = v
		}
	}

	//Custom transport for IceCast protocol
	tr := &http.Transport{
		Dial: func(network, a string) (net.Conn, error) {
			realConn, err := net.Dial(network, a)
			if err != nil {
				return nil, err
			}
			hijack := &IcyConnWrapper{
				deadline: time.Second * 5,
				Conn:     realConn,
			}
			if *noskip {
				hijack.deadline = time.Second * 15
			}
			return hijack, nil
		},
	}
	client := &http.Client{
		Transport: tr,
		Timeout:   time.Hour * 6,
	}
	http.DefaultClient = client

	for {
		stream, err := http.Get(station.stream)
		if err != nil {
			log.Println("GET:", err)
			time.Sleep(time.Second * 5)
			continue
		}
		pipe, err := getPipe(station.codec)
		if err != nil {
			log.Println("GETPIPE:", err)
			time.Sleep(time.Second * 5)
			continue
		}
		//Skip the buffer for more realtime playback
		if !*noskip {
			err := skipBytes(stream.Body, station.bitrate/8*1024*station.buffer)
			if err != nil {
				log.Println("SKIP BYTES:", err)
				stream.Body.Close()
				pipe.Close()
				time.Sleep(time.Second * 5)
				continue
			}
		}
		//Fetch only half a second at a time
		buf := make([]byte, station.bitrate/8*1024/2)
		_, err = io.CopyBuffer(pipe, stream.Body, buf)
		if err != nil {
			log.Println("COPY BUFFER:", err)
			stream.Body.Close()
			pipe.Close()
			time.Sleep(time.Second * 5)
		}
	}
}

//IcyConnWrapper makes IceCast to work with net/http package
type IcyConnWrapper struct {
	net.Conn
	haveReadAny bool
	deadline    time.Duration
}

func (i *IcyConnWrapper) Read(b []byte) (int, error) {
	i.Conn.SetReadDeadline(time.Now().Add(i.deadline))
	if i.haveReadAny {
		return i.Conn.Read(b)
	}
	i.haveReadAny = true
	//bounds checking ommitted. There are a few ways this can go wrong.
	//always check array sizes and returned n.
	n, err := i.Conn.Read(b[:3])
	if err != nil {
		return n, err
	}
	if string(b[:3]) == "ICY" {
		//write Correct http response into buffer
		copy(b, []byte("HTTP/1.1"))
		return 8, nil
	}
	return n, nil
}

func skipBytes(i io.Reader, target int) error {
	buf := make([]byte, 128)
	total := 0
	for {
		n, err := i.Read(buf)
		if err != nil {
			return err
		}
		total += n
		if total > target {
			return nil
		}
	}
}

func getPipe(codec string) (pipe io.WriteCloser, err error) {
	cmd := exec.Command("mpv", "-", "--profile=low-latency", "--quiet",
		"--cache=no", "--demuxer-lavf-buffersize=512",
		"--demuxer-lavf-format="+codec, "--demuxer=+lavf")

	pipe, err = cmd.StdinPipe()
	if err != nil {
		return
	}

	statusOut, err := cmd.StdoutPipe()
	if err != nil {
		return
	}
	statusErr, err := cmd.StderrPipe()
	if err != nil {
		return
	}

	go func(src io.ReadCloser) {
		io.Copy(os.Stdout, src)
	}(statusOut)

	go func(src io.ReadCloser) {
		io.Copy(os.Stderr, src)
	}(statusErr)

	err = cmd.Start()
	if err != nil {
		return
	}
	go func(c *exec.Cmd) {
		c.Wait()
		log.Println("REALEASING MPV")
	}(cmd)
	return
}

Last edited by ugjka (2019-07-05 22:30:47)


ENTER

~ Arch is not Arch unless it is Arch ~
~ It’s better to be good at something, instead of mediocre at everything ~

Offline

#3261 2019-06-29 21:22:19

N_BaH
Member
Registered: 2015-12-30
Posts: 52

Re: Post your handy self made command line utilities

no shebang ?
is the kernel able to read your mind ?
wink

Offline

#3262 2019-06-29 21:28:45

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,327

Re: Post your handy self made command line utilities

N_BaH wrote:

no shebang ?
is the kernel able to read your mind ?
wink

Go compiles to a statically linked binary executable

Last edited by ugjka (2019-06-29 21:29:18)


ENTER

~ Arch is not Arch unless it is Arch ~
~ It’s better to be good at something, instead of mediocre at everything ~

Offline

#3263 2019-06-30 05:24:33

N_BaH
Member
Registered: 2015-12-30
Posts: 52

Re: Post your handy self made command line utilities

ah, go !
you should have given a link (where) to go.

Offline

#3264 2019-06-30 10:57:14

quequotion
Member
From: Oita, Japan
Registered: 2013-07-29
Posts: 694
Website

Re: Post your handy self made command line utilities

I emulated the rotate cube "screensaver" functionality compiz (0.9.x) lost many cycles ago, with xdotool and xprintidle:

#!/bin/bash

#Compiz rotate plugin's "initiate" button
xdotool mousedown 9

#Pixels to move mouse (rotates cube)
step=1

#Start rotation left or right at random
direction=$RANDOM; let "direction >>= 14"
[[ "$direction" -eq 1 ]] && direction=90 || direction=270

#Cube rotation screensaver
while [ $(xprintidle) -gt 0 ]; do
  xdotool mousemove_relative --polar "$direction" "$step"
  #Approximately 60fps
  sleep 0.0165
done

#Compiz rotate plugin's "initiate" button
xdotool mouseup 9

Last edited by quequotion (2019-06-30 15:14:37)

Offline

#3265 2019-06-30 15:17:34

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,327

Re: Post your handy self made command line utilities

N_BaH wrote:

ah, go !
you should have given a link (where) to go.

There is a way to execute it as a script though: https://blog.cloudflare.com/using-go-as … -in-linux/


ENTER

~ Arch is not Arch unless it is Arch ~
~ It’s better to be good at something, instead of mediocre at everything ~

Offline

#3266 2019-07-09 18:04:40

kokoko3k
Member
Registered: 2008-11-14
Posts: 1,829

Re: Post your handy self made command line utilities

Vpn via ssh with routed traffic, it could be done a lot better, and seems to require superuser rights on the local and remote side ; of course key authentication is needed, but it is still definitely nice:

TUNIP_L=192.168.244.2 #local vpn ip
TUNIP_R=192.168.244.1 #remote vpn ip
DEF_GW=192.168.15.1  #default local gateway
SSHD=<remote ip> #remote ssh server
SSHDP=22 #remote ssh server port

sh -c "sleep 5 ; \
        route add -net 0.0.0.0 netmask 0.0.0.0 gw $TUNIP_R ; \
        route add $SSHD gw $DEF_GW ; \
        echo routed" &

ssh \
  -o PermitLocalCommand=yes \
  -o LocalCommand="sudo ifconfig tun5 $TUNIP_L pointopoint $TUNIP_R netmask 255.255.255.0" \
  -o ServerAliveInterval=60 \
  -w 5:5 root@$SSHD -p $SSHDP  \
  "sudo ifconfig tun5 $TUNIP_R pointopoint $TUNIP_R netmask 255.255.255.0;   echo tun ready"

route del -net 0.0.0.0 netmask 0.0.0.0 gw $TUNIP_R ; route del $SSHD gw $DEF_GW
echo unrouted

on the ssh server (it could be done even in the previous command):

iptables -t nat -A POSTROUTING -s 192.0.0.0/24 -o etha0a9 -j MASQUERADE

Do i really need openvpn?

Last edited by kokoko3k (2019-07-09 18:07:01)

Offline

#3267 2019-07-10 11:06:05

Awebb
Member
Registered: 2010-05-06
Posts: 5,269

Re: Post your handy self made command line utilities

N_BaH wrote:

ah, go !
you should have given a link (where) to go.

He said it's a go script. Dunno what else you'd need to know.

Offline

Board footer

Powered by FluxBB