You are not logged in.

#3251 2019-05-19 05:24:33

pellertson
Member
Registered: 2018-03-16
Posts: 5

Re: Post your handy self made command line utilities

I'm a bit of an odd duck and I use Rob Pike's acme as my text editor/general programming environment.  Recently, I wrote an RSS feed for it by converting all the RSS feeds to JSON feeds and using the following to display a summary of my subscriptions:

#!/bin/sh
# RSS/Atom/JSON Feed reader for acme
# This code is in the public domain
for url in $(cat $URLS)
do
	temp=$(mktemp)
	curl -s $url > $temp
	jq -r '.title' $temp
	echo
	jq -r ' .items | .[] | [.title, .url] | join("\t") ' $temp | head -n 5
	echo
	echo =====
	echo
	rm $temp
done | 9p write acme/new/body

writetoacme JSON-Feeds

"writetoacme" is also a script I wrote to help with writing acme screens.

#!/bin/sh
# Shortcut for common writing acme applications
# USAGE: writetoacme $1
# $1: name of window

last=$(9p ls acme | sort -g | tail -n 1)
echo "name $1" | 9p write acme/$last/ctl
echo -n "clean" | 9p write acme/$last/ctl
echo -n "0,0" | 9p write acme/$last/addr
echo -n "dot=addr" | 9p write acme/$last/ctl
echo -n "show" | 9p write acme/$last/ctl

EDIT: before anyone asks, I simply use a web-service called feed2json to do all the converting for me.  I'll probably write my own program to do that later on.  But this works for now.

Last edited by pellertson (2019-05-19 05:31:47)

Offline

#3252 2019-05-19 12:21:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 21,555
Website

Re: Post your handy self made command line utilities

for url in $(cat $URLS)
do
# ...
done

Save the cats!

while read url
do
# ...
done < $URLS

You also may be interested in python's feedparser which would drastically simplify the whole process.

Last edited by Trilby (2019-05-19 12:21:35)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Online

#3253 2019-05-19 18:37:20

escondida
Trusted User (TU)
Registered: 2008-04-03
Posts: 157

Re: Post your handy self made command line utilities

Props for the acme rss script, pellertson! You've inspired me to finish putting together my own version (sans the Web service), something I've been meaning to do for a while.

Offline

#3254 2019-05-20 01:01:14

pellertson
Member
Registered: 2018-03-16
Posts: 5

Re: Post your handy self made command line utilities

Trilby wrote:

You also may be interested in python's feedparser which would drastically simplify the whole process.

That could be useful, but I've found through experience that shell scripts are truly the best way to write to acme screens.  As readable as python is, I cannot go without pipes.  Plus I wanted to learn how to use "jq".  But thanks for the tip on removing cat!

Offline

#3255 2019-05-21 18:36:26

pellertson
Member
Registered: 2018-03-16
Posts: 5

Re: Post your handy self made command line utilities

escondida wrote:

Props for the acme rss script, pellertson! You've inspired me to finish putting together my own version (sans the Web service), something I've been meaning to do for a while.

Thanks!  Can I see your version?

Offline

#3256 2019-06-05 00:39:02

Alad
Wiki Admin/IRC Op/TU
From: Bagelstan
Registered: 2014-05-04
Posts: 1,950
Website

Re: Post your handy self made command line utilities

I was looking to watch websites for changes, but all I found was a paid service. So I decided to host my own solution.

The script is run from cron, and supports three methods of comparison:

1. direct HTML comparison (using curl). This may result in false positives, especially with javascript-heavy sites.
2. text rendering (using w3m). This does not necessarily render all content (again, javascript).
3. full rendering (using cutycapt/qtwebengine). Heavy-handed but reliable approach for most websites. An image overlay is sent as email attachment.

#!/bin/bash
# requires: diff mail file nice imagemagick curl w3m cutycapt xvfb-run
argv0=webwatcher
XDG_CACHE_HOME=${XDG_CACHE_HOME:-$HOME/.cache}

# default arguments
destination=your@face.com
mode=exact

while getopts :d:m: opt; do
    case $opt in
	d) destination=$OPTARG
	   ;;
	m) mode=$OPTARG
	   ;;
	*) echo "usage: ${0##*/} [+-mi} [--] ARGS..."
	   exit 2
    esac
done
shift $(( OPTIND - 1 ))
OPTIND=1

if [[ ! $1 ]]; then
    printf >&2 '%s\n' "$argv0: error: website not specified"
    exit 2
fi

uri=$1
uri_id=$(sha256sum <<< "$1" | cut -d' ' -f1)

case $mode in
    exact)
        tmp_extension=.htm
	# "small changes", may have false positives	
	ww_fetch() { curl -fsS "$1" --output "$2"; }
	;;
    text)
	tmp_extension=.txt
	# text-browser mode, may not render all contents
	ww_fetch() { w3m -dump "$1" > "$2"; }
	;;
    image)
	tmp_extension=.png
	# browser mode, will render full page
	# xvfb-run -a is used to allow multiple processes
	ww_fetch() { nice -n 19 xvfb-run -a --server-args "-screen 0, 1024x768x24" \
			  cutycapt --url="$1" --out="$2"; }
	;;
esac

# xdg base directory
known=$XDG_CACHE_HOME/$argv0/$uri_id$tmp_extension
mkdir -p "$XDG_CACHE_HOME/$argv0"

if [[ -s $known ]]; then
    tmp=$(mktemp --suffix "$tmp_extension")
    ww_fetch "$uri" "$tmp"

    case $mode in
	exact|text)
	    if ! diff -q "$known" "$tmp" >/dev/null; then
		diff -u "$known" "$tmp" | mail -s "$argv0: changes for $1" "$destination"
		mv "$tmp" "$known"
	    else
		rm "$tmp"
	    fi ;;
	image)
	    # when comparing screenshots, metadata has to be taken into account. See:
	    # http://www.libpng.org/pub/png/spec/1.2/PNG-Chunks.html#C.Anc-text (PNG meta-data)
	    # http://www.imagemagick.org/Usage/compare/ ("IM Image Signatures", imagemagick)
	    image_signatures=()
	    while read -r sig; do
		image_signatures+=("$REPLY")
	    done < <(identify -quiet -format '%#\n' "$tmp" "$known")

	    if [[ ${image_signatures[0]} != "${image_signatures[1]}" ]]; then
		compare "$tmp" "$known" -compose src diff.png
		mail -s "$argv0: changes for $1" "$destination" -A diff.png
		mv "$tmp" "$known"
	    else
		rm "$tmp"
	    fi ;;
    esac
else
    printf >&2 '%s\n' "$argv0: retrieving website for the first time"
    ww_fetch "$uri" "$known"
fi

Last edited by Alad (2019-06-05 00:39:41)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
Honest Alad's Package Emporium—Now with added bugs! (Grand reopening: December 1st 2018)

Offline

#3257 2019-06-05 01:41:29

bulletmark
Member
From: Brisbane, Australia
Registered: 2013-10-22
Posts: 433

Re: Post your handy self made command line utilities

Reading Alad's post above and seeing how he uses email for his notification system from his script has prompted me to recommend pushover.net which I have been using for a few years. E.g I have a personal script "notify-phone <message>" which I can call from scripts like the above (PS, I long ago wrote a similar "site-monitor" script myself). Pushover sends notifications direct to your phone, is extremely fast and reliable, and can be configured with different messages and priorities, i.e. with/without sound/pop up etc. You can create groups, so e.g my wife and I are in a "home alarm" group so our phones get instantly notified by my home alarm system (running on a Raspberry Pi using Arch ARM of course). That one is a high priority notification with a distinct sound and popup, which we must instantly deal with.

I normally avoid recommending any paid service but pushover only costs a single one-off "cup of coffee" for lifetime use of the client app. So it is a cheap, easy, and modern but reliable way to add phone notifications to your scripts and programs.

Offline

#3258 2019-06-05 07:00:30

boban_dj
Member
Registered: 2015-03-17
Posts: 127

Re: Post your handy self made command line utilities

@Alad,
CutyCapt is executable name instead cutycapt, found in AUR: https://aur.archlinux.org/packages/cutycapt-qt5-git/
also the website url argument must be full eg: https://example.com

thanks for the script!

Offline

Board footer

Powered by FluxBB