Post your handy self made command line utilities

studentik · 2019-11-12 21:41:04

When dealing with timestamped files, it is sometimes nice to be able to cleanup unnecessary files while keeping some for archive purposes. Could not find better option than to write small script for that. It searches directory content for timestamped files and lists only files for last N days/months/years. Optionally it allows to --invert filter and --delete filtered files:

https://gitlab.com/student/timestamps-archive

# list top 10 timestamped files in current directory
timestamps-archive.py --format "archive_%Y-%m-%d.zip"  --top 10

# first timestamped file from last 12 months  
timestamps-archive.py --format "%Y-%m-%d"  --months 12 

# delete timestamped files except latest 5 and last 10 years
timestamps-archive.py --format "%Y-%m-%d"  --top 5 --years 10 --invert --delete

# print help
timestamps-archive.py -h 
usage: timestamps-archive.py [-h] [--path PATH] [--format FORMAT] [--top TOP]
                             [--days DAYS] [--months MONTHS] [--years YEARS]
                             [--invert] [--delete]

Daily/monthly/yearly archive filter for timestamped files/directories

optional arguments:
  -h, --help       show this help message and exit
  --path PATH      files path, default is current directory
  --format FORMAT  date/time format, default is %Y-%m-%dT%H:%M:%S
  --top TOP        include TOP files
  --days DAYS      include number of DAYS
  --months MONTHS  include number of MONTHS
  --years YEARS    include number of YEARS
  --invert         invert filter
  --delete         delete filtered files/directories

Trilby · 2019-11-13 00:29:57

studentik, is this just on the filenames, or do you look at mtime/atime? If the latter, you should really just use find as it has flags for file times in certain ranges.

studentik · 2019-11-13 05:50:16

Trilby wrote:

studentik, is this just on the filenames, or do you look at mtime/atime? If the latter, you should really just use find as it has flags for file times in certain ranges.

This looks for timestamp in names. Regarding find, I don't know if that is possible, since whether or not file should end up in list depends on existance/absence of another files, not current time. Let's say we don't have archive for January 1st, but do have for January 2nd, filtering --years should keep this January 2nd since it is first existing timestamp for particular year.

Trilby · 2019-11-13 12:53:48

Well I was thinking find piped to head/sed or the like for restricting the number of results, but yes, if this is just about dates in the names of files, find wouldn't help.

quequotion · 2019-12-01 22:46:05

I lied, I have a little AUR helper; this is all it does:

aur pkgname:

#!/bin/bash

git clone https://aur.archlinux.org/$1.git

I also have a little ABS helper,

abs pkgname:

case $1 in
  -c|-m)
    git clone git://git.archlinux.org/svntogit/community.git --{single-,}branch {packages/,}$2
  ;;
  *)
    git clone git://git.archlinux.org/svntogit/packages.git --{single-,}branch {packages/,}$1
  ;;
  "")
  ;;
esac

edit: [community] and [multiarch] use a different git url, so use "-m" or "-c" for those packages.

edit more: one of these days, I am thinking of combining these and adding functionality to make a real pacman-wrapping aur/abs helper--as if anyone needs that.

Last edited by quequotion (2020-10-18 11:55:04)

Alad · 2019-12-02 13:47:16

How many AUR packages do you manage with your "little" helper?

Just wondering.

dimich · 2019-12-02 18:50:57

quequotion wrote:

git clone https://aur.archlinux.org/$1.git

I have almost the same:

#!/bin/sh

if [ -z "$1" ]; then
    exit 1
fi

git clone "https://aur.archlinux.org/$1"

quequotion · 2019-12-02 20:03:17

Alad wrote:

How many AUR packages do you manage with your "little" helper?

lol, "manage" may be too strong a word; it's downloaded a few dozen packages.

EDIT: If you were asking about the packages I maintain in the AUR; I "manage" those in a couple of git repositories of their own. All my published AUR packages have been managed with git since.. aur 4?.

dimich wrote:

if [ -z "$1" ]; then
    exit 1
fi

ooh, mine just breaks!

Last edited by quequotion (2019-12-03 10:31:58)

Alad · 2019-12-03 00:15:28

Suggested improvement:

#!/bin/bash
if [[ -z $1 ]]; then
  echo "The trash heap of Arch. Dig around enough and maybe you will find some gold."
else
  git clone https://aur.archlinux.org/"$1".git
fi

Trilby · 2019-12-03 00:32:22

The quotes around $1 don't really help much - they just change the type of problem you'd face if you pass a first parameter with spaces in it - but it really doesn't matter as package names can't have spaces.

Alad · 2019-12-03 01:04:04

That reminds me of an AUR pkgbuild I saw which had depends like 'foo bar'. I guess that's for the grr thread...

quequotion · 2019-12-03 05:49:40

Alad wrote:

The trash heap of Arch. Dig around enough and maybe you will find some gold.

Is that a proposal for the AUR FAQ?

I support this proposal. It makes clear the risks involved; the dumpster diving metaphor is excellent: this may give you cholera, or hepatitis, not to mention a sore stomach (and honestly a few headaches).

Also fine documentation for a little AUR helper.

Trilby wrote:

they just change the type of problem you'd face if you pass a first parameter with spaces in it

It just does one pkgname at a time; someone may attempt multiple pkgnames in a single call. That won't work without a loop; and is going to cause some unspecified behavior as is.

Maybe mandatory single pkgname mode is best anyway ($1 without quotes; relies on user not supplying $1 as a string with a space in it using their own quotes; silently drops other parameters if say, someone tried to aur multiple-bzr aurpkgs-git atonce-hg)?

Last edited by quequotion (2019-12-03 13:20:36)

Alad · 2019-12-03 12:58:39

If someone quoted multiple packages in one string they only have themselves to blame...

Roken · 2019-12-22 09:05:32

Inspired by a thread on the forums, I threw this together to show the number of packages installed from the different repos in a nice tabular format.

#!/bin/bash

TOT=`pacman -Q | wc -l`
printf "REPOSITORY \t\t PACKAGES\n"
for REPO in 'core' 'extra' 'community' 'multilib' 'testing' 'community-testing' 'multilib-testing';
do
	case $REPO in
		core|extra)
			TABS="\t\t\t"
			;;

		community|multilib|testing)
			TABS="\t\t"
			;;

		*)
			TABS="\t"
			;;
	esac

	MANY=`paclist $REPO | wc -l`
	printf "$REPO $TABS %6s \n" "$MANY"
	OFFICIAL=$(($OFFICIAL + $MANY ))
done

AUR=$(($TOT - $OFFICIAL))
printf "AUR/Other \t\t %6s \n" "$AUR"
printf "Total \t\t\t %6s \n" "$TOT"

Trilby · 2019-12-22 13:28:35

Why do you mix tabs and spaces in your column separators? And why varied number of tabs? That looks like you are trying to manually align based on assumptions about the tab width. Just use `columns` to format the output. Also you assume all those repos are in use, but ignore any unofficial repos. Why not just check which repos are actually in use on the system? You can also do the entire operation with 2 calls to pacman rather than N+1 for N repos:

#!/bin/sh

pacman -Sl | \
sed -n '/\[installed\]/s/ .*//p' | \
uniq -c | \
{
        total=0
        while read count repo; do
                printf '%s %d\n' $repo $count
                total=$((total+count))
        done
        foreign=$(pacman -Qm | wc -l)
        printf 'AUR/Other %d\n' $foreign
        printf 'Total %d\n' $((total+foreign))
} | \
column --table -N REPOSITORY,PACKAGES

Here's an awk-based version that saves a few subshells and is just *slightly* faster on my system:

#!/bin/sh

pacman -Sl | \
awk -v n=$(pacman -Qm | wc -l) '
	/\[installed\]/ {
		repos[$1] += 1
		total += 1
	}
	END {
		for (repo in repos) {
			print repo, repos[repo]
		}
		print "AUR/Other", n
		print "Total", n+total
	}
' | \
column --table -N REPOSITORY,PACKAGES

Last edited by Trilby (2019-12-22 13:36:49)

ugjka · 2019-12-26 11:07:00

I'm rocking 2 laptops on my desk now. The new one is my main machine and the old one is a dedicated IRC machine that stays on 24/7. I made the IRC machine completely silent by disabling multicore support and intel speedstep so it is running with one core only and on the lowest cpu clock speed.

Now I had 2 problems: links posted in irc and the clipboard.
I wanted the links in Konversation to open on my main machine and the clipboard to be completely in sync with both machines. I looked for some solutions online but i didn't find anything simple and/or the solution were nodejs crap that i tried and found unreliable.

So I decided to code it myself in Go which turned out pretty easy job

FF/FFD

//ff: send url's across the network to the destination machine
package main

import "flag"

import "net"

import "fmt"

import "log"

import "io"

const port = 45654

func main() {
	dest := flag.String("dest", "archee.local", "destination adress")
	flag.Parse()
	url := flag.Args()[0]
	conn, err := net.Dial("tcp", fmt.Sprintf("%s:%d", *dest, port))
	if err != nil {
		log.Fatal(err)
	}
	_, err = io.WriteString(conn, url)
	if err != nil {
		conn.Close()
		log.Fatal(err)
	}
	conn.Close()
}

// ffd: listen's for incoming url's and launches them in Firefox
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"os/exec"
)

const port = 45654

func main() {
	l, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
	if err != nil {
		log.Fatal(err)
	}
	for {
		conn, err := l.Accept()
		if err != nil {
			log.Printf("Accept error: %v", err)
			continue
		}
		data, err := ioutil.ReadAll(conn)
		if err != nil {
			conn.Close()
			log.Printf("Read error: %v", err)
			continue
		}
		conn.Close()
		url := string(data)
		cmd := exec.Command("firefox", url)
		cmd.Start()
		cmd.Wait()
	}
}

CLIPPER

// clipper: sync clipboard between 2 machines
// first arg must be the client's address
package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net"
	"os"
	"os/exec"
	"strings"
	"time"
)

const port = 43344

var client string

func main() {
	client = os.Args[1]
	data, _ := exec.Command("xclip", "-selection", "clipboard", "-out").Output()
	current := string(data)
	incoming := make(chan string)
	go listener(incoming)
	for {
		select {
		case txt := <-incoming:
			c := exec.Command("xclip", "-selection", "clipboard", "-in")
			c.Stdin = strings.NewReader(txt)
			err := c.Run()
			if err != nil {
				log.Printf("Xclip write error: %v", err)
				break
			}
			current = txt
		default:
			time.Sleep(time.Second)
			c := exec.Command("xclip", "-selection", "clipboard", "-out")
			data, err := c.Output()
			if err != nil {
				log.Printf("Xclip read error: %v", err)
				break
			}
			txt := string(data)
			if txt == current {
				break
			}
			current = txt
			err = send(txt)
			if err != nil {
				log.Printf("Send error: %v", err)
			}
		}
	}
}

func send(data string) error {
	conn, err := net.Dial("tcp", fmt.Sprintf("%s:%d", client, port))
	if err != nil {
		return err
	}
	_, err = io.WriteString(conn, data)
	conn.Close()
	return err
}

func listener(t chan<- string) {
	l, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
	if err != nil {
		log.Fatal(err)
	}
	for {
		conn, err := l.Accept()
		if err != nil {
			log.Printf("Accept error: %v", err)
			continue
		}
		data, err := ioutil.ReadAll(conn)
		if err != nil {
			conn.Close()
			log.Printf("Read error: %v", err)
			continue
		}
		conn.Close()
		t <- string(data)
	}

}

Last edited by ugjka (2019-12-26 11:09:05)

linux-mate · 2020-01-04 19:02:42

"Message" is a required field in this form.

Mod comment: Post blanked by the author. Author has been banned

Last edited by ewaller (2020-01-05 22:19:53)

qinohe · 2020-01-04 19:58:30

Wouldn't this be a beautiful example to setup an array for?

I don't know any python so forgive me if the syntax is not completely correct, but something like:

...
showtime = ["sherlock","the-flash","arrow"]

for i in range(len(showtime)):

get_time_until_next_episode("http://next-episode.net/[i]", print(showtime[i]))

edit: forgot a closing ')'

Last edited by qinohe (2020-01-04 20:17:52)

Trilby · 2020-01-04 20:28:00

You also forgot that lists are iterable, that python requires blocks to be indented, and that there's no reason to call print on a string just to get a string as doing so will actually not return a string but will result in the called function failing.

Last edited by Trilby (2020-01-04 20:38:35)

qinohe · 2020-01-04 20:52:07

Told you so, don't know any python(at all) looked up some example;)
Hope that this is somehow closer to the truth:

showtime = ["sherlock","the-flash","arrow"]

for i in range(len(showtime)):

get_time_until_next_episode("http://next-episode.net/(showtime[i]"))

Hope to learn so something too of course;)

Trilby · 2020-01-04 21:10:17

No, you still missed the two main points from my comment:

for show in showtime:
   get_time_until_next_episode("http://next-episode.net/" + show)

Though there's really no reason for that base url to not be in the function too, so this could just be:

for show in showtime:
   get_time_until_next_episode(show)

Some of python's greatest strengths are the abilities to write simple, concise, yet expressive code. Don't sacrifice that to write code that looks like C code. I love C, but if you are going to write something that looks like C, write C, and it will be far faster than python. If you are going to write python, be sure to use it's strengths or you are sacrificing speed for nothing.

And learn python. It's worth it. I was skeptical and avoided it for a long time; I regret that. It's not right for everything, but what it is right for, it's really right for.

Last edited by Trilby (2020-01-04 21:14:55)

qinohe · 2020-01-04 21:20:14

Believe me when I say; I only know some shell/bash scripting...
Will look into your example tommorow, time's up for today, thanks for the lesson;)

Will do a python course.

Last edited by qinohe (2020-01-04 21:21:59)

eschwartz · 2020-01-05 04:52:18

Given the original script actually used a "human readable" name combined with an url suffix, this would actually be a good place to use a dictionary:

shows = {
    "http://next-episode.net/sherlock": "Sherlock",
    "http://next-episode.net/the-flash": 'The Flash',
    "http://next-episode.net/arrow": "Arrow",
    "http://next-episode.net/travelers": "Travelers",
    "http://next-episode.net/shooter": "Shooter",
    "http://next-episode.net/the-walking-dead": "Walking dead",
    "http://next-episode.net/supergirl": "Supergirl",
    "http://next-episode.net/marvels-daredevil": "Daredevil",
    "http://next-episode.net/lucifer": "Lucifer",
    "http://next-episode.net/vikings": "Vikings",
    "http://next-episode.net/marvels-agents-of-s.h.i.e.l.d.": "SHIELD",
    "http://next-episode.net/gotham": "Gotham",
    "http://next-episode.net/supernatural": "Supernatural",
    "http://next-episode.net/dcs-legends-of-tomorrow": "Legends",
    "http://next-episode.net/timeless": "Timeless",
    "http://next-episode.net/doctor-who": "Doctor who",
}

for url, displayname in shows.items():
    get_time_until_next_episode(url, displayname)

Or, if moving the url prefix out of the dictionary and into the get_time_until_next_episode function...

shows = {
    "sherlock": "Sherlock",
    "the-flash": 'The Flash',
    "arrow": "Arrow",
    "travelers": "Travelers",
    "shooter": "Shooter",
    "the-walking-dead": "Walking dead",
    "supergirl": "Supergirl",
    "marvels-daredevil": "Daredevil",
    "lucifer": "Lucifer",
    "vikings": "Vikings",
    "marvels-agents-of-s.h.i.e.l.d.": "SHIELD",
    "gotham": "Gotham",
    "supernatural": "Supernatural",
    "dcs-legends-of-tomorrow": "Legends",
    "timeless": "Timeless",
    "doctor-who": "Doctor who",
}

for urlsuffix, displayname in shows.items():
    get_time_until_next_episode(urlsuffix, displayname)

Last edited by eschwartz (2020-01-05 04:55:55)

linux-mate · 2020-01-05 11:14:22

"Message" is a required field in this form.

Mod comment: Post blanked by the author. Author has been banned

Last edited by ewaller (2020-01-05 22:20:15)

eschwartz · 2020-01-05 19:15:08

linux-mate wrote:

Why would i complicate things? Why you need dictionary to then iterate on it and still call function with url and tvshow name for each of them? I already do it in original script without dictionary and iteraration.
No matter what, you still have to copy url and add somewhere it's easier in original script then in dictionary.

Because many people find it helps them organize and visualize data. No one will try to hurt you for not doing so, though... although personally I think it's in very bad taste to respond so negatively to a thread about sharing and offering suggestions to each other. Maybe someone else would like to use your script, but prefers visualizing the shows via a dictionary, and would appreciate our discussion.

That being said, if you ever wanted to make the script capable of reading shows from a text or json configuration file, you would need to use a similar approach anyway.

Arch Linux

#3276 2019-11-12 21:41:04

Re: Post your handy self made command line utilities

#3277 2019-11-13 00:29:57

Re: Post your handy self made command line utilities

#3278 2019-11-13 05:50:16

Re: Post your handy self made command line utilities

#3279 2019-11-13 12:53:48

Re: Post your handy self made command line utilities

#3280 2019-12-01 22:46:05

Re: Post your handy self made command line utilities

#3281 2019-12-02 13:47:16

Re: Post your handy self made command line utilities

#3282 2019-12-02 18:50:57

Re: Post your handy self made command line utilities

#3283 2019-12-02 20:03:17

Re: Post your handy self made command line utilities

#3284 2019-12-03 00:15:28

Re: Post your handy self made command line utilities

#3285 2019-12-03 00:32:22

Re: Post your handy self made command line utilities

#3286 2019-12-03 01:04:04

Re: Post your handy self made command line utilities

#3287 2019-12-03 05:49:40

Re: Post your handy self made command line utilities

#3288 2019-12-03 12:58:39

Re: Post your handy self made command line utilities

#3289 2019-12-22 09:05:32

Re: Post your handy self made command line utilities

#3290 2019-12-22 13:28:35

Re: Post your handy self made command line utilities

#3291 2019-12-26 11:07:00

Re: Post your handy self made command line utilities

#3292 2020-01-04 19:02:42

Re: Post your handy self made command line utilities

#3293 2020-01-04 19:58:30

Re: Post your handy self made command line utilities

#3294 2020-01-04 20:28:00

Re: Post your handy self made command line utilities

#3295 2020-01-04 20:52:07

Re: Post your handy self made command line utilities

#3296 2020-01-04 21:10:17

Re: Post your handy self made command line utilities

#3297 2020-01-04 21:20:14

Re: Post your handy self made command line utilities

#3298 2020-01-05 04:52:18

Re: Post your handy self made command line utilities

#3299 2020-01-05 11:14:22

Re: Post your handy self made command line utilities

#3300 2020-01-05 19:15:08

Re: Post your handy self made command line utilities

Board footer