Nifty scroll script still not fast enough

Machiavelli · 2011-07-28 07:49:41

I started writing this python script because I wanted to have a scrolling, fixed-width mpc status line in my dwm top bar:

#!/usr/bin/python3

from time import time
from subprocess import getstatusoutput
import sys

_ERROR_MSG = "Usage: scrollout [-l (int)] (cmd)"

def input_cmd(cmd,l):
        if cmd == "":
            s = sys.stdin.read()
        else:
            e,s = getstatusoutput(cmd)
            if e: sys.exit(1)
        s = s.replace("\n"," ")
        s += (" " * int(l/3))
        k = len(s)
        t = int(k*(time()/k-int(time()/k)))
        if k <= l: # static
            return s
        else: # scrolling
            while len(s[t:]) < l: s += s
            return s[t:t+l]


def run():
    _arg = -1
    length = 20
    cmd = ""
    enter_cmd = False
    flags = ("-l")
    for arg in sys.argv[1:]:
        if enter_cmd or (arg not in flags and _arg not in flags):
            cmd += arg + " "
            enter_cmd = True
        elif _arg == "-l":
            try:
                length = int(arg)
            except:
                print(_ERROR_MSG)
                exit(1)
        _arg = arg
    output=input_cmd(cmd,length)
    print(output)

run()

Since it uses time since Epoch to determine how much to scroll the text at any given moment it doesn't have to have any internal loops or run any longer than it takes to compute the output. You can simply call it like so:

scrollout -l 20 mpc current

alternatively

mpc current | scrollout -l 20

For example, in my statusbar script I have the following line:

mpc_status="|$(scrollout -l 20 mpc current) ($(mpc | grep -Eo '[0-9]*\:[0-9]*\/[0-9]*\:[0-9]*')) "

I find it works nicely, but is a bit slow. Not surprisingly, since it has to re-import all of it's libraries every time it gets launched. To be a truly useful script I imagine it would have to be able to finish it's task in less than one second. Right now it's just above that, ofcourse somewhat depending on what command is being piped through it. The big timestealer, however, is the module imports (according to the python profiler). Having a pretty limited experience of programming, I would like to know if anyone knows of a good workaround for this. Perhaps some way to speed up the module imports or to mimic some of the functionality of these modules without having to import them?

Cheers,
Mach

juster · 2011-07-30 21:39:27

Why not avoid executing a command from the script and just settle on piping a command into it? Then you would have one less module to import.

Here is a cheap ripoff in hackish perl:

#!/usr/bin/perl

$/ = undef;
$\ = "\n";
$_ = <STDIN>;
s/\n/ /g;
$w = shift || 20;
if (length > $w) {
    $_ .= q{|};
    $i = time % length;
    $_ = substr(substr($_, $i) . substr($_, 0, $i), 0, $w);
}
print

PS I would avoid using "l" (lowercase L) as a variable name because it looks so much like a 1 (number one).

Machiavelli · 2011-07-30 22:21:12

Thanks for the input! You're probably right, I could do without subprocess... as well as the -l argument (which besides from looking like the number one looks a lot like capital I in many fonts; always a problem for me with includes/libraries in makefiles).

However, I've found that the greatest amount of time the script was running, it was simply initiating the python interpreter. So I built a cx_freeze version which is a lot less sluggish. Got so excited about it I had to make a PKGBUILD:

http://www.bolotin.se/scrollout/arch/sc … 1/PKGBUILD (depends on python-cx_freeze from AUR, not cx_freeze from repositories... off topic, I think the AUR package should state the repository one in the supplies array)

Please try it out and tell me what you think! I'd love for this to be of use to someone other than me as well.

EDIT: Btw, just noticed that the "cheap" ripoff is still doing the same job in one third of the time... amazing. Perhaps python is simply ill-suited to be called regularily with one second intervals. But unfortunately, I don't know any perl.

Last edited by Machiavelli (2011-07-30 22:35:44)

rowdog · 2011-07-31 08:46:59

I don't know python but perhaps an overly commented version of Juster's code would give more insight into the underlying algorithm he's trying to explain. In Perl, the rule of thumb is that everything after # is a comment (not always, but its true here).

#!/usr/bin/perl

$/ = undef;  # slurp the whole file instead of one line at a time
$\ = "\n";  # when we print, append \n at the end
$_ = <STDIN>;   # read everything waiting on STDIN into the variable $_
s/\n/ /g;  # remove all newlines from $_ (it's the implied variable, might be unique to Perl)
$w = shift || 20;  # set variable $w to be the command line argument if there is one, 20 otherwise
if (length > $w) { # if the length of $_ is greater than $w
    $_ .= q{|};  # append a pipe symbol to the string $_, aka $_ = "$_|"
    $i = time % length; # nice trick to get time as an index into the string
    $_ = substr(substr($_, $i) . substr($_, 0, $i), 0, $w); # chop and chop and chop the string to get it right
}
print # print the contents of $_

juster · 2011-07-31 17:03:28

Thanks rowdog! Very nicely done. Machiavelli why don't you write this in C? This is actually pretty simple. Before you say "I don't know C", this could be a relatively painless way to try learning it. A little C goes a long way. I really like simple little C programs for these sort of things.

Here is a version in awk, because I've been practicing awk lately. The R[ecord]S[plitter] is kind of funky. Sadly, gawk is slower than perl!?! GNUUUUUUUUU! Other awks are not too bad...

#!/usr/bin/awk -f
# printf 'Hello\nWorld\n' | ./scrawk -v width=5

BEGIN {
    FS = "\n"; RS = "\000" # avoid splitting records
    if (!width) width = 20
}

{
    out = $1;
    for (i = 2; i <= NF; i++) if ($i) out = out " " $i
    # ignoring empty fields compresses newlines
}

length(out) > width {
    out = out " " # separate begin/end

    "date +%s" | getline epoch
    # there is only one record so I don't close("date +%s")

    i = epoch % length(out)
    out = substr(out, i + 1) substr(out, 1, i) # rotate around i
    out = substr(out, 1, width)
}

{ print out }

Machiavelli · 2011-07-31 20:52:34

Thanks a lot guys! Glad to see such enthusiasm. :-)

juster: But I don't know C! :-) Oh well, I'm working nights, so I have time to learn.

Rewrote the script in python based on juster's algorithm:

#!/usr/bin/python3

from time import time
import sys

w = 20
if len(sys.argv) > 1: w = sys.argv[1]
s = sys.stdin.read().replace("\n","") + (" " * int(w/3))
if len(s) > w:
    i = int(time() % w)
    while len(s[i:]) < w: s += s
    s = s[i:i+w]
print(s)

Then wrote a small bash script to run each of the different variants a hundred times and output their average execution time. Brace yourselves...

The script originally posted:

# mpc current | runtime10 scrollout.py | grep nanoseconds
Average run time: 0 seconds (205726798 nanoseconds)

Roughly the same script, frozen with cx_freeze:

# mpc current | runtime10 scrollout | grep nanoseconds
Average run time: 0 seconds (166651145 nanoseconds)

The perl script submitted by juster:

# mpc current | runtime10 ./perlscrollout | grep nanoseconds
Average run time: 0 seconds (24679770 nanoseconds)

The python script from this post:

# mpc current | runtime10 ./pythonscrollout | grep nanoseconds
Average run time: 0 seconds (190167249 nanoseconds)

So, the perl script still clearly stands out. Also, the new python script doesn't stand a chance compared to the old frozen executable, even though the script itself is proven faster than the original one. A frozen version of the new script will only get me so far in optimization - obviously, this is not a job for python.

Anyone care to prove me wrong?

falconindy · 2011-07-31 22:06:51

Both of these register as fast as the perl script (~3s for 1000 iterations)... I think calls to the RTC are just slow.

#!/bin/bash

read input
maxlen=$1

if (( ${#input} > maxlen )); then
  printf -v i '%(%s)T' -1
  (( i %= maxlen ))
  printf -v out "%s " "${input:i+1}" "${input:0:i}"
else
  out=$input
fi

printf '%s\n' "$out"

And c99:

#define _GNU_SOURCE
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main(int argc, char *argv[]) {
  int maxlen, i;
  char buf[128];
  size_t len;

  maxlen = atoi(argv[1]);
  if (maxlen == 0) {
    return 2;
  }
  char out[maxlen];

  len = fread(buf, sizeof(char), 128, stdin);
  if (len == 0) {
    return 3;
  }

  if (buf[len - 1] = '\n') {
    buf[len - 1] = '\0';
  }

  if (len > maxlen) {
    snprintf(out, maxlen, "%s %s", &buf[(time(NULL) % len) + 1], buf);
  } else {
    sprintf(out, "%s", buf);
  }

  printf("%s\n", out);

  return 0;
}

Machiavelli · 2011-08-01 12:54:14

falconindy: Oooh... yummy. Now, if someone would only care to give me a line-by-line description of what is happening in the C code, like rowdog did with juster's perl script, I'd be a very happy man. :-)

Btw: Froze the new python script, it averages at 71440570 nanoseconds runtime, which is a huge improvement but still three times slower than the perl/C/bash ones. Will start trying to learn C shortly... ;-)

juster · 2011-08-01 21:15:08

Everything that's happening is more or less the same thing as the earlier examples. Get argument, read stdin, replace newlines, print output.

I finally figured out why you you do

while len(s[i:]) < w: s += s

. for scrolling text smaller than the width. Here is another C version: https://gist.github.com/1118935

I'll probably use this myself now... I like your idea. The only problem is it can't handle UTF8 ;-).

Note that in falconindy's C line 23 has a small mistake. I think the code should also null terminate the string in any case. On line 28 you don't need "+ 1", this will go beyond the buffer. On line 28 you will also have to use "maxlen+1" because snprintf saves room for the null char. This means you make "out" an array of "maxlen+1" size.

falconindy · 2011-08-01 22:01:38

juster wrote:

Note that in falconindy's C line 23 has a small mistake. I think the code should also null terminate the string in any case.

Probably true. I originally had out allocated with calloc, which would mean free null termination, assuming the correct size is allocated.

Machiavelli · 2011-08-01 23:21:13

juster: Dang, you beat me to it... :-) The utf-8 thing isn't a problem with python (though in cx_freeze I had to redefine sys.stdin and sys.stdout with the correct encoding). Guess it's just on of the niceties that makes the makes the interpreter a little slower on the start than perl etc.

My biggest problem with the script right now is that with all that's going on in my statusbar, I have to chose between a certain irregularity in the update frequency (by using "sleep 1") or accepting a pretty high cpu load (I think ~3-4% for my statusbar script). So I thought, what if there was a "snooze" script that clocked the loop and made the system rest for just the right amount of time; for example "snooze 1" would mean that the system would rest until exactly one second had passed since "snooze 1" was last called. Does this make any sense to you? Is there some tool like this, or is it worth looking into?

juster · 2011-08-03 02:53:42

Sounds cool. I imagine you would have to store the time in microseconds that the last call to "snooze" slept at in a file or something. Then you just sleep for last_time+1000000 - now microseconds... avoiding a negative number. I updated the gist to allow for locales and just so you know you can use utf8 in the perl script by prepending at the top:

binmode STDIN, ':utf8';
binmode STDOUT, ':utf8';

Arch Linux

#1 2011-07-28 07:49:41

Nifty scroll script still not fast enough

#2 2011-07-30 21:39:27

Re: Nifty scroll script still not fast enough

#3 2011-07-30 22:21:12

Re: Nifty scroll script still not fast enough

#4 2011-07-31 08:46:59

Re: Nifty scroll script still not fast enough

#5 2011-07-31 17:03:28

Re: Nifty scroll script still not fast enough

#6 2011-07-31 20:52:34

Re: Nifty scroll script still not fast enough

#7 2011-07-31 22:06:51

Re: Nifty scroll script still not fast enough

#8 2011-08-01 12:54:14

Re: Nifty scroll script still not fast enough

#9 2011-08-01 21:15:08

Re: Nifty scroll script still not fast enough

#10 2011-08-01 22:01:38

Re: Nifty scroll script still not fast enough

#11 2011-08-01 23:21:13

Re: Nifty scroll script still not fast enough

#12 2011-08-03 02:53:42

Re: Nifty scroll script still not fast enough

Board footer