You are not logged in.

#26 2012-10-22 09:52:32

dickey
Member
Registered: 2009-06-01
Posts: 35

Re: RGB values of current terminal color settings?

Trilby wrote:
Lux Perpetua wrote:
$ echo -en "\033]4;132;?\007"

Dear lord.  That's much easier!  I'll have to remember that one.

putty and konsole do not respond to this sequence.
Also, while the replies from xterm, rxvt, vte (gnome-terminal) are similar, their prefixes/suffixes differ.

Offline

#27 2012-10-22 11:15:20

Lux Perpetua
Member
From: The Local Group
Registered: 2009-02-22
Posts: 69

Re: RGB values of current terminal color settings?

Xyne wrote:

I see that all of the terminal initialization code is in the main routine for now. Would it make sense to encapsulate that in a terminal class and then use that for queries?

That makes a lot of sense, actually, and I restructured my code with a class encapsulating the terminal operations. I believe the best way to handle this in Python is as a context manager: then the client instantiates it via a "with" clause, ensuring that all the proper setup/cleanup code is executed when necessary.  I should probably put this on github so I don't have to keep pasting my code here, but here it is for now:

#!/usr/bin/python

'''
This is a Python script to show off your terminal ANSI colors (or more
colors, if your terminal has them).  It works on both Python 2.7 and
Python 3.

This script must be run from the terminal whose colors you want to
showcase.  Not all terminal types are supported (see below).  At the
very minimum, 16-color support is required. 

Fully supported terminals:
    
    xterm
    urxvt

For these terminals, this script can show a color table with correct RGB
values for each color.  It queries the RGB values from the terminal
itself, so it works even if you change the colors after starting the
terminal.

Mostly supported terminals: pretty much all VTE-based terminals. This
includes:

    vte
    Terminal (XFCE)
    gnome-terminal
    terminator
    tilda

and many more.  These are on "mostly" status because I don't know how to
query their foreground and background colors.  Everything else works,
though, albeit with noticeable slowness (which may be beyond this
script's control).

Somewhat supported terminals: pretty much all other X-client terminals
I've tried.  These include:

    konsole (KDE)
    terminology (Enlightenment)
    Eterm (Enlightenment)
    (etc.)

For these terminals, the script can output a color table just fine, but
without RGB values.

Unsupported terminals:

    ajaxterm
    Linux virtual console (i.e., basic TTY without X-windows)

Warning: do not run this script on the Linux virtual console unless you
want a garbled TTY!  That said, you can still type `tput reset<Enter>'
afterward to get back to a usable console. :-)  The situation with
ajaxterm is similar, but not as bad.

If a terminal isn't mentioned here, I probably haven't tried it.  Attempt
at your own risk!

Note regarding screen/tmux: this script can theoretically be run from a
screen or tmux session, but you will not get any RGB values in the
output (indeed, a screen session can be opened on multiple terminals
simultaneously, so there typically isn't a well defined color value for
a given index).  However, it's interesting to observe that screen and
tmux emulate a 256 color terminal independently of the terminal(s)
to which they are attached, which is very apparent if you run the script
with 256-color output on a screen session attached to a terminal with 8-
or 16-color terminfo (or with $TERM set to such).

This code is licensed under the terms of the GNU General Public License:
    http://www.gnu.org/licenses/gpl-3.0.html

and with absolutely no warranty.  All use is strictly at your own risk.

'''

import os
from sys import stdin, stdout, stderr, version_info
import re
import select
import termios
from collections import defaultdict
from argparse import (ArgumentParser, ArgumentError)


# Operating system command
osc = "\033]"

# String terminator
#  ("\033\\" is another option, but "\007" seems to be understood by
#  more terminals.  Terminology, for example, doesn't seem to like
#  "\033\\".)
st = "\007"

# Control sequence introducer
csi = "\033["

# ANSI SGR0
reset = csi + 'm'


# This is what we expect the terminal's response to a query for a color
# to look like.  If we didn't care about urxvt, we could get away with a
# simpler implementation here, since xterm and vte seem to give pretty 
# consistent and systematic responses.  But I actually use urxvt most of
# the time, so....
ndec = "[0-9]+"
nhex = "[0-9a-fA-F]+"
crgb = ("\033\\]({ndec};)+rgba?:" +
        "({nhex})/({nhex})/({nhex})(/({nhex}))?").format(**vars())

re_response = re.compile(crgb)


#######################################################################
# Query-related error conditions

class TerminalSetupError(Exception):

    '''
    We couldn't set up the terminal properly.

    '''

    def __init__(self, fd):
        Exception.__init__(self, "Couldn't set up terminal on file " +
                           ("descriptor %d" % fd))


class InvalidResponseError(Exception):

    '''
    The terminal's response couldn't be parsed.

    '''

    def __init__(self, q, r):
        Exception.__init__(self, "Couldn't parse response " + repr(r) +
                           " to query " + repr(q))


class NoResponseError(Exception):

    '''
    The terminal didn't respond, or we were too impatient.

    '''

    def __init__(self, q):
        Exception.__init__(self, "Timeout on query " + repr(q))


########################################################################

class TerminalQueryContext(object):

    '''
    Context manager for terminal RGB queries.

    '''

    def __init__(self, fd):
        '''
        fd: open file descriptor referring to the terminal we care
        about.

        '''
        self.tc_save = None
        self.fd = fd

        self.P = select.poll()
        self.P.register(self.fd, select.POLLIN)

        self.num_errors = 0


    def __enter__(self):
        '''
        Set up the terminal for queries.

        '''
        self.tc_save = termios.tcgetattr(self.fd)

        tc = termios.tcgetattr(self.fd)

        # Don't echo the terminal's responses
        tc[3] &= ~termios.ECHO

        # Noncanonical mode (i.e., disable buffering on the terminal
        # level)
        tc[3] &= ~termios.ICANON

        # Make input non-blocking
        tc[6][termios.VMIN] = 0
        tc[6][termios.VTIME] = 0

        termios.tcsetattr(self.fd, termios.TCSANOW, tc)

        # Check if it succeeded
        if termios.tcgetattr(self.fd) != tc:
            termios.tcsetattr(self.fd, termios.TCSANOW, self.tc_save)
            raise TerminalSetupError(self.fd)

        return self


    def __exit__(self, exc_type, exc_value, traceback):
        '''
        Reset the terminal to its original state.

        '''
        self.flush_input()

        if self.tc_save is not None:
            termios.tcsetattr(self.fd, termios.TCSANOW, self.tc_save)


    # Wrappers for xterm & urxvt operating system controls.
    #
    # These codes are all common to xterm and urxvt. Their responses
    # aren't always in the same format (xterm generally being more
    # consistent), but the regular expression used to parse the
    # responses is general enough to work for both.
    #
    # Note: none of these functions is remotely thread-safe.


    def get_fg(self, timeout):
        '''
        Get the terminal's foreground (text) color as a 6-digit
        hexadecimal string.

        '''
        return self.rgb_query([10], timeout)


    def get_bg(self, timeout):
        '''
        Get the terminal's background color as a 6-digit hexadecimal
        string.

        '''
        return self.rgb_query([11], timeout)


    def get_indexed_color(self, a, timeout):
        '''
        Get color a as a 6-digit hexadecimal string.

        '''
        return self.rgb_query([4, a], timeout)


    def test_fg(self, timeout):
        '''
        Return True if the terminal responds to the "get foreground"
        query within the time limit and False otherwise.

        '''
        return self.test_rgb_query([10], timeout)


    def test_bg(self, timeout):
        '''
        Return True if the terminal responds to the "get background"
        query within the time limit and False otherwise.

        '''
        return self.test_rgb_query([11], timeout)


    def test_color(self, timeout):
        '''
        Return True if the terminal responds to the "get color 0" query
        within the time limit and False otherwise.

        '''
        return self.test_rgb_query([4, 0], timeout)


    def test_rgb_query(self, q, timeout):
        '''
        Determine if the terminal supports query q.

        Arguments: `q' and `timeout' have the same interpretation as in
            rgb_query().

        Return: True if the terminal gives a valid response within the
            time limit and False otherwise.

        This function will not raise InvalidResponseError or
        NoResponseError, but any other errors raised by rgb_query will
        be propagated. 

        '''
        try:
            self.rgb_query(q, timeout)
            return True
        except (InvalidResponseError, NoResponseError):
            return False


    def flush_input(self):
        '''
        Discard any input that can be read at this moment.

        '''
        repeat = True
        while repeat:
            evs = self.P.poll(0)
            if len(evs) > 0:
                os.read(self.fd, 4096)
                repeat = True
            else:
                repeat = False


    # The problem I'm attempting to work around with this complicated
    # implementation is that if you supply a terminal with a query that
    # it does not recognize or does not have a good response to, it will
    # simply not respond *at all* rather than signaling the error in any
    # way.  Moreover, there is a large variation in how long terminals
    # take to respond to valid queries, so it's difficult to know
    # whether the terminal has decided not to respond at all or it needs
    # more time.  This is why rgb_query has a user-settable timeout.


    def rgb_query(self, q, timeout=-1):
        '''
        Query a color-valued terminal parameter. 

        Arguments:
            q: The query code as a sequence of nonnegative integers,
                i.e., [q0, q1, ...] if the escape sequence in
                pseudo-Python is

                    "\033]{q0};{q1};...;?\007"

            timeout: how long to wait for a response.  (negative means
                wait indefinitely if necessary)

        Return: the color value as a 6-digit hexadecimal string.

        Errors:
            NoResponseError will be raised if the query times out.

            InvalidResponseError will be raised if the terminal's
            response can't be parsed.

        See 
            http://invisible-island.net/xterm/ctlseqs/ctlseqs.html

        ("Operating System Controls") to see the various queries
        supported by xterm.  Urxvt supports some, but not all, of them,
        and has a number of its own (see man -s7 urxvt). 

        Warning: before calling this function, make sure the terminal is
        in noncanonical, non-blocking mode.  This can be done easily by
        calling self.__enter__() or instantiating this instance in a
        "with" clause, which will do that automatically.

        '''
        query = osc + ';'.join([str(k) for k in q]) + ';?' + st

        self.flush_input()
        os.write(self.fd, query.encode())

        # This is addmittedly flawed, since it assumes the entire
        # response will appear in one shot.  It seems to work in
        # practice, though.

        evs = self.P.poll(timeout)
        if len(evs) == 0:
            self.num_errors += 1
            raise NoResponseError(query)

        r = os.read(self.fd, 4096)
        if version_info.major >= 3:
            r = r.decode()

        m = re_response.search(r)

        if not m:
            self.num_errors += 1
            raise InvalidResponseError(query, r)

        # (possibly overkill, since I've never seen anything but 4-digit
        # RGB components in responses from terminals, in which case `nd'
        # is 4 and `u' is 0xffff, and the following can be simplified as
        # well (and parse_component can be eliminated))
        nd = len(m.group(2))
        u = (1 << (nd << 2)) - 1

        # An "rgba"-type reply (for urxvt) is apparently actually
        #
        #    rgba:{alpha}/{alpha * red}/{alpha * green}/{alpha * blue}
        #
        # I opt to extract the actual RGB values by eliminating alpha.
        # (In other words, the alpha value is discarded completely in
        # the reported color value, which is a compromise I make in
        # order to get an intuitive and compact output.)

        if m.group(5):
            # There is an alpha component
            alpha = float(int(m.group(2), 16))/u
            idx = [3, 4, 6]
        else:
            # There is no alpha component
            alpha = 1.0
            idx = [2, 3, 4]

        c_fmt = '%0' + ('%d' % nd) + 'x'

        components = [int(m.group(i), 16) for i in idx]
        t = tuple(parse_component(c_fmt % (c/alpha)) 
                  for c in components)

        return "%02X%02X%02X" % t


    def test_num_colors(self, timeout):
        '''
        Attempt to determine the number of colors we are able to query
        from the terminal.  timeout is measured in milliseconds and has
        the same interpretation as in rgb_query.  A larger timeout is
        safer but will cause this function to take proportionally more
        time.

        '''
        # We won't count failed queries in this function, since we're
        # guaranteed to fail a few.
        num_errors = self.num_errors

        if not self.test_color(timeout):
            return 0
        
        a = 0
        b = 1
        while self.test_rgb_query([4, b], timeout):
            a = b
            b += b

        while b - a > 1:
            c = (a + b)>>1
            if self.test_rgb_query([4, c], timeout):
                a = c
            else:
                b = c

        self.num_errors = num_errors
        return b



def parse_component(s):
    '''
    Take a string representation of a hexadecimal integer and transorm
    the two most significant digits into an actual integer (or double
    the string if it has only one character).

    '''
    n = len(s)

    if n == 1:
        s += s
    elif n > 2:
        s = s[:2]

    return int(s, 16)


########################################################################

class ColorDisplay(TerminalQueryContext):

    '''
    Class for producing a colored display of terminal RGB values.  It's
    best to use this class as a context manager, which will properly set
    and reset the terminal's attributes.

    '''

    def __init__(self, tty_fd,
                 timeout=100, color_level=3, do_query=True):
        '''
        tty_fd: open file descriptor connected to a terminal.

        timeout: same interpretation as in rgb_query. A larger timeout
            will be used a small number of times to test the
            capabilities of the terminal.

        color_level: how much color should be in the output. Use 0 to
            suppress all color and 3 or greater for maximum coloredness.

        do_query: whether to attempt to query RGB values from the
            terminal or just use placeholders everywhere

        '''
        TerminalQueryContext.__init__(self, tty_fd)

        self.timeout = timeout
        self.color_level = color_level
        self.do_query = do_query

        def none_factory():
            return None

        # colors for highlighting
        self.hi = defaultdict(none_factory)

        self.hi['['] = 10
        self.hi[']'] = 10
        self.hi['+'] = 9
        self.hi['/'] = 9

        for c in '0123456789ABCDEF':
            self.hi[c] = 12

        # String to use for color values that couldn't be determined
        self.rgb_placeholder = '??????'


    def __enter__(self):
        TerminalQueryContext.__enter__(self)

        # try getting the rgb value for color 0 to decide whether to
        # bother trying to query any more colors.
        self.do_query = self.do_query and self.test_color(self.timeout*5)

        if self.color_level >= 1:
            stdout.write(reset)

        return self


    def __exit__(self, exc_type, exc_value, traceback):
        if self.color_level >= 1:
            stdout.write(reset)

        TerminalQueryContext.__exit__(self, exc_type, exc_value,
                                      traceback)


    def show_fgbg(self):
        '''
        Show the foreground and background colors.

        '''
        if self.do_query:
            try:
                bg = self.get_bg(timeout=self.timeout)
            except (InvalidResponseError, NoResponseError):
                bg = self.rgb_placeholder

            try:
                fg = self.get_fg(timeout=self.timeout)
            except (InvalidResponseError, NoResponseError):
                fg = self.rgb_placeholder
        else:
            bg = self.rgb_placeholder
            fg = self.rgb_placeholder

        stdout.write("\n    Background: %s\n" % bg)
        stdout.write("    Foreground: %s\n\n" % fg)


    def show_ansi(self):
        '''
        Show the 16 ANSI colors (colors 0-15).

        '''
        color_order = [0, 1, 3, 2, 6, 4, 5, 7]

        names = ['   Black ', '    Red  ', '   Green ', '  Yellow ',
                 '   Blue  ', '  Magenta', '   Cyan  ', '   White ']

        stdout.write(self.fgcolor('15', 3))

        for k in range(8):
            a = color_order[k]
            stdout.write(names[a])

        stdout.write('\n')
        stdout.write(self.fgcolor(None, 3))

        c = None
        for k in range(8):
            a = color_order[k]
            c = self.hiprint('   [%X/%X] ' % (a, 8 + a), c)
        stdout.write('\n')

        self.show_color_table([0,8], color_order)


    def show_color_cube(self, n):
        '''
        Show the "RGB cube" (xterm colors 16-231 (256-color) or 16-79
        (88-color)).  The cube has sides of length 6 or 4 (for 256-color
        or 88-color, respectively).

        '''
        base = {256:6, 88:4}[n]

        c = None
        c = self.hiprint('[ + ]   ', c)
        for w in range(base):
            c = self.hiprint('[%X]      ' % w, c)
        stdout.write('\n\n' + self.fgcolor(None, 3))

        for u in range(base):
            for v in range(base):
                stdout.write(' '*v)

                x = (u*base + v)*base
                self.hiprint('  [%02X]  ' % (16 + x))
                stdout.write(self.fgcolor(None, 3))

                for w in range(base):
                    self.show_color(x + w + 16)
                stdout.write('\n')
            stdout.write('\n\n')


    def show_grayscale_ramp(self, end):
        '''
        Show the "grayscale ramp" (xterm colors 232-255 (256-color) or
        80-87 (88-color)).

        '''
        start = {256:232, 88:80}[end]
        n = end - start

        vals = [self.get_color(a) for a in range(start, end)]

        #stdout.write(reset)
        c = None

        c = self.hiprint('[ ', c)
        for v in range(n):
            c = self.hiprint('%02X ' % (start + v), c)
        c = self.hiprint(']\n', c)

        stdout.write('\n ' + self.fgcolor(None, 3))

        for v in range(n):
            stdout.write(' ' + self.block(start + v, 2))
        stdout.write('\n ')

        for u in range(3):
            for v in range(n):
                stdout.write(' ')
                stdout.write(self.fgcolor(start + v, 2))
                stdout.write(vals[v][2*u : 2*(u + 1)])
                stdout.write(self.fgcolor(None, 2))
            stdout.write('\n ')
        stdout.write('\n')


    def show_colors(self, n):
        '''
        Make a table showing colors 0 through n-1.

        '''
        self.show_color_table(range(0,n,8), range(8), n, True)


    def show_color_table(self, rows, cols, stop=-1, label=False):
        '''
        Make a color table with all possible color indices of the form
        rows[k] + cols[j] that are less than `stop' (if `stop' is not
        negative). If label is True, then print row and column labels.

        '''
        if label:
            self.hiprint('[ + ]')
            stdout.write(self.fgcolor(None, 3))

            for a in cols:
                stdout.write('   ' + self.octal(a) + '  ')
            stdout.write('\n' + self.fgcolor(None, 1))

        if label:
            stdout.write('     ')

        stdout.write('\n')

        for b in rows:
            if label:
                stdout.write(self.octal(b) + ' ' +
                             self.fgcolor(None, 1))

            for a in cols:
                c = a + b
                if stop < 0 or c < stop:
                    self.show_color(b + a)
                else:
                    stdout.write('         ')
            stdout.write('\n')
        stdout.write('\n')


    def show_color(self, a):
        '''
        Make a pretty display of color number `a', showing a block of
        that color followed by the 6-character hexadecimal code for the
        color.

        '''
        stdout.write(' ' + self.block(a) + ' ')
        stdout.write(self.fgcolor(a, 2) + (self.get_color(a)))
        stdout.write(self.fgcolor(None, 2))


    def hiprint(self, s, last_color=-1):
        '''
        Print s to stdout, highlighting digits, brackets, etc. if the
        color level allows it.

        Arguments:
            s: the string to print.

            last_color: the current terminal foreground color.  This
                should be `None' if no color is set, or the current
                color index, or something else (like a negative integer)
                if the color isn't known.  (The last option is always
                safe and will force this function to do the right
                thing.)

        Return: the current foreground color, which can be passed as
            last_color to the next call if the color isn't changed in
            between.

        '''
        for c in s:
            if c == ' ':
                color = last_color
            else:
                color = self.hi[c]

            if color != last_color:
                stdout.write(self.fgcolor(color, 3))

            stdout.write(c)
            last_color = color

        return last_color


    def octal(self, x):
        '''
        Return a base-8 string for the integer x, highlighted if the
        color level allows it.

        '''
        return self.fgcolor(self.hi['+'], 3) + '0' + \
               self.fgcolor(self.hi['0'], 3) + ('%03o' % x)


    def block(self, c, n=1):
        '''
        Return a string that prints as a block of color `c' and size `n'.

        '''
        return self.bgcolor(c, 1) + ' '*n + self.bgcolor(None, 1)


    # Changing the foreground and background colors.
    #
    # While the 38;5 and 48;5 SGR codes are less portable than the usual
    # 30-37 and 40-47, these codes seem to be fairly widely implemented (on
    # X-windows terminals, screen, and tmux) and support the whole color
    # range, as opposed to just colors 0-8.  They also make it very easy to
    # set the background to a given color without needing to mess around
    # with bold or reverse video (which are hardly portable themselves).
    # This is useful even for the 16 ANSI colors.


    def fgcolor(self, a=None, level=-1):
        '''
        Return a string designed to set the foreground color to `a' when 
        printed to the terminal. None means default.

        '''
        if self.color_level >= level:
            if a is None:
                return csi + '39m'
            else:
                return csi + '38;5;' + str(a) + 'm'
        else:
            return ''


    def bgcolor(self, a=None, level=-1):
        '''
        Return a string designed to set the background color to `a' when 
        printed to the terminal. None means default.

        '''
        if self.color_level >= level:
            if a is None:
                return csi + '49m'
            else:
                return csi + '48;5;' + str(a) + 'm'
        else:
            return ''


    def get_color(self, a):
        if self.do_query:
            try:
                return self.get_indexed_color(a, timeout=self.timeout)
            except (InvalidResponseError, NoResponseError):
                return self.rgb_placeholder
        else:
            return self.rgb_placeholder


########################################################################
# Command-line arguments

timeout_dft = 200

parser = ArgumentParser(
        description="Python script to show off terminal colors.",
        epilog="Run this script from the terminal whose colors " +
               "you want to showcase.  " +
               "For a brief synopsis of which terminal types are " +
               "supported, see the top of the source code.")

mode_group = parser.add_mutually_exclusive_group()

p_choices = [16, 88, 256]

arg_p = mode_group.add_argument(
        '-p', '--pretty',
        action='store_true', default=False,
        help="show colors 0 through N-1 in a pretty format.  " +
             ("N must belong to %r.  " % p_choices) +
             "If N > 16, it should be the actual number of colors " +
             "supported by the terminal, or the output will almost " +
             "certainly not be pretty.")

mode_group.add_argument(
        '-f', '--flat',
        action='store_true', default=False,
        help="show a simple table with colors 0 through N-1.  ")

parser.add_argument(
        'n', nargs='?', metavar='N',
        type=int, default=16,
        help="number of colors to show.  " +
             "Unless you explicitly supply -p/--pretty or -f/--flat, " +
             "--pretty is used if possible and --flat is used " +
             "otherwise.  " +
             "N defaults to 16, showing the ANSI colors 0-15.  " +
             "If N is 0, the script will attempt to determine the " +
             "maximum number of colors automatically " +
             "(which may be slow).")

parser.add_argument(
        '--no-fgbg',
        action='store_false', dest='fgbg', default=True,
        help="suppress display of foreground/background colors.")

parser.add_argument(
        '--no-query',
        action='store_false', dest='do_query', default=True,
        help="don't try to query any RGB values from the terminal " +
             "and just use placeholders.")

parser.add_argument(
        '-t', '--timeout', metavar='T',
        type=int, default=timeout_dft,
        help="how long to wait for the terminal to "
             "respond to a query, in milliseconds  " +
             "[default: {0}].  ".format(timeout_dft) +
             "If your output has '?' characters " +
             "instead of RGB values " +
             "or junk printed after the script runs, " +
             "increasing this value may or may not " +
             "help, depending on the terminal.  " +
             "A negative T will behave like infinity.")

parser.add_argument(
        '-l', '--level', metavar='L',
        type=int, default=3,
        help="choose how much color to use in the output.  " +
             "(0 = no color; 3 = most color [default])")


########################################################################

def color_display(*args):
    return ColorDisplay(*args)


if __name__ == '__main__':
    args = parser.parse_args()

    assert not (args.pretty and args.flat)

    if args.pretty:
        if args.n not in p_choices:
            raise ArgumentError(
                    arg_p,
                    "N must belong to %r" % p_choices)

    with ColorDisplay(0, args.timeout, args.level, args.do_query) as C:
        if args.n == 0:
            args.n = C.test_num_colors(args.timeout)

        if not (args.pretty or args.flat):
            if args.n in p_choices:
                args.pretty = True
            else:
                args.flat = True

        if args.level >= 1:
            stdout.write(reset)

        if args.fgbg:
            C.show_fgbg()

        if args.pretty:
            assert args.n in p_choices

            stdout.write('\n    ANSI colors:\n\n')
            C.show_ansi()

            if args.n > 16:
                stdout.write('\n    RGB cube:\n\n')
                C.show_color_cube(args.n)

                stdout.write('    Grayscale ramp:\n\n')
                C.show_grayscale_ramp(args.n)
        else:
            C.show_colors(args.n)

        if C.num_errors > 0:
            stderr.write("Warning: not all queries succeeded\n" +
                         "Warning:     (output contains " + 
                         "placeholders and may be inaccurate)\n")
Xyne wrote:

My main interest is converting to and from RGB values. With the above code I can get all available term colors and their matching rgb values. I thought I could use

\033[38;2;<r>;<g>;<b>m

to go the other way, but it doesn't seem to work. I may well be doing something wrong. It's not a show-stopper as I can use vector projections to go the other way. I already have code for that.

Your escape sequence seems to work on xterm but not urxvt. Unfortunately, each terminal pretty much seems to do its own thing with its escape sequences.

Xyne wrote:

I don't fully understand what's going on with the polling. I noticed the comments in rgb_query function and I've restructured the code to try to address the issue in ansi_to_rbg above. I am may well be messing with something that I shouldn't.

Well, you certainly seem to have addressed the issue of the response not being available in a single read. I don't know if that's a realistic situation or not (which is why I wasn't terribly concerned), but you certainly didn't make it any less robust.

Xyne wrote:

My alternative solution avoids string conversions, if you're interested:

  width = len(m.group(3))
  base = float(0x10**width-1)

Yes, I see I was converting it immediately from str to int, so your version is better in a way. Just to be cute, I rewrote the expression as

(1 << (width << 2)) - 1

(And now it's probably worse than my original version. Look what you made me do! smile)

dickey wrote:

putty and konsole do not respond to this sequence.
Also, while the replies from xterm, rxvt, vte (gnome-terminal) are similar, their prefixes/suffixes differ.

Yep, as I mentioned, each terminal pretty much does its own thing. I doubt there's any clean and truly portable solution, and I suspect that the terminals that don't respond to such sequences don't allow their colors to be queried, period. Even the ones that try to emulate xterm to some degree, like urxvt, don't seem to do it consistently. For example, with that sequence "\033]4;132;?\007", xterm replies "\033]4;132;rgb:afaf/5f5f/8787\007", while urxvt replies "\033]4;rgb:afaf/5f5f/8787\007". xterm's reply follows some kind of logic, since it is the exact escape sequence you could use to set that color to that RGB value. (Note: all the colors on a running xterm can be changed dynamically.) urxvt's response looks roughly similar but clearly doesn't use that logic.

Offline

#28 2012-10-22 16:53:11

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: RGB values of current terminal color settings?

Lux Perpetua wrote:

That makes a lot of sense, actually, and I restructured my code with a class encapsulating the terminal operations. I believe the best way to handle this in Python is as a context manager: then the client instantiates it via a "with" clause, ensuring that all the proper setup/cleanup code is executed when necessary.  I should probably put this on github so I don't have to keep pasting my code here, but here it is for now:

I really like the new approach. I hope that you will move the TerminalQueryContext class to a separate module and package it (or at least host it somewhere so that I can package it).

It gave me some ideas too. The global variables at the top for the different escape codes and regular expressions could be moved into the class. This would facilitate subclassing TerminalQueryContext for different terminals.

As the user will want to automatically detect the terminal in most cases, the following globals could be added:

# This would be instantiated as a dictionary mapping valid $TERM strings for
# recognized terminals to their corresponding subclasses.
#
# For example, all of the urxvt and xterm $TERM strings would map to
# TerminalQueryContext
terms = dict()

def register_terminal(term, cls):
  """
  Register a terminal string and associate it with a TerminalQueryContext
  (sub)class.
  """
  global terms
  terms[term] = cls

def get_terminal_query_context(term=None, default=None):
  """
  Return the matching TerminalQueryContext for the given term.

  if term is None:
    term = os.getenv('TERM')
  """

  try:
    return terms[term]
  except KeyError:
    # An alternative approach to the parameterized default would be to set
    # terms[None] and/or terms[''].
    if default:
      # maybe warn the user
      return default
    else:
      # raise exception
  """

The reasoning behind this approach is that it would make it easy to subclass TerminalQueryContext in the user's own code and still have it detected globally. A subclass declaration would look something like this:

class MyObscureTerminalQueryContext(TerminalQueryContext):
  ...

# These are possible $TERM values for this terminal context.
# This will enable automatic detection of the terminal using the $TERM
# environment variable via the global get_terminal_query_context function.
for term in ('Footerm', 'Footerm-256', 'Footerm-mod'):
  register_terminal(term, MyObscureTerminalQueryContext)


I think it makes sense to have a function that returns an RGB triple as well. colorsys and other modules tend to use RGB as the chromatic lingua franca for conversion functions. The current query function uses intermediate RGB values, so you could just use the end of my ansi_to_rgb function and then create a wrapper around that to return the hex string:

  width = len(m.group(2))
  base = float(0x10**width-1)

  # Group 5 is only present if there is an alpha channel. The value apparently
  # precedes the RGB values.
  if m.group(5):
    rgb_indices = [3, 4, 6]
  else:
    rgb_indices = [2, 3, 4]

  return tuple(int(m.group(i), 16)/base for i in rgb_indices)

The only issue with the wrapper may be the number of characters for each field, but that can be made configurable via a parameter. In most cases the desired width will probably be a function of the application, not the terminal (e.g. even if the terminal returns rrrr/gggg/bbbb,  you may only want #rrggbb).



I still haven't looked into what the polling is actually doing, but given that this is now a context, can it be assumed that the query function is the only thing that will be generating output from stdout? If so, would it suffice to flush stdout once on __enter__? Or is there a risk that something might be there even with my method above for collecting all of the output after a query?



The following are just a few things that I have done differently. I submit them for consideration and critique. Your Python-fu is clearly stronger than mine so just take the following as a friendly "hey, would this work?".

Could

    def flush_input(self):
        '''
        Discard any input that can be read at this moment.
        '''
        repeat = True
        while repeat:
            evs = self.P.poll(0)
            if len(evs) > 0:
                os.read(self.fd, 4096)
                repeat = True
            else:
                repeat = False

be changed to the following?

    def flush_input(self):
        '''
        Discard any input that can be read at this moment.
        '''
        while self.P.poll(0):
            os.read(self.fd, 4096)

Incidentally, I use the same direct boolean evaluation in my adapted query function to avoid the extra call to len().


I don't understand what test_num_colors is doing either. I use the equivalent of the following to determine the number of colors (I also collect the RGB values into a list at the same time (index=term color, value=RGB tuple), but have omitted that for simplicity):

i = 0
while query([4, i], timeout=1000)
  i += 1

I get the expected 256 for urxvt and an unexpected 260 for xterm. I have tested the resulting colors and they all work as expected. Does this fail for some pathological case?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#29 2012-10-22 17:26:24

dickey
Member
Registered: 2009-06-01
Posts: 35

Re: RGB values of current terminal color settings?

Xyne wrote:
i = 0
while query([4, i], timeout=1000)
  i += 1

I get the expected 256 for urxvt and an unexpected 260 for xterm. I have tested the resulting colors and they all work as expected. Does this fail for some pathological case?

That sounds right.  xterm assigns the next 4 colors to colorBD (bold), colorUL (underline), colorBL (blink) and colorRV (reverse).

Offline

#30 2012-10-23 11:45:03

Lux Perpetua
Member
From: The Local Group
Registered: 2009-02-22
Posts: 69

Re: RGB values of current terminal color settings?

Xyne, I incorporated a bunch of your suggestions, which were all good. The current state of it is here on my github page: https://github.com/dranjan/termcolors The TerminalQueryContext class is in its own file: https://github.com/dranjan/termcolors/b … l_query.py Regarding subclassing it for specific terminals: in theory, this sounds good, and I think the current version is somewhat amenable to this.  That said, I don't actually know of any terminals whose colors can be queried and which don't mostly support the xterm-style queries implemented by TerminalQueryContext.

Xyne wrote:

I still haven't looked into what the polling is actually doing, but given that this is now a context, can it be assumed that the query function is the only thing that will be generating output from stdout? If so, would it suffice to flush stdout once on __enter__? Or is there a risk that something might be there even with my method above for collecting all of the output after a query?

If we're talking about reading a reply to a query and the issue of junk already being there on stdin: in short, yes, I think your suggestion is reasonable, mainly because the "flush_input()" call in rgb_query doesn't work well for its intended purpose anyway.  The intended purpose: if the timeout is too small, it's possible to miss a valid reply completely.  Then that reply is there on the terminal when we do the next query, and ideally we'd like to discard that missed reply so we have a chance of receiving the correct reply to the new query.  My "flush_input" method for flushing the old characters out doesn't really work well, though.  I'm guessing this happens because if the timeout is small enough and the terminal is slow enough, it's possible that we time out of one query, go to do the next query, and call "flush_input" all before the terminal gets around to replying to the first query.  Since the flushing doesn't really accomplish what it's supposed to do anyway, I think it's okay to get rid of it.  Flushing once before the whole process is probably a good idea, though, since we have no control over what happened to the terminal before this code started.

I have an idea for modifying the query function to alleviate these timeout issues a bit, but (surprise) it's not extremely portable.  The idea is to take a query string that most terminals "should" respond to in a predictable way, a possible example being "\033[6n" ("device status report").  Then immediately send this "safe" query string after every "questionable" query.  Now we expect the terminal to reply even if the questionable query fails, and it's possible to test for failure when the reply comes.  Unfortunately, not every terminal responds to "\033[6n", but a lot of them do.

Xyne wrote:

I don't understand what test_num_colors is doing either. I use the equivalent of the following to determine the number of colors (I also collect the RGB values into a list at the same time (index=term color, value=RGB tuple), but have omitted that for simplicity):

i = 0
while query([4, i], timeout=1000)
  i += 1

My version gets the number of available colors without necessarily querying every single color.  If the terminal has N colors, your version does N+1 queries, while mine does about 2*ceiling(log_2(N))+1.  If N=256, that's 257 queries for your version vs. 17 queries for mine, which will query colors 0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 192, 224, 240, 248, 252, 254, and 255, in that order.  This optimization is only beneficial if you don't want the color values, just the number of colors, so since you're saving and using the color values, your version is preferable for that use.

Offline

#31 2012-10-23 18:07:06

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: RGB values of current terminal color settings?

Lux Perpetua wrote:

Xyne, I incorporated a bunch of your suggestions, which were all good. The current state of it is here on my github page: https://github.com/dranjan/termcolors The TerminalQueryContext class is in its own file: https://github.com/dranjan/termcolors/b … l_query.py Regarding subclassing it for specific terminals: in theory, this sounds good, and I think the current version is somewhat amenable to this.  That said, I don't actually know of any terminals whose colors can be queried and which don't mostly support the xterm-style queries implemented by TerminalQueryContext.

Nice, thanks! I'll try to package it tonight (no promises, I'm tired and likely to get sidetracked).

Regarding subclassing, the current code is fine. I think the global functions that I suggested would make sense, but these can easily be provided elsewhere along with default mappings of $TERM strings to the base class for terminals that it supports.

Lux Perpetua wrote:

I have an idea for modifying the query function to alleviate these timeout issues a bit, but (surprise) it's not extremely portable.  The idea is to take a query string that most terminals "should" respond to in a predictable way, a possible example being "\033[6n" ("device status report").  Then immediately send this "safe" query string after every "questionable" query.  Now we expect the terminal to reply even if the questionable query fails, and it's possible to test for failure when the reply comes.  Unfortunately, not every terminal responds to "\033[6n", but a lot of them do.

I had the same idea as I was reading the paragraph just before this one.  You could then flush once on entry and then disable the timeout for each query as it would always return some fixed reply afterwards. This comes back to the idea of explicitly supporting terminals that are known to work. You could create a separate "query" method that could be overwritten in subclasses. The query method would accept a query, input it, input the safe command (relative to the terminal), then read everything up to the safe response without a timeout. It could then return the output to the given command if there was any, otherwise None.

Whether or not it would be necessary to flush stdout at the beginning of the method prior to the query would depend on the context. The method could simply accept an additional parameter (e.g. "flush=False").

Lux Perpetua wrote:

My version gets the number of available colors without necessarily querying every single color.  If the terminal has N colors, your version does N+1 queries, while mine does about 2*ceiling(log_2(N))+1.  If N=256, that's 257 queries for your version vs. 17 queries for mine, which will query colors 0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 192, 224, 240, 248, 252, 254, and 255, in that order.  This optimization is only beneficial if you don't want the color values, just the number of colors, so since you're saving and using the color values, your version is preferable for that use.

Clever. I should have looked at it longer. hmm



edit
I have uploaded PKGBUILDs and setup.py files for two packages here. The are still incomplete.

The setup files should be included upstream so that you can manage the internal versions. As for overall file organization, I do not know what you would prefer. From a packaging standpoint it would be convenient if terminal_query were in its own repo or equivalent. My git-fu is weak, but there seems to be various ways to do this, such as git-submodule, which also mentions subtrees (not sure how they work, but may be relevant). As I understand it, that would allow you to develop everything in the same place instead of jumping back and forth between independent repos.

Regardless of whether terminal_query is moved into its own repo and packaged separately, I think termcolors should be a proper Python package with its own module hierarchy..  The preliminary setup file for termcolors expects that.

Check the details in each file (in particular, license, author, author email, description) and update as necessary. Once you know how you want to organize upstream, you or I can submit the updated PKGBUILDs to the AUR.

Last edited by Xyne (2012-10-23 20:28:28)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#32 2012-10-24 10:28:24

Lux Perpetua
Member
From: The Local Group
Registered: 2009-02-22
Posts: 69

Re: RGB values of current terminal color settings?

Xyne wrote:

I had the same idea as I was reading the paragraph just before this one.  You could then flush once on entry and then disable the timeout for each query as it would always return some fixed reply afterwards. This comes back to the idea of explicitly supporting terminals that are known to work. You could create a separate "query" method that could be overwritten in subclasses. The query method would accept a query, input it, input the safe command (relative to the terminal), then read everything up to the safe response without a timeout. It could then return the output to the given command if there was any, otherwise None.

Whether or not it would be necessary to flush stdout at the beginning of the method prior to the query would depend on the context. The method could simply accept an additional parameter (e.g. "flush=False").

This is pretty close to what I implemented.  I added flush=True as a parameter to the main query function (guarded_query).  I can't think of many situations where one would definitely not want to flush the input (besides reducing the number of polls, but I imagine this is very cheap compared to generating responses from a typical terminal), but the option is there.

Here's the main part of the updated code.  (I took out some of the comments for this forum post, but they're still there in the repo: https://github.com/dranjan/termcolors)

    ndec = "[0-9]+"
    nhex = "[0-9a-fA-F]+"

    # The "guard" query and its response pattern
    q_guard = csi + "6n"

    str_guard = "(.*)\033\\[{ndec};{ndec}R".format(**vars())
    re_guard = re.compile(str_guard)

    str_rgb = ("\033\\]({ndec};)+rgba?:(({nhex})/)?" +
               "({nhex})/({nhex})/({nhex})").format(**vars())

    re_rgb = re.compile(str_rgb)
    
    
    def rgb_query(self, q, timeout=-1):
        query = (self.osc +
                 ';'.join([str(k) for k in q]) + ';?' +
                 self.st)

        try:
            response = self.guarded_query(query, timeout)
        except NoResponseError:
            return None

        m = self.re_rgb.match(response)

        if not m:
            self.num_errors += 1
            return None

        nd = len(m.group(4))
        u = (1 << (nd << 2)) - 1

        alpha = float(int(m.group(3), 16))/u if m.group(3) else 1.0

        return RGBColor(*tuple(int(m.group(i), 16)/(alpha*u) 
                               for i in [4, 5, 6]))


    def guarded_query(self, q, timeout=-1, flush=True):
        '''
        Send the terminal query string `q' and return the terminal's
        response.

        Arguments:
            q: the query string to send to the terminal.

            timeout: how many milliseconds to wait for a response, a
                negative number meaning "infinite".  

            flush: whether to discard waiting input before sending the
                query.  It usually makes sense to do this, but note that
                the terminal may still send seemingly unsolicited data
                (possibly in response to an earlier query) after the
                input is flushed, so flushing the input does not
                provide any guarantees.

        Return: The terminal's response to the query as a string.

        Errors:
            NoResponseError will be raised if the query times out.

            TerminalUninitializedError will be raised if this instance's
            context has not been __enter__-ed.

        '''
        if not hasattr(self, "P"):
            raise TerminalUninitializedError(self.fd)

        query = q + self.q_guard

        if flush:
            self.flush_input()

        os.write(self.fd, query.encode())

        response = ""

        while self.P.poll(timeout):
            s = os.read(self.fd, 4096)
            if version_info.major >= 3:
                s = s.decode()
            response += s

            m = self.re_guard.match(response)

            if m:
                return m.group(1)
        else:
            self.num_errors += 1
            raise NoResponseError(query)

I haven't figured out the whole packaging aspect to this yet, but I'm on board.  I consider this still at the stage where if people were using this, each commit would break everybody's code, but I guess proper versioning deals with this.  I'll look at your PKGBUILDs and setup.py files properly when I have a bit more time (soon).

Offline

#33 2012-10-24 11:03:29

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: RGB values of current terminal color settings?

Lux Perpetua wrote:

This is pretty close to what I implemented.  I added flush=True as a parameter to the main query function (guarded_query).  I can't think of many situations where one would definitely not want to flush the input (besides reducing the number of polls, but I imagine this is very cheap compared to generating responses from a typical terminal), but the option is there.

I like the way that you've implemented it and it looks very easy to subclass.

I thought that flush=False would be the norm as most calls would be made within the context, which would flush once on entry. The guarded_query method should then collect all new output in response to each query so that the output is empty for the next one in the context.


Lux Perpetua wrote:

I haven't figured out the whole packaging aspect to this yet, but I'm on board.  I consider this still at the stage where if people were using this, each commit would break everybody's code, but I guess proper versioning deals with this.  I'll look at your PKGBUILDs and setup.py files properly when I have a bit more time (soon).

There's no rush on the packaging (or the rest of the code for that matter), so deal with it when you have the time and inclination and let me know what I can do to help. Thanks for all your work so far. I've really enjoyed discussing the code and different approaches btw.

I wouldn't worry about breakage too much, especially not for *-git packages. Just add a caveat in the README clarifying that the code is still experimental and that backwards compatibility is not a concern (at the moment). If stability is important, someone could include the necessary modules in their own package, which I may do for the colorsys package, provided that you have no objections.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#34 2012-10-28 00:36:10

Lux Perpetua
Member
From: The Local Group
Registered: 2009-02-22
Posts: 69

Re: RGB values of current terminal color settings?

Xyne wrote:

There's no rush on the packaging (or the rest of the code for that matter), so deal with it when you have the time and inclination and let me know what I can do to help. Thanks for all your work so far. I've really enjoyed discussing the code and different approaches btw.

I wouldn't worry about breakage too much, especially not for *-git packages. Just add a caveat in the README clarifying that the code is still experimental and that backwards compatibility is not a concern (at the moment). If stability is important, someone could include the necessary modules in their own package, which I may do for the colorsys package, provided that you have no objections.

It has been and continues to be a fun project, and the code has improved a lot through our discussion.  I certainly have no objections to your including any of this in your own packages, and it's all officially GPL-licensed now.  (It was before, but it wasn't clearly indicated.)

Offline

Board footer

Powered by FluxBB