Arch Linux

flying sheep

Member

Registered: 2012-02-29

Posts: 93

pygments strangeness

hi, i’m trying to use pygmentize from the python3 pygments package, but have an extremely strange error:

$ python3 /usr/lib/python3.3/site-packages/pygments/cmdline.py   
Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/pygments/cmdline.py", line 17, in <module>
    from pygments.lexers import get_all_lexers, get_lexer_by_name, get_lexer_for_filename, \
  File "/usr/lib/python3.3/site-packages/pygments/lexers/__init__.py", line 18, in <module>
    from pygments.plugin import find_plugin_lexers
  File "/usr/lib/python3.3/site-packages/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python3.3/site-packages/pkg_resources.py", line 21, in <module>
    import imp
  File "/usr/lib/python3.3/imp.py", line 28, in <module>
    import tokenize
  File "/usr/lib/python3.3/tokenize.py", line 38, in <module>
    __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
AttributeError: 'module' object has no attribute '__all__'

ok, so there’s a bug in pygments right? let’s exectute the lines manually and see…

$ head -n 18 /usr/lib/python3.3/site-packages/pygments/cmdline.py
# -*- coding: utf-8 -*-
"""
    pygments.cmdline
    ~~~~~~~~~~~~~~~~

    Command line interface.

    :copyright: Copyright 2006-2013 by the Pygments team, see AUTHORS.
    :license: BSD, see LICENSE for details.
"""
import sys
import getopt
from textwrap import dedent

from pygments import __version__, highlight
from pygments.util import ClassNotFound, OptionError, docstring_headline
from pygments.lexers import get_all_lexers, get_lexer_by_name, get_lexer_for_filename, \
     find_lexer_class, guess_lexer, TextLexer
$ ipython                                                        
Python 3.3.4 (default, Feb 11 2014, 15:56:08) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import sys

In [2]: import getopt

In [3]: from textwrap import dedent

In [4]: 

In [4]: from pygments import __version__, highlight

In [5]: from pygments.util import ClassNotFound, OptionError, docstring_headline

In [6]: from pygments.lexers import get_all_lexers, get_lexer_by_name, get_lexer_for_filename, \
   ...:      find_lexer_class, guess_lexer, TextLexer

huh? everything works. wtf. let’s see that __all__ it’s complaining about

In [7]: import token

In [8]: token.__all__
Out[8]: 
['tok_name',
 'ISTERMINAL',
 'ISNONTERMINAL',
 'ISEOF',
 ...

it’s there. wat? ok, let’s execute it with python -i and use pdb…

$ python3 -i /usr/lib/python3.3/site-packages/pygments/cmdline.py bin/trypy
Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/pygments/cmdline.py", line 17, in <module>
    from pygments.lexers import get_all_lexers, get_lexer_by_name, get_lexer_for_filename, \
  File "/usr/lib/python3.3/site-packages/pygments/lexers/__init__.py", line 18, in <module>
    from pygments.plugin import find_plugin_lexers
  File "/usr/lib/python3.3/site-packages/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python3.3/site-packages/pkg_resources.py", line 21, in <module>
    import imp
  File "/usr/lib/python3.3/imp.py", line 28, in <module>
    import tokenize
  File "/usr/lib/python3.3/tokenize.py", line 38, in <module>
    __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
AttributeError: 'module' object has no attribute '__all__'
>>> import pdb; pdb.post_mortem()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/pdb.py", line 75, in <module>
    import code
  File "/usr/lib/python3.3/code.py", line 9, in <module>
    import traceback
  File "/usr/lib/python3.3/traceback.py", line 3, in <module>
    import linecache
  File "/usr/lib/python3.3/linecache.py", line 10, in <module>
    import tokenize
  File "/usr/lib/python3.3/tokenize.py", line 38, in <module>
    __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
AttributeError: 'module' object has no attribute '__all__'

oh…

well…

$ ls /usr/lib/python3.3/site-packages/pygments/
filters/     lexers/       styles/     console.py  formatter.py  lexer.py   scanner.py  token.py      util.py
formatters/  __pycache__/  cmdline.py  filter.py   __init__.py   plugin.py  style.py    unistring.py

there’s a token.py that obviously ends up in front of sys.path.

damn, so i have to do…

$ ipython                             
Python 3.3.4 (default, Feb 11 2014, 15:56:08) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from pygments.cmdline import main

In [2]: main()
Usage: /usr/bin/ipython [-l <lexer> | -g] [-F <filter>[:<options>]] [-f <formatter>]
          [-O <options>] [-P <option=value>] [-o <outfile>] [<infile>]

       /usr/bin/ipython -S <style> -f <formatter> [-a <arg>] [-O <options>] [-P <option=value>]
       /usr/bin/ipython -L [<which> ...]
       /usr/bin/ipython -N <filename>
       /usr/bin/ipython -H <type> <name>
       /usr/bin/ipython -h | -V
...

damn, that could have been easier.

it would be very nice to have pygmentize on the $PATH to save users from this.

Last edited by flying sheep (2014-03-01 14:25:12)