Arch Linux

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Lost in python..

I'm not a python programmer but i have a problem with a plugin I'm trying to solve.. I can't quite navigate the syntax but perhaps the solution is easy to one who knows the language... here it the snippet in question.

        hooray = os.popen("find " + location + " -type f -not -name '.svn*'").readlines()
        for hip in hooray:
          string += " '%s'" % hip[:-1]
        
        # str_case_operator will hold the "case insensitive" command if necessary
        str_case_operator = ""
        if (not self.case_sensitive):
            str_case_operator = " -i"

        # Create a pipe and call the grep command, then read it            
        pipe = os.popen("grep -n -H" + str_case_operator + " %s %s" % (self.search_form.get_text(), string))
        data = pipe.read()
        results = data.split("\n")

The problem in the above code is that, while it works for the most part, certain characters cause it to barf.  Specifically any special shell characters like <>.  What I'd like to do is escape those characters.. I believe the os.popen commands need to be written using submodules or popen3 but I keep getting weird errors like "AttributeError: 'tuple' object has no attribute 'read'" or "TypeError: popen() argument 1 must be string, not list"

Can anyone provide any pointers?

Thanks in advance..

Trent

Member

From: Baltimore, MD (US)

Registered: 2009-04-16

Posts: 990

Re: Lost in python..

Hmm, I recently went through a similar issue, but I can't seem to find it right now.  You probably want to use the subprocess module to replace os.popen, and construct it with an argument list.  Something like

import subprocess
cmd=['find', location, '-type', 'f', '-not', '-name', '*.svn']
p = subprocess.Popen(cmd)
p.wait()
hooray=p.stdout.readlines()

Untested, but it should get you started.

EDIT: forgot to add my reference material: http://docs.python.org/library/subprocess.html

Last edited by Trent (2009-06-25 01:53:30)

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Re: Lost in python..

I get an error that says

hooray=p.stdout.readlines()
     AttributeError: 'NoneType' object has no attribute 'readlines'

But shouldn't we be modifying the popen that has the grep command in it?  I think that's the one where it searches through the files found by the find command and searches for the string in question.

ralvez

Member

From: Canada

Registered: 2005-12-06

Posts: 1,718

Website

Re: Lost in python..

A rough work around could be:

try:
   hooray=p.stdout.readlines()
except AttributeError:
   pass

That piece of code would just catch and ignore  that particular error, therefore, you would not get the
AttributeError: 'NoneType' object has no attribute 'readlines'

R.

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Re: Lost in python..

The code now looks like this:

 
        cmd=['find', location, '-type', 'f', '-not', '-name', '*.svn']
        p = subprocess.Popen(cmd)
        p.wait()
        try:
          hooray=p.stdout.readlines()
        except AttributeError:
          pass

        for hip in hooray:
          string += " '%s'" % hip[:-1]
        
        # str_case_operator will hold the "case insensitive" command if necessary
        str_case_operator = ""
        if (not self.case_sensitive):
            str_case_operator = " -i"

        # Create a pipe and call the grep command, then read it            
        pipe = os.popen("grep -n -H" + str_case_operator + " %s %s" % (self.search_form.get_text(), string))
        data = pipe.read()
        results = data.split("\n")

and gives this error:

for hip in hooray:
UnboundLocalError: local variable 'hooray' referenced before assignment

I played around in case it was a whitespace issue but my indentation seems ok..  Am still wondering if it should be the grep command that gets escaped anyway..

ralvez

Member

From: Canada

Registered: 2005-12-06

Posts: 1,718

Website

Re: Lost in python..

Well, the problem now with that change is that in case of an error the variable `hooray` has no value assigned effectively storing a None (think null) value; so you end up with "UnboundLocalError: local variable 'hooray' referenced before assignment"

In all truthfulness it is considered a bad programming style to have an unassigned variable at any point in your program because, as you can see, you get unexpected errors out of that... and that's why I said it was a rough work around.

BetterLeftUnsaid

Member

From: My Happy Place

Registered: 2007-11-04

Posts: 78

Re: Lost in python..

Couldn't you use communicate()?

cmd=['find', location, '-type', 'f', '-not', '-name', '*.svn']
# communicate()  returns tuple: (stdout, stderr).  Using [0] gives only stdout
hooray = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0]
# hooray is now the complete stdout of cmd
for hip in hooray.splitlines():   # Create an array from the newlines of hooray
    # Do stuff

I've never seen readlines() used with subprocess before, and something similar could be done for the grep command.

Source:  http://docs.python.org/library/subproce … ess-module

Last edited by BetterLeftUnsaid (2009-06-25 03:45:18)

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Re: Lost in python..

Communicate definitely gets me closer! It actually worked and I could search inside the files of my current directory but there was some wackiness in searching subdirectories.  No worries though because it confirmed that it should be the grep command I'm escaping and not the find command.

I tried to apply communicate to the grep command but was unsuccessful.  Here's what I tried:

        # Create a pipe and call the grep command, then read it            
        search_text = self.search_form.get_text(), string
        
        cmd=['grep -n -H', str_case_operator, ' %s %s']
        pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0]
        
        data = pipe.read()
        results = data.split("\n")

iphitus

Forum Fellow

From: Melbourne, Australia

Registered: 2004-10-09

Posts: 4,927

Re: Lost in python..

the find command could be replaced with this... better to use python than call an external program

import os
import os.path

hooray = [ f for f in os.listdir(location) if not os.path.isdir(f) and not f[0:4] == ".svn" ]

If you need subdirectories, have a look at os.walk() or os.path.walk() (can't remember which is deprecated).

Python has very good string processing abilities, grep isn't necessary either and could be replaced with native python, thus avoiding any shell related issues.

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Re: Lost in python..

Can you give me a hint as to how to do the grep command using python?  The find thing is pretty much a non-issue so really I either need to escape the grep command or (preferably?) find a way to do this code in pure python instead:

        search_text = self.search_form.get_text(), string
        pipe = os.popen("grep -n -H" + str_case_operator + " %s %s" % (search_text))
        data = pipe.read()
        results = data.split("\n")

Thanks in advance

The root problem, btw is that I can currently search for "bold" but not "<bold>"

Last edited by viniosity (2009-06-25 05:22:37)

Trent

Member

From: Baltimore, MD (US)

Registered: 2009-04-16

Posts: 990

Re: Lost in python..

My mistake, the constructor for subprocess.Popen should have had stdout=subprocess.PIPE in it, which tells the object to capture standard output rather than sending it on.  That's why you had an AttributeError (Popen.stdout is None unless you explicitly capture stdout).

You should be able to do the same thing with the grep.

cmd = ['grep', '-n', '-H', str_case_operator] + search_text
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE)
data = pipe.stdout.read()
results = data.split('\n')

(edit: corrected code typo)

Last edited by Trent (2009-06-25 22:13:20)

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Re: Lost in python..

Hi Trent,

Getting closer I think.  I'm getting this error:

data = pipe.read()
AttributeError: 'Popen' object has no attribute 'read'

Thank you!

Trent

Member

From: Baltimore, MD (US)

Registered: 2009-04-16

Posts: 990

Re: Lost in python..

darn... should be pipe.stdout.read

viniosity

Member

From: New York, NY

Registered: 2005-01-22

Posts: 404

Website

Re: Lost in python..

Unfortunately that didn't work, but I did cobble together enough from the previous posts and google to find something that did:

        search_text = self.search_form.get_text()
        cmd=['grep', '-R', '-n', '-H', search_text, location]
        output = subprocess.Popen(cmd, stdout=subprocess.PIPE)
        data = output.stdout.read()
        results = data.split('\n')

Thanks everyone for your patience and help!

Yannick_LM

Member

Registered: 2008-12-22

Posts: 142

Re: Lost in python..

the find command could be replaced with this... better to use python than call an external program

Sorry to be back on this, but big +1 for this.
(If you're developping a tool related to svn, maybe you would like to be able to use this tool on other platforms ... which come has an easy bonus while writing in python)