(Python) Using io.BufferedReader object in place of text file

alphaniner · 2014-01-14 22:38:47

I have a tar file which contains a ConfigParser-style ini file. I need to be able to parse the ini without extracting it to disk. AFAICT, the only way to do this is with extractfile, which returns an io.BufferedReader object. From what I understand this a raw binary stream, so it needs to be decoded to text before it can be of any use to ConfigParser.

I couldn't find a way to decode the entire object, only individual lines, so the first method I came up with is config.read_file(get_ini(foo, bar)) with get_ini() defined as:

def get_ini(archive, inifile):
    with tarfile.open(archive, 'r') as a:
        f = a.extractfile(a.getmember(inifile))
        for l in f:
            yield l.strip().decode('utf-8')

Frankly, I'm in over my head with this binary stream stuff, so I'm sure there are better ways of doing this. I'm not looking for fully formed solutions, just suggestions, etc. Thanks.

firecat53 · 2014-01-14 23:25:38

Your way seems to work fine. I was able to decode an entire file at once using read(). Returns a string instead of a list.

def get_file(arch, fn):
    with tarfile.open(arch, 'r') as a:
        f = a.extractfile(a.getmember(fn))
        return f.read().decode()

Scott

alphaniner · 2014-01-16 15:16:13

Thanks. I tried decoding f directly, didn't think of involving read().

Your method is definitely more elegant, but if I understand yield correctly, mine involves less memory usage right?

firecat53 · 2014-01-16 17:00:06

That would be my understanding as well. I gathered from the fact that you were just extracting a config file that size/memory probably wouldn't be an issue. If it is, your way definitely falls in the 'more elegant' category! I've worked with some pretty good sized CSV files imported into big lists and haven't had a memory issue yet (ok, unless maybe you're running on a 256MB VPS ).

Scott

alphaniner · 2014-01-16 17:43:53

You're right, the ini file is miniscule and the function's memory usage really isn't an issue.

Well, as long as the memory is freed once it terminates. The bigger picture is that the function will be run on an indeterminate number of files.

Arch Linux

#1 2014-01-14 22:38:47

(Python) Using io.BufferedReader object in place of text file

#2 2014-01-14 23:25:38

Re: (Python) Using io.BufferedReader object in place of text file

#3 2014-01-16 15:16:13

Re: (Python) Using io.BufferedReader object in place of text file

#4 2014-01-16 17:00:06

Re: (Python) Using io.BufferedReader object in place of text file

#5 2014-01-16 17:43:53

Re: (Python) Using io.BufferedReader object in place of text file

Board footer