Efficient Python file IO help needed

skottish · 2008-05-10 04:53:41

I have a GIMP-Python script that I wrote a while back that I want to finish. The functionality of the script itself is more or less complete, but I believe that it has severe penalties in the way I implemented the file IO.

Before I start, please ignore the divide by zero test. I'll refine that later. Also, the redundancy of the math up front is only to make the input values seem more reasonable to the user. Never mind these for now.

What this script does is it takes any GIMP readable image and converts it gray scale. Then it maps every single pixel of the picture to it's previous x and y coordinates, as well as a new z coordinate based on a combination of its new value between black and white, and an arbitrary user defined depth in either the positive or negative direction. The greater the depth, the higher or lower the z coordinate. This effectively takes any two dimensional object and converts it into 3D space. These coordinates are written into a Wavefront .obj text file that can be imported into 3D modeling programs such as Blender -- the first pass the vertices, the second the faces.

This is the script:

#! /usr/bin/env python
from gimpfu import *

def pic_to_obj(file_name, depth):
    
    if depth==0:
        depth = 1
    
    depth = float(1000/depth)
    
    FILE = open(file_name + ".obj", "w")
    firstline = "g " + file_name + ".obj\n\n"
    FILE.writelines( firstline )
            
    image = pdb.gimp_file_load(file_name, file_name)
    drawable = pdb.gimp_image_get_active_layer(image)
        
    if pdb.gimp_drawable_is_gray(drawable)!= True:
        pdb.gimp_image_convert_grayscale(image)
            
    width = pdb.gimp_image_width(image)
    height = pdb.gimp_drawable_height(drawable)
        
    for j in range(height):
        for i in range(width):
            num_channels, pixel = pdb.gimp_drawable_get_pixel(drawable, i, j)
            list = "\nv " + str(float(i)/10) + " " + str(float(j)/10) + " " + str(float(pixel[0])/depth)
            FILE.writelines(list) 
            
        
    FILE.writelines( "\n" )
        
    for l in range((width)*(height-1)-1):
        if((l+1)%width)!=0:
            list = "\nf " + str(l+1) + " " + str(l+2) + " " + str(l+width+2) + " " + str(l+width+1)
            FILE.writelines(list)
            
    

register(
  "pic_to_obj", "", "", "", "", "",
  "<Toolbox>/Xtns/Languages/Python-Fu/pic_to_obj", "",
  [
  (PF_STRING, "file_name", "file_name", "file_name"),
  (PF_FLOAT, "depth", "depth", "depth")
  ],
  [],
  pic_to_obj
  )

main()

If you look at the first for loop, you'll see that it's calling FILE.writeline() for every single coordinate in the entire image to create the vertex array. Then it does it again for the faces array. Am I right in feeling like that there is a much more efficient way of doing this?

N30N · 2008-05-11 01:03:25

Python only writes to the file when it's closed/flushed not each time you call the writelines method, so I don't there's any think wrong with the loops.

skottish · 2008-05-11 02:02:59

N30N wrote:

Python only writes to the file when it's closed/flushed not each time you call the writelines method, so I don't there's any think wrong with the loops.

Thanks for the interesting post. At no time did I explicitly close the file. I probably should learn how to do that before I cram 4000+ images through the script.

I guess then that the inefficiency that I'm perceiving is simply from the large number of elements I'm running through the arrays. Does this seem right? Of course having redundant math all through the script doesn't help...

N30N · 2008-05-11 04:13:11

skottish wrote:

At no time did I explicitly close the file. I probably should learn how to do that before I cram 4000+ images through the script.

Sounds like a cool project.

FILE.close() #saves and closes the file

FILE.flush() #saves buffer to file - could be useful if your working on high resolution images

If your wanting to squeezing every bit of performance out you could cache the FILE.writelines lookup like so...

writelines = FILE.writelines
...
writelines(list)

Also changeing the string concatenation may help a bit.

#Slower
list = "\nv " + str(float(i)/10) + " " + str(float(j)/10) + " " + str(float(pixel[0])/depth)
#Faster
list = "\nv %s %s %s" % (float(i)/10, float(j)/10, float(pixel[0])/depth)

Have you tried using the PIL (Python Imaging Library) it' probably more efficient what using GIMP?

I'm sure some of the more experienced python programmers around here will have more helpful tips for ya.

skymt · 2008-05-11 04:16:26

I noticed you're using file.writelines() to write single strings to the file. That function iterates over a set and prints each item. In Python, when you iterate over a string, you get each individual character. The end result is that Python is individually writing each character to the file. Larger chunks should be more efficient, so use file.write() instead. (Note that this only applies to Python's internal buffer for the file: it isn't doing a disk write for each character unless writelines() is calling flush() [it shouldn't].)

skymt · 2008-05-11 04:24:32

N30N wrote:

If your wanting to squeezing every bit of performance out you could cache the FILE.writelines lookup like so...
writelines = FILE.writelines
...
writelines(list)

That's interesting. Do you happen to have a source for that? Looking up a key in a specific namespace should be exactly as fast as looking in the local scope (and one lookup faster than checking global).

(Also, it's 'file' not 'FILE'. )

skottish · 2008-05-11 04:29:48

N30N wrote:

Have you tried using the PIL (Python Imaging Library) it' probably more efficient what using GIMP?
I'm sure some of the more experienced python programmers around here will have more helpful tips for ya.

Thanks N30N, this is all great! I really appreciate it.

No, I haven't even thought about NIL. I came to this script in a inspired flurry of creative work. My head was so wrapped up in GIMP scripting that I didn't even consider any other options. I "learned" Python in a matter of a few days because Tiny Scheme was making me insane. Any code that ends like this should be considered unreadable:

)))))))))))))))))))))))))))

I'll definitely look into it.

...Then it will be time to get to the Blender scripting phase... The information there sucks even worse than the GIMP-Python stuff when I took on that.

skottish · 2008-05-11 04:31:09

skymt wrote:

(Also, it's 'file' not 'FILE'. )

I got FILE from the Python website. I guess they both work.

skymt · 2008-05-11 04:39:25

Oh wait, never mind. Late-night brain fart. 'FILE' is a variable you created, 'file' is a built-in class.

That said, it's considered poor form to differentiate between variables with case. It can create the sort of confusion I just had. In this context, I'd use something slightly more verbose, like 'objfile'.

Edit: Also, either one would work to cache the lookup, as file.writelines and <any file object>.writelines are the same object. If lookups are indeed cached.

Last edited by skymt (2008-05-11 04:40:52)

kumico · 2008-05-11 06:07:38

#!/usr/bin/env python
from gimpfu import *

def pic_to_obj(file_name, depth):
    if depth == 0:
        depth = 1000.0
    else:
        depth = 1000.0 // depth
    
    image = pdb.gimp_file_load(file_name, file_name)
    buf = []
    file_name = "%s.obj" % os.path.splitext(file_name)[0]
    buf.append("g %s\n\n" % file_name)
    
    drawable = pdb.gimp_image_get_active_layer(image)
    
    if pdb.gimp_drawable_is_gray(drawable) != True:
        pdb.gimp_image_convert_grayscale(image)
    
    width = pdb.gimp_image_width(image)
    height = pdb.gimp_drawable_height(drawable)
    gimp_drawable_get_pixel = pdb.gimp_drawable_get_pixel
    
    for j in xrange(height):
        for i in xrange(width):
            num_channels, pixel = gimp_drawable_get_pixel(drawable, i, j)
            list = "\nv %s %s %s" % (i / 10.0, j / 10.0, pixel[0] / depth)
            buf.append(list)
    buf.append("\n")
    
    for l in xrange((width * (height - 1)) - 1):
        p = l + 1
        q = l + width
        if (p % width) != 0:
            list = "\nf %d %d %d %d" % (p, p + 1, q + 2, q + 1)
            buf.append(list)
    fo = open(file_name, "w")
    fo.writelines(buf)
    fo.close()

register(
  "pic_to_obj", "", "", "", "", "",
  "<Toolbox>/Xtns/Languages/Python-Fu/pic_to_obj", "",
  [
  (PF_STRING, "file_name", "file_name", "file_name"),
  (PF_FLOAT, "depth", "depth", "depth")
  ],
  [],
  pic_to_obj
  )

main()

some of the ideas used in the mod of your code is already mentioned here.
one change i made was to use a list to buffer the output. you could also have used StringIO here.
i personally don't know how the python write/flush process works, but to imo,
if it was going to cache everything then there'd be no need for StringIO in the first place.

i've also removed all the explicit casting, and string concatenation.
this doesn't make much if any difference through most of the code, but in loops it counts,, believe me it does.
so i rewrote the rest of the code to make it all consistent.

and i changed to using xrange, this may also make little difference in the first set of loops,
but you never known how big width * height is going to be so it makes sense for it to be used in the second loop
again i used it elsewhere for consistency

and there may be a little over optimization in the second loop where i cached the repeated calculations.

if you wanted to get even more out of this,
you could push one of the loops onto another thread then join them up at the end.

Last edited by kumico (2008-05-11 09:39:58)

skottish · 2008-05-11 07:30:19

kumico,

Thank you so much for coming in.

On line 11, there is:

file_name = "%s.obj" % file_name

It's not an '.obj' file coming in; That's added later after it becomes the new format (I didn't know how to strip extensions. '.png' became '.png.obj'. Bad form on my part). It's simply a GIMP readable bitmap image. The script fails here first because GIMP has no idea how to handle any "alien" file types.

If '.obj' is dropped from the script the process seems to continue, but the output file then has the extension of '.png'. It's unreadable as an '.obj' file and it overwrites the source. Also, it picks up around 7 MB of more information for a 720x576 image then it would have originally. There's a whole lot of extra stuff going into the output. It should be slightly under 18 MB and it's ending up around 25 MB.

Last edited by skottish (2008-05-11 07:34:16)

kumico · 2008-05-11 07:41:18

fixed above,
load the image before resetting file_name, file_name's original extention is removed and replace with .obj.

i'll go download gimpfu and see what's up
also, now uses writelines instead of write,, more efficient(saves building another huge string)

;;;

and fixed out[ut issue, strformats output the trailign zeros,
for now it's quite fast compared to the original,,
i'll try to get some more speed in there later

Last edited by kumico (2008-05-11 09:36:53)

N30N · 2008-05-11 10:05:57

skottish wrote:

...Then it will be time to get to the Blender scripting phase... The information there sucks even worse than the GIMP-Python stuff when I took on that.

Are you planning on getting it to join the models together or is it a obj sequence for an animation (if you dont mind me asking)?

skymt wrote:

N30N wrote:
If your wanting to squeezing every bit of performance out you could cache the FILE.writelines lookup like so...
writelines = FILE.writelines
...
writelines(list)
That's interesting. Do you happen to have a source for that? Looking up a key in a specific namespace should be exactly as fast as looking in the local scope (and one lookup faster than checking global).

I'm not sure where I picked it up from (it's common practice in JavaScript). I did a quick search and it features among others tips on the python wiki.

skottish · 2008-05-11 15:55:44

N30N wrote:

Are you planning on getting it to join the models together or is it a obj sequence for an animation (if you dont mind me asking)?

I don't mind. In fact, I was going to post a before and after shot of an actual frame last night. Then I found out that my free host stopped being free. I also found out that they are a new company all together. Time to find a new host!

The frames are part of an ongoing project. I have two different but similar very psychedelic kaleidoscopes that were produced through GIMP. These things sort of morph and melt into themselves. Very beautiful if you like stuff like this.

The kaleidoscopes combined have somewhere around 8000 frames and took approximately 350 hours to render. One day I thought it would be fun to bring one of them into 3D space and this script was born. Realistically speaking, the original script was good enough to do this. It takes about 20 seconds to produce one .obj file and I can run two instances of GIMP to use both cores of my processor. And, I currently have the hard drive space to do this. 4000 frames at 18 MB a piece just for the .obj files!

I'm going to take each frame, turn it into a .obj file, import it into Blender, texture the object with the original picture, render it, and output the file as a .png. The trick now is figuring out scripting at this level with Blender. I know it's capable of doing it, but Python scripting for Blender is nowhere near as easy as it is for GIMP (and less documented!). The GIMP designers did an awesome job with their PDB, and should be the model for anyone wanting to open up the functionality of their media programs.

Oh yeah, the last trick is lighting. I never got to the point with 3D to learn lighting. Anyone that's ever worked with 3D can confirm how tricky lighting can be.

Last edited by skottish (2008-05-11 17:54:58)

skottish · 2008-05-11 16:41:57

kumico wrote:

i'll go download gimpfu and see what's up

All you need is GIMP. Both the Python and Tiny Scheme (Script-FU) back ends are part of the default installation. To see how the PDB works in GIMP 2.4, start it, go to Xtns on the main tool bar, Python-FU, then Console. If you then hit Browse, you can search for any GIMP function. Type Sharpen in the Search box, and double click on plug-in-sharpen. In the console window there will be the generic form of the function as it's called in the script. Cool, yes?

N30N · 2008-05-11 22:14:02

skottish wrote:

The frames are part of an ongoing project. I have two different but similar very psychedelic kaleidoscopes that were produced through GIMP. These things sort of morph and melt into themselves. Very beautiful if you like stuff like this.
The kaleidoscopes combined have somewhere around 8000 frames and took approximately 350 hours to render. One day I thought it would be fun to bring one of them into 3D space and this script was born. Realistically speaking, the original script was good enough to do this. It takes about 20 seconds to produce one .obj file and I can run two instances of GIMP to use both cores of my processor. And, I currently have the hard drive space to do this. 4000 frames at 18 MB a piece just for the .obj files!
I'm going to take each frame, turn it into a .obj file, import it into Blender, texture the object with the original picture, render it, and output the file as a .png. The trick now is figuring out scripting at this level with Blender. I know it's capable of doing it, but Python scripting for Blender is nowhere near as easy as it is for GIMP (and less documented!).

Wouldn't using your kaleidoscopes sequence as a displacement map be sufficient (also check the displace modifier if you not aware of it)?

skottish · 2008-05-11 22:34:15

N30N wrote:

Wouldn't using your kaleidoscopes sequence as a displacement map be sufficient (also check the displace modifier if you not aware of it)?

I'm not sure that displacement maps will change much here, but I'm certainly willing to try. The problem is that I'd have to map the image to some object (a plane in this case). In order for the object to reflect the detail of the original image, I'd have to subdivide it to the resolution of the original image (I don't care about poly count). So either I'm going to ask Blender to subdivide the object into 414000+ segments, texture it, and then apply the displacement map, or I can let GIMP create the object and have Blender texture it. Admittedly I'm curious to see how Blender handles subsurf, textures, and displacement maps though.

Last edited by skottish (2008-05-11 22:49:42)

skottish · 2008-05-11 23:22:54

N30N,

Over and over again in this thread you have broken my single mindedness towards this. Awesome! Thank you so much.

Arch Linux

#1 2008-05-10 04:53:41

Efficient Python file IO help needed

#2 2008-05-11 01:03:25

Re: Efficient Python file IO help needed

#3 2008-05-11 02:02:59

Re: Efficient Python file IO help needed

#4 2008-05-11 04:13:11

Re: Efficient Python file IO help needed

#5 2008-05-11 04:16:26

Re: Efficient Python file IO help needed

#6 2008-05-11 04:24:32

Re: Efficient Python file IO help needed

#7 2008-05-11 04:29:48

Re: Efficient Python file IO help needed

#8 2008-05-11 04:31:09

Re: Efficient Python file IO help needed

#9 2008-05-11 04:39:25

Re: Efficient Python file IO help needed

#10 2008-05-11 06:07:38

Re: Efficient Python file IO help needed

#11 2008-05-11 07:30:19

Re: Efficient Python file IO help needed

#12 2008-05-11 07:41:18

Re: Efficient Python file IO help needed

#13 2008-05-11 10:05:57

Re: Efficient Python file IO help needed

#14 2008-05-11 15:55:44

Re: Efficient Python file IO help needed

#15 2008-05-11 16:41:57

Re: Efficient Python file IO help needed

#16 2008-05-11 22:14:02

Re: Efficient Python file IO help needed

#17 2008-05-11 22:34:15

Re: Efficient Python file IO help needed

#18 2008-05-11 23:22:54

Re: Efficient Python file IO help needed

Board footer