You are not logged in.

#1 2010-02-10 21:38:02

mrbug
Member
Registered: 2007-07-17
Posts: 221

[SOLVED!] PYTHON: sequential subprocesses

I'm working on a python script that calls two different external programs; one is called multiple times in a loop, the other is called once after the loop has finished.

I'm using subprocess.call() due to its modern-ness and security over the old os library. However, it seems to fork the processes in the background rather than waiting for each process to end before starting the next step in the script.


Any idea how I can fix this? Is there a better way to do it in general? The commandlines have variables, so it's not just subprocess.call("program",shell=True). Instead, it's something like subprocess.call("program " + inputfile,shell=True)

EDIT: Got it! I used a while loop for the "download each video part" to prevent it from moving onto the "transcode and build dvd" part.

Additionally, I'm using the PIPE method (or whatever it's called) of subprocess to hide the "noisy" output. I'm only showing things like "checking page (url)" "found video: (video)" "downloading video #(num)" and so on.

Now that the script is mostly done, where should I share it? It'd be nice if others could contribute and make it better, so should I use Google Code? It does download all videos from a YouTube user page. Is that a "problem" with the "YouTube rules" or anything like the DMCA?

It's purely for educational purposes as a proof-of-concept, of course.

Last edited by mrbug (2010-02-12 02:21:32)


dvdtube - download all uploads from a YouTube user and then optionally create a DVD.
(Regular version AUR link / SVN version AUR link)

Offline

#2 2010-02-10 22:01:13

Grazz256
Member
Registered: 2009-06-28
Posts: 69

Re: [SOLVED!] PYTHON: sequential subprocesses

how about something more like this

from subprocess import *
subproc = Popen(["program", param1, param2, etc..], stdout=PIPE, stderr=PIPE)
(out, err) = subproc.communicate()

the communicate function doesn't return until the subprocess has completed.

Cheers

Offline

#3 2010-02-10 22:24:21

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: [SOLVED!] PYTHON: sequential subprocesses

If you're familiar with old libraries,  http://docs.python.org/library/subprocess.html this page can be very useful
(have a look on the sections called "replacing ...")


From the same page:

Popen.wait():

    Wait for child process to terminate. Set and return returncode attribute.

    Warning:

    This will deadlock if the child process generates enough output to a stdout or stderr pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

Sim be careful with wait() ....

Offline

#4 2010-02-10 23:58:48

mrbug
Member
Registered: 2007-07-17
Posts: 221

Re: [SOLVED!] PYTHON: sequential subprocesses

I've tried wait(), but I don't think that it worked... The same behavior happened.

I did the communicate() trick as well, but it complained about PIPE not existing.

Did I miss something with PIPE like declaring it somewhere?



When I'm done with the code, I'll post it. I plan to put it in the AUR -- and I believe that people will like it!


dvdtube - download all uploads from a YouTube user and then optionally create a DVD.
(Regular version AUR link / SVN version AUR link)

Offline

#5 2010-02-11 00:00:12

Peasantoid
Member
Registered: 2009-04-26
Posts: 928
Website

Re: [SOLVED!] PYTHON: sequential subprocesses

mrbug wrote:

When I'm done with the code, I'll post it. I plan to put it in the AUR -- and I believe that people will like it!

Care to describe what 'it' is...?

Offline

#6 2010-02-11 00:02:33

mrbug
Member
Registered: 2007-07-17
Posts: 221

Re: [SOLVED!] PYTHON: sequential subprocesses

All of the videos for a single user on Youtube -> DVD converter (with menu).

I can download the videos without any problems, but it tries to convert the videos and make the DVD (+menu) before they're done downloading.


dvdtube - download all uploads from a YouTube user and then optionally create a DVD.
(Regular version AUR link / SVN version AUR link)

Offline

#7 2010-02-11 01:07:11

mikesd
Member
From: Australia
Registered: 2008-02-01
Posts: 788
Website

Re: [SOLVED!] PYTHON: sequential subprocesses

mrbug wrote:

I've tried wait(), but I don't think that it worked... The same behavior happened.

I did the communicate() trick as well, but it complained about PIPE not existing.

Did I miss something with PIPE like declaring it somewhere?

When I'm done with the code, I'll post it. I plan to put it in the AUR -- and I believe that people will like it!

Did you import subprocess as from subprocess import *?

If you just used import subprocess you would need to use subprocess.PIPE in place of PIPE.

Offline

#8 2010-02-11 09:49:30

mrbug
Member
Registered: 2007-07-17
Posts: 221

Re: [SOLVED!] PYTHON: sequential subprocesses

mikesd wrote:

Did you import subprocess as from subprocess import *?

If you just used import subprocess you would need to use subprocess.PIPE in place of PIPE.

Weird, I didn't know that there was a difference! I'm still pretty new with Python, so there's a lot that I don't know yet...

I just tried that and it's at least not showing that the two programs are running at the same time. Whether they are or not, I can't tell.

Also, is it possible that ffmpeg can't (currently) transcode h264 flv files to m2v/vob?


EDIT: Maybe I should go ahead and post my code now... It would be freely available once it works anyway. Any takers?

Last edited by mrbug (2010-02-11 09:57:32)


dvdtube - download all uploads from a YouTube user and then optionally create a DVD.
(Regular version AUR link / SVN version AUR link)

Offline

#9 2010-02-12 02:24:02

mrbug
Member
Registered: 2007-07-17
Posts: 221

Re: [SOLVED!] PYTHON: sequential subprocesses

I think that it would be helpful to make a post to let people know the following things about Python:

1. The subprocess library will FORK processes, so they'll run independently of the rest of the script.
2. If you don't want the script to continue until the processes are complete, use PIPE and the communicate() method.
3. You'll likely need to use some kind of a loop to contain the processes until they're actually complete.


dvdtube - download all uploads from a YouTube user and then optionally create a DVD.
(Regular version AUR link / SVN version AUR link)

Offline

#10 2010-02-14 14:27:25

Grazz256
Member
Registered: 2009-06-28
Posts: 69

Re: [SOLVED!] PYTHON: sequential subprocesses

mrbug wrote:

I think that it would be helpful to make a post to let people know the following things about Python:

1. The subprocess library will FORK processes, so they'll run independently of the rest of the script.
2. If you don't want the script to continue until the processes are complete, use PIPE and the communicate() method.
3. You'll likely need to use some kind of a loop to contain the processes until they're actually complete.

You don't need to use a loop to contain the process if you use communicate as it doesn't return until the sub process exits

Cheers

Offline

Board footer

Powered by FluxBB