youtube-dl called from python script fails...why ? [solved]

Darksoul71 · 2013-06-06 08:08:37

Hi there,

since downloading youtube stuff with Browser plugins has been pretty unstable for me lately (read: mostly incomplete videos) I switched over to youtube-dl again. Unfortunately youtube-dl often stops downloading as well...

I wrote a small Python script which looks for incomplete downloads in my Youtube folder and restarts the download again. The code is quite simple:

#!/usr/bin/python
# -*- coding: utf8 -*-

# Modules used:
import sys, os

part_files = os.popen ("ls *.part")

for part_file in part_files:
  target_name = part_file.strip()[:-5]
  target_name = target_name.split(".")[0]
  yt_id = target_name [len(target_name)-11:]
  cli = 'youtube-dl -t http://www.youtube.com/watch?v=' + yt_id
  print cli
  os.popen (cli)

Unfortunately youtube-dl bombs out with the following stack trace and I have got no clue why...

youtube-dl -t http://www.youtube.com/watch?v=XNVaEoQMarg
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/bin/youtube-dl/__main__.py", line 17, in <module>
  File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 516, in main
    
  File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 500, in _real_main
    
  File "/usr/bin/youtube-dl/youtube_dl/FileDownloader.py", line 507, in download
  File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 92, in extract
  File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 87, in initialize
  File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 313, in _real_initialize
  File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 197, in report_lang
  File "/usr/bin/youtube-dl/youtube_dl/FileDownloader.py", line 192, in to_screen
IOError: [Errno 32] Broken pipe

Does anybody have an idea why this happens ? Using youtube-dl without the "resume download" script works perfectly though...

TIA,
D$

Last edited by Darksoul71 (2013-06-06 11:00:51)

mychris · 2013-06-06 10:51:28

I think the problem is that you are using os.popen.
The function returns a file object to which the pipe of the new spawned process is connected. If you don't catch that file object, python might garbage collect it. (In addition, your script terminates, so the file object will be garbage collected).
Check out the subprocess module from the standard lib. I think there you can redirect the pipe of the new spawned processs to /dev/null or something.
http://docs.python.org/2/library/subprocess.html

Darksoul71 · 2013-06-06 10:57:58

@mychris:
Thanks a lot ! Using the subprocess module fixed my issue.

The working code looks like this:

#!/usr/bin/python
# -*- coding: utf8 -*-

# Modules used:
import sys, os, subprocess

part_files = os.popen ("ls *.part")

for part_file in part_files:
  target_name = part_file.strip()[:-5]
  target_name = target_name.split(".")[0]
  yt_id = target_name [len(target_name)-11:]
  cli = 'youtube-dl -t http://www.youtube.com/watch?v=' + yt_id
  subprocess.call(cli, shell=True)

Perfect !

mychris · 2013-06-06 11:00:15

No problem.
Please mark the thread as [SOLVED].

Darksoul71 · 2013-06-06 11:11:59

Done !

WorMzy · 2013-06-06 11:16:50

Just a note: you may want to exchange the ambiguous shebang with the more portable '#!/usr/bin/env python3', or at least '#!/usr/bin/python3', if you ever want to share the script across multiple distros.

Darksoul71 · 2013-06-06 11:40:26

Thanks for the hint !
I will consider this as change for my scripts. I am always happy to use Python because it often runs under other distros / platforms (e.g. Raspberry Pi).

whitie · 2013-06-07 10:01:27

Hello,
maybe a little bit more pythonic:

#!/usr/bin/env python3

import glob
import os
import subprocess


def main():
    part_files = glob.glob('./*.part')
    for part_file in part_files:
        target_name = os.path.splitext(part_file)[0]
        yt_id = target_name[len(target_name)-11:]
        cmd = ['youtube-dl', '-t',
               'http://www.youtube.com/watch?v={}'.format(yt_id)]
        subprocess.call(cmd)


if __name__ == '__main__':
    main()

Whitie

Darksoul71 · 2013-06-07 10:05:15

Whitie,

thanks for posting ! I know that my scripts often look like a bad reinterpretation of shell script in Python
Honestly what counts for me is that things work. They do not have to be pretty or well implemented, but just work.

Regards,
D$

Mr.Elendig · 2013-06-08 09:33:33

https://github.com/metal3d/python-libquvi

Darksoul71 · 2013-06-09 09:31:11

Thanks...one never stops to learn

I wasn't even aware of quvi.

Arch Linux

#1 2013-06-06 08:08:37

youtube-dl called from python script fails...why ? [solved]

#2 2013-06-06 10:51:28

Re: youtube-dl called from python script fails...why ? [solved]

#3 2013-06-06 10:57:58

Re: youtube-dl called from python script fails...why ? [solved]

#4 2013-06-06 11:00:15

Re: youtube-dl called from python script fails...why ? [solved]

#5 2013-06-06 11:11:59

Re: youtube-dl called from python script fails...why ? [solved]

#6 2013-06-06 11:16:50

Re: youtube-dl called from python script fails...why ? [solved]

#7 2013-06-06 11:40:26

Re: youtube-dl called from python script fails...why ? [solved]

#8 2013-06-07 10:01:27

Re: youtube-dl called from python script fails...why ? [solved]

#9 2013-06-07 10:05:15

Re: youtube-dl called from python script fails...why ? [solved]

#10 2013-06-08 09:33:33

Re: youtube-dl called from python script fails...why ? [solved]

#11 2013-06-09 09:31:11

Re: youtube-dl called from python script fails...why ? [solved]

Board footer