You are not logged in.

#1 2012-12-31 03:16:11

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Greg -- a command-line podcast aggregator

UPDATED: 13 Oct 2013

The latest release (0.4.0) can filter feeds, and has a number of bugs fixed.

Hi, I've been putting together a little podcast aggregator in Python. It (only) depends on python-feedparser. Just in case someone else finds it useful. From the README:

greg
====

A command-line podcast aggregator, written in python 3. It basically exposes some of the functionality of the excellent [feedparser](http://pypi.python.org/pypi/feedparser) -- feedparser and python are its only non-optional dependencies.

# Usage

Let's start by adding a feed (RSS or Atom versions will do):

    greg add PhilosophyBites http://philosophybites.com/atom.xml

The `add` command expects a name and a url of an RSS or Atom feed. You will use this name to refer to the feed whenever you interact with it.

If you were to run `greg sync` now, it would download the latest episode of the podcast to the default directory (which is `~/Podcasts`; you can change how many episodes are dowloaded in the first sync, and the download directory, in the config file; see below). But maybe we just want to check out what this podcast is all about, so we download a list of available entries:

    greg check -f PhilosophyBites

(the `-f` flag means that "PhilosophyBites" is the name of a feed. `greg check` also accepts urls directly, using the `-u` flag.)

This will give you the following kind of info:
   

    0: Tom Sorell on Surveillance (2013-01-25T13:43:46+00:00)
    1: John Campbell on Schizophrenia (2013-01-08T12:41:27+00:00)
    2: Kendall Walton on Photography (2012-12-23T12:33:09+00:00)
    3: Twitter Competition: Who's Your Favourite Philosopher? (2012-12-11T07:24:51+00:00)
    4: Alan Ryan on Freedom and Its History (2012-12-08T11:16:45+00:00)
    5: Nigel Warburton at Blackwell's Bookshop, Oxford 7pm Wed. Dec. 5th (2012-12-01T11:19:09+00:00)
    6: Who's Your Favourite Philosopher? (2012-11-30T18:33:56+00:00)
    7: Peter Adamson on Avicenna's Flying Man Thought Experiment (2012-11-26T15:57:18+00:00)
    8: Links to Past Episodes (2012-12-01T11:53:03+00:00)
    9: Tim Bayne on the Unity of  Consciousness (2012-11-11T22:20:17+00:00)
    10: Galen Strawson on the Sense of Self (2012-05-05T12:56:05+01:00)
    11: Liane Young on Mind and Morality (2012-10-27T12:39:22+01:00)
    12: Gary L. Francione on Animal Abolitionism (2012-10-13T13:48:32+01:00)
    13: Richard Sorabji on Mahatma Gandhi as Philosopher (2012-09-28T13:18:08+01:00)
    14: Tim Crane on Non-Existence (2012-09-15T18:50:32+01:00)
    15: Michael Tye on Pain (2012-08-31T20:51:01+01:00)
    16: Daniel Dennett on Free Will Worth Wanting (2012-08-18T08:58:24+01:00)
    17: Pat Churchland on What Neuroscience Can Teach Us About Morality (2012-08-03T22:52:12+01:00)
    18: Rae Langton on Hate Speech (2012-07-28T20:14:27+01:00)
    19: Molly Crockett on Brain Chemistry and Moral-Decision Making (originally on Bioethics Bites) (2012-07-22T21:14:35+01:00)

Interesting stuff. We'll download a couple of episodes, just to make sure that it's really worth it:

    greg download 1, 5-7

and Greg says

    Downloading John Campbell on Schizophrenia -- John_Campbell_on_Schizophrenia.mp3
    Done
    Downloading John Campbell on Schizophrenia -- John_Campbell_on_Berkeleys_Puzzle_1.mp3
    Done
    Downloading Who's Your Favourite Philosopher? -- Whos_Your_Favourite_Philosopher_.mp3
    Done
    Downloading Peter Adamson on Avicenna's Flying Man Thought Experiment -- Peter_Adamson_on_Avicennas_Flying_Man.mp3
    Done
    Downloading Peter Adamson on Avicenna's Flying Man Thought Experiment -- AdamsonMixSes.MP3
    Done
    Downloading Peter Adamson on Avicenna's Flying Man Thought Experiment -- Peter_Adamson_on_Plotinus_on_Evil.mp3
    Done

As you can see, `greg download` accepts a range of episodes of the kind `a, b, c-f, h, ...`. The numbers make reference to the numbers at the beginning of each entry provided by `greg check`. `check` creates a persistent file (`feeddump` in the data directory, `~/.local/share/greg/data by default, but you can change that in the config file, or passing a different path with the `--datadirectory` flag), so `download` will keep on working, and referring to the last `check` ever done.

All of these podcasts will be downloaded to the default download directory for the feed (if you used the `-f` flag) or the general default download directory (again, `~/Podcasts` if you don't tell Greg otherwise. We'll learn how to change that soon), inside a subdirectory named after the podcast (we can change that default too.) After listening to them we decide that this podcast is well worth our time, and keep it, or we decide that it's not, and

    greg remove PhilosophyBites

If we keep it, we might want to start sync'ing from, say, the 30th of April, 2013, on. So we edit the feed information

    greg edit PhilosophyBites -d 2013-4-30

`-d` or `--downloadfrom` change the date after which Greg should start downloading episodes when it syncs. Currently, the only two things one can `edit` in a feed are the download-from date and `--url` -- but many more things can be changed by editing the config file. `greg edit -h` will give help you with the `edit` options and syntax -- likewise for the rest of Greg subcommands.

All right. Let's add a second feed:

    greg add History http://feeds.feedburner.com/historyofph … format=xml

If you want to keep track of the feeds you have added, you can ask Greg:

    greg info

which returns

    PhilosophyBites
    ---------------
        url: http://philosophybites.com/atom.xml
        Next sync will download from: 30 Apr 2013 00:00:00.

    History
    -------
        url: http://feeds.feedburner.com/historyofph … format=xml

Let us add another feed:

    greg add MusicaAntigua http://www.rtve.es/api/programas/23353/audios.rss

This is a great program on ancient music at the Spanish public radio. The thing is, these guys do not tag their episodes, which is bad for most portable media players. Greg uses [stagger](http://pypi.python.org/pypi/stagger/0.4.2) (as an optional dependency) to tag podcasts, if one so wishes. By default, it uses the podcast name for the *artist* tag, and the entry title for the *title* tag. To enable tagging for MusicaAntigua, copy the system-wide config file locally:

    cp /etc/greg.conf ~/.config/greg/greg.conf

and add a section for MusicaAntigua:

    [MusicaAntigua]

    Tag = yes

In fact, you can fill out any tag however you see fit. For example,

    tag_genre = Ancient Music
    tag_comment = {date}

will fill the *genre* tag with the string "Ancient Music", and the *comment* tag with the download date.
   
Let's add a video podcast

    greg add TEDTalks http://feeds.feedburner.com/TEDTalks_video

By default, Greg only donwloads audio files (in fact, files that have "audio" as part of their type). In order to download the right file in TEDTalks, then, you need to change that in the config file. Again, add a section:

    [TEDTalks]

    mime = video

You could also have a couple of types there, as in `mime = audio, video`; or any other type, `mime = torrent`, or whatever.

Another useful thing that you can change in the config file is the download handler; Greg by default uses `urllib.request.urlretrieve`, but you can use whatever you want. I, for example, have

    downloadhandler = wget {link} -P {directory}

in my local `greg.conf`. You can do all sorts of nice things with this. For example, when `check`ing a podcast, you don't need to download it, but maybe just stream it, like this:

    greg download 0 --downloadhandler "mplayer {link}"

One last thing: if you subscribe to a very active feed, and you are only interested in some of the entries, you can filter the feed. For example, if you only want to watch TED talks about Google, say, you can add the following line to the `[TEDTalks]` section:

    filter = "Google" in "{title}"

(You need the quotes around {title} if the string you are filtering by has spaces, for example; they are strictly unnecessary here.)

For information about the {placeholders}, take a look at [greg.conf](https://github.com/manolomartinez/greg/ … /greg.conf). In `greg.conf` you can also change the download directory, and some other things. It should be self-explanatory.

I have also created two AUR packages: https://aur.archlinux.org/packages/greg-git/ and https://aur.archlinux.org/packages/greg/

Cheers,
Manolo

Last edited by manolomartinez (2013-10-13 11:13:18)

Offline

#2 2012-12-31 11:35:31

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: Greg -- a command-line podcast aggregator

Just a heads up: feedparser is broken for python3 right now.

The patch that fixes it already on the master branch:

http://code.google.com/p/feedparser/iss … ail?id=384

I've also open an issue in Arch's bug tracker:

https://bugs.archlinux.org/task/33254

Edit: The bug is fixed in python-feedparser-5.1.3-2

That was quick smile

Last edited by Yannick_LM (2012-12-31 13:19:10)

Offline

#3 2012-12-31 13:01:44

Paul-S
Member
From: Wales
Registered: 2008-02-04
Posts: 343

Re: Greg -- a command-line podcast aggregator

Hi Manolo, I get a couple of errors on a few of my feeds either a - Something has gone wrong: object has no attribute 'parsed' or Something has gone wrong: object has no attribute 'title'. I assume these are problems with the rss feeds but they work ok in podget.

Cheers
Paul-S

Offline

#4 2012-12-31 14:08:54

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Hi, Paul,

Thanks a lot for the feedback. Please let me know which are those feeds, and I'll look into them right away.

Yannick, I guess I've been only using the subset of feedparser that works? I've been having no (general) problems with it. Thanks for the heads up, though.

Manolo

Offline

#5 2012-12-31 15:54:55

bslackr
Member
Registered: 2012-01-27
Posts: 131

Re: Greg -- a command-line podcast aggregator

I was actually looking for something like this, I'll check it out. One thing though; I'm somewhat OCD about my directory names and prefer that every folder in ~ is in lowercase. I'll hack the source myself to change that, but could you make the directory configurable in a future version?

Edit: I really need to learn to read better, that is already a feature.

Last edited by bslackr (2012-12-31 15:57:51)

Offline

#6 2012-12-31 16:13:48

bslackr
Member
Registered: 2012-01-27
Posts: 131

Re: Greg -- a command-line podcast aggregator

Found a bug, greg doesn't seem to create the .local/share/greg directory.

Traceback (most recent call last):
  File "/usr/bin/greg", line 18, in <module>
    main()
  File "/usr/lib/python3.3/site-packages/greg/gregparser.py", line 90, in main
    args.func(args)
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 126, in add
    with open(DATA_FILENAME, 'w') as configfile:
FileNotFoundError: [Errno 2] No such file or directory: '/home/brendan/.local/share/greg/data/data'

It works properly after I manually create the ~/.local/share/greg/data directory.

Offline

#7 2012-12-31 17:59:10

stealthy
Member
Registered: 2011-05-02
Posts: 67

Re: Greg -- a command-line podcast aggregator

Looks good, I feel like I'm reading a commercial though. Your a little bit ahead of clipodder, as greg can download based off date.


clipodder-git A small simple cron-friendly podcast downloader, with support for arbitrary user defined media types (pdf, html, etc...)

Offline

#8 2012-12-31 18:29:01

Paul-S
Member
From: Wales
Registered: 2008-02-04
Posts: 343

Re: Greg -- a command-line podcast aggregator

Hi Manolo, http://www.hwhq.com/rssOGG.xml fails

Cheers
Paul-S

Last edited by Paul-S (2012-12-31 19:08:48)

Offline

#9 2013-01-01 17:05:13

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

bslackr wrote:

Found a bug, greg doesn't seem to create the .local/share/greg directory.

Traceback (most recent call last):
  File "/usr/bin/greg", line 18, in <module>
    main()
  File "/usr/lib/python3.3/site-packages/greg/gregparser.py", line 90, in main
    args.func(args)
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 126, in add
    with open(DATA_FILENAME, 'w') as configfile:
FileNotFoundError: [Errno 2] No such file or directory: '/home/brendan/.local/share/greg/data/data'

It works properly after I manually create the ~/.local/share/greg/data directory.

Indeed! I wonder why you are the only one that have had this issue. It seems to me that everybody should have seen the same error. This is now fixed in the latest commit. Thanks.

Manolo

Offline

#10 2013-01-01 17:06:27

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

stealthy wrote:

Looks good, I feel like I'm reading a commercial though. Your a little bit ahead of clipodder, as greg can download based off date.

Do you mean the README? It's maybe a bit too silly, yes; I'll think about it smile

Cheers,
Manolo

Offline

#11 2013-01-01 17:07:28

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Paul-S wrote:

Hi Manolo, http://www.hwhq.com/rssOGG.xml fails

Cheers
Paul-S

I'm on it. Once the new year social commitments are over, I'll push a fix for this feed and, hopefully, a more general one.

M

Offline

#12 2013-01-01 17:28:26

mamamia88
Member
Registered: 2012-08-29
Posts: 483

Re: Greg -- a command-line podcast aggregator

pretty cool.   i'll stick with my gpodder but it's always nice to have an alternative.

Offline

#13 2013-01-14 06:13:26

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Paul-S wrote:

Hi Manolo, http://www.hwhq.com/rssOGG.xml fails

Cheers
Paul-S

If you can, please reinstall Greg, and let me know if it deals with this feed (and any other) satisfactorily now.

Cheers,
Manolo

Offline

#14 2013-01-14 17:22:06

Paul-S
Member
From: Wales
Registered: 2008-02-04
Posts: 343

Re: Greg -- a command-line podcast aggregator

Hi Manolo, I now get 'I cannot parse the time information of this feed' errors on multiple feeds.

Cheers
Paul-S

Offline

#15 2013-01-14 18:52:32

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Paul-S wrote:

Hi Manolo, I now get 'I cannot parse the time information of this feed' errors on multiple feeds.

Cheers
Paul-S

That is a (possibly obnoxious, but) merely informative message. I have changed it to a less alarming one smile

The feeds should be downloading all the same, though. Are they?

Thanks,

Manolo

Last edited by manolomartinez (2013-01-15 15:21:55)

Offline

#16 2013-01-14 19:02:24

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Hi, just a summary of changes in the latest commits:

* greg check now can retrieve the information for a feed we have not added to our list, using the --url flag. This should be useful if one just want to check a certain podcast out.

* There is a new config option, firstsync, which tells Greg what to do in a first sync of a podcast: download 1, 2, ... n issues, or download all.

* Greg retains its ability to sync from a certain date (the --downloadfrom flag in greg edit), but can now deal with feeds which lack date information.

* There are a bunch of new sanity checks, to avoid choking on defective feeds.

Updating greg-git should be enough to have these new features. I have tried to make sure that no user intervention was necessary. But let me know if I'm wrong.

More soon!
Manolo

Offline

#17 2013-01-28 03:12:43

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Hi, some new things that Greg can do. as of right now:

* It can now download any kind of enclosures (video, audio, torrents, whatever)
* It allows the user to specify custom download handlers. For example, if you are partial to wget, you can now use "wget {link} -P {directory}" to retrieve your podcasts. If you don't want to download, just listen, you can specify "mplayer {link}" instead.

I've also fixed a bunch of bugs. I'm learning as I go along, so there will surely be a bunch more smile

Cheers,
Manolo

Offline

#18 2013-02-18 21:15:48

oloꟼ
Member
From: France
Registered: 2012-01-05
Posts: 15

Re: Greg -- a command-line podcast aggregator

Hi Manolo,

I like to rename the downloaded files to something like "podcast_name/date - title.mp3". I have:

#greg.conf
downloadhandler = wget {link} -O {directory}/{title}.mp3

but greg expects the original filename for tags setting (Tag option set to yes):

2013-02-18 21:55:54 (464 KB/s) - «/home/polo/Podcasts/gerra/J'ai attrapé le chameau du terroriste !.mp3» saved [8890018/8890018]

Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 144, in tag
    stagger.util.set_frames(podpath, {"artist":podcast.feed.title})
  File "/usr/lib/python3.3/site-packages/stagger/util.py", line 93, in set_frames
    tag = stagger.read_tag(filename)
  File "/usr/lib/python3.3/site-packages/stagger/tags.py", line 80, in read_tag
    with fileutil.opened(filename, "rb") as file:
  File "/usr/lib/python3.3/contextlib.py", line 48, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.3/site-packages/stagger/fileutil.py", line 54, in opened
    file = open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/polo/Podcasts/gerra/273073201584.mp3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/greg", line 18, in <module>
    main()
  File "/usr/lib/python3.3/site-packages/greg/gregparser.py", line 101, in main
    args.func(vars(args))
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 419, in sync
    tag(feed, entry, podcast, podpath)
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 146, in tag
    stagger.util.set_frames(podpath, {"artist":feed})
  File "/usr/lib/python3.3/site-packages/stagger/util.py", line 93, in set_frames
    tag = stagger.read_tag(filename)
  File "/usr/lib/python3.3/site-packages/stagger/tags.py", line 80, in read_tag
    with fileutil.opened(filename, "rb") as file:
  File "/usr/lib/python3.3/contextlib.py", line 48, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.3/site-packages/stagger/fileutil.py", line 54, in opened
    file = open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/polo/Podcasts/gerra/273073201584.mp3'

I'd also love a {date} placeholder (should use the international format yyyy-mm-dd for sorting). I also have problems when the title contains spaces, " or '. I think that filesystem-safe {title} and {directory} could be very helpful.

Thank you for your time and great project.

EDIT:

shlex.quote seems to solve the filename issue:

instruction = value.format(link = link, filename = filename, directory = shlex.quote(directory), fullpath = fullpath, title = shlex.quote(title))

but I don't know if it's a clean way to deal with it.

Last edited by oloꟼ (2013-02-18 21:45:36)

Offline

#19 2013-02-20 03:02:20

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

oloꟼ wrote:

Hi Manolo,

I like to rename the downloaded files to something like "podcast_name/date - title.mp3". I have:

#greg.conf
downloadhandler = wget {link} -O {directory}/{title}.mp3

but greg expects the original filename for tags setting (Tag option set to yes).

Hi, Polo. Indeed, tagging support in greg can (and will) be greatly improved. Right now I am away from my usual workplace, but in a couple of weeks I hope to push changes that will allow for the user to control much more how and what to tag.

I'd also love a {date} placeholder (should use the international format yyyy-mm-dd for sorting). I also have problems when the title contains spaces, " or '. I think that filesystem-safe {title} and {directory} could be very helpful.

For the time being,

downloadhandler = wget {link} -O "{directory}/{title}.mp3"

should work, I think. Would you mind giving that a try? As for the {date} placeholder, I'm on it!

Thank you for your time and great project.

Thanks for using it, and the very helpful feedback!

Cheers,
Manolo

Offline

#20 2013-02-20 09:32:10

oloꟼ
Member
From: France
Registered: 2012-01-05
Posts: 15

Re: Greg -- a command-line podcast aggregator

downloadhandler = wget {link} -O "{directory}/{title}.mp3"

doesn't work if the directory or title contains double quotes. I've also tried with simple quotes, and escaped simple and double quotes. Also when the title contains spaces, greg considers each word as a new url.

I've found the shlex.quote solution at stackoverflow. Looks like it solves everything here. (tested with http://www.rtl.fr/podcast/laurent-gerra.xml , sorry but in french).

Offline

#21 2013-04-19 18:53:09

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Dear Polo, all,

The tip of the [development-main branch](https://github.com/manolomartinez/greg/ … pment-main) now has a much more complete tagging support, a new {date} placeholder, and has fixed the bug reported in #18 above.

I was wondering if some of you guys would care to checkout this branch and use it for some time, then report problems and bugs? The new greg.conf explains how to use the new functionality.

Thanks in advance,
Manolo

Offline

#22 2013-05-03 18:21:52

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

Hi, greg 0.3.0 is now in the aur (as greg, or greg-git). The changelog:

* Much better support for tagging -- many more tags can be now filled out, using arbitrary strings with placeholders (see greg.conf for details)
* The specification of the download directory is now much more flexible -- again, arbitrary strings with placeholders.
* A new {date} placeholder, which substitutes for the podcast entry date in a format specified by the user (the default is YYYY-MM-DD)
* There is a new file_to_tag field in the config file with which the user can specify under which name it is downloading her podcasts (necessary for tagging files when the download is not handled by greg itself).
* A number of bugs have been fixed.
* The code has been quite extensively refactored, and is now, I think, better and more readable.
* Narrat has helped improve the pkgbuild files.

Enjoy, and please let me know (here, or over at github.com/manolomartinez/greg) if there are new bugs or regressions.

Manolo

Offline

#23 2013-05-28 20:36:43

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

If you pull greg-git, you'll find a new option in greg.conf to filter in or out podcast episodes. From greg.conf:

# Finally, you don't need to download every issue of a podcast. You can specify
# a condition such that greg will download an issue if and only if the issue
# meets it. For example,
#
# filter = "BBC" in "{title}" and "Lennon" not in "{title}"
#
# The syntax here is Python's, but it should be straightforward: "and", "or" and
# "not" mean what they customarily mean, and "in" means that the string to its
# left is contained in the string to its right -- i.e., "BBC" is in "BBC News".

Offline

#24 2013-10-04 12:39:09

MekkaGodzilla
Member
Registered: 2012-01-22
Posts: 9

Re: Greg -- a command-line podcast aggregator

Just a reply to tell you that I love using greg!

Offline

#25 2013-10-04 20:13:10

manolomartinez
Member
Registered: 2011-04-29
Posts: 64
Website

Re: Greg -- a command-line podcast aggregator

MekkaGodzilla wrote:

Just a reply to tell you that I love using greg!

Thanks smile It's encouraging to know of people who find greg useful!

Remember, if there's anything that you'd like to see implemented, or something that doesn't quite work, to drop me a line here or over at github.

Cheers,
Manolo

Offline

Board footer

Powered by FluxBB