Greg -- a command-line podcast aggregator

manolomartinez · 2012-12-31 03:16:11

UPDATED: April 5 2021

The latest release (0.4.8) has moved to EyeD3 for tagging. I've also fixed a bunch of bugs.

Hi, I've been putting together a little podcast aggregator in Python. It (only) depends on python-feedparser. Just in case someone else finds it useful. From the README:

# greg
A command-line podcast aggregator, written in python. It basically exposes some
of the functionality of the excellent
[feedparser](http://pypi.python.org/pypi/feedparser).
## Installation
Use [pip](https://pip.pypa.io/en/stable/):
`pip install --user greg`
If you run [Arch Linux](https://archlinux.org/), there is also an [AUR
package](https://aur.archlinux.org/packages/greg-git/).
## Configuration
To edit the configuration for greg, copy the system-wide [greg.conf](https://github.com/manolomartinez/greg/ … /greg.conf) file to your local config folder:
```
mkdir -p ~/.config/greg && cp `greg retrieveglobalconf` ~/.config/greg/greg.conf
```
Then open and edit `~/.config/greg/greg.conf` in a text editor. The configuration file is self-explanatory.
## Usage
Let's start by adding a feed (RSS or Atom versions will do):
greg add PhilosophyBites http://philosophybites.com/atom.xml
The `add` command expects a name and a url of an RSS or Atom feed. You will use this name to refer to the feed whenever you interact with it.
If you were to run `greg sync` now, it would download the latest episode of the podcast to the default directory (which is `~/Podcasts`; you can change how many episodes are dowloaded in the first sync, and the download directory, in the config file; see below). But maybe we just want to check out what this podcast is all about, so we download a list of available entries:
greg check -f PhilosophyBites
(the `-f` flag means that "PhilosophyBites" is the name of a feed. `greg check` also accepts urls directly, using the `-u` flag.)
This will give you the following kind of info:

0: Tom Sorell on Surveillance (2013-01-25T13:43:46+00:00)
1: John Campbell on Schizophrenia (2013-01-08T12:41:27+00:00)
2: Kendall Walton on Photography (2012-12-23T12:33:09+00:00)
3: Twitter Competition: Who's Your Favourite Philosopher? (2012-12-11T07:24:51+00:00)
4: Alan Ryan on Freedom and Its History (2012-12-08T11:16:45+00:00)
5: Nigel Warburton at Blackwell's Bookshop, Oxford 7pm Wed. Dec. 5th (2012-12-01T11:19:09+00:00)
6: Who's Your Favourite Philosopher? (2012-11-30T18:33:56+00:00)
7: Peter Adamson on Avicenna's Flying Man Thought Experiment (2012-11-26T15:57:18+00:00)
8: Links to Past Episodes (2012-12-01T11:53:03+00:00)
9: Tim Bayne on the Unity of Consciousness (2012-11-11T22:20:17+00:00)
10: Galen Strawson on the Sense of Self (2012-05-05T12:56:05+01:00)
11: Liane Young on Mind and Morality (2012-10-27T12:39:22+01:00)
12: Gary L. Francione on Animal Abolitionism (2012-10-13T13:48:32+01:00)
13: Richard Sorabji on Mahatma Gandhi as Philosopher (2012-09-28T13:18:08+01:00)
14: Tim Crane on Non-Existence (2012-09-15T18:50:32+01:00)
15: Michael Tye on Pain (2012-08-31T20:51:01+01:00)
16: Daniel Dennett on Free Will Worth Wanting (2012-08-18T08:58:24+01:00)
17: Pat Churchland on What Neuroscience Can Teach Us About Morality (2012-08-03T22:52:12+01:00)
18: Rae Langton on Hate Speech (2012-07-28T20:14:27+01:00)
19: Molly Crockett on Brain Chemistry and Moral-Decision Making (originally on Bioethics Bites) (2012-07-22T21:14:35+01:00)
Interesting stuff. We'll download a couple of episodes, just to make sure that
it's really worth it:
greg download 1, 5-7
and Greg says
Downloading John Campbell on Schizophrenia -- John_Campbell_on_Schizophrenia.mp3
Done
Downloading John Campbell on Schizophrenia -- John_Campbell_on_Berkeleys_Puzzle_1.mp3
Done
Downloading Who's Your Favourite Philosopher? -- Whos_Your_Favourite_Philosopher_.mp3
Done
Downloading Peter Adamson on Avicenna's Flying Man Thought Experiment -- Peter_Adamson_on_Avicennas_Flying_Man.mp3
Done
Downloading Peter Adamson on Avicenna's Flying Man Thought Experiment -- AdamsonMixSes.MP3
Done
Downloading Peter Adamson on Avicenna's Flying Man Thought Experiment -- Peter_Adamson_on_Plotinus_on_Evil.mp3
Done
As you can see, `greg download` accepts a range of episodes of the kind `a, b,
c-f, h, ...`. The numbers make reference to the numbers at the beginning of
each entry provided by `greg check`. `check` creates a persistent file
(`feeddump` in the data directory, `~/.local/share/greg/data by` default, but
you can change that in the config file, or passing a different path with the
`--datadirectory` flag), so `download` will keep on working, and referring to
the last `check` ever done.
All of these podcasts will be downloaded to the default download directory for
the feed (if you used the `-f` flag) or the general default download directory
(again, `~/Podcasts` if you don't tell Greg otherwise. We'll learn how to
change that soon), inside a subdirectory named after the podcast (we can change
that default too.) After listening to them we decide that this podcast is well
worth our time, and keep it, or we decide that it's not, and
greg remove PhilosophyBites
If we keep it, we might want to start `sync`ing from, say, the 30th of April,
2013, on. So we edit the feed information
greg edit PhilosophyBites -d 2013-4-30
We may also use the `now` keyword to instruct greg to start syncing from now
on:
greg edit PhilosophyBites -d now
`-d` or `--downloadfrom` change the date after which Greg should start
downloading episodes when it syncs. Currently, the only two things one can
`edit` in a feed are the download-from date and `--url` -- but many more things
can be changed by editing the config file. `greg edit -h` will give help you
with the `edit` options and syntax -- likewise for the rest of Greg
subcommands.
All right. Let's add a second feed:
greg add History http://podcast.ulcc.ac.uk/accounts/king … dcasts.xml
If you want to keep track of the feeds you have added, you can ask Greg:
greg info
which returns
PhilosophyBites
---------------
url: http://philosophybites.com/atom.xml
Next sync will download from: 30 Apr 2013 00:00:00.
History
-------
url: http://podcast.ulcc.ac.uk/accounts/king … dcasts.xml
Let us add another feed:
greg add MusicaAntigua http://www.rtve.es/api/programas/23353/audios.rss
This is a great program on ancient music at the Spanish public radio. The thing
is, these guys do not tag their episodes, which is bad for most portable media
players. Greg uses [EyeD3](https://github.com/nicfit/eyeD3) (as an optional
dependency) to tag podcasts, if one so wishes. By default, it uses the podcast
name for the *artist* tag, and the entry title for the *title* tag. To enable
tagging for MusicaAntigua, copy the system-wide config file locally. (see
[Configuration](#configuration) above)
Then, add a section for MusicaAntigua:
[MusicaAntigua]
Tag = yes
In fact, you can fill out any tag however you see fit. For example,
tag_genre = Ancient Music
tag_comment = {date}
will fill the *genre* tag with the string "Ancient Music", and the *comment*
tag with the download date.
Let's add a video podcast
greg add TEDTalks http://feeds.feedburner.com/TEDTalks_video
By default, Greg only donwloads audio files (in fact, files that have "audio"
as part of their type). In order to download the right file in TEDTalks, then,
you need to change that in the config file. Again, add a section:
[TEDTalks]
mime = video
You could also have a couple of types there, as in `mime = audio, video`; or
any other type, `mime = torrent`, or whatever.
Another useful thing that you can change in the config file is the download
handler; Greg by default uses `urllib.request` from the Python standard
library, but you can use whatever you want. Indeed, it's advisable that you use
an external program for this: the default downloader just downloads,
period---it doesn't [follow
redirects](https://github.com/manolomartinez/greg/issues/98) or does any of the
other fancy stuff that, e.g., `wget` or `curl` do.
I, for example, have
downloadhandler = wget {link} -P {directory}
in my local `greg.conf`. You can do all sorts of nice things with this. For
example, when `check`ing a podcast, you don't need to download it, but maybe
just stream it, like this:
greg download 0 --downloadhandler "mplayer {link}"
One last thing: if you subscribe to a very active feed, and you are only
interested in some of the entries, you can filter the feed. For example, if you
only want to watch TED talks about Google, say, you can add the following line
to the `[TEDTalks]` section:
filter = "Google" in "{title}"
(You need the quotes around {title} if the string you are filtering by has
spaces, for example; they are strictly unnecessary here.)
For information about the {placeholders}, take a look at
[greg.conf](https://github.com/manolomartinez/greg/ … /greg.conf).
In `greg.conf` you can also change the download directory, and some other
things. It should be self-explanatory.

Cheers,
Manolo

Last edited by manolomartinez (2021-04-05 16:39:44)

Yannick_LM · 2012-12-31 11:35:31

Just a heads up: feedparser is broken for python3 right now.

The patch that fixes it already on the master branch:

http://code.google.com/p/feedparser/iss … ail?id=384

I've also open an issue in Arch's bug tracker:

https://bugs.archlinux.org/task/33254

Edit: The bug is fixed in python-feedparser-5.1.3-2

That was quick

Last edited by Yannick_LM (2012-12-31 13:19:10)

Paul-S · 2012-12-31 13:01:44

Hi Manolo, I get a couple of errors on a few of my feeds either a - Something has gone wrong: object has no attribute 'parsed' or Something has gone wrong: object has no attribute 'title'. I assume these are problems with the rss feeds but they work ok in podget.

Cheers
Paul-S

manolomartinez · 2012-12-31 14:08:54

Hi, Paul,

Thanks a lot for the feedback. Please let me know which are those feeds, and I'll look into them right away.

Yannick, I guess I've been only using the subset of feedparser that works? I've been having no (general) problems with it. Thanks for the heads up, though.

Manolo

bslackr · 2012-12-31 15:54:55

I was actually looking for something like this, I'll check it out. One thing though; I'm somewhat OCD about my directory names and prefer that every folder in ~ is in lowercase. I'll hack the source myself to change that, but could you make the directory configurable in a future version?

Edit: I really need to learn to read better, that is already a feature.

Last edited by bslackr (2012-12-31 15:57:51)

bslackr · 2012-12-31 16:13:48

Found a bug, greg doesn't seem to create the .local/share/greg directory.

Traceback (most recent call last):
  File "/usr/bin/greg", line 18, in <module>
    main()
  File "/usr/lib/python3.3/site-packages/greg/gregparser.py", line 90, in main
    args.func(args)
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 126, in add
    with open(DATA_FILENAME, 'w') as configfile:
FileNotFoundError: [Errno 2] No such file or directory: '/home/brendan/.local/share/greg/data/data'

It works properly after I manually create the ~/.local/share/greg/data directory.

stealthy · 2012-12-31 17:59:10

Looks good, I feel like I'm reading a commercial though. Your a little bit ahead of clipodder, as greg can download based off date.

Paul-S · 2012-12-31 18:29:01

Hi Manolo, http://www.hwhq.com/rssOGG.xml fails

Cheers
Paul-S

Last edited by Paul-S (2012-12-31 19:08:48)

manolomartinez · 2013-01-01 17:05:13

bslackr wrote:

Found a bug, greg doesn't seem to create the .local/share/greg directory.

Traceback (most recent call last):
  File "/usr/bin/greg", line 18, in <module>
    main()
  File "/usr/lib/python3.3/site-packages/greg/gregparser.py", line 90, in main
    args.func(args)
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 126, in add
    with open(DATA_FILENAME, 'w') as configfile:
FileNotFoundError: [Errno 2] No such file or directory: '/home/brendan/.local/share/greg/data/data'

It works properly after I manually create the ~/.local/share/greg/data directory.

Indeed! I wonder why you are the only one that have had this issue. It seems to me that everybody should have seen the same error. This is now fixed in the latest commit. Thanks.

Manolo

manolomartinez · 2013-01-01 17:06:27

stealthy wrote:

Looks good, I feel like I'm reading a commercial though. Your a little bit ahead of clipodder, as greg can download based off date.

Do you mean the README? It's maybe a bit too silly, yes; I'll think about it

Cheers,
Manolo

manolomartinez · 2013-01-01 17:07:28

Paul-S wrote:

Hi Manolo, http://www.hwhq.com/rssOGG.xml fails
Cheers
Paul-S

I'm on it. Once the new year social commitments are over, I'll push a fix for this feed and, hopefully, a more general one.

M

mamamia88 · 2013-01-01 17:28:26

pretty cool. i'll stick with my gpodder but it's always nice to have an alternative.

manolomartinez · 2013-01-14 06:13:26

Paul-S wrote:

Hi Manolo, http://www.hwhq.com/rssOGG.xml fails
Cheers
Paul-S

If you can, please reinstall Greg, and let me know if it deals with this feed (and any other) satisfactorily now.

Cheers,
Manolo

Paul-S · 2013-01-14 17:22:06

Hi Manolo, I now get 'I cannot parse the time information of this feed' errors on multiple feeds.

Cheers
Paul-S

manolomartinez · 2013-01-14 18:52:32

Paul-S wrote:

Hi Manolo, I now get 'I cannot parse the time information of this feed' errors on multiple feeds.
Cheers
Paul-S

That is a (possibly obnoxious, but) merely informative message. I have changed it to a less alarming one

The feeds should be downloading all the same, though. Are they?

Thanks,

Manolo

Last edited by manolomartinez (2013-01-15 15:21:55)

manolomartinez · 2013-01-14 19:02:24

Hi, just a summary of changes in the latest commits:

* greg check now can retrieve the information for a feed we have not added to our list, using the --url flag. This should be useful if one just want to check a certain podcast out.

* There is a new config option, firstsync, which tells Greg what to do in a first sync of a podcast: download 1, 2, ... n issues, or download all.

* Greg retains its ability to sync from a certain date (the --downloadfrom flag in greg edit), but can now deal with feeds which lack date information.

* There are a bunch of new sanity checks, to avoid choking on defective feeds.

Updating greg-git should be enough to have these new features. I have tried to make sure that no user intervention was necessary. But let me know if I'm wrong.

More soon!
Manolo

manolomartinez · 2013-01-28 03:12:43

Hi, some new things that Greg can do. as of right now:

* It can now download any kind of enclosures (video, audio, torrents, whatever)
* It allows the user to specify custom download handlers. For example, if you are partial to wget, you can now use "wget {link} -P {directory}" to retrieve your podcasts. If you don't want to download, just listen, you can specify "mplayer {link}" instead.

I've also fixed a bunch of bugs. I'm learning as I go along, so there will surely be a bunch more

Cheers,
Manolo

oloꟼ · 2013-02-18 21:15:48

Hi Manolo,

I like to rename the downloaded files to something like "podcast_name/date - title.mp3". I have:

#greg.conf
downloadhandler = wget {link} -O {directory}/{title}.mp3

but greg expects the original filename for tags setting (Tag option set to yes):

2013-02-18 21:55:54 (464 KB/s) - «/home/polo/Podcasts/gerra/J'ai attrapé le chameau du terroriste !.mp3» saved [8890018/8890018]

Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 144, in tag
    stagger.util.set_frames(podpath, {"artist":podcast.feed.title})
  File "/usr/lib/python3.3/site-packages/stagger/util.py", line 93, in set_frames
    tag = stagger.read_tag(filename)
  File "/usr/lib/python3.3/site-packages/stagger/tags.py", line 80, in read_tag
    with fileutil.opened(filename, "rb") as file:
  File "/usr/lib/python3.3/contextlib.py", line 48, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.3/site-packages/stagger/fileutil.py", line 54, in opened
    file = open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/polo/Podcasts/gerra/273073201584.mp3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/greg", line 18, in <module>
    main()
  File "/usr/lib/python3.3/site-packages/greg/gregparser.py", line 101, in main
    args.func(vars(args))
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 419, in sync
    tag(feed, entry, podcast, podpath)
  File "/usr/lib/python3.3/site-packages/greg/greg.py", line 146, in tag
    stagger.util.set_frames(podpath, {"artist":feed})
  File "/usr/lib/python3.3/site-packages/stagger/util.py", line 93, in set_frames
    tag = stagger.read_tag(filename)
  File "/usr/lib/python3.3/site-packages/stagger/tags.py", line 80, in read_tag
    with fileutil.opened(filename, "rb") as file:
  File "/usr/lib/python3.3/contextlib.py", line 48, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.3/site-packages/stagger/fileutil.py", line 54, in opened
    file = open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/polo/Podcasts/gerra/273073201584.mp3'

I'd also love a {date} placeholder (should use the international format yyyy-mm-dd for sorting). I also have problems when the title contains spaces, " or '. I think that filesystem-safe {title} and {directory} could be very helpful.

Thank you for your time and great project.

EDIT:

shlex.quote seems to solve the filename issue:

instruction = value.format(link = link, filename = filename, directory = shlex.quote(directory), fullpath = fullpath, title = shlex.quote(title))

but I don't know if it's a clean way to deal with it.

Last edited by oloꟼ (2013-02-18 21:45:36)

manolomartinez · 2013-02-20 03:02:20

oloꟼ wrote:

Hi Manolo,
I like to rename the downloaded files to something like "podcast_name/date - title.mp3". I have:
#greg.conf
downloadhandler = wget {link} -O {directory}/{title}.mp3
but greg expects the original filename for tags setting (Tag option set to yes).

Hi, Polo. Indeed, tagging support in greg can (and will) be greatly improved. Right now I am away from my usual workplace, but in a couple of weeks I hope to push changes that will allow for the user to control much more how and what to tag.

I'd also love a {date} placeholder (should use the international format yyyy-mm-dd for sorting). I also have problems when the title contains spaces, " or '. I think that filesystem-safe {title} and {directory} could be very helpful.

For the time being,

downloadhandler = wget {link} -O "{directory}/{title}.mp3"

should work, I think. Would you mind giving that a try? As for the {date} placeholder, I'm on it!

Thank you for your time and great project.

Thanks for using it, and the very helpful feedback!

Cheers,
Manolo

oloꟼ · 2013-02-20 09:32:10

downloadhandler = wget {link} -O "{directory}/{title}.mp3"

doesn't work if the directory or title contains double quotes. I've also tried with simple quotes, and escaped simple and double quotes. Also when the title contains spaces, greg considers each word as a new url.

I've found the shlex.quote solution at stackoverflow. Looks like it solves everything here. (tested with http://www.rtl.fr/podcast/laurent-gerra.xml , sorry but in french).

manolomartinez · 2013-04-19 18:53:09

Dear Polo, all,

The tip of the [development-main branch](https://github.com/manolomartinez/greg/ … pment-main) now has a much more complete tagging support, a new {date} placeholder, and has fixed the bug reported in #18 above.

I was wondering if some of you guys would care to checkout this branch and use it for some time, then report problems and bugs? The new greg.conf explains how to use the new functionality.

Thanks in advance,
Manolo

manolomartinez · 2013-05-03 18:21:52

Hi, greg 0.3.0 is now in the aur (as greg, or greg-git). The changelog:

* Much better support for tagging -- many more tags can be now filled out, using arbitrary strings with placeholders (see greg.conf for details)
* The specification of the download directory is now much more flexible -- again, arbitrary strings with placeholders.
* A new {date} placeholder, which substitutes for the podcast entry date in a format specified by the user (the default is YYYY-MM-DD)
* There is a new file_to_tag field in the config file with which the user can specify under which name it is downloading her podcasts (necessary for tagging files when the download is not handled by greg itself).
* A number of bugs have been fixed.
* The code has been quite extensively refactored, and is now, I think, better and more readable.
* Narrat has helped improve the pkgbuild files.

Enjoy, and please let me know (here, or over at github.com/manolomartinez/greg) if there are new bugs or regressions.

Manolo

manolomartinez · 2013-05-28 20:36:43

If you pull greg-git, you'll find a new option in greg.conf to filter in or out podcast episodes. From greg.conf:

# Finally, you don't need to download every issue of a podcast. You can specify
# a condition such that greg will download an issue if and only if the issue
# meets it. For example,
#
# filter = "BBC" in "{title}" and "Lennon" not in "{title}"
#
# The syntax here is Python's, but it should be straightforward: "and", "or" and
# "not" mean what they customarily mean, and "in" means that the string to its
# left is contained in the string to its right -- i.e., "BBC" is in "BBC News".

MekkaGodzilla · 2013-10-04 12:39:09

Just a reply to tell you that I love using greg!

manolomartinez · 2013-10-04 20:13:10

MekkaGodzilla wrote:

Just a reply to tell you that I love using greg!

Thanks It's encouraging to know of people who find greg useful!

Remember, if there's anything that you'd like to see implemented, or something that doesn't quite work, to drop me a line here or over at github.

Cheers,
Manolo

Arch Linux

#1 2012-12-31 03:16:11

Greg -- a command-line podcast aggregator

#2 2012-12-31 11:35:31

Re: Greg -- a command-line podcast aggregator

#3 2012-12-31 13:01:44

Re: Greg -- a command-line podcast aggregator

#4 2012-12-31 14:08:54

Re: Greg -- a command-line podcast aggregator

#5 2012-12-31 15:54:55

Re: Greg -- a command-line podcast aggregator

#6 2012-12-31 16:13:48

Re: Greg -- a command-line podcast aggregator

#7 2012-12-31 17:59:10

Re: Greg -- a command-line podcast aggregator

#8 2012-12-31 18:29:01

Re: Greg -- a command-line podcast aggregator

#9 2013-01-01 17:05:13

Re: Greg -- a command-line podcast aggregator

#10 2013-01-01 17:06:27

Re: Greg -- a command-line podcast aggregator

#11 2013-01-01 17:07:28

Re: Greg -- a command-line podcast aggregator

#12 2013-01-01 17:28:26

Re: Greg -- a command-line podcast aggregator

#13 2013-01-14 06:13:26

Re: Greg -- a command-line podcast aggregator

#14 2013-01-14 17:22:06

Re: Greg -- a command-line podcast aggregator

#15 2013-01-14 18:52:32

Re: Greg -- a command-line podcast aggregator

#16 2013-01-14 19:02:24

Re: Greg -- a command-line podcast aggregator

#17 2013-01-28 03:12:43

Re: Greg -- a command-line podcast aggregator

#18 2013-02-18 21:15:48

Re: Greg -- a command-line podcast aggregator

#19 2013-02-20 03:02:20

Re: Greg -- a command-line podcast aggregator

#20 2013-02-20 09:32:10

Re: Greg -- a command-line podcast aggregator

#21 2013-04-19 18:53:09

Re: Greg -- a command-line podcast aggregator

#22 2013-05-03 18:21:52

Re: Greg -- a command-line podcast aggregator

#23 2013-05-28 20:36:43

Re: Greg -- a command-line podcast aggregator

#24 2013-10-04 12:39:09

Re: Greg -- a command-line podcast aggregator

#25 2013-10-04 20:13:10

Re: Greg -- a command-line podcast aggregator

Board footer