Voracious: customizable web feed aggregator for your browser

Xyne · 2009-11-22 05:41:58

Info page: http://xyne.archlinux.ca/info/voracious

I came across an interesting site the other day and started to add its rss feeds to my Firefox bookmarks when I realized that I have too many of them. Even with a nice hierarchy there are just too many branches and it's too annoying to jump between them all the time. I thought it would be nice if I could "merge" multiple feeds in a way that would give me a nice overview and let me quickly navigate to what I want.

I ended up with voracious.

The man page explains it in a bit more detail, but basically voracious is a web feed aggregator in a small server that you can access through your browser. It's very customizable both in content and appearance and is meant to provide a single location for all of your web feeds. For example, you could use it to do the following:

merge all of your
general news feeds under "http://localhost:8080/news"
tech news feeds under "http://localhost:8080/news/tech"
science news feeds under "http://localhost:8080/news/science"
web comics under "http://localhost:8080/comics"
words of the day under ...

You get the point.

Quick list of features:
fully customizable html/css template (so don't bitch if you think the default is ugly)
use of threads to get feeds faster
reloads the config and the template files on the fly
in-browser config editing
only retrieves what you actually look at (better privacy through not letting every single server know you're online even when you're not looking at their content)

I've posted a simple example config and template on the info page to show some per-feed colors. It should give you some ideas of what you can do with it. If you come up with a nice template or interesting config, post it here.

Nezmer · 2009-11-22 12:48:22

Didn't try this yet. But I want to thank you in advance for your effort.

I was looking for a Google Reader alternative the other day. Something I can host in my own server. I only found Tiny Tiny RSS and this blog post. So we definitely need more alternatives.

Nezmer · 2009-11-22 13:16:14

Just tried it.
Holy crap , Voracious is awesome.

Xyne · 2009-11-22 14:00:25

Thanks

Nezmer · 2009-11-22 15:09:58

Unfortunately , python-feedparser seems to have a lot of bugs (Or maybe It's just my setup). Some Arch feeds are not working for me.

[paths]
/ArchFeeds
  http://aur.archlinux.org/rss2.php
  http://bugs.archlinux.org/feed.php?feed_type=rss2&project=1
  http://planet.archlinux.org/atom.xml
  http://www.archlinux.org/feeds/news/
  http://www.archlinux.org/feeds/packages/
/example
  http://xkcd.com/rss.xml
  http://rss.slashdot.org/Slashdot/slashdot

[classes]
arch
  http://aur.archlinux.org/rss2.php
  http://bugs.archlinux.org/feed.php?feed_type=rss2&project=1
  http://planet.archlinux.org/atom.xml
  http://www.archlinux.org/feeds/news/
  http://www.archlinux.org/feeds/packages/
slashdot
  http://rss.slashdot.org/Slashdot/slashdot
xkcd
  http://xkcd.com/rss.xml

Bugs,Aur and packages feeds are not working for me.

Xyne · 2009-11-22 15:42:41

I've uploaded a quick fix for now to let it gracefully ignore the errors. I'm going to clean up a bit of code and see if I can figure out what's causing them.

Xyne · 2009-11-22 19:15:58

Ok, I've cleaned up the code and added some more options (download intervals, accept malformed xml, custom templates per path , etc).

@Nezmer
The errors that you received are due to malformed xml (according to feedparser) in the following feeds:
http://aur.archlinux.org/rss2.php
http://bugs.archlinux.org/feed.php?feed … &project=1

You can add the "--nobozo" option to ignore feedparser's "bozo" flag and thus accept the malformed xml (seems to work).

*edit*
[classes] and [templates] entries are now regular expressions, e.g.

[classes]
arch
  http://aur.archlinux.org/rss2.php
  http://bugs.archlinux.org/feed.php?feed_type=rss2&project=1
  http://planet.archlinux.org/atom.xml
  http://www.archlinux.org/feeds/news/
  http://www.archlinux.org/feeds/packages/

could be changed to

[classes]
arch
  archlinux

Last edited by Xyne (2009-11-22 20:15:12)

Nezmer · 2009-11-22 20:17:49

Every thing seems to be working now.
Apparently the Aur changed the feed link from ..../rss2.php to .../rss.php last month.

Implementing simple authentication would make this perfect If users want to host their feeds in remote servers.

Thanks again for this great web based aggregator.

Xyne · 2009-11-22 20:38:06

Nezmer wrote:

Implementing simple authentication would make this perfect If users want to host their feeds in remote servers.

That crossed my mind as well. I think the easiest way will be to simply disable the in-browser editor and let users edit the config file via some ssh, for example (which would still get reloaded on the fly).

Nezmer · 2009-11-22 20:45:46

Xyne wrote:

Nezmer wrote:
Implementing simple authentication would make this perfect If users want to host their feeds in remote servers.
That crossed my mind as well. I think the easiest way will be to simply disable the in-browser editor and let users edit the config file via some ssh, for example (which would still get reloaded on the fly).

Yeah , that would solve the write(edit) part but not the read(view) part.

Last edited by Nezmer (2009-11-22 20:46:10)

Xyne · 2009-11-22 20:54:25

Nezmer wrote:

Yeah , that would solve the write(edit) part but not the read(view) part.

I don't think BaseHTTPServer supports any type of authentication. I would have to move it over to something a bit more robust in that case.
I'll leave it as-is for now and give it some more thought.

iphitus · 2009-11-23 05:02:45

For someone wanting to host it, I used 'rawdog' for years. It ran in a cronjob generating the static html hourly and uploaded it to my web host.

Voracious could output to static html and do the same thing - then there's no need for a http server in voracious and you can use the authentication mechanisms provided by an existing web server. Or have voracious just print it all to to stdout and you could run it as CGI if you want it generated on demand...

Last edited by iphitus (2009-11-23 05:11:02)

Gen2ly · 2009-11-23 05:19:53

I remember using Gregarius about a year ago and really liked it. Unfortunately Gregarius isn't being developed anymore and broke on one of my updates. Gregarius required Apache or lighttpd to be installed, looking at the pkgbuild I'm guessing this doesn't? I've thought about setting up a RSS server again but haven't been too excited too because Gregarius took a lot of work.

Xyne · 2009-11-23 23:05:58

Voracious was written to manage rss feeds locally and easily. I see it as one step up from bookmarking rss fees and I wanted to be able to do everything from within the browser as I could before. The codebase could be tweaked for server deployment (I don't even think you would need to change much... the Aggregator class should be independent of HTTPServer) but I don't want to refocus it on that so I'm not going to make too many changes or any major changes in that direction.

As it is now, you could easily get the html using something as simple as "curl" and then pipe or send that wherever you want. If you leave it bound to localhost then you should be able to run it securely on a public server.

In other news, I've included support for local content such as external css and javascript files, images, etc. You will need to specify a directory with "--include-dir" (there is no default content directory, to prevent accidental sharing). This should make template editing more versatile.

I've also changed the default replacement for "$NAVLINKS$" to only include links to children and ancestors of the current path. "$ALLNAVLINKS$" can now be used for a complete URL list.

I've also made it possible to include "dummy" paths just for the sake of navigation, e.g.:

[paths]
/foo
/foo/bar
  http://example.com/1.xml
  http://example.com/2.xml
/foo/baz
  http://example.com/3.xml
/foo/whatever
  http://example.com/4.xml

In the example above, "$NAVLINKS$ for "/foo" would be replaced with links to "bar", "baz" and "whatever", but there would be no feeds on that page.

Xyne · 2009-12-13 19:46:14

I've added an optional shutdown button to the browser interface and a "--noedit" option to block editing and shutdown from the browser interface as well.

fflarex · 2010-01-13 17:27:28

This is a great piece of software, but I seem to be having some trouble. The default template file only shows the feed summary, how can I get it to show the full content for feeds which include both? I tried just switching $entries.summary$ with $entries.content$, but I suddenly started getting errors for this feed:

Exception happened during processing of request from ('127.0.0.1', 35820)
Traceback (most recent call last):
  File "/usr/lib/python2.6/SocketServer.py", line 281, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 307, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
    self.handle()
  File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
    method()
  File "/usr/bin/voracious", line 185, in do_GET
    content, found = agg.get_html(self.path)
  File "/usr/bin/voracious", line 472, in get_html
    item_html = item_html.replace('$' + '.'.join(tag) + '$', replacement)
TypeError: expected a character buffer object

Now my page is just blank. If I remove that feed everything is mostly fine, except that the other feeds don't provide any content (it just shows the feed title, entry title, date; but no content).

Last edited by fflarex (2010-01-13 17:29:31)

Xyne · 2010-01-13 21:05:54

I've added some code to handle the nested attributes. Try "$entries.content.value$" with the latest version and let me know if that works as expected.

x0rg · 2010-01-13 21:09:03

could you post a screenshot of Voracious running (design doesn't matter) ?

fflarex · 2010-01-13 21:33:42

It works, thanks!

By the way, does voracious do any caching or support proxies?

Xyne · 2010-01-13 21:51:38

x0rg wrote:

could you post a screenshot of Voracious running (design doesn't matter) ?

I've added some screenshots but considering that the user can fully change all aspects of the display and content, they don't really mean anything, especially considering my lack of artistic skill when it comes to web design.

fflarex wrote:

By the way, does voracious do any caching or support proxies?

It caches feeds internally the first time that they're requested and updates them during a subsequent request if they're older than the retrieval interval, which can be set from the command line.

I'll look into adding proxy support.

*edit*
I've added a "--proxy" option. I've only tested it locally with privoxy and tor though, but it seems to work. Please try it and confirm that it does.

Last edited by Xyne (2010-01-13 22:19:10)

x0rg · 2010-01-13 23:06:58

If I make my own RSS-Feed and save it to /home/user/feed.xml, is it possible to fetch that feed with Voracious?
Because that way I could use it as notifier for finished torrents and other alerts.

Last edited by x0rg (2010-01-13 23:07:36)

Xyne · 2010-01-13 23:34:31

@xOrg
I've just tested that and it works.

onearm · 2010-01-29 08:41:40

First of all, great idea Xyne!

I'm testing voracious as a Google Reader replacement and it seems to miss (or I haven't looked well enough) an important feature: limit the age of the showed items. There seems to not be a way to make the it retrieves only the latest (let's say 1 or 2 days old) items in the feeds and it makes impossible to know what is read and what is new.
I studied the template but I don't know if it is supported an if...else based on entries.published

timetrap · 2010-01-29 11:45:14

This is very cool. I am such a google reader junkie ...

Xyne · 2010-01-29 23:31:58

@onearm

I've added an "--age-limit" option that lets you set the age limit in minutes (I thought about using hours first, but minutes is more flexible). I might move that option to the configuration file to enable individual limits per feed and maybe some other options (show only x most recent items, etc).

I'll give it some more thought when I have more time.

Btw, has anyone come up with some nice templates that they'd like to share? If we get a small collection of them, I can host them on my site for others to download.

Arch Linux

#1 2009-11-22 05:41:58

Voracious: customizable web feed aggregator for your browser

#2 2009-11-22 12:48:22

Re: Voracious: customizable web feed aggregator for your browser

#3 2009-11-22 13:16:14

Re: Voracious: customizable web feed aggregator for your browser

#4 2009-11-22 14:00:25

Re: Voracious: customizable web feed aggregator for your browser

#5 2009-11-22 15:09:58

Re: Voracious: customizable web feed aggregator for your browser

#6 2009-11-22 15:42:41

Re: Voracious: customizable web feed aggregator for your browser

#7 2009-11-22 19:15:58

Re: Voracious: customizable web feed aggregator for your browser

#8 2009-11-22 20:17:49

Re: Voracious: customizable web feed aggregator for your browser

#9 2009-11-22 20:38:06

Re: Voracious: customizable web feed aggregator for your browser

#10 2009-11-22 20:45:46

Re: Voracious: customizable web feed aggregator for your browser

#11 2009-11-22 20:54:25

Re: Voracious: customizable web feed aggregator for your browser

#12 2009-11-23 05:02:45

Re: Voracious: customizable web feed aggregator for your browser

#13 2009-11-23 05:19:53

Re: Voracious: customizable web feed aggregator for your browser

#14 2009-11-23 23:05:58

Re: Voracious: customizable web feed aggregator for your browser

#15 2009-12-13 19:46:14

Re: Voracious: customizable web feed aggregator for your browser

#16 2010-01-13 17:27:28

Re: Voracious: customizable web feed aggregator for your browser

#17 2010-01-13 21:05:54

Re: Voracious: customizable web feed aggregator for your browser

#18 2010-01-13 21:09:03

Re: Voracious: customizable web feed aggregator for your browser

#19 2010-01-13 21:33:42

Re: Voracious: customizable web feed aggregator for your browser

#20 2010-01-13 21:51:38

Re: Voracious: customizable web feed aggregator for your browser

#21 2010-01-13 23:06:58

Re: Voracious: customizable web feed aggregator for your browser

#22 2010-01-13 23:34:31

Re: Voracious: customizable web feed aggregator for your browser

#23 2010-01-29 08:41:40

Re: Voracious: customizable web feed aggregator for your browser

#24 2010-01-29 11:45:14

Re: Voracious: customizable web feed aggregator for your browser

#25 2010-01-29 23:31:58

Re: Voracious: customizable web feed aggregator for your browser

Board footer