Voracious: customizable web feed aggregator for your browser

onearm · 2010-01-30 08:15:18

Xyne wrote:

@onearm
I've added an "--age-limit" option that lets you set the age limit in minutes (I thought about using hours first, but minutes is more flexible). I might move that option to the configuration file to enable individual limits per feed and maybe some other options (show only x most recent items, etc).
I'll give it some more thought when I have more time.

That would be great but:

 >: voracious --age-limit 2800
Usage: voracious [options]

voracious: error: no such option: --age-limit

Minutes are fine for me as long as that span of time applies to every feed (meaning, there won't be any 1 month old entries just because that is the last update from a feed).

About the template, this is mine, still need to tweak it a bit but have changed the colours to go accordingly with my awesomewm theme and added both summary and entries content because some feed show enough in the summary and others haven't got it at all:

<html>
  <head>
    <style type="text/css">
    body {
      margin-left: 205px;
      background: #2e3436;
    }
    #navbar {
      color: #2e3436;
      background: #ef8764;
      margin-left: -200px;
      width: 190px;
      position: absolute;
      padding-bottom: 5px;

    }
    .navlink {
      color: inherit;
      display: block;
      margin: 2px 10px;
    }
    #editlink {
      color: inherit;
      display: block;
      margin: 2px 10px;
    }
    .item {
      background: #d8d8d8;
      margin: 20px auto;
      text-align: justify;
    }
    .item_title {
      color: #ef8764;
      background: #2e3436;
      padding: 5px 10px;
      font-weight: bold;
      border-bottom: solid 2px #000000;
    }
    .item_meta {
      font-weight: bold;
      padding: 5px 10px;
    }
    .item_title a {
      color: inherit;
      text-decoration: none;
    }
    .item_summary {
      padding: 10px;
    }
    .item_content {
        padding: 10px;
    }
    </style>
  </head>
  <body>
    <div id="navbar">
      <div class="item_title">List</div>
      $NAVLINKS$
      <br/>
      $EDITLINK$
    </div>
    $ITEM$
      <div class="item $CLASS$">
        <div class="item_title"><a href="$entries.link$">$feed.title$ : $entries.title$</a></div>
        <div class="item_meta">$entries.date$</div>
        <div class="item_summary">$entries.summary$</div>
        <div class="item_content">$entries.content.value$</div>
      </div>
    $/ITEM$
  </body>
</html>

Xyne · 2010-01-30 08:31:27

Did you update to the latest version? (>=1.2)

I just downloaded it again from my repo to check, and the option is there and working. The AUR also has the latest version.

bauerbill -Ss --aur voracious
xyne-any/voracious 1.2-1
    A customizable web feed aggregator for your browser.

AUR/voracious 1.2-1
    A customizable web feed aggregator for your browser.

onearm · 2010-01-30 09:02:28

I was sure to have the latest version but apparently yaourt pulled the 1.1 instead of 1.2. Now it's working. Sorry, my fault.

firecat53 · 2010-10-01 17:58:40

Hey xyne, is there an easy way to add a daemonize feature to this for running on startup from /etc/rc.d ? I can't quite seem to get the right combo of redirections and backgrounding to get it running cleanly in the background.

Thanks!
Scott

Xyne · 2010-10-01 18:08:39

If you just want to suppress all the messages then you could try this:

voracious [args] > /dev/null 2>&1

I'll try to remember to look into properly daemonizing it when I have time.

firecat53 · 2010-10-01 20:27:31

Ah, that one worked. I used another rc.d script as a template and got it to work properly. Thanks!

Works great on my desktop browsers...on my mobile browser it takes a looooong time to load each page, even with only one or two items in a category (or no items, which is the root category). About 1/2 the time, it won't load at all. Not sure what the difference is (ok besides the obvious ) I tested it both from outside the network using my dyndns hostname and from inside the network on wifi using the server ip directly and it responded the same way. Bummer, 'cause I was attempting to replace google reader!

Scott

Xyne · 2010-10-01 23:49:24

firecat53 wrote:

Works great on my desktop browsers...on my mobile browser it takes a looooong time to load each page
...
I tested it both from outside the network using my dyndns hostname and from inside the network on wifi using the server ip directly

Are you running voracious on the same device as the mobile browser or are you connecting to the voracious server on your desktop? The second part implies the latter but the first part implies the former.

The steps involved when browsing the voracious server are the following:

client: send request to server
server: check if request is cached... retrieve rss if not and parse it... return html
client: parse html and render page

If you're running the server on your desktop then the only thing that could be slowing down the mobile browser is the html rendering. That depends both on the voracious template that you use (which is really simple by default and shouldn't slow anything down unless there are hundreds of items on the page) and on embedded html formatting in the rss feed (which has nothing to do with voracious). If you don't need the summary then you could remove that from the template, otherwise you could reduce the number of items displayed, but that doesn't seem to be the issue as you mentioned that it was slow with only two items in a category.

If you're actually running the server on the mobile device then maybe you can try running it on your desktop instead. Voracious provides both HTTP authentication and HTTPS support so you should be able to access it securely via dyndns when you're out and about.

I use the feedparser module to parse rss feeds and the server makes use of threading when retrieving multiple feeds... it may be that the combination of the two (simultaneous parsing of multiple, potentially huge xml documents) is too much for a mobile device, but the feeds would need to be ridiculously huge (or the feedparser module would have to be horribly inefficient).

*several minutes later*

I've restructured the threading of the feed retrieval. Instead of spawning threads on demand it now spawns a fixed pool of worker threads at startup. You can control the size of the pool with the "--threads" option. Hopefully that will reduce overhead enough to make it run smoothly on mobile devices.

firecat53 · 2010-10-02 02:39:50

Ah, to clarify -- I'm running Voracious on my server and accessing it through either my laptop/desktop or my mobile phone browser. From the laptop/desktop there is a slight pause when clicking on a category when I haven't accessed it in awhile (expected, as that triggers voracious to reload those feeds if it's been longer than the default 15 min), but it's almost instant response after the feeds are cached. As a note -- the pages appear to never stop loading (even on desktop/laptop), even when viewing a cached feed category and even when all the data appears to be loaded on the page. Even the root page, which has no assigned feeds, will continue the 'loading' phase for a very long time -- I've never actually seen it completed, now that I think about it.

I wonder if that's what causes the issue with the mobile browser -- total guess, but maybe the mobile browser is waiting for a completed page load before rendering the page.

Let me know if I can help at all with more testing. Oh, and I'm just using the default template for now, FYI.

Thanks!
Scott

edit: I clicked back to the empty root category about five minutes ago and the page is still showing as loading...

Last edited by firecat53 (2010-10-02 02:41:01)

Xyne · 2010-10-02 04:34:44

Ah.

I think I've resolved this. I ran into the same problem with quickserve and after several "WTF?" moments I realized that it was due to a missing "Content-Length" header. When I split out the common backend (python-xynehttpserver) I added convenience methods to handle common requests (HTML, plaintext & files) but I forgot to insert them into voracious when I updated it.

Try the latest version and let me know how it works.

Barghest · 2010-10-02 15:25:33

Just a stupid question: I use the examples form your site but I don't have a navbar. Am I missing something. I though the navbar automatically contains the feeds under [path].

Oh...and thanks for that nice rss aggregator

firecat53 · 2010-10-02 16:24:59

Thanks, Xyne! Works perfectly now. Now I just need to work on the stylesheet to adjust it for the mobile browser and add 'next item' links for easier scrolling down.

Scott

edit:
1. Ok, almost perfect -- for some reason the http authentication doesn't work on the mobile browser. Works fine on desktop browsers. It asks for the username and password, but never accepts them -- just keeps asking for name/password over and over.
2. I also noticed that the username/password show up in 'ps aux' on the server. Security issue (not for me, as mine is single user )?

Last edited by firecat53 (2010-10-02 16:39:59)

kinhodder · 2010-10-02 17:52:26

This is great - thanks Xyne.

I rustled up a quick theme: http://gist.github.com/607810

Screenshot: http://db.tt/TwwEXMH

Update: the theme now works well on iPhones, iPads and (hopefully!) other things with small screens.

Last edited by kinhodder (2010-10-02 21:49:51)

Xyne · 2010-10-02 19:35:11

@Barghest
It should create the navbar automatcially. Post your config and the html output of the server so I can see what's wrong. Post your template too if you've changed it.

@firecat53
That was one of the problems that I solved with the "Content-Length" header. I don't understand why the mobile browser is failing to authenticate and I can't test it myself.

You could try opening XyneHTTPServer.py and change

  def authenticate(self, method='GET'):

to

  def authenticate(self, method='GET'):
    print self.headers

to get it to print the headers sent by the browser. That might give me an idea of what's wrong.

Are you able to auth on other sites? If so, can you check what type of HTTP authentication they use? (basic, digest)

kinhodder
I like the theme... very simple and elegant. It's a bit bright for me, but it's definitely given me some inspiration. I've added a "themes" section to the project page and linked to your post (so that you can update it if need be).

I hope your theme will also inspire others to create their own.

*edit*
I've wrapped the nav links in a proper list (<ul>) and added a doctype to the default template. (I wrote that before I cared about W3C validation).

Last edited by Xyne (2010-10-02 21:48:06)

cesura · 2010-10-02 19:58:52

I was just working on something very very similar a week or so ago!

firecat53 · 2010-10-02 21:33:36

@Xyne -- as requested, headers showing authentication.

However, I think this may be a browser issue with my phone, because I have another site setup using digest authentication (directly via lighttpd.conf) and my mobile browser exhibits the same behavior -- never logs in. I guess when I have some time I'll see about changing to plain instead of digest authentication and testing if that works.

Thanks,
Scott

edit: and of course a quick google search leads to the android bug report for this very issue.

edit2: if anyone's interested, here's a rc.d script that works to start/stop voracious. Just edit for the username you want to use. I'm sure there's probably a cleaner way, but it works

#!/bin/sh
#
# Startup script for the Voracious news reader
# Requires sudo

daemon_user="firecat53"
daemon_name="voracious"
daemon_conf="/home/$daemon_user/.config/voracious/config.txt"
daemon_options="--noedit --age-limit=2880 --bind=192.168.0.101 --port=6001"
daemon_path=`which $daemon_name`
daemon_pid="/var/run/$daemon_name/$daemon_name.pid"
daemon_log="/var/log/$daemon_name/$daemon_name.log"

. /etc/rc.conf
. /etc/rc.d/functions

get_pid() {
        pgrep -f "$daemon_path"
}

case "$1" in
  start)
    stat_busy "Starting $daemon_name daemon"

    PID=$(get_pid)
    if [ -z "$PID" ]; then
      [ -f "$daemon_pid" ] && rm -f "$daemon_pid"
      # RUN
      if [ ! -f "$daemon_log" ];then
        mkdir /var/log/$daemon_name
        touch $daemon_log
      fi
      chown $daemon_user: $daemon_log
      sudo -u $daemon_user $daemon_path $daemon_options > /dev/null 2>&1 &
      #
      if [ $? -gt 0 ]; then
        stat_fail
        exit 1
      else
        echo $(get_pid) > $daemon_pid
        add_daemon $daemon_name
        stat_done
      fi
    else
      stat_fail
      exit 1
    fi
    ;;

  stop)
    stat_busy "Stopping $daemon_name daemon"
    PID=$(get_pid)
    # KILL
    [ ! -z "$PID" ] && kill $PID &> /dev/null
    #
    if [ $? -gt 0 ]; then
      stat_fail
      exit 1
    else
      rm -f $daemon_pid &> /dev/null
      rm_daemon $daemon_name
      stat_done
    fi
    ;;

  restart)
    $0 stop
    sleep 3
    $0 start
    ;;

  status)
    stat_busy "Checking $daemon_name status";
    ck_status $daemon_name
    ;;

  *)
    echo "usage: $0 {start|stop|restart|status}"
esac

exit 0

Last edited by firecat53 (2010-10-02 21:46:41)

kinhodder · 2010-10-02 21:53:36

Xyne wrote:

I like the theme... very simple and elegant.

Thanks, glad you like it! And thanks for the linkage.

Xyne · 2010-10-02 23:39:09

@firecat
Well, that's out of my hands then but I've implemented something else that might help... optional requirement of a client certificate

The python-xynehttpserver page lists simple commands for generating the PKCS#12 client certificate. I've only done some basic testing but it seems to work (access is blocked until the certificate is imported in Firefox, after which the site becomes accessible as usual).

firecat53 · 2010-10-03 02:43:36

Meh, another quick google search, and adding custom client certificates to android browsers is another unresolved issue. Well thanks for trying! Guess I'll just turn off the edit/stop server functions and go with security by obscurity for now. :-)

Scott

Barghest · 2010-10-03 11:16:18

Xyne wrote:

@Barghest
It should create the navbar automatcially. Post your config and the html output of the server so I can see what's wrong. Post your template too if you've changed it.

$ powerpill -Qs voracious
local/voracious 2010.10.02.4-1

I didn't touch the template:

$ cat .config/voracious/template.htm
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <style type="text/css">
    body {
      /*margin-left: 205px;*/
      background: #333;
    }
    #navbar {
      color: black;
      background: #d8d8d8;
      /*margin-left: -200px;*/
      width: 190px;
      /*position: fixed;*/
      float: left;
      padding-bottom: 5px;
      border-right: solid #333 10px;
      border-bottom: solid #333 10px;
      margin-right: 10px;
    }

    #navbar ul {
      list-style-type: none;
      display: inline-block;
      margin: 0;
      padding: 0;
    }

    #navbar li {
      list-style-type: none;
      display: block;
      margin: 0;
      padding: 0;
    }

    #navbar .item_title {
      background: #c00;
    }

    .navlink {
      color: inherit;
      display: block;
      margin: 2px 10px;
    }
    #editlink {
      color: inherit;
      display: block;
      margin: 2px 10px;
    }
    .item {
      background: #d8d8d8;
      margin: 20px auto;
      text-align: justify;
      max-width: 800px;
    }
    .item_title {
      color: #eee;
      background: #666;
      padding: 5px 10px;
      font-weight: bold;
      border-bottom: solid 2px black;
    }
    .item_meta {
      font-weight: bold;
      padding: 5px 10px;
    }
    .item_title a {
      color: inherit;
      text-decoration: none;
    }
    .item_summary {
      padding: 20px;
    }
    </style>
  </head>
  <body>
    <div id="navbar">
      <div class="item_title">Navbar</div>
      $NAVLINKS$
      <br/>
      $EDITLINK$
    </div>
    $ITEM$
      <div class="item $CLASS$">
        <div class="item_title"><a href="$entry.link$">$feed.title$ : $entry.title$</a></div>
        <div class="item_meta">$entry.date$</div>
        <div class="item_summary">$entry.summary$</div>
      </div>
    $/ITEM$
  </body>
</html>

$ cat .config/voracious/config.txt 
[paths]
/example
  http://feeds.arstechnica.com/arstechnica/index
  http://www.archlinux.org/feeds/news/
  http://www.merriam-webster.com/word/index.xml
  http://xkcd.com/rss.xml



[classes]
arch
  archlinux
ars
  arstechnica
slashdot
  slashdot
xkcd
  xkcd

console output:

$ voracious 
Starting voracious...
  address: all interfaces
  port: 8080
  pid: 10624
Press <Ctrl-C> to exit.

retrieving http://feeds.arstechnica.com/arstechnica/index
retrieving http://www.archlinux.org/feeds/news/
retrieving http://www.merriam-webster.com/word/index.xml
retrieving http://xkcd.com/rss.xml
localhost.localdomain - - [03/Oct/2010 13:13:20] "GET /example HTTP/1.1" 200 -
localhost.localdomain - - [03/Oct/2010 13:13:20] "GET /favicon.ico HTTP/1.1" 200 -

Last edited by Barghest (2010-10-03 11:16:50)

Xyne · 2010-10-03 15:17:19

@firecat
Try this.

@Barghest
I seem to have left out the root path in the example. Try changing

[paths]
/example
  http://feeds.arstechnica.com/arstechnica/index
  http://www.archlinux.org/feeds/news/
  http://www.merriam-webster.com/word/index.xml
  http://xkcd.com/rss.xml

to

[paths]
/
/example
  http://feeds.arstechnica.com/arstechnica/index
  http://www.archlinux.org/feeds/news/
  http://www.merriam-webster.com/word/index.xml
  http://xkcd.com/rss.xml

I have added the root path to the example on the project page.

firecat53 · 2010-10-03 16:46:05

LOL, Xyne you work fast! I don't remember quickproxy being on that page 2 days ago

It works perfectly, by the way -- however it's not going to help me, because I'm needing to access my news from outside my network, too, and whether I use quickproxy + voracious w/ http auth or just voracious w/out auth, I can still view the news site from an 'unsecured' outside URL. I can see how it might be useful in certain situations, though.

I did discover that the opera mini mobile browser does support http auth -- too bad I don't like it as much as the stock android browser Just never been an opera fan, but I can't put my finger on why.

Anyone have an idea that doesn't involve tinkering with the source to put a 'next item' link as part of each item title? All I can think of is creating a counter in the code and adding a numbered id or anchor and an href= link to the next item as part of each item title.

Thanks!
Scott

Last edited by firecat53 (2010-10-03 16:46:45)

Barghest · 2010-10-03 17:53:06

Xyne wrote:

@Barghest
I seem to have left out the root path in the example.
[...]
I have added the root path to the example on the project page.

Thanks Xyne, now it works.

Don't know if I already said it: Thanks for your great projects and your efforts and contributions for the community!

Xyne · 2010-10-03 23:46:52

@firecat53
What I had in mind was running voracious on your server with e.g. https and client-side certificates. You can then import those certs in browsers that support them. For those that don't (e.g. your mobile browser), you would run quickproxy locally (i.e. on the mobile device, bound to localhost) and connect to it with the browser. Quickproxy's connection to voracious would be secure (e.g. using a client-side cert) and it would only be accessible locally. That should let you securely connect to voracious from outside your network. Quickproxy itself should be very light so you shouldn't even notice it on the mobile device. If you already have python then the size is insignificant as well.

@Barghest
Thanks

firecat53 · 2010-10-04 00:35:58

@Xyne: ahhh, I see now. <Off to figure out python on Android> Thanks!!

Scott

tladuke · 2011-03-31 01:28:17

I stared at the code for a few minutes and couldn't figure out how to make it support ETag and Last-Modified headers. Basically, if you already have the feed, next time you request it, you do

feedparser.parse('http://feedparser.org/docs/examples/atom10.xml', modified=feed.modified,etag=feed.etag)

And you'll get back a 304 Not Modified

my brain hasn't quite learned to comprehend python yet, especially that Thread stuff in there

http://www.feedparser.org/docs/http-etag.html

Arch Linux

#26 2010-01-30 08:15:18

Re: Voracious: customizable web feed aggregator for your browser

#27 2010-01-30 08:31:27

Re: Voracious: customizable web feed aggregator for your browser

#28 2010-01-30 09:02:28

Re: Voracious: customizable web feed aggregator for your browser

#29 2010-10-01 17:58:40

Re: Voracious: customizable web feed aggregator for your browser

#30 2010-10-01 18:08:39

Re: Voracious: customizable web feed aggregator for your browser

#31 2010-10-01 20:27:31

Re: Voracious: customizable web feed aggregator for your browser

#32 2010-10-01 23:49:24

Re: Voracious: customizable web feed aggregator for your browser

#33 2010-10-02 02:39:50

Re: Voracious: customizable web feed aggregator for your browser

#34 2010-10-02 04:34:44

Re: Voracious: customizable web feed aggregator for your browser

#35 2010-10-02 15:25:33

Re: Voracious: customizable web feed aggregator for your browser

#36 2010-10-02 16:24:59

Re: Voracious: customizable web feed aggregator for your browser

#37 2010-10-02 17:52:26

Re: Voracious: customizable web feed aggregator for your browser

#38 2010-10-02 19:35:11

Re: Voracious: customizable web feed aggregator for your browser

#39 2010-10-02 19:58:52

Re: Voracious: customizable web feed aggregator for your browser

#40 2010-10-02 21:33:36

Re: Voracious: customizable web feed aggregator for your browser

#41 2010-10-02 21:53:36

Re: Voracious: customizable web feed aggregator for your browser

#42 2010-10-02 23:39:09

Re: Voracious: customizable web feed aggregator for your browser

#43 2010-10-03 02:43:36

Re: Voracious: customizable web feed aggregator for your browser

#44 2010-10-03 11:16:18

Re: Voracious: customizable web feed aggregator for your browser

#45 2010-10-03 15:17:19

Re: Voracious: customizable web feed aggregator for your browser

#46 2010-10-03 16:46:05

Re: Voracious: customizable web feed aggregator for your browser

#47 2010-10-03 17:53:06

Re: Voracious: customizable web feed aggregator for your browser

#48 2010-10-03 23:46:52

Re: Voracious: customizable web feed aggregator for your browser

#49 2010-10-04 00:35:58

Re: Voracious: customizable web feed aggregator for your browser

#50 2011-03-31 01:28:17

Re: Voracious: customizable web feed aggregator for your browser

Board footer