You are not logged in.

#1 2012-09-06 19:50:02

aecepoglu
Member
Registered: 2012-09-05
Posts: 16
Website

Minimal rss reader - rssproc

Hello fellow arch users,
I have written a very minimal program to read rss data with
The program reads rss data from stdin, and prints each item in the desired format.

For example, the following is the output of
(%t means "the-title-of-item")

$ curl https://www.archlinux.org/feeds/news/ | rssproc $'_%t_\n'
_Fontconfig 2.10.1 update - manual intervention required_
_netcfg-2.8.9 drops deprecated rc.conf compatibility_
_Install media 2012.08.04 available_
_pkgstats now collects modules usage_
_Changes to rc.conf and crypttab_
_Install media 2012.07.15 released_
_GRUB legacy no longer supported_
_The /lib directory becomes a symlink_
_filesystem upgrade - manual intervention required_
_Having pacman verify packages_

I currently handle a few number of sequences
%t : item title
%T : feed title
%l : item link
%d : item description
%e : item enclosure url

Or this script can generate the news in HTML

./rssproc "<h4>%t</h4>
        %d
        <a href=\"%l\">Read the rest of the story</a>
        <hr>
" < archlinux.rss

Like so:

<h4>Binaries move to /usr/bin requiring update intervention</h4>
        ...
        ...
        <a href="https://www.archlinux.org/news/binaries-move-to-usrbin-requiring-update-intervention/">Read the rest of the story</a>
        <hr>
<h4>netctl is now in [core]</h4>
        ...
        ...
        <a href="https://www.archlinux.org/news/netctl-is-now-in-core/">Read the rest of the story</a>
        <hr>
<h4>MariaDB replaces MySQL in repositories</h4>
        ...
        ....
        <a href="https://www.archlinux.org/news/mariadb-replaces-mysql-in-repositories/">Read the rest of the story</a>
        <hr>
<h4>qt4 replaces qt</h4>
        ...
        <a href="https://www.archlinux.org/news/qt4-replaces-qt/">Read the rest of the story</a>
        <hr>
<h4>Changes to LVM</h4>
        ...
        <a href="https://www.archlinux.org/news/changes-to-lvm/">Read the rest of the story</a>
        <hr>
<h4>Final sysvinit deprecation warning</h4>
        ...
        <a href="https://www.archlinux.org/news/final-sysvinit-deprecation-warning/">Read the rest of the story</a>
        <hr>
<h4>Update filesystem-2013.01-1 and glibc-2.17-2 together</h4>
        ...
        <a href="https://www.archlinux.org/news/update-filesystem-201301-1-and-glibc-217-2-together/">Read the rest of the story</a>
        <hr>
<h4>December: time for a new install medium</h4>
        ...
        <a href="https://www.archlinux.org/news/december-time-for-a-new-install-medium/">Read the rest of the story</a>
        <hr>
<h4>End of initscripts support</h4>
        ...
        <a href="https://www.archlinux.org/news/end-of-initscripts-support/">Read the rest of the story</a>
        <hr>
<h4>November release of install media available</h4>
        ...
        <a href="https://www.archlinux.org/news/november-release-of-install-media-available/">Read the rest of the story</a>
        <hr>

I very much appreciate all feedback.
You can contact me here on the forums, or via my email written on top of AUR/PKGBUILD and the source codes. You can also message me from the Code Homepage.
Cheers

AUR Package
Code Homepage

Last edited by aecepoglu (2013-06-25 19:54:32)

Offline

#2 2013-06-25 17:04:34

moebius_eye
Member
Registered: 2013-02-24
Posts: 6

Re: Minimal rss reader - rssproc

Hi. I have installed your package.

I think it is a very good idea. It makes manipulating rss feed much much easier.
I will try it for some experiments, and report any bugs here.

Or maybe you want me to post somewhere else?

Offline

#3 2013-06-25 19:58:12

aecepoglu
Member
Registered: 2012-09-05
Posts: 16
Website

Re: Minimal rss reader - rssproc

moebius_eye wrote:

Hi. I have installed your package.

I think it is a very good idea. It makes manipulating rss feed much much easier.
I will try it for some experiments, and report any bugs here.

Or maybe you want me to post somewhere else?

I have updated the first post, and added some contact information.

Does that work fine?
I check my emails very regularly, so, you can email me and I will get back to you ASAP.

I too find the code quite useful smile Sadly, it's only you and me

Offline

#4 2013-06-28 14:41:52

moebius_eye
Member
Registered: 2013-02-24
Posts: 6

Re: Minimal rss reader - rssproc

aecepoglu wrote:
moebius_eye wrote:

Hi. I have installed your package.

...

I have updated the first post, and added some contact information.

Does that work fine?
I check my emails very regularly, so, you can email me and I will get back to you ASAP.

I too find the code quite useful :) Sadly, it's only you and me

I prefer to keep the conversation on the forum if you don't mind.
If some other people find your tool useful, they might want to know what went well/wrong.
If I need to send you more consequent data about an issue, I'll email it.

For now, I have found one thing that bugs me: the errors are not sent to stderr but to stdout!
When I use your program to pull titles from an rss list and post them to twitter, I get this kind of stuff.

So, I need to do this in order to ignore error messages:

curl $url | rssproc $'%T > %t > %l\n' | grep -v "Parse error at line" | grep -v "no element found" | grep -v "Exit code"

I hope that I will find some time to look into your code and maybe suggest some patches. :)

Offline

#5 2013-06-28 16:29:26

aecepoglu
Member
Registered: 2012-09-05
Posts: 16
Website

Re: Minimal rss reader - rssproc

Yes of course! You can choose whichever medium you like. I get email notifications when people post in this thread, so it is perfectly fine by me.

Ah, I forgot to change that to stderr. Well, that's embarrassing.
I have earlier today made a commit and fixed it. Updating to the latest version should do it.

Sure, feel free to play around with the code.
make debug will compile the code with debug-outputs enabled.
Also, the code is actually an xml-processor. You can modify the "code.h"  to deal with any type of xml data.

Last edited by aecepoglu (2013-06-28 17:30:36)

Offline

#6 2013-06-28 19:49:01

moebius_eye
Member
Registered: 2013-02-24
Posts: 6

Re: Minimal rss reader - rssproc

Oh, also your program works on armv6h architectures. (I installed it on my Raspberry pi)
So, you can add armv6h in PKGBUILD

Offline

#7 2013-06-29 07:36:26

Army
Member
Registered: 2007-12-07
Posts: 1,784

Re: Minimal rss reader - rssproc

Bug:

# rssproc
Improper usage. See "rssproc -h" for help
Quitting...

Then trying rssproc -h, but nothing happens.

Offline

#8 2013-06-29 12:54:47

moebius_eye
Member
Registered: 2013-02-24
Posts: 6

Re: Minimal rss reader - rssproc

Army wrote:

Bug:

# rssproc
Improper usage. See "rssproc -h" for help
Quitting...

Then trying rssproc -h, but nothing happens.

Same here.

Offline

#9 2013-06-30 14:51:15

aecepoglu
Member
Registered: 2012-09-05
Posts: 16
Website

Re: Minimal rss reader - rssproc

Aha! My bad. I am going to do some automated testing before each commit.

Anyway, bug fixed.

Offline

#10 2013-07-05 14:40:56

moebius_eye
Member
Registered: 2013-02-24
Posts: 6

Re: Minimal rss reader - rssproc

So, here's my result for this script:

#!/bin/bash

while read feed_url
do
    [ -z "$feed_url" ] && continue
    echo "[[$feed_url]]"
    echo "[[$feed_url]]" 1>&2
    curl $feed_url 2> /dev/null | rssproc $'<(%T|#|%t|#|%l|#|%e)>\n'
done < url_list

and url_list contains the following:

http://grab-dev.tumblr.com/rss
http://moebiuseye.jeannedhack.org/totoro/rss
http://moebiuseye.soup.io/rss
http://podcast.jeannedhack.org/feed/atom
http://trium.swordarmor.fr/news.xml
http://wiki.jeannedhack.org/index.php?title=Sp%C3%A9cial:Nouvelles_pages&feed=atom&namespace=0
http://identi.ca/api/statusnet/groups/timeline/50296.atom
http://sybix.jeannedhack.org/feed/
http://archlinux.fr/feed
http://www.serverporn.fr/feeds/all.rss.xml
http://www.avaland.org/feed/rss2
http://www.laquadrature.net/fr/rss.xml

Here is the stdout result:

http://ix.io/6wj

And here is the stderr result:

http://ix.io/6wi

Basically, the atom feeds don't work and I don't understand the last error.
If the URL http://wiki.jeannedhack.org/index.php?t … amespace=0 happens to work with you, please tell me.
It is very important that I get my bot to pull the news from this feed.

Last edited by moebius_eye (2013-07-05 14:41:56)

Offline

#11 2013-07-05 15:34:02

aecepoglu
Member
Registered: 2012-09-05
Posts: 16
Website

Re: Minimal rss reader - rssproc

Oh yes, it doesn't handle atom-feeds, but code.h can be easily modified to add support for them.

As a temporary fix, you can use "http://wiki.jeannedhack.org/index.php?t … amespace=0" while I'm trying to figure out where the error is coming from.

**edit**

Okay, problem fixed.
So... there is a buffer where data is kept temporarily. That had the size 'BUFSIZ' which in my system is 8192. I haven't considered the possiblity that it could overflow.
This case is now properly handled, and the buffer is enlarged if it is full.

Let me know if any more errors come up. Also let me know if you want Atom feed support by default.

Also, the code will be inaccesible for a few hours due to some planned maintainance at the repository server. The start and end dates of this event is written here http://www.timeanddate.com/worldclock/f … 1=224&ah=6

Last edited by aecepoglu (2013-07-05 17:46:27)

Offline

#12 2013-07-07 05:45:46

prazzb
Member
Registered: 2012-10-12
Posts: 14

Re: Minimal rss reader - rssproc

I got bit confused. DEPENDS file in the source says hg.... everything else says git.... change it sometime

Offline

#13 2013-07-08 17:06:05

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Minimal rss reader - rssproc

aecepoglu wrote:

Thanks for pointing that out.
Fixed.

I'm assuming that you meant to post that here rather than send a report. wink


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#14 2013-07-09 05:29:37

aecepoglu
Member
Registered: 2012-09-05
Posts: 16
Website

Re: Minimal rss reader - rssproc

Xyne wrote:
aecepoglu wrote:

Thanks for pointing that out.
Fixed.

I'm assuming that you meant to post that here rather than send a report. wink

smile Yes, I meant to quote him,  not report him.

Offline

Board footer

Powered by FluxBB