[SOLVED] rss command line downloader.. no viable solution found yet

lpallard · 12-25-2011, 08:02 AM

Hello all and Merry Christmas! !

I have been trying for the longest time to find a solution to automatically download attachments to RSS posts such as podcasts (mp3) and pictures from sites like the NASA astronomy picture of the day.

I want something command line based because I will be running this from a headless server and I want to automate it as much as feasible.

I've tried it a lot of different programs but none of them are satisfying me. I tried armangil podcast downloader, for whatever reason it doesn't run (it says stopped due to an internal error) and it seems that its not a popularprogram since Googling revealed nothing and their support forum is very lean..

I have tried dozens of PHP, Ruby and bash scripts programmed by all kind of people, most of them crash, don't do anything or download the wrong things..

I have also tried rsstail, it seems to work but doesn't actually download the attachments but display only to headlines in the terminal which would be useful if I was doing text analysis or keyword search but I am not interested in the headlines, descriptions or text..

I also tried rssdler. So far is this one have provided the best results but still it's not working. It end up downloading not the attachment but the html page wherr the attachment is located. I believe this happens because it's meant to download torrent files. The configuration file has a field for regular expressions but I am too new to these to know what to put there to only download the jpeg or mp3 files..

There must be a script or something somewhere to automate this kind of stuff rss news has been around forever and I can't imagine people downloading attachments 1 by 1 especially if you're not updating for a few weeks....

Any humble suggestions for me?

Thanks again and Merry Christmas again!

TV RSS seems to be for torrents

teckk · 12-25-2011, 01:02 PM

I can't speak about any of these. But here are some.
http://www.buzzmaven.com/podcast-software-list.html
http://www.peapodpy.org.uk/using-peapod/
http://www.newsbeuter.org/
http://blog.davidron.com/2009/08/acc...d-line_09.html
https://github.com/mackers/theyoke/
https://kiza.kcore.de/software/snownews/
http://raggle.org/about/
http://www.makeuseof.com/tag/3-light...anagers-linux/

lpallard · 12-25-2011, 03:06 PM

Mmmm its sad..

None of them worked, or I should say, do what I want.

Newsbeuter actually compiled, installed and runs OK, but doesnt allow to download *only* the attachments, but seems to download only the article and text. Useless for what I want to do. Also automation seems limited.. Will try more but I have a bad feeling.

Peapod installed, but doesnt run:

Code:

bash-4.1$ ./peapod.py --addnew=http://apod.nasa.gov/apod.rss
/home/lpallard/peapod-0.7.2/Peapod/tagging.py:21: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  from popen2 import Popen3
Traceback (most recent call last):
  File "./peapod.py", line 7, in <module>
    from Peapod.peapod import getConfig, newTracks, feedLog
  File "/home/lpallard/peapod-0.7.2/Peapod/peapod.py", line 35, in <module>
    from Peapod.download import downloadURL
  File "/home/lpallard/peapod-0.7.2/Peapod/download.py", line 14, in <module>
    from urlgrabber.grabber import URLGrabber
ImportError: No module named urlgrabber.grabber

theyoke seems to work but I cant figure out how to use it to retrieve only attachments. I run it with my feed, and this is what I get:

Code:

bash-4.1$ ./theyoke http://apod.nasa.gov/apod.rss
APOD: M1: The Crab Nebula from Hubble
APOD: Eclipsed Moon in the Morning
APOD: Shell Galaxy NGC 7600
APOD: Through a Sun Tunnel
APOD: A Horseshoe Einstein Ring from Hubble
APOD: NGC 253: The Sculptor Galaxy
APOD: A Geminid Meteor Over Iran

Subsequent runs are only saying:

Code:

bash-4.1$ ./theyoke -d
theyoke: Getting "http://apod.nasa.gov/apod.rss" -  got a 304, skipping

Which I believe means "this feed has not changed since last time therefore we skip to avoid abusing the server and getting banned"...

Snownews just like Newsbeuter seems to work well, except doesnt seems to be able to handle attachments but only displays the article in plain text with a placeholder for the picture or podcast like [picture01].. I've looked into the extensions that are available at https://kiza.kcore.de/software/snown...ts/extensions/ and nothing to handle attachments..

Raggle totally explodes..

Code:

bash-4.1$ raggle
/usr/local/bin/raggle:460: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
        when 'amp':   '&'
                   ^
/usr/local/bin/raggle:461: syntax error, unexpected keyword_when, expecting keyword_end
        when 'nbsp'   ' '
            ^
/usr/local/bin/raggle:462: syntax error, unexpected keyword_when, expecting keyword_end
        when /^quot$/ni:  '"'
            ^
/usr/local/bin/raggle:462: syntax error, unexpected ':', expecting keyword_end
        when /^quot$/ni:  '"'
                        ^
/usr/local/bin/raggle:463: syntax error, unexpected keyword_when, expecting keyword_end
        when /^lt$/ni:    '<'
            ^
/usr/local/bin/raggle:463: syntax error, unexpected ':', expecting keyword_end
        when /^lt$/ni:    '<'
                      ^
/usr/local/bin/raggle:464: syntax error, unexpected keyword_when, expecting keyword_end
        when /^gt$/ni:    '>'
            ^
/usr/local/bin/raggle:464: syntax error, unexpected ':', expecting keyword_end
        when /^gt$/ni:    '>'
                      ^
/usr/local/bin/raggle:465: syntax error, unexpected keyword_when, expecting keyword_end
        when /^copy/:     '(c)'
            ^
/usr/local/bin/raggle:465: syntax error, unexpected ':', expecting keyword_end
        when /^copy/:     '(c)'
                     ^
/usr/local/bin/raggle:466: syntax error, unexpected keyword_when, expecting keyword_end
        when /^trade/:    '(tm)'
            ^
/usr/local/bin/raggle:466: syntax error, unexpected ':', expecting keyword_end
        when /^trade/:    '(tm)'
                      ^
/usr/local/bin/raggle:467: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#8212$/n:  ","
            ^
/usr/local/bin/raggle:467: syntax error, unexpected ':', expecting keyword_end
        when /^#8212$/n:  ","
                        ^
/usr/local/bin/raggle:468: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#8217$/n:  "'"
            ^
/usr/local/bin/raggle:468: syntax error, unexpected ':', expecting keyword_end
        when /^#8217$/n:  "'"
                        ^
/usr/local/bin/raggle:469: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#8218$/n:  ","
            ^
/usr/local/bin/raggle:469: syntax error, unexpected ':', expecting keyword_end
        when /^#8218$/n:  ","
                        ^
/usr/local/bin/raggle:470: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#(\d+)$/n
            ^
/usr/local/bin/raggle:477: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#x([0-9a-f]+)$/ni
            ^
/usr/local/bin/raggle:485: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:546: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:551: module definition in method body
/usr/local/bin/raggle:555: module definition in method body
/usr/local/bin/raggle:597: dynamic constant assignment
    BROWSERS = %w{links elinks w3m lynx iexplore.exe explorer.exe}
              ^
/usr/local/bin/raggle:630: module definition in method body
/usr/local/bin/raggle:702: module definition in method body
/usr/local/bin/raggle:712: class definition in method body
/usr/local/bin/raggle:713: class definition in method body
/usr/local/bin/raggle:777: dynamic constant assignment
    TAG_SET = TagSet.new 
             ^
/usr/local/bin/raggle:827: class definition in method body
/usr/local/bin/raggle:1042: module definition in method body
/usr/local/bin/raggle:1043: dynamic constant assignment
      NO_ATTRIBUTES = {}.freeze
                     ^
/usr/local/bin/raggle:1044: dynamic constant assignment
      ATTRIBUTE_LIST_RE = /\s*([^>=\s]+)\s*(?:=\s*(?:(...
                         ^
/usr/local/bin/raggle:1045: dynamic constant assignment
      PARSER_RE = %r!<(/?\w+[^>]*?/?)>|([^<]*)!m
                 ^
/usr/local/bin/raggle:1097: module definition in method body
/usr/local/bin/raggle:1149: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
... outline.attributes['refresh']: opml_refresh = outline.attri...
...                               ^
/usr/local/bin/raggle:1150: syntax error, unexpected keyword_else, expecting keyword_end
            else opml_refresh = nil; end  # fixes clobberation
                ^
/usr/local/bin/raggle:1183: warning: else without rescue is useless
/usr/local/bin/raggle:1188: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
        if $wins: Interfaces::NcursesInterface...
                 ^
/usr/local/bin/raggle:1225: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
        if $wins: Interfaces::NcursesInterface...
                 ^
/usr/local/bin/raggle:1228: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:1261: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:1269: class definition in method body
/usr/local/bin/raggle:1271: dynamic constant assignment
    VERSION = '0.1.0'
             ^
/usr/local/bin/raggle:1313: module definition in method body
/usr/local/bin/raggle:1317: dynamic constant assignment
    Item = Struct.new :title, :link, :desc, :date
          ^
/usr/local/bin/raggle:1323: class definition in method body
/usr/local/bin/raggle:1620: class definition in method body
/usr/local/bin/raggle:1863: module definition in method body
/usr/local/bin/raggle:2028: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--config':  $config['config_path'] = arg
                            ^
/usr/local/bin/raggle:2029: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--force':   ret['force'] = true
                           ^
/usr/local/bin/raggle:2030: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--list':    ret['mode'] = 'list'
                          ^
/usr/local/bin/raggle:2031: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--title':   ret['title'] = arg
                           ^
/usr/local/bin/raggle:2032: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--url':     ret['url'] = arg
                         ^
/usr/local/bin/raggle:2033: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--refresh': ret['refresh'] = arg.to_i
                             ^
/usr/local/bin/raggle:2037: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--priority':      ret['priority'] = arg.to_i
                              ^
/usr/local/bin/raggle:2038: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--purge':         ret['mode'] = 'purge'
                           ^
/usr/local/bin/raggle:2041: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--lock-title':    ret['lock_title?'] = true
                                ^
/usr/local/bin/raggle:2042: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--unlock-title':  ret['lock_title?'] = false
                                  ^
/usr/local/bin/raggle:2043: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--save-items':    ret['save_items?'] = true
                                ^
/usr/local/bin/raggle:2047: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--unsave-items':  ret['save_items?'] = false
                                  ^
/usr/local/bin/raggle:2048: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--verbose':       $config['verbose'] = true
                             ^
/usr/local/bin/raggle:2055: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--diag':          $config['diag'] = true
                          ^
/usr/local/bin/raggle:2058: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--ascii':         ret['ascii'] = true
                           ^
/usr/local/bin/raggle:2223: module definition in method body
/usr/local/bin/raggle:2228: module definition in method body
/usr/local/bin/raggle:2232: module definition in method body
/usr/local/bin/raggle:2336: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
...sInterface::get_win_id('find'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2337: syntax error, unexpected keyword_when, expecting keyword_end
          when NcursesInterface::get_win_id...
              ^
/usr/local/bin/raggle:2337: syntax error, unexpected ':', expecting keyword_end
...esInterface::get_win_id('cat'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2338: syntax error, unexpected keyword_when, expecting keyword_end
          when NcursesInterface::get_win_id...
              ^
/usr/local/bin/raggle:2338: syntax error, unexpected ':', expecting keyword_end
...sInterface::get_win_id('keys'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2339: syntax error, unexpected keyword_when, expecting keyword_end
          when NcursesInterface::get_win_id...
              ^
/usr/local/bin/raggle:2339: syntax error, unexpected ':', expecting keyword_end
...sInterface::get_win_id('edit'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2727: class definition in method body
/usr/local/bin/raggle:2925: class definition in method body
/usr/local/bin/raggle:3103: class definition in method body
/usr/local/bin/raggle:3408: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
            if str =~ /true/i: newopts = {feedopt => true}
                              ^
/usr/local/bin/raggle:3410: syntax error, unexpected keyword_when, expecting keyword_end
          when 'priority', 'refresh'
              ^
/usr/local/bin/raggle:3410: syntax error, unexpected ',', expecting keyword_end
          when 'priority', 'refresh'
                          ^
/usr/local/bin/raggle:3412: syntax error, unexpected keyword_when, expecting keyword_end
          when 'max_items'
              ^
/usr/local/bin/raggle:3413: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
            if str =~ /none|nil/i: newopts = {feedopt => nil}
                                  ^
/usr/local/bin/raggle:3415: syntax error, unexpected keyword_when, expecting keyword_end
          when 'category'
              ^
/usr/local/bin/raggle:3416: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
            if str =~ /\bnone\b|\bnil\b/i: newopts = {feedopt => nil}
                                          ^
/usr/local/bin/raggle:3417: warning: else without rescue is useless
/usr/local/bin/raggle:3419: warning: else without rescue is useless
/usr/local/bin/raggle:3428: syntax error, unexpected keyword_else, expecting '}'
        else  # if no change
            ^
/usr/local/bin/raggle:3430: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3578: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
              when (/^(\w+):\/\//): m
                                   ^
/usr/local/bin/raggle:3595: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3602: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when (/(\w+):\/\//):  item_url
                                ^
/usr/local/bin/raggle:3629: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3668: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
          when /%s/:  cmd_part % url.escape
                    ^
/usr/local/bin/raggle:3669: syntax error, unexpected keyword_when, expecting '}'
          when '${browser}':  $config['browser']
              ^
/usr/local/bin/raggle:3669: syntax error, unexpected ':', expecting '}'
          when '${browser}':  $config['browser']
                            ^
/usr/local/bin/raggle:3670: syntax error, unexpected keyword_else, expecting '}'
          else cmd_part
              ^
/usr/local/bin/raggle:3691: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3731: dynamic constant assignment
      DIRECTION_BACKWARD = -1
                          ^
/usr/local/bin/raggle:3732: dynamic constant assignment
      DIRECTION_FORWARD = 1
                         ^
/usr/local/bin/raggle:3836: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
          if title: added = Engine::add_feed({ '...
                   ^
/usr/local/bin/raggle:3841: warning: else without rescue is useless
/usr/local/bin/raggle:3851: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4115: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
          when 'DC':        key = 'Delete'
                    ^
/usr/local/bin/raggle:4116: syntax error, unexpected keyword_when, expecting '}'
          when ' ':         key = 'Space'
              ^
/usr/local/bin/raggle:4116: syntax error, unexpected ':', expecting '}'
          when ' ':         key = 'Space'
                   ^
/usr/local/bin/raggle:4117: syntax error, unexpected keyword_when, expecting '}'
          when 'NPAGE':     key = 'Page Down'
              ^
/usr/local/bin/raggle:4117: syntax error, unexpected ':', expecting '}'
          when 'NPAGE':     key = 'Page Down'
                       ^
/usr/local/bin/raggle:4118: syntax error, unexpected keyword_when, expecting '}'
          when 'PPAGE':     key = 'Page Up'
              ^
/usr/local/bin/raggle:4118: syntax error, unexpected ':', expecting '}'
          when 'PPAGE':     key = 'Page Up'
                       ^
/usr/local/bin/raggle:4119: syntax error, unexpected keyword_when, expecting '}'
          when '
                ':        key = 'Control-L'
              ^
/usr/local/bin/raggle:4119: syntax error, unexpected ':', expecting '}'
          when '
                ':        key = 'Control-L'
                   ^
/usr/local/bin/raggle:4120: syntax error, unexpected keyword_when, expecting '}'
          when '	':  key = 'Tab'
              ^
/usr/local/bin/raggle:4120: syntax error, unexpected ':', expecting '}'
          when '	':  key = 'Tab'
                	  ^
/usr/local/bin/raggle:4121: syntax error, unexpected keyword_when, expecting '}'
          when /^\d$/:      key = '1-9'
              ^
/usr/local/bin/raggle:4121: syntax error, unexpected ':', expecting '}'
          when /^\d$/:      key = '1-9'
                      ^
/usr/local/bin/raggle:4122: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4254: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when /feed/:  cl = NcursesInterface::ListWindow
                        ^
/usr/local/bin/raggle:4255: syntax error, unexpected keyword_when, expecting '}'
            when /item/:  cl = NcursesInterface::ListWindow
                ^
/usr/local/bin/raggle:4255: syntax error, unexpected ':', expecting '}'
            when /item/:  cl = NcursesInterface::ListWindow
                        ^
/usr/local/bin/raggle:4256: syntax error, unexpected keyword_when, expecting '}'
            when /desc/:  cl = NcursesInterface::TextWindow
                ^
/usr/local/bin/raggle:4256: syntax error, unexpected ':', expecting '}'
            when /desc/:  cl = NcursesInterface::TextWindow
                        ^
/usr/local/bin/raggle:4257: syntax error, unexpected keyword_else, expecting '}'
/usr/local/bin/raggle:4259: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4314: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4320: module definition in method body
/usr/local/bin/raggle:4324: class definition in method body
/usr/local/bin/raggle:4665: module definition in method body
/usr/local/bin/raggle:4666: class definition in method body
/usr/local/bin/raggle:4704: class definition in method body
/usr/local/bin/raggle:4732: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4738: module definition in method body
/usr/local/bin/raggle:5574: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:6658: syntax error, unexpected $end, expecting '}'

Apparently (https://bbs.archlinux.org/viewtopic.php?pid=711844) due to changes in syntax between Ruby 1.8 (which I believe raggle was built on) and Ruby 1.9 which I use.. I am not going to downgrade because I have numerous services depending on it.

As you can see, I havent had much success so far.. To be honest, I am pretty shocked because I really thought automating attachment download from RSS feeds was an "old" technology like Usenet.. Maybe I am doing/thinking the wrong way?

lpallard · 12-29-2011, 06:33 PM

Bump!? No one?

lpallard · 01-03-2012, 09:37 AM

OK so far I've been able to use cast get to retrieve podcasts such as the Linux action show, but somehow automatically downloading pictures via RSS feeds remains a mystery .. I am pretty sure its possible but in light of my posts above you see I haven't had much success....

I always wanted to follow some photography RSS sites to build a huge database of high Res pictures for a screen saver on my xbmc htpc ... APOD has that kind of pictures but I have other sites in mind. Not to mention some friends that are in professional photography ..

That's why I am that interested in doing this

So again I am asking this community for kind help! Anybody doing this? Its part of several programs as a nice functionality so I imagine with Linux flexibility and tools like curl, wget and others its feasible. Even a bash script would do. Download the pics via RSS, dump to a specific folder, get xbmc to play the slideshow from that folder and off you go!

Anyways thanks !