LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-25-2011, 09:02 AM   #1
lpallard
Member
 
Registered: Nov 2008
Posts: 987

Rep: Reputation: Disabled
rss command line downloader.. no viable solution found yet


Hello all and Merry Christmas! !

I have been trying for the longest time to find a solution to automatically download attachments to RSS posts such as podcasts (mp3) and pictures from sites like the NASA astronomy picture of the day.

I want something command line based because I will be running this from a headless server and I want to automate it as much as feasible.

I've tried it a lot of different programs but none of them are satisfying me. I tried armangil podcast downloader, for whatever reason it doesn't run (it says stopped due to an internal error) and it seems that its not a popularprogram since Googling revealed nothing and their support forum is very lean..

I have tried dozens of PHP, Ruby and bash scripts programmed by all kind of people, most of them crash, don't do anything or download the wrong things..

I have also tried rsstail, it seems to work but doesn't actually download the attachments but display only to headlines in the terminal which would be useful if I was doing text analysis or keyword search but I am not interested in the headlines, descriptions or text..

I also tried rssdler. So far is this one have provided the best results but still it's not working. It end up downloading not the attachment but the html page wherr the attachment is located. I believe this happens because it's meant to download torrent files. The configuration file has a field for regular expressions but I am too new to these to know what to put there to only download the jpeg or mp3 files..

There must be a script or something somewhere to automate this kind of stuff rss news has been around forever and I can't imagine people downloading attachments 1 by 1 especially if you're not updating for a few weeks....

Any humble suggestions for me?

Thanks again and Merry Christmas again!


TV RSS seems to be for torrents
 
Old 12-25-2011, 04:06 PM   #3
lpallard
Member
 
Registered: Nov 2008
Posts: 987

Original Poster
Rep: Reputation: Disabled
Mmmm its sad..

None of them worked, or I should say, do what I want.

Newsbeuter actually compiled, installed and runs OK, but doesnt allow to download *only* the attachments, but seems to download only the article and text. Useless for what I want to do. Also automation seems limited.. Will try more but I have a bad feeling.

Peapod installed, but doesnt run:

Code:
bash-4.1$ ./peapod.py --addnew=http://apod.nasa.gov/apod.rss
/home/lpallard/peapod-0.7.2/Peapod/tagging.py:21: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  from popen2 import Popen3
Traceback (most recent call last):
  File "./peapod.py", line 7, in <module>
    from Peapod.peapod import getConfig, newTracks, feedLog
  File "/home/lpallard/peapod-0.7.2/Peapod/peapod.py", line 35, in <module>
    from Peapod.download import downloadURL
  File "/home/lpallard/peapod-0.7.2/Peapod/download.py", line 14, in <module>
    from urlgrabber.grabber import URLGrabber
ImportError: No module named urlgrabber.grabber
theyoke seems to work but I cant figure out how to use it to retrieve only attachments. I run it with my feed, and this is what I get:
Code:
bash-4.1$ ./theyoke http://apod.nasa.gov/apod.rss
APOD: M1: The Crab Nebula from Hubble
APOD: Eclipsed Moon in the Morning
APOD: Shell Galaxy NGC 7600
APOD: Through a Sun Tunnel
APOD: A Horseshoe Einstein Ring from Hubble
APOD: NGC 253: The Sculptor Galaxy
APOD: A Geminid Meteor Over Iran
Subsequent runs are only saying:

Code:
bash-4.1$ ./theyoke -d
theyoke: Getting "http://apod.nasa.gov/apod.rss" -  got a 304, skipping
Which I believe means "this feed has not changed since last time therefore we skip to avoid abusing the server and getting banned"...

Snownews just like Newsbeuter seems to work well, except doesnt seems to be able to handle attachments but only displays the article in plain text with a placeholder for the picture or podcast like [picture01].. I've looked into the extensions that are available at https://kiza.kcore.de/software/snown...ts/extensions/ and nothing to handle attachments..

Raggle totally explodes..
Code:
bash-4.1$ raggle
/usr/local/bin/raggle:460: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
        when 'amp':   '&'
                   ^
/usr/local/bin/raggle:461: syntax error, unexpected keyword_when, expecting keyword_end
        when 'nbsp'   ' '
            ^
/usr/local/bin/raggle:462: syntax error, unexpected keyword_when, expecting keyword_end
        when /^quot$/ni:  '"'
            ^
/usr/local/bin/raggle:462: syntax error, unexpected ':', expecting keyword_end
        when /^quot$/ni:  '"'
                        ^
/usr/local/bin/raggle:463: syntax error, unexpected keyword_when, expecting keyword_end
        when /^lt$/ni:    '<'
            ^
/usr/local/bin/raggle:463: syntax error, unexpected ':', expecting keyword_end
        when /^lt$/ni:    '<'
                      ^
/usr/local/bin/raggle:464: syntax error, unexpected keyword_when, expecting keyword_end
        when /^gt$/ni:    '>'
            ^
/usr/local/bin/raggle:464: syntax error, unexpected ':', expecting keyword_end
        when /^gt$/ni:    '>'
                      ^
/usr/local/bin/raggle:465: syntax error, unexpected keyword_when, expecting keyword_end
        when /^copy/:     '(c)'
            ^
/usr/local/bin/raggle:465: syntax error, unexpected ':', expecting keyword_end
        when /^copy/:     '(c)'
                     ^
/usr/local/bin/raggle:466: syntax error, unexpected keyword_when, expecting keyword_end
        when /^trade/:    '(tm)'
            ^
/usr/local/bin/raggle:466: syntax error, unexpected ':', expecting keyword_end
        when /^trade/:    '(tm)'
                      ^
/usr/local/bin/raggle:467: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#8212$/n:  ","
            ^
/usr/local/bin/raggle:467: syntax error, unexpected ':', expecting keyword_end
        when /^#8212$/n:  ","
                        ^
/usr/local/bin/raggle:468: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#8217$/n:  "'"
            ^
/usr/local/bin/raggle:468: syntax error, unexpected ':', expecting keyword_end
        when /^#8217$/n:  "'"
                        ^
/usr/local/bin/raggle:469: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#8218$/n:  ","
            ^
/usr/local/bin/raggle:469: syntax error, unexpected ':', expecting keyword_end
        when /^#8218$/n:  ","
                        ^
/usr/local/bin/raggle:470: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#(\d+)$/n
            ^
/usr/local/bin/raggle:477: syntax error, unexpected keyword_when, expecting keyword_end
        when /^#x([0-9a-f]+)$/ni
            ^
/usr/local/bin/raggle:485: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:546: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:551: module definition in method body
/usr/local/bin/raggle:555: module definition in method body
/usr/local/bin/raggle:597: dynamic constant assignment
    BROWSERS = %w{links elinks w3m lynx iexplore.exe explorer.exe}
              ^
/usr/local/bin/raggle:630: module definition in method body
/usr/local/bin/raggle:702: module definition in method body
/usr/local/bin/raggle:712: class definition in method body
/usr/local/bin/raggle:713: class definition in method body
/usr/local/bin/raggle:777: dynamic constant assignment
    TAG_SET = TagSet.new 
             ^
/usr/local/bin/raggle:827: class definition in method body
/usr/local/bin/raggle:1042: module definition in method body
/usr/local/bin/raggle:1043: dynamic constant assignment
      NO_ATTRIBUTES = {}.freeze
                     ^
/usr/local/bin/raggle:1044: dynamic constant assignment
      ATTRIBUTE_LIST_RE = /\s*([^>=\s]+)\s*(?:=\s*(?:(...
                         ^
/usr/local/bin/raggle:1045: dynamic constant assignment
      PARSER_RE = %r!<(/?\w+[^>]*?/?)>|([^<]*)!m
                 ^
/usr/local/bin/raggle:1097: module definition in method body
/usr/local/bin/raggle:1149: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
... outline.attributes['refresh']: opml_refresh = outline.attri...
...                               ^
/usr/local/bin/raggle:1150: syntax error, unexpected keyword_else, expecting keyword_end
            else opml_refresh = nil; end  # fixes clobberation
                ^
/usr/local/bin/raggle:1183: warning: else without rescue is useless
/usr/local/bin/raggle:1188: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
        if $wins: Interfaces::NcursesInterface...
                 ^
/usr/local/bin/raggle:1225: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
        if $wins: Interfaces::NcursesInterface...
                 ^
/usr/local/bin/raggle:1228: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:1261: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:1269: class definition in method body
/usr/local/bin/raggle:1271: dynamic constant assignment
    VERSION = '0.1.0'
             ^
/usr/local/bin/raggle:1313: module definition in method body
/usr/local/bin/raggle:1317: dynamic constant assignment
    Item = Struct.new :title, :link, :desc, :date
          ^
/usr/local/bin/raggle:1323: class definition in method body
/usr/local/bin/raggle:1620: class definition in method body
/usr/local/bin/raggle:1863: module definition in method body
/usr/local/bin/raggle:2028: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--config':  $config['config_path'] = arg
                            ^
/usr/local/bin/raggle:2029: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--force':   ret['force'] = true
                           ^
/usr/local/bin/raggle:2030: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--list':    ret['mode'] = 'list'
                          ^
/usr/local/bin/raggle:2031: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--title':   ret['title'] = arg
                           ^
/usr/local/bin/raggle:2032: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--url':     ret['url'] = arg
                         ^
/usr/local/bin/raggle:2033: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--refresh': ret['refresh'] = arg.to_i
                             ^
/usr/local/bin/raggle:2037: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--priority':      ret['priority'] = arg.to_i
                              ^
/usr/local/bin/raggle:2038: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--purge':         ret['mode'] = 'purge'
                           ^
/usr/local/bin/raggle:2041: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--lock-title':    ret['lock_title?'] = true
                                ^
/usr/local/bin/raggle:2042: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--unlock-title':  ret['lock_title?'] = false
                                  ^
/usr/local/bin/raggle:2043: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--save-items':    ret['save_items?'] = true
                                ^
/usr/local/bin/raggle:2047: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--unsave-items':  ret['save_items?'] = false
                                  ^
/usr/local/bin/raggle:2048: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--verbose':       $config['verbose'] = true
                             ^
/usr/local/bin/raggle:2055: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--diag':          $config['diag'] = true
                          ^
/usr/local/bin/raggle:2058: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when '--ascii':         ret['ascii'] = true
                           ^
/usr/local/bin/raggle:2223: module definition in method body
/usr/local/bin/raggle:2228: module definition in method body
/usr/local/bin/raggle:2232: module definition in method body
/usr/local/bin/raggle:2336: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
...sInterface::get_win_id('find'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2337: syntax error, unexpected keyword_when, expecting keyword_end
          when NcursesInterface::get_win_id...
              ^
/usr/local/bin/raggle:2337: syntax error, unexpected ':', expecting keyword_end
...esInterface::get_win_id('cat'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2338: syntax error, unexpected keyword_when, expecting keyword_end
          when NcursesInterface::get_win_id...
              ^
/usr/local/bin/raggle:2338: syntax error, unexpected ':', expecting keyword_end
...sInterface::get_win_id('keys'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2339: syntax error, unexpected keyword_when, expecting keyword_end
          when NcursesInterface::get_win_id...
              ^
/usr/local/bin/raggle:2339: syntax error, unexpected ':', expecting keyword_end
...sInterface::get_win_id('edit'): $wins[$a_win].close(true)
...                               ^
/usr/local/bin/raggle:2727: class definition in method body
/usr/local/bin/raggle:2925: class definition in method body
/usr/local/bin/raggle:3103: class definition in method body
/usr/local/bin/raggle:3408: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
            if str =~ /true/i: newopts = {feedopt => true}
                              ^
/usr/local/bin/raggle:3410: syntax error, unexpected keyword_when, expecting keyword_end
          when 'priority', 'refresh'
              ^
/usr/local/bin/raggle:3410: syntax error, unexpected ',', expecting keyword_end
          when 'priority', 'refresh'
                          ^
/usr/local/bin/raggle:3412: syntax error, unexpected keyword_when, expecting keyword_end
          when 'max_items'
              ^
/usr/local/bin/raggle:3413: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
            if str =~ /none|nil/i: newopts = {feedopt => nil}
                                  ^
/usr/local/bin/raggle:3415: syntax error, unexpected keyword_when, expecting keyword_end
          when 'category'
              ^
/usr/local/bin/raggle:3416: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
            if str =~ /\bnone\b|\bnil\b/i: newopts = {feedopt => nil}
                                          ^
/usr/local/bin/raggle:3417: warning: else without rescue is useless
/usr/local/bin/raggle:3419: warning: else without rescue is useless
/usr/local/bin/raggle:3428: syntax error, unexpected keyword_else, expecting '}'
        else  # if no change
            ^
/usr/local/bin/raggle:3430: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3578: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
              when (/^(\w+):\/\//): m
                                   ^
/usr/local/bin/raggle:3595: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3602: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when (/(\w+):\/\//):  item_url
                                ^
/usr/local/bin/raggle:3629: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3668: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
          when /%s/:  cmd_part % url.escape
                    ^
/usr/local/bin/raggle:3669: syntax error, unexpected keyword_when, expecting '}'
          when '${browser}':  $config['browser']
              ^
/usr/local/bin/raggle:3669: syntax error, unexpected ':', expecting '}'
          when '${browser}':  $config['browser']
                            ^
/usr/local/bin/raggle:3670: syntax error, unexpected keyword_else, expecting '}'
          else cmd_part
              ^
/usr/local/bin/raggle:3691: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:3731: dynamic constant assignment
      DIRECTION_BACKWARD = -1
                          ^
/usr/local/bin/raggle:3732: dynamic constant assignment
      DIRECTION_FORWARD = 1
                         ^
/usr/local/bin/raggle:3836: syntax error, unexpected ':', expecting keyword_then or ';' or '\n'
          if title: added = Engine::add_feed({ '...
                   ^
/usr/local/bin/raggle:3841: warning: else without rescue is useless
/usr/local/bin/raggle:3851: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4115: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
          when 'DC':        key = 'Delete'
                    ^
/usr/local/bin/raggle:4116: syntax error, unexpected keyword_when, expecting '}'
          when ' ':         key = 'Space'
              ^
/usr/local/bin/raggle:4116: syntax error, unexpected ':', expecting '}'
          when ' ':         key = 'Space'
                   ^
/usr/local/bin/raggle:4117: syntax error, unexpected keyword_when, expecting '}'
          when 'NPAGE':     key = 'Page Down'
              ^
/usr/local/bin/raggle:4117: syntax error, unexpected ':', expecting '}'
          when 'NPAGE':     key = 'Page Down'
                       ^
/usr/local/bin/raggle:4118: syntax error, unexpected keyword_when, expecting '}'
          when 'PPAGE':     key = 'Page Up'
              ^
/usr/local/bin/raggle:4118: syntax error, unexpected ':', expecting '}'
          when 'PPAGE':     key = 'Page Up'
                       ^
/usr/local/bin/raggle:4119: syntax error, unexpected keyword_when, expecting '}'
          when '
                ':        key = 'Control-L'
              ^
/usr/local/bin/raggle:4119: syntax error, unexpected ':', expecting '}'
          when '
                ':        key = 'Control-L'
                   ^
/usr/local/bin/raggle:4120: syntax error, unexpected keyword_when, expecting '}'
          when '	':  key = 'Tab'
              ^
/usr/local/bin/raggle:4120: syntax error, unexpected ':', expecting '}'
          when '	':  key = 'Tab'
                	  ^
/usr/local/bin/raggle:4121: syntax error, unexpected keyword_when, expecting '}'
          when /^\d$/:      key = '1-9'
              ^
/usr/local/bin/raggle:4121: syntax error, unexpected ':', expecting '}'
          when /^\d$/:      key = '1-9'
                      ^
/usr/local/bin/raggle:4122: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4254: syntax error, unexpected ':', expecting keyword_then or ',' or ';' or '\n'
            when /feed/:  cl = NcursesInterface::ListWindow
                        ^
/usr/local/bin/raggle:4255: syntax error, unexpected keyword_when, expecting '}'
            when /item/:  cl = NcursesInterface::ListWindow
                ^
/usr/local/bin/raggle:4255: syntax error, unexpected ':', expecting '}'
            when /item/:  cl = NcursesInterface::ListWindow
                        ^
/usr/local/bin/raggle:4256: syntax error, unexpected keyword_when, expecting '}'
            when /desc/:  cl = NcursesInterface::TextWindow
                ^
/usr/local/bin/raggle:4256: syntax error, unexpected ':', expecting '}'
            when /desc/:  cl = NcursesInterface::TextWindow
                        ^
/usr/local/bin/raggle:4257: syntax error, unexpected keyword_else, expecting '}'
/usr/local/bin/raggle:4259: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4314: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4320: module definition in method body
/usr/local/bin/raggle:4324: class definition in method body
/usr/local/bin/raggle:4665: module definition in method body
/usr/local/bin/raggle:4666: class definition in method body
/usr/local/bin/raggle:4704: class definition in method body
/usr/local/bin/raggle:4732: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:4738: module definition in method body
/usr/local/bin/raggle:5574: syntax error, unexpected keyword_end, expecting '}'
/usr/local/bin/raggle:6658: syntax error, unexpected $end, expecting '}'
Apparently (https://bbs.archlinux.org/viewtopic.php?pid=711844) due to changes in syntax between Ruby 1.8 (which I believe raggle was built on) and Ruby 1.9 which I use.. I am not going to downgrade because I have numerous services depending on it.

As you can see, I havent had much success so far.. To be honest, I am pretty shocked because I really thought automating attachment download from RSS feeds was an "old" technology like Usenet.. Maybe I am doing/thinking the wrong way?
 
Old 12-29-2011, 07:33 PM   #4
lpallard
Member
 
Registered: Nov 2008
Posts: 987

Original Poster
Rep: Reputation: Disabled
Bump!? No one?
 
Old 01-03-2012, 10:37 AM   #5
lpallard
Member
 
Registered: Nov 2008
Posts: 987

Original Poster
Rep: Reputation: Disabled
OK so far I've been able to use cast get to retrieve podcasts such as the Linux action show, but somehow automatically downloading pictures via RSS feeds remains a mystery .. I am pretty sure its possible but in light of my posts above you see I haven't had much success....

I always wanted to follow some photography RSS sites to build a huge database of high Res pictures for a screen saver on my xbmc htpc ... APOD has that kind of pictures but I have other sites in mind. Not to mention some friends that are in professional photography ..

That's why I am that interested in doing this

So again I am asking this community for kind help! Anybody doing this? Its part of several programs as a nice functionality so I imagine with Linux flexibility and tools like curl, wget and others its feasible. Even a bash script would do. Download the pics via RSS, dump to a specific folder, get xbmc to play the slideshow from that folder and off you go!

Anyways thanks !
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Command Line RSS Reader and Weather Forecaster? Baryonic Being Linux - Software 5 07-26-2010 12:41 PM
Automatic downloader for command line alpha_lt Linux - Software 1 06-23-2010 11:08 AM
Command line RSS,PLS,M3U player. almost000 Linux - Software 6 10-06-2007 12:39 AM
command-line RSS client? prell Linux - Software 1 07-18-2005 10:58 AM
command line RSS? DJOtaku Linux - Software 2 05-26-2005 12:19 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration