LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-30-2019, 02:05 AM   #16
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,223
Blog Entries: 9

Rep: Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617

Quote:
Originally Posted by jamtat View Post
I do use newsboat for reading news feeds. But I'm hoping to automate this, hopefully as a cron job I can trigger each weekday. If you have particular suggestions for how a cli RSS utility might more easily automate things, I'd be happy to hear about that.
A Quick Search (tm) seems to indicate that utilities to parse RSS feeds do not exist as such. of course you can still check if one of the more advanced CLI RSS readers has capabilities that come very close to what you want (i.e. is scriptable).

if not, I still strongly recommend an XML tool to actually parse the feed.
have a look at the search.

edit:
oops, just noticed that the first search result recreates an XML parser with bash. not what i would've recommended. nevertheless, probably more robust than yours.

Last edited by ondoho; 03-30-2019 at 02:07 AM.
 
Old 03-30-2019, 03:04 PM   #17
jamtat
Member
 
Registered: Oct 2004
Distribution: Debian/Ubuntu, Arch, Gentoo, Void
Posts: 132

Original Poster
Rep: Reputation: 23
Thanks for the further input, ondoho. Since the cURL/grep/sed combination I've tested so far seems to be performing adequately I'm inclined to stick with it. But I will look further into your suggestions just in case.

Further reading and testing has led me to believe that an until loop should be most appropriate and elegant for this use case. What that should do is begin testing, by running the below script at a time specified in my system's cron file, whether a file with today's date is listed as being available. If it's not listed and thus grep does not find it, the script will sleep for 300 seconds, then check again. It will keep iterating like that until the designated pattern is found. Once the grep'd-for date pattern is found, the until loop exits, causing the wget command seen at the end of the script to run. That command will download, using a link gotten via the same date logic as was used in the until loop, the newly-uploaded episode to the local machine.

An example of how the new and improved script looks is as follows:
Code:
#!/bin/sh
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
until curl http://my.podcast.url/12345/show/ | grep $moday >/dev/null
do
sleep 300
#echo "sleeping 5 minutes"
done
wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
Testing of this script done so far on the local machine seems to indicate it should work as envisioned. A real-life test will have to wait until next week, when the podcast episodes begin becoming available again.

LATER EDIT: here's an updated version based on ondoho's remarks in #22 below, one that may work in cases where the URL for the desired podcast has not yet become available at the time the script was first run (the one above will apparently not work in such a case since the URL variable will be blank):
Code:
#!/bin/sh
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
until curl http://my.podcast.url/12345/show/ | grep $moday >/dev/null
do
sleep 300
#echo "sleeping 5 minutes"
done
URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
(some lines were moved or repeated after the "done" line)

Last edited by jamtat; 04-01-2019 at 06:34 PM.
 
Old 03-31-2019, 03:08 AM   #18
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,223
Blog Entries: 9

Rep: Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617
Quote:
Originally Posted by jamtat View Post
Code:
#!/bin/sh
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
until curl http://my.podcast.url/12345/show/ | grep $moday >/dev/null
do
sleep 300
#echo "sleeping 5 minutes"
done
wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
this doesn't make sense.
the whole until loop does nothing to the final command - $URL is the same before the loop, your script could just as well be:
Code:
#!/bin/sh
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
!!!

didn't you rather mean something like this:
Code:
#!/bin/bash
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
URL=""

while true
do
    URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
    [[ "$URL" != "" ]] && break
    sleep 300
    #echo "sleeping 5 minutes"
done

wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
???

I also replaced the shebang in your script's first line.
since your title asks about a bash script, make sure it is one!

please also run the script through shellcheck and see what else it complains about.
 
Old 03-31-2019, 02:05 PM   #19
jamtat
Member
 
Registered: Oct 2004
Distribution: Debian/Ubuntu, Arch, Gentoo, Void
Posts: 132

Original Poster
Rep: Reputation: 23
Quote:
Originally Posted by ondoho View Post
this doesn't make sense.
the whole until loop does nothing to the final command
My intention was not for the until loop to do anything to the final command. The until loop, as I envisioned it, is supposed to more or less stall the running of the final command so that it will execute only after the URL for today's episode has appeared in the list of episode URL's located at http://my.podcast.url/12345/show/. Does that make sense?
Quote:
$URL is the same before the loop, your script could just as well be:
Code:
#!/bin/sh
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
As I noted toward the beginning of this thread, that's pretty much what my original script looked like. And it works great if the URL for today's episode has been posted in the list already. But if it has not, the script does nothing. And then I have to run it again manually--sometimes repeatedly--until the URL for today's episode does appear in the list. I'm trying to make it more automated. The script, if it does not find the URL for today's episode in the list, should try again later--repeatedly--until it finds the URL for today's episode. The until loop I've created does, according to testing I've tried so far, allow the script to do just that: if the URL is not found it checks for it again in 5 minutes and and at 5-minute intervals thereafter. Once the date snippet indicating the presence of today's episode has been found in the list, the loop exits. Once the loop exits, the final wget command can execute, right?
Quote:
didn't you rather mean something like this:
Code:
#!/bin/bash
moday=$(date +%m-%d-)
date=$(date +%m-%d-%Y)
URL=""

while true
do
    URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D')
    [[ "$URL" != "" ]] && break
    sleep 300
    #echo "sleeping 5 minutes"
done

wget --user=myuname --password=mypass $URL -O /home/user/mydir/myfile-$date.mp3
???
I suppose that might work, though I haven't yet tested it. Could you explain how it is an improvement over the one I posted and that seems to be accomplishing the aim for which it was designed (caveat: again, I can only do a real-life test to confirm that it works tomorrow)? Please understand that I'm not trying to say that what I've come up with is superior or preferable or anything like that, even though it does seem to be effective. Your sample looks a lot more sophisticated and is likely more cleanly designed and written. I would only point out here that the sed stuff is not really needed in order to test whether the date snippet I'm grep'ing for is present among the URL's listed at the site; sed only needs to be brought into play later so as to strip off extraneous characters from the found URL (the URL is embedded in xml code), thereby putting it into a form suitable for feeding to wget.
Quote:
I also replaced the shebang in your script's first line.
since your title asks about a bash script, make sure it is one!
Thanks. It's a newbie oversight. And thanks for the shellcheck link.

Last edited by jamtat; 03-31-2019 at 05:20 PM.
 
Old 03-31-2019, 03:57 PM   #20
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ Win10
Posts: 9,510

Rep: Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036Reputation: 2036
if you move this around, putting your message about going to sleep before it gores to sleep then you will see it before it wakes up
Code:
echo "sleeping 5 minutes"
sleep 300
that is if you decide you still want a message to update what it is doing, so you will know.
 
Old 03-31-2019, 05:38 PM   #21
jamtat
Member
 
Registered: Oct 2004
Distribution: Debian/Ubuntu, Arch, Gentoo, Void
Posts: 132

Original Poster
Rep: Reputation: 23
Let me try posting an itemized list of what I would like the script to do. Maybe it will help to better clarify the aim.

1. cURL and grep the file found at the (fake) URL http://my.podcast.url/12345/show/ for the presence of a string matching the output of the command date +%m-%d-

2. if such a string is not found, i.e., grep returns exit code 1, discard the grep output, sleep for 5 minutes and try step 1. at 5-minute intervals thereafter until grep finds the string

3. when such a string is found (grep returns exit code 0), pipe the grep output through sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D' to get rid of extraneous characters the output contains and then read that into the variable $URL.

4. finally, use wget $URL to grab the file (podcast episode) the URL is pointing to. Done

NOTE that in this itemized iteration of the script I've made a slight change in how $URL is defined: in previous revisions it was a variable defined at the script's beginning, whereas in this rendition the variable is actually being created more on-the-fly from output produced by other parts of the script.

Last edited by jamtat; 03-31-2019 at 06:09 PM.
 
Old 04-01-2019, 02:09 PM   #22
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,223
Blog Entries: 9

Rep: Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617
Quote:
Originally Posted by jamtat View Post
My intention was not for the until loop to do anything to the final command. The until loop, as I envisioned it, is supposed to more or less stall the running of the final command so that it will execute only after the URL for today's episode has appeared in the list of episode URL's located at http://my.podcast.url/12345/show/. Does that make sense?
no, sorry.
what your script in that version did:
  • look if the desired file is available in the RSS feed, and store the result of that query in the URL variable (i.e. whether it actually returned a usable URL or not).
  • then keep checking every 5min or so if the desired file is available in the RSS feed
  • when it is, break the loop and download the result of the initial query, which is stored in the URL variable
That doesn't make sense!
 
Old 04-01-2019, 03:54 PM   #23
jamtat
Member
 
Registered: Oct 2004
Distribution: Debian/Ubuntu, Arch, Gentoo, Void
Posts: 132

Original Poster
Rep: Reputation: 23
Ok, thanks for that explanation ondoho. I don't have sufficient knowledge of scripting to dispute or confirm what you're saying, so I've got at least that opportunity to learn something more about it now.

Obviously, I was under the impression that the URL variable only gets defined once the script runs a command containing that variable, whereas I gather from your comments that it gets defined as soon as it is declared--correct? As you point out, that's right at the very beginning of the script, before I know for sure whether the link I'm aiming to capture using that variable is even available yet. That would clearly not accomplish the intended aim of the script.

So, back to the drawing board. Now I have to more carefully analyze the sample you've given so I can try to better understand it.

PS One implication of the defining of the URL variable as outlined above is that, were I to move the line URL=$(curl http://my.podcast.url/12345/show/ | grep $moday | sed -n '/"/!{/\n/{P;b}};s/"/\n/g;D') after the line consisting in the word "done," then the script should work as intended, correct? If so, would I also need to essentially re-define the moday variable after the "done" line, too?

LATER EDIT: it's beginning to seem to me that the moday variable does not need to be repeated: subsequent reading indicates to me that the loop executes in a subshell and that, once it exits, the process returns to the parent shell in which it was executing and so all variables that were defined there should still be in effect.

Last edited by jamtat; 04-01-2019 at 06:37 PM.
 
Old 04-02-2019, 01:53 AM   #24
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,223
Blog Entries: 9

Rep: Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617Reputation: 3617
it's not so much about "knowledge of scripting", it's about logic.
your script is syntactically correct but logically wrong.

btw, it would be nice if you showed us the final, working version.

Last edited by ondoho; 04-02-2019 at 01:58 AM.
 
Old 04-07-2019, 05:04 PM   #25
jamtat
Member
 
Registered: Oct 2004
Distribution: Debian/Ubuntu, Arch, Gentoo, Void
Posts: 132

Original Poster
Rep: Reputation: 23
Why is it more logical to assume that a variable will get expanded on parse as opposed to when it gets used within a command? It doesn't seem so sensible to me and was not at all what I was expecting. I don't see how I, as a relative newcomer to scripting, was in any way violating the dictates of logic in assuming that variables get expanded only when used in a command. For now it looks to me like maybe the decision about when to expand is either somewhat arbitrary--like the decision about whether cars traveling in opposite directions down the same road should drive past one another on the right or left sides--or else it relies on some convention(s) defined elsewhere in programing practice or experience (the "knowledge of scripting" to which I refered).

My modified version of the script I outlined in #17 above (the one given after "LATER EDIT" in that post) is the one that is currently working for me. It has so far proved effective in several tests. So long as the URL variable gets defined in the second half of the script, following the "done" line that marks the point at which the until loop exits, it is working as intended. Thanks for the input in this thread. I guess I'll go ahead and mark it "solved" now.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Ways to prevent cron e-mails from particular bash script located in cron.daily? postcd Linux - General 2 08-22-2017 09:55 AM
Scripts in cron.daily are not running daily abefroman Linux - Server 1 12-23-2014 01:11 PM
LXer: Bash Script To Get You Your Daily Farmer's Almanac Information LXer Syndicated Linux News 0 10-13-2008 07:30 AM
Poll: (Without the poll) - How is Linux used in your workplace? SlowCoder General 13 09-12-2007 12:03 AM
Advice? Best way to move files daily to a daily "date" named directory ziphem Linux - Newbie 2 04-15-2007 09:03 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:44 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration