LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-29-2012, 12:33 AM   #1
theBowler
LQ Newbie
 
Registered: Feb 2012
Posts: 6

Rep: Reputation: Disabled
wget the same page every 5 minutes


So I'm trying to write a script that will get me the stock prices for GE from yahoo finance. I can get the information with a one-off page I used wget to store in my home directory. My problem is that I need to it to run every 5 minutes, wget the page, rip the stock out and append it to a file. I can do everything except wget the page and use the file name in the script. Every time I try and wget the page again, it gives me filename.1, .2, .3, etc. Is there any way I can get around this?
 
Old 02-29-2012, 12:38 AM   #2
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
1. What command specific options are you using for wget? From the wget man page:
Quote:
When running Wget with `-r' or `-p', but without `-N', `-nd', or `-nc', re-downloading a file will result in the new copy simply overwriting the old.
2. If you are appending the data for the stock price (after parsing the downloaded page) to another file, why are you not simply deleting the downloaded page (and any associated files) when the script finishes? That would clean up disk space and avoid the file-overwrite problem to begin with.
 
Old 02-29-2012, 12:46 AM   #3
theBowler
LQ Newbie
 
Registered: Feb 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
Red face

Quote:
Originally Posted by Dark_Helmet View Post
1. What command specific options are you using for wget? From the wget man page:


2. If you are appending the data for the stock price (after parsing the downloaded page) to another file, why are you not simply deleting the downloaded page (and any associated files) when the script finishes? That would clean up disk space and avoid the file-overwrite problem to begin with.
1. only -o to redirect it. (why did I say rename? )

2. I didn't think about that, to be honest. I guess I've been working on this too long, it didn't pop in my head as an option.

Last edited by theBowler; 02-29-2012 at 12:53 AM.
 
Old 02-29-2012, 12:55 AM   #4
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
My apologies if #2 came off as harsh. I just re-read it, and it could be taken that way. I was more interested in finding out if there was some purpose beyond parsing the file that would require you to keep the file.

Anyway, if you want to try an approach for option #1, you probably want to use the -p option.

If you want to go with something for option #2, then you might want to try (ironically) the -P option--which, if I read the man page correctly, will place everything downloaded in a directory specified. That should make it easy for an "rm -r" on the directory after parsing.

Last edited by Dark_Helmet; 02-29-2012 at 12:56 AM.
 
Old 02-29-2012, 01:01 AM   #5
theBowler
LQ Newbie
 
Registered: Feb 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
I think I'm going to try number two; that seems like a fairly easy option to write. Thanks a lot!
 
Old 02-29-2012, 02:43 AM   #6
theBowler
LQ Newbie
 
Registered: Feb 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
Does someone mind double checking this crontab for me?

*/5 9-1630 * * 1-5 command

This is supposed to run every 5 minutes from 9am-430pm Monday through Friday every month, every week. Can you specify 1630 in the hour column?
 
Old 02-29-2012, 03:08 AM   #7
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
Quote:
Originally Posted by theBowler
Can you specify 1630 in the hour column?
Nope. According to the man page:
Quote:
Code:
              field          allowed values
              -----          --------------
              minute         0-59
              hour           0-23
              day of month   1-31
              month          1-12 (or names, see below)
              day of week    0-7 (0 or 7 is Sun, or use names)
Two options come to mind:
1. Have your script check the time, and if it's spawned at an undesirable time, do nothing. Then set your cron command to run every 5 minutes from 9AM to 5PM.

2. Create two cron jobs that launch the same script. One job is every 5 minutes from 9AM to 4PM. The other is a cron job that runs every 5 minutes from 4PM to 4:30PM.

With #2, you may get one "overlap" job (at 4PM) that you might have to fiddle with the start/stop times of the jobs to avoid.
 
Old 02-29-2012, 03:11 AM   #8
theBowler
LQ Newbie
 
Registered: Feb 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
I didn't think I could, but no harm in asking. Thanks again, Dark Helmet!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] wget failed to download a html page moebus Linux - General 11 01-31-2012 10:58 PM
A shell script that will wget for 30 minutes, then stop? Rotwang Linux - General 7 06-12-2008 03:28 AM
wget and links2 can't access web page. fakie_flip Programming 6 01-11-2008 05:34 PM
a script that runs wget every 3 minutes Menestrel Programming 3 07-26-2005 12:22 PM
Reloading a page every 3 minutes in lynx Menestrel Linux - Newbie 1 05-21-2005 11:42 AM


All times are GMT -5. The time now is 05:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration