LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (http://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   check if a website file has changed (http://www.linuxquestions.org/questions/linux-newbie-8/check-if-a-website-file-has-changed-782954/)

qwertyjjj 01-18-2010 08:36 AM

check if a website file has changed
 
I have a site that I login to to check updates. It does not have RSS because users need to authenticate themselves before getting access to the page.
Is there a way to write a script that can login to the page and check whether the HTML has changed and then send me an email?

nuwen52 01-18-2010 09:40 AM

You could maybe script wget to do this? If wget asks for a log and password for the website, then you could automate this using expect. Another way to do this is to use the urllib or urllib2 in python. Depending on how you have to log into the page, these should do what you are looking for.

qwertyjjj 01-18-2010 09:55 AM

Quote:

Originally Posted by nuwen52 (Post 3830840)
You could maybe script wget to do this? If wget asks for a log and password for the website, then you could automate this using expect. Another way to do this is to use the urllib or urllib2 in python. Depending on how you have to log into the page, these should do what you are looking for.

I cannot get wget to successfully login to the page. I try to submit the details by POST to a login form but it doesn't pick up the correct page afterwards, just the standard page when not logged in.
I picked up the form name from the source on this page, not sure if it's the correct one or not: https://www.inthemoneystocks.com/pro...watch_list.php

Code:

[root@serve~]# wget --post-data='username=DUMMYUSER&password=DUMMYPASSWORD'--save-cookies=my-cookies.txt --keep-session-cookies https://www.inthemoneystocks.com/swing_trade_month.php
--2010-01-18 14:51:32--  https://www.inthemoneystocks.com/swing_trade_month.php
Resolving www.inthemoneystocks.com... 72.29.80.60
Connecting to www.inthemoneystocks.com|72.29.80.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15559 (15K) [text/html]
Saving to: `swing_trade_month.php'

100%[=======================================================================================================================================>] 15,559      63.6K/s  in 0.2s

2010-01-18 14:51:33 (63.6 KB/s) - `swing_trade_month.php' saved [15559/15559]

[root@server ~]# wget https://www.inthemoneystocks.com/pro...watch_list.php
--2010-01-18 14:52:16--  https://www.inthemoneystocks.com/pro...watch_list.php
Resolving www.inthemoneystocks.com... 72.29.80.60
Connecting to www.inthemoneystocks.com|72.29.80.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17008 (17K) [text/html]
Saving to: `pro_trader_watch_list.php'

100%[=======================================================================================================================================>] 17,008      72.3K/s  in 0.2s

2010-01-18 14:52:17 (72.3 KB/s) - `pro_trader_watch_list.php' saved [17008/17008]

[root@server ~]# nano pro_trader_watch_list.php
[root@server ~]#


qwertyjjj 01-18-2010 02:24 PM

any thoughts on the login part?

qwertyjjj 01-18-2010 05:17 PM

After the initial request, it is rerouted through to login again :

Code:

[root@server ~]# wget --save-cookies cookies.txt --keep-session-cookies --post-data 'username=MYUSERN&currentpassword=MYPASS' http://www.inthemoneystocks.com/login.php
--2010-01-18 22:15:12--  http://www.inthemoneystocks.com/login.php
Resolving www.inthemoneystocks.com... 72.29.80.60
Connecting to www.inthemoneystocks.com|72.29.80.60|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16397 (16K) [text/html]
Saving to: `login.php.1'

100%[=======================================================================================================================================>] 16,397      62.9K/s  in 0.3s

2010-01-18 22:15:13 (62.9 KB/s) - `login.php.1' saved [16397/16397]

[root@server ~]# wget --load-cookies cookies.txt -p https://www.inthemoneystocks.com/pro_trader_watch_list_prem.php                                                --2010-01-18 22:15:45--  https://www.inthemoneystocks.com/pro_trader_watch_list_prem.php
Resolving www.inthemoneystocks.com... 72.29.80.60
Connecting to www.inthemoneystocks.com|72.29.80.60|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://www.inthemoneystocks.com/login.php [following]
--2010-01-18 22:15:45--  https://www.inthemoneystocks.com/login.php
Connecting to www.inthemoneystocks.com|72.29.80.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16254 (16K) [text/html]
Saving to: `www.inthemoneystocks.com/login.php'

100%[=======================================================================================================================================>] 16,254      68.3K/s  in 0.2s

2010-01-18 22:15:46 (68.3 KB/s) - `www.inthemoneystocks.com/login.php' saved [16254/16254]

Loading robots.txt; please ignore errors.
--2010-01-18 22:15:46--  https://www.inthemoneystocks.com/robots.txt
Reusing existing connection to www.inthemoneystocks.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 191 [text/plain]
Saving to: `www.inthemoneystocks.com/robots.txt'

100%[=======================================================================================================================================>] 191        --.-K/s  in 0s

2010-01-18 22:15:46 (6.77 MB/s) - `www.inthemoneystocks.com/robots.txt' saved [191/191]

--2010-01-18 22:15:46--  https://www.inthemoneystocks.com/main.css
Reusing existing connection to www.inthemoneystocks.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 19172 (19K) [text/css]
Saving to: `www.inthemoneystocks.com/main.css'

100%[=======================================================================================================================================>] 19,172      --.-K/s  in 0.1s

2010-01-18 22:15:47 (159 KB/s) - `www.inthemoneystocks.com/main.css' saved [19172/19172]

--2010-01-18 22:15:47--  https://www.inthemoneystocks.com/js/mootools1.11.js
Reusing existing connection to www.inthemoneystocks.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 43715 (43K) [application/javascript]
Saving to: `www.inthemoneystocks.com/js/mootools1.11.js'

100%[=======================================================================================================================================>] 43,715      183K/s  in 0.2s

2010-01-18 22:15:47 (183 KB/s) - `www.inthemoneystocks.com/js/mootools1.11.js' saved [43715/43715]

--2010-01-18 22:15:47--  https://www.inthemoneystocks.com/js/fValidator.js
Reusing existing connection to www.inthemoneystocks.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 8966 (8.8K) [application/javascript]
Saving to: `www.inthemoneystocks.com/js/fValidator.js'

100%[=======================================================================================================================================>] 8,966      --.-K/s  in 0.001s

2010-01-18 22:15:47 (10.4 MB/s) - `www.inthemoneystocks.com/js/fValidator.js' saved [8966/8966]

--2010-01-18 22:15:47--  https://www.inthemoneystocks.com/js/purchaseform.js
Reusing existing connection to www.inthemoneystocks.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 7248 (7.1K) [application/javascript]
Saving to: `www.inthemoneystocks.com/js/purchaseform.js'

100%[=======================================================================================================================================>] 7,248      --.-K/s  in 0.001s

2010-01-18 22:15:47 (8.26 MB/s) - `www.inthemoneystocks.com/js/purchaseform.js' saved [7248/7248]

--2010-01-18 22:15:47--  https://www.inthemoneystocks.com/ticker.swf
Reusing existing connection to www.inthemoneystocks.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 43872 (43K) [application/x-shockwave-flash]
Saving to: `www.inthemoneystocks.com/ticker.swf'

100%[=======================================================================================================================================>] 43,872      --.-K/s  in 0.1s

2010-01-18 22:15:47 (361 KB/s) - `www.inthemoneystocks.com/ticker.swf' saved [43872/43872]

FINISHED --2010-01-18 22:15:47--
Downloaded: 7 files, 136K in 0.7s (193 KB/s)
[root@server~]#


chrism01 01-18-2010 05:57 PM

For that sort of stuff I use Perl + http://search.cpan.org/~petdance/WWW...W/Mechanize.pm

qwertyjjj 01-18-2010 07:07 PM

Quote:

Originally Posted by chrism01 (Post 3831397)
For that sort of stuff I use Perl + http://search.cpan.org/~petdance/WWW...W/Mechanize.pm

I just thought wget would do it simply.
Could I use PHP as well? I'm more mailiar with that than perl.
What lib do I need for PHP?

qwertyjjj 01-18-2010 07:29 PM

Here is my current curl file but it doesn't seem to download the file:

Code:

#! /usr/bin/php

<?php
// INIT CURL
$ch = curl_init();

// SET URL FOR THE POST FORM LOGIN
curl_setopt($ch, CURLOPT_URL,
'https://www.inthemoneystocks.com/login.php');

// ENABLE HTTP POST
curl_setopt ($ch, CURLOPT_POST, 1);

// SET POST PARAMETERS : FORM VALUES FOR EACH FIELD
curl_setopt ($ch, CURLOPT_POSTFIELDS,
'username=xxxxx&currentpassword=xxxxx');

// IMITATE CLASSIC BROWSER'S BEHAVIOUR : HANDLE COOKIES
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

# Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
# not to print out the results of its query.
# Instead, it will return the results as a string return value
# from curl_exec() instead of the usual true/false.
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

// EXECUTE 1st REQUEST (FORM LOGIN)
$store = curl_exec ($ch);

// SET FILE TO DOWNLOAD
curl_setopt($ch, CURLOPT_URL,
'https://www.inthemoneystocks.com/pro_trader_watch_list_prem.php');

// EXECUTE 2nd REQUEST (FILE DOWNLOAD)
$content = curl_exec ($ch);

// CLOSE CURL
curl_close ($ch);

?>

I guess I need to write $content to a file? How do I do that?

qwertyjjj 01-18-2010 09:17 PM

I've got it into a file but I am noe using file_get_contents to check the new file every 10mins and if it changes then I email it to myself.
Unfortunately, curl or something is changing the file size by a few bytes even if nothing has changed.
Any ideas?

nuwen52 01-19-2010 08:39 AM

Is there a date or perhaps a number of hits counter on the page? Curl should just get contents unmodified, so it's probably the page itself that is changing. I would do a diff on two versions of the page and see what changed. Then you can remove those sections from your checks. Just a thought.

qwertyjjj 01-19-2010 09:08 AM

Quote:

Originally Posted by nuwen52 (Post 3832135)
Is there a date or perhaps a number of hits counter on the page? Curl should just get contents unmodified, so it's probably the page itself that is changing. I would do a diff on two versions of the page and see what changed. Then you can remove those sections from your checks. Just a thought.

Any ideas on how to do a diff in PHP?

nuwen52 01-19-2010 09:58 AM

Well, what I meant was do a diff manually just to see what's different. If what's different is minor, then you can program your php script to ignore those minor types of changes. Like, if the line that's different looks like "DATE: Thu Feb 27 2010", then you can tell the script to ignore that line when determining if the file changed. But, other than that...
Code:

exec("diff <file1> <file2> > /tmp/difftmp.txt");
diff_file=fopen("/tmp/difftmp.txt", "r");

and interpret the file. I don't know of a PHP library for diff.


All times are GMT -5. The time now is 02:05 AM.