Wget and membership required
 

I am trying to get a csv file from barchart.com but it is only accessable if you are logged in to the site. I have a membership and can easily download it with lynx or Firefox, but I am looking to automate the download.

I have tried --http-user and --http-password but to no avail. I always get the "you need to log in" page. The best I can tell is it is a multistep process.
How do I access the site with my account so I may download the csv?


Thanks,
Rogue

Hi, welcome to LQ!

Investigate the -load-cookies & -save-cookies options.


Cheers,
Tink

That did the job...just had to export cookie out of Firefox and then the --load-cookie switch.

Sweet. ;}


Cheers,
Tink

Hi rogueit,

For the benefit of others (like me) searching the same info, would you mind posting your solution (with the details of the wget commandline options and what needs to be copied from firefox folder and to where)? Thanks.

-JB

I'm not rogueit, and I didn't use his solution; but I can try to outline the way *I*
would try this ...

Step 1:
Log into your site with wget using
wget --http-user USER --http-password PASSWORD -save-cookies ~/.my_wget_jar http://site.of.choice

Step 2:
wget -load-cookies ~/.my_wget_jar http://site.of.choice/tucked/away/pages/for/auth_users



Should work - is untested.



Cheers,
Tink