LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-08-2011, 08:06 AM   #1
rogueit
LQ Newbie
 
Registered: Jul 2011
Posts: 4

Rep: Reputation: Disabled
Wget and membership required


I am trying to get a csv file from barchart.com but it is only accessable if you are logged in to the site. I have a membership and can easily download it with lynx or Firefox, but I am looking to automate the download.

I have tried --http-user and --http-password but to no avail. I always get the "you need to log in" page. The best I can tell is it is a multistep process.
How do I access the site with my account so I may download the csv?


Thanks,
Rogue

Last edited by Tinkster; 07-08-2011 at 05:20 PM. Reason: URL removed... that was what got the post auto-moderated
 
Old 07-08-2011, 05:32 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Hi, welcome to LQ!

Investigate the -load-cookies & -save-cookies options.


Cheers,
Tink
 
Old 07-08-2011, 08:35 PM   #3
rogueit
LQ Newbie
 
Registered: Jul 2011
Posts: 4

Original Poster
Rep: Reputation: Disabled
That did the job...just had to export cookie out of Firefox and then the --load-cookie switch.
 
Old 07-08-2011, 08:39 PM   #4
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Sweet. ;}


Cheers,
Tink
 
Old 03-06-2012, 01:33 AM   #5
bcj01
LQ Newbie
 
Registered: Nov 2007
Posts: 2

Rep: Reputation: 0
Hi rogueit,

For the benefit of others (like me) searching the same info, would you mind posting your solution (with the details of the wget commandline options and what needs to be copied from firefox folder and to where)? Thanks.

-JB
 
Old 03-08-2012, 08:21 PM   #6
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
I'm not rogueit, and I didn't use his solution; but I can try to outline the way *I*
would try this ...

Step 1:
Log into your site with wget using
wget --http-user USER --http-password PASSWORD -save-cookies ~/.my_wget_jar http://site.of.choice

Step 2:
wget -load-cookies ~/.my_wget_jar http://site.of.choice/tucked/away/pages/for/auth_users



Should work - is untested.



Cheers,
Tink
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to resume an interrupted wget using wget.log? misterJ Linux - Software 2 06-19-2011 02:21 PM
[SOLVED] Problem by using wget: ERROR 407: Proxy Authentication Required thomas2004ch Linux - Software 4 09-01-2010 04:38 PM
Q. How much is membership? Sepero LQ Suggestions & Feedback 19 11-13-2006 10:59 AM
Lifetime membership Dirty_Ink Linspire/Freespire 7 05-13-2004 12:41 AM
Contributing Membership... flapjackboy LQ Suggestions & Feedback 5 09-05-2003 10:25 AM


All times are GMT -5. The time now is 04:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration