Using wget or curl command to download from dynamic urls

pratap.iisc · 07-25-2009, 07:04 PM

Dear all,
I want to use two of the fantastic commands available in linux e.g. wget and curl which help me download files with direct and straightforward links right from the command prompt without going into any gui browser!
But unfortunately, I am not being able to download the files which would be linked with dynamic urls (urls containing "?" in them which is probably used for redirection).
The link to all the the IEEE papers also fall in this category...Eg. http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5173739&isnumber=5173690
Also some times some of the useful software also come in this format E.g. "http://www.mozilla.com/products/download.html?product=firefox-3.5.1&os=win&lang=en-US"
While using wget to download files like this I get the error...(I am attaching the complete log...)

Command: user@machine:~> wget "http://www.mozilla.com/products/download.html?product=firefox-3.5.1&os=win&lang=en-US"

--04:33:52-- http://www.mozilla.com/products/down...win&lang=en-US
=> `download.html?product=firefox-3.5.1&os=win&lang=en-US'
Resolving iiscproxy.serc.iisc.ernet.in... aa.bb.cc.dd, aa.bb.cc.ee, aa.bb.cc.fe, ...
Connecting to iiscproxy.serc.iisc.ernet.in|aa.bb.cc.dd|:3128... connected.
Proxy request sent, awaiting response... 302 Moved Temporarily
Location: http://www.mozilla.com/en-US/product...win&lang=en-US [following]
--04:33:53-- http://www.mozilla.com/en-US/product...win&lang=en-US
=> `download.html?product=firefox-3.5.1&os=win&lang=en-US'
Connecting to iiscproxy.serc.iisc.ernet.in|aa.bb.cc.dd|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
download.html?product=firefox-3.5.1&os=win&lang=en-US: File name too long

Cannot write to `download.html?product=firefox-3.5.1&os=win&lang=en-US' (File name too long).

Command: user@machine:~> wget "http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5173739&isnumber=5173690"

--04:58:39-- http://ieeexplore.ieee.org/stamp/sta...number=5173690
=> `stamp.jsp?tp=&arnumber=5173739&isnumber=5173690'
Resolving iiscproxy.serc.iisc.ernet.in... aa.bb.cc.dd, 10.66.70.81, 10.16.40.12, ...
Connecting to iiscproxy.serc.iisc.ernet.in|aa.bb.cc.dd|:3128... connected.
Proxy request sent, awaiting response... 302 Moved Temporarily
Location: http://ieeexplore.ieee.org/stamp/sta...number=5173690 [following]
--04:58:40-- http://ieeexplore.ieee.org/stamp/sta...number=5173690
=> `stamp.jsp?tp=&arnumber=5173739&isnumber=5173690'
Connecting to iiscproxy.serc.iisc.ernet.in|aa.bb.cc.dd|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 612 [text/html]
stamp.jsp?tp=&arnumber=5173739&isnumber=5173690: File name too long

Cannot write to `stamp.jsp?tp=&arnumber=5173739&isnumber=5173690' (File name too long).

Can anybody suggest any workaround for these type of links?

Waiting for some positive responses,
-Pratap

JulianTosh · 07-26-2009, 02:52 AM

for wget, you could use the -o option to specify the output file...

wget -o /tmp/myDownload "http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5173739&isnumber=517369"

archtoad6 · 07-26-2009, 10:55 PM

I think you mean the -O option, from the wget man page:

Quote:

-o logfile
--output-file=logfile
Log all messages to logfile. The messages are normally reported to standard error.

-O file
--output-document=file
The documents will not be written to the appropriate files, but all will be concatenated together and written to file. If - is used as file, documents will be printed to standard output, disabling link conversion. (Use ./- to print to a file literally named -.)