wget command access forbidden problem pls help
 

hi,

The site is http://files.brothersoft.com
when I tried to wget it, I got the following error message,


[root@localhost b]# wget -t1 -r -np --level=1 http://files.brothersoft.com
--18:10:53-- http://files.brothersoft.com/
=> `files.brothersoft.com/index.html'
Resolving files.brothersoft.com... done.
Connecting to files.brothersoft.com[70.84.95.234]:80... connected.
HTTP request sent, awaiting response... 403 Access Forbidden
18:10:54 ERROR 403: Access Forbidden.


FINISHED --18:10:54--
Downloaded: 0 bytes in 0 files
[root@localhost b]#


There are lot of exe files under http://files.brothersoft.com (Eg. http://files.brothersoft.com/i_bpk_lite.exe


I can download the above i_bpk_lite.exe file using wget http://files.brothersoft.com/i_bpk_lite.exe. At this time I can download the file.

But there are a number of exe files on the above site. I want to download all of them using single wget command.

I tried :

root@localhost b]# wget -t1 -r -np --level=1 http://files.brothersoft.com/[*]*
Warning: wildcards not supported in HTTP.
--18:19:28-- http://files.brothersoft.com/%5B*%5D*
=> `files.brothersoft.com/[*]*'
Resolving files.brothersoft.com... done.
Connecting to files.brothersoft.com[70.84.95.234]:80... connected.
HTTP request sent, awaiting response... 404 Object Not Found
18:19:29 ERROR 404: Object Not Found.


FINISHED --18:19:29--
Downloaded: 0 bytes in 0 files
[root@localhost b]#


But it is not working. I tried to use wild-characters to download all files on the site.

How can I do this??? please help.
Thanks in Advance.

This is a completely wild guess, but I'm going to bet that the site is using Apache with mod_security and there are rules in mod_security that prevent the use of tools like wget and curl. Unless you have administrator privileges to that site, you probably can't use wget to get everything.

Is there anyway around this is? I am looking at the same problem on a different site. The site is a company that I do business with and I have an account that allows me access to the entire site. Is there a way to use WGET and a username and password?

You can use the --user and --password option to wget. Note, however, that this only works if the site is using "vanilla" http encryption (and not some added on system like a CMS to control access). In that case you might have to script wget to use the session key returned by the site to download files.

As always, be sure that downloading with a tool like wget is allowed in the site's TOS before proceeding. Some webmasters don't like it for various reasons.