wget - using --user-agent option still results in 403/forbidden error
 

I'm hoping if someone can tell me what I might be doing wrong.

A number of websites have directories I want to download or graphics I want to DL. I can almost always access them through a browser or by directly downloading the file using wget:

wget http://www.somesite.com/dir1/dir2/pic1.jpg.

But if I try to use -r or -m, I get the dreaded "403/Forbidden" error, despite being able to open the file in my browser.

I've tried many combinations of -U options; -U firefox -U Mozilla, -U "Mozilla, platform, blah blah" and they NEVER work.

Is there something else I can do? Most of the time when I Google this issue, the solutions stop with forging a user agent. That never seems to work for me.

What am I doing wrong?

What's the site in question?

Can you browse to http://www.somesite.com/dir1/dir2/pic1.jpg traditionally?

I can see the page/graphics through a browser.
 

This is the company I work for, and like I said in my original post, I can get the link through a browser:

http://www.barclayvouchers.co.jp/ima...mainvisual.jpg

Naturally, if I wget the graphic directly, it also works, but that defeats the purpose of using wget (addressing the filenname specifically).

But that isn't the question - this happens to me on a LOT of websites. Not only my own company's website, but MOST of the time I get a 403/Forbidden error.

So the original question remains: am I doing something wrong with wget? I've tried using -m and -U with all types of descriptions after -U and they never work; I always end up at 403/Forbidden.

wrt "almost always"...what do the logs say about these events specifically?

You should be able to find 4 events in the logs, 2 for the browser (1 each Success and Fail) and 2 for wget (1 each Success and Fail)

Again, all 4 details should be in the logs.

I'm assuming apache error 403 is from apache,so ...
What are the owner:group perms on /path/to/dir1/dir2/pic1.jpg?
What are, if any .htaccess files (or comparable httpd.conf inclusions)?
Have you tried from another host to use the dreaded -r or -m options?
wget version? terminal >
output please. Thanks.

Please let us know.

It is likely the htaccess file is set up to prevent document/image "leeching" by direct download.

.htaccess file
 

@NyteOwl:

I guess that's what I'm asking; will an .htaccess file block wget:

- even if the user agent is forged, and
- even if the files are accessible through a web browser?

Every time I get a 403 error, both of the above conditions are met.

It happens with a number of sites and from various machines, so that's why I said "almost always". One time I was able to actually download an entire site when all I wanted was some PDFs, but other than that I get 403/Forbidden errors. :)

Why would I find a fail entry in the browser logs, and why would I find a success in the wget logs? It works in a browser and always gets a 403/Forbidden using wget.

It is from Apache.
The most recent problem I've had is on my own company's site, but I've had it at other sites, so I don't have access to the .htaccess files. Obviously, the .htaccess files are preventing wget from downloading, I was trying to figure out if it's a limitation of wget or if I wasn't using it correctly.

I have tried from another host, same results.
And I don't dread the -r and -m options, I dread the 403/Forbidden error. :)

I can do that once I'm back at work. At home I only have access to wget on *cough* Windows *cough*.

I guess the limitation is that wget is going to be stopped by an .htaccess file, regardless of changing the user-agent.

For the site(s) that you have control over, you will have references to the 403 "error" and the GET statement from the browser session.
or similar.

There is no "browser logs" or "wget log".
Both are clients asking apache (server) for the file and hence all requests should be logged in the apache log file.

Now, as for wget options, this works for me here:
wrt: "403/Forbidden", this can happen 2 ways that I know of, the /path/to/dir1/dir2/pic1.jpg has permissions that the apache software/daemon doesn't have access to, OR, the robots.txt and/or .htaccess prevents it.

I hope this helps.

You answered my question, thanks.
 

That's why I was confused; I was talking about the client side and you were talking about the server side.

Yeah, that works for me too, but the original problem remains. I can't just use the directory and get the files, I have to specify the file, which defeats the purpose of using wget.


Okay, that's pretty much what I was looking for. The fact that wget is stopped by a .htaccess file. I don't think it's file permissions since it's available through a browser or specifying directly via wget. Either way, that's my answer; wget is limited.

It did. Thanks for your answers.

I beg to differ:
That implies you know what your are asking the server for and therefore non-interactive.

The googletubes are chock-full of people who have (probably, for years) wondered why wget doesn't seem to grab arbitrary directories.

You'd be correct on the perms...Good Eye!

More like the system admin has done a good job of keeping things tight.

Good points
 

Sorry, let me clarify; it defeats my purpose for using wget (like I said earlier, if I have to access said files through a
browser, using wget does not save steps.)

In this case (and others when I use it) I know what the directories are, I just was hoping to save time over right-click/save as by using wget.


True dat.

Name one.

If the problem is referrer, try
If you use -r option you very likely can point to an index file and all linked content will be downloaded.

One step ahead of you.
 

Already did. But as I said earlier, it happens on multiple sites.


That's what I was trying to find out; was the problem the referrer, or something else?
Thanks for the tip, tho', I'll try it.

"Very likely" - based on actual experience, or the way it's supposed to work? As I mentioned earlier, I've tried both -r and -m (with -r I usually add -l1 as well) and still get the error.

I did find that by loading a whole boatload of options such as wget -e robots=off -m -r -l3 -np -nd -U Mozilla (site) AND by modifying the .wgetrc file, it will load the entire site, even if you're just hunting for one or two directories. Kind of silly to restrict a directory via an .htaccess file to prevent people from doing mass downloads but downloading the entire site works. :)

Thanks for the tips.

It's not a matter of blocking wget per se. It's a matter of how the files are accessed. If they are called in reference to an http request by the server as part of the web page then they can be displayed. You can also prevent them from being displayed if they are not so accessed.

For example:

Any request that doesn't come through domain.tld will return a 403 error.