LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Networking (https://www.linuxquestions.org/questions/linux-networking-3/)
-   -   Googlebot blocked - can't figure out why (https://www.linuxquestions.org/questions/linux-networking-3/googlebot-blocked-cant-figure-out-why-4175430874/)

cnmoore 10-06-2012 02:40 PM

Googlebot blocked - can't figure out why
 
Please help us, our forum very much depends on being indexed by Google.

As of October 1, Googlebot suddenly cannot access our forum at http://www.spywareinfoforum.com.
I know of nothing that changed on the server that day.

We do not seem to have ModSecurity - locate doesn't report any.

I don't find any likely address banned in iptables. Adding this line at the top of INPUT hasn't helped:
ACCEPT all -- 66.249.64.0/19 anywhere

The error that Googlebot gets (using Fetch as Googlebot in Webmaster Tools):
Quote:

HTTP/1.1 403 Forbidden
Date: Fri, 05 Oct 2012 22:43:32 GMT
Server: Apache/2
Content-Length: 406
Keep-Alive: timeout=1, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /index.php
on this server.</p>
<p>Additionally, a 403 Forbidden
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache/2 Server at www.spywareinfoforum.com Port 80</address>
</body></html>
The messages in /var/log/httpd/domains/spywareinfoforum.com.error.log say:
Quote:

[Sat Oct 06 14:07:32 2012] [error] [client 66.249.71.38] client denied by server configuration: /home/mike/domains/spywareinfoforum.com/public_html/index.php
What does that mean? What server configuration? Where? I believe it is IP specific since I have no problem accessing our forum as guest. I have no idea where to look.

We have CentOS 5.5

cnmoore 10-06-2012 02:56 PM

Solved
 
Solved - I feel foolish - 66.249.71.38 was denied in ~/.htaccess file on 02-21-2010 with my note Mediapartners-Google.

Apparently Googlebot didn't use that IP until October 1, 2012.

Not a total waste of time posting here as writing it somehow clarified my fuzzy thinking.
I love LinuxQuestions.


All times are GMT -5. The time now is 02:59 AM.