LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 10-06-2012, 02:40 PM   #1
cnmoore
Member
 
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89

Rep: Reputation: 0
Googlebot blocked - can't figure out why


Please help us, our forum very much depends on being indexed by Google.

As of October 1, Googlebot suddenly cannot access our forum at http://www.spywareinfoforum.com.
I know of nothing that changed on the server that day.

We do not seem to have ModSecurity - locate doesn't report any.

I don't find any likely address banned in iptables. Adding this line at the top of INPUT hasn't helped:
ACCEPT all -- 66.249.64.0/19 anywhere

The error that Googlebot gets (using Fetch as Googlebot in Webmaster Tools):
Quote:
HTTP/1.1 403 Forbidden
Date: Fri, 05 Oct 2012 22:43:32 GMT
Server: Apache/2
Content-Length: 406
Keep-Alive: timeout=1, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /index.php
on this server.</p>
<p>Additionally, a 403 Forbidden
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache/2 Server at www.spywareinfoforum.com Port 80</address>
</body></html>
The messages in /var/log/httpd/domains/spywareinfoforum.com.error.log say:
Quote:
[Sat Oct 06 14:07:32 2012] [error] [client 66.249.71.38] client denied by server configuration: /home/mike/domains/spywareinfoforum.com/public_html/index.php
What does that mean? What server configuration? Where? I believe it is IP specific since I have no problem accessing our forum as guest. I have no idea where to look.

We have CentOS 5.5
 
Old 10-06-2012, 02:56 PM   #2
cnmoore
Member
 
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89

Original Poster
Rep: Reputation: 0
Solved

Solved - I feel foolish - 66.249.71.38 was denied in ~/.htaccess file on 02-21-2010 with my note Mediapartners-Google.

Apparently Googlebot didn't use that IP until October 1, 2012.

Not a total waste of time posting here as writing it somehow clarified my fuzzy thinking.
I love LinuxQuestions.

Last edited by cnmoore; 10-06-2012 at 02:58 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: ModSecurity: Allowing GoogleBot LXer Syndicated Linux News 0 04-19-2010 04:11 PM
Mod recent blocked related question (netfilter). WHO IS BLOCKED CarLost Linux - Security 6 07-29-2008 03:53 PM
GoogleBot and .htaccess bzzz Programming 3 06-30-2008 11:05 AM
googlebot unstoppable? randomx General 7 11-16-2005 06:22 AM
sockets blocked on local network, can't figure out why Dooberdoink Linux - Networking 5 07-07-2005 01:16 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 10:50 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration