LXer: Robots.txt Tips For Deailing With Bots

		LinuxQuestions.org > Forums > Linux Forums > Linux - News > Syndicated Linux News
LXer: Robots.txt Tips For Deailing With Bots

Syndicated Linux News This forum is for the discussion of Syndicated Linux News stories.

Notices

Welcome to LinuxQuestions.org, a friendly and active Linux Community.

You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!

Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.

Are you new to LinuxQuestions.org? Visit the following links:
Site Howto | Site FAQ | Sitemap | Register Now

If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.

Having a problem logging in? Please visit this page to clear all LQ-related cookies.

Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.

Exclusive for LQ members, get up to 45% off per month. Click here for more info.

Search this Thread

01-13-2010, 06:40 AM	#1
LXer LXer NewsBot Registered: Dec 2005 Posts: 128,363 Rep:	LXer: Robots.txt Tips For Deailing With Bots Published at LXer: A few tips I put together while re-creating the robots.txt file on my Linux web server. The robots.txt is used to provide crawling instructions to web robots using the Robots Exclusion Protocol. When a web robots visits your site it will check this file, robots.txt, to discover any directories or pages you want to exclude from the web robot listing on the search engine. This is an important file which determines SEO for search engines and can help rankings. Read More...

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
robots.txt file ooops!...or bot ooops?!?!	vous	Linux - Software	1	05-15-2007 10:39 AM
LXer: Robots for the rest of us	LXer	Syndicated Linux News	0	04-27-2007 04:31 AM
LXer: Do-It-Yourself Robots with Linux	LXer	Syndicated Linux News	0	08-25-2006 08:03 AM
robots.txt	paleogryph	Linux - Software	1	11-11-2005 02:32 PM
configuring robots.txt	jc materi	Linux - Security	1	04-09-2005 10:37 AM

All times are GMT -5. The time now is 11:56 AM.

Main Menu

(Con't)

My LQ

Write for LQ

LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.

Main Menu

Syndicate

Latest Threads

LQ News

Twitter: @linuxquestions