Home Messages Index
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index

Re: robots.txt

CarolW. wrote:

> On Wed, 12 Jan 2005 22:56:54 GMT, "star green" <spam@verizon.net>
> wrote:
> 
>>Ive been trying to figure out how to write a robots.txt file that will
>>allow the robots to access the homepage (index.html), but no other page on
>>the site.
>>
>>Short of putting in a "disallow" line for every webpage on the site,
>>except for the homepage, is there some easy way to do this?
> 
> One suggestion: Could put the other pages into a subdirectory folder
> then disallow the spiders from those folders.
> 
> Carol

This can be a big overhaul in the absence of _relative_ links.

I used the following excellent source for all my robots.txt needs:

http://www.robotstxt.org/wc/faq.html#log

-- 
Roy Schestowitz
http://schestowitz.com

[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index