Home Messages Index
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index

Re: Robots txt

__/ [ Borek ] on Friday 24 March 2006 13:19 \__

> On Fri, 24 Mar 2006 12:12:16 +0100, Roy Schestowitz
> <newsgroups@xxxxxxxxxxxxxxx> wrote:
> 
>>> Since we're on the topic of robots.txt: I looked at my website stats as
>>> provided by my host, and in the list of pages not found, 404, etc, it
>>> shows
>>> the robots.txt as not found VERY often!  I haven't even tried to find
>>> out
>>> any more about it, but maybe this could be causing a problem?
>>
>>
>> No. It shouldn't be a cause for concern unless there are pages you want
>> to
>> exclude from search engine endices. To suppress these 404 errors, I
>> strongly
>> advise you to create a file called robots.txt, ensure it _is empty_ and
>> leave it in your main directory.
> 
> Roy, not questioning your idea about empty robots.txt - why do you think
> empty == better, instead of
> 
> User-agent: *
> Disallow:

You are right. Yours is a better suggestion.

Best wishes,

Roy

-- 
Roy S. Schestowitz      |    "Software sucks. Open Source sucks less."
http://Schestowitz.com  |    SuSE Linux    ¦     PGP-Key: 0x74572E8E
  1:55pm  up 16 days  6:32,  9 users,  load average: 0.15, 0.50, 0.42
      http://iuron.com - next generation of search paradigms

[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index