__/ [ Borek ] on Friday 24 March 2006 13:19 \__
> On Fri, 24 Mar 2006 12:12:16 +0100, Roy Schestowitz
> <newsgroups@xxxxxxxxxxxxxxx> wrote:
>
>>> Since we're on the topic of robots.txt: I looked at my website stats as
>>> provided by my host, and in the list of pages not found, 404, etc, it
>>> shows
>>> the robots.txt as not found VERY often! I haven't even tried to find
>>> out
>>> any more about it, but maybe this could be causing a problem?
>>
>>
>> No. It shouldn't be a cause for concern unless there are pages you want
>> to
>> exclude from search engine endices. To suppress these 404 errors, I
>> strongly
>> advise you to create a file called robots.txt, ensure it _is empty_ and
>> leave it in your main directory.
>
> Roy, not questioning your idea about empty robots.txt - why do you think
> empty == better, instead of
>
> User-agent: *
> Disallow:
You are right. Yours is a better suggestion.
Best wishes,
Roy
--
Roy S. Schestowitz | "Software sucks. Open Source sucks less."
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
1:55pm up 16 days 6:32, 9 users, load average: 0.15, 0.50, 0.42
http://iuron.com - next generation of search paradigms
|
|