__/ [John Bokma] on Sunday 27 November 2005 01:27 \__
> "John" <nospam@xxxxxxxxx> wrote:
>
>>
>> "John Bokma" <john@xxxxxxxxxxxxxxx> wrote in message
>> news:Xns971A6E1C2536Dcastleamber@xxxxxxxxxxxxxx
>>> "Clint" <pepmax@xxxxxxxxxxxx> wrote:
>>>
>>>> After a year, my website www.FreeSpiritGallery.ca is finally
>>>> tracking decent on the search engines. I have never used a robot
>>>> file though since I don't mind the search engines spidering my
>>>> entire site. My question - is there any value for me to add in a
>>>> robot file even if I don't mind my entire site being indexed?
>>>
>>> Yes, it keeps the error_log clean :-D
>>>
>>> For a nice default version, see
>>>
>>> http://johnbokma.com/robots.txt
>>
>> Would an empty robots.txt file do the same thing? I think MSN
>> suggests it and that's what I use but is it ok with other SE's?
>
> The one suggested above means: everything is allowed, so is identical to
> none. No idea if an empty file works similar.
An empty file should be fine. Keeps the 404's away and implies no exclusions.
I must have read this somewhere because I had an empty file before I began
to list particular pages.
Hope it helps,
Roy
--
Roy S. Schestowitz | Data lacking semantics is currency in an island
http://Schestowitz.com | SuSE Linux | PGP-Key: 0x74572E8E
6:05am up 24 days 1:59, 7 users, load average: 2.23, 1.73, 1.79
http://iuron.com - next generation of search paradigms
|
|