Home Messages Index
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index

Re: Do I need robot file?

  • Subject: Re: Do I need robot file?
  • From: Roy Schestowitz <newsgroups@xxxxxxxxxxxxxxx>
  • Date: Sun, 27 Nov 2005 06:09:24 +0000
  • Newsgroups: alt.internet.search-engines
  • Organization: schestowitz.com / MCC / Manchester University
  • References: <ea0if.39408$5o6.712665@wagner.videotron.net> <Xns971A6E1C2536Dcastleamber@> <n%4if.11601$mM2.4298@fe09.lga> <Xns971AC5E6092ABcastleamber@>
  • Reply-to: newsgroups@xxxxxxxxxxxxxxx
  • User-agent: KNode/0.7.2
__/ [John Bokma] on Sunday 27 November 2005 01:27 \__

> "John" <nospam@xxxxxxxxx> wrote:
>> "John Bokma" <john@xxxxxxxxxxxxxxx> wrote in message
>> news:Xns971A6E1C2536Dcastleamber@xxxxxxxxxxxxxx
>>> "Clint" <pepmax@xxxxxxxxxxxx> wrote:
>>>> After a year, my website www.FreeSpiritGallery.ca is finally
>>>> tracking decent on the search engines.  I have never used a robot
>>>> file though since I don't mind the search engines spidering my
>>>> entire site.  My question - is there any value for me to add in a
>>>> robot file even if I don't mind my entire site being indexed?
>>> Yes, it keeps the error_log clean :-D
>>> For a nice default version, see
>>> http://johnbokma.com/robots.txt
>> Would an empty robots.txt file do the same thing?  I think MSN
>> suggests it and that's what I use but is it ok with other SE's?
> The one suggested above means: everything is allowed, so is identical to
> none. No idea if an empty file works similar.

An empty file should be fine. Keeps the 404's away and implies no exclusions.
I must have read this somewhere because I had an empty file before I began
to list particular pages.

Hope it helps,


Roy S. Schestowitz      | Data lacking semantics is currency in an island
http://Schestowitz.com  |    SuSE Linux     |     PGP-Key: 0x74572E8E
  6:05am  up 24 days  1:59,  7 users,  load average: 2.23, 1.73, 1.79
      http://iuron.com - next generation of search paradigms

[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index