Paul Burke wrote:
> On Fri, 15 Jul 2005 20:41:44 +0100, mogga <di@NOSPAMPLEASEmogga.com>
>>Robots/Spiders visitors (Top 25)
>>In my stats (awstats)
>>I have a section about robots and spiders.
>>* Robots shown here gave hits or traffic "not viewed" by visitors, so
>>they are not included in other charts. Numbers after + are successful
>>hits on "robots.txt" files
>>Why does it count the successful hits on robots.txt and what purpose
>>does it have?
> It could be human. I always look for a robots.txt when someone asks
> for a link exchange, or me checking a site out to exchange links with.
I have had various user agents access non-existent URL's like robot.txt or
robots1.txt. It usually raises questions and leads to a short investigation
or research on the Web, which is a total waste of time. There are also some
experimental bots that make nothing but a mess in the error logs. Always
bear that in mind if you see suspicious stuff going on. It is nothing to
Roy S. Schestowitz