Home Messages Index
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index

Re: Alexa Internet archive crawler gone wild?

__/ [Luigi Donatello Asero] on Tuesday 03 January 2006 02:49 \__

> 
> "Marco Schmidt" <marcoschmidt@xxxxxxxxxxxxx> skrev i meddelandet
> news:41u6opF1g7kd9U1@xxxxxxxxxxxxxxxxx
>> The Alexa Internet Archive crawler has visited each of my pages listed
>> at DMOZ 50 times each day for at least the last couple of days.
>> They're all static content and change s rarely. Is anyone else
>> experiencing this? I don't want to exclude their crawler right away,
>> but why on earth would they want to have 50 copies per page per day
>> for archiving purposes? Requests are all coming from one host
>> (crawl23-public.alexa.com).
>>
>> Regards,
>> Marco
> 
> I am not experiencing this with Alexa.
> But I have  experienced long visits from Google (several robots)
>  for a long time although these visits do not seem to  show significant
> changes in the ranking of my pages. So I am wondering what is Google
> actually doing?

On  my main site, Alexa snatched 250 MB of pages in the past 2 days.  They
never  consumed  that much traffic, not even in a whole month.  I  haven't
looked for any patterns yet as the server can cope, so it's not worrisome.

What's  the  benefit of permitting Alexa to crawl though? Having the  site
archived  for someone to look back at deleted content in the future?  Have
we not learned the lesson yet?

http://web.archive.org/web/20021030152640/www-db.stanford.edu/~sergey/photos/drag96.jpg

Roys

[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index