__/ [www.1-script.com] on Monday 05 September 2005 01:14 \__
> Roy Schestowitz wrote:
>
>> Can you sniff the agent, e.g. crawler/human, and respond appropriately?
>
> Don't want to cloak if possible to avoid. The server is pretty busy as it
> is, and I would not want to analyze every request.
Cloaking in this case is not unethical, especially if you cloak to better
serve the crawler that introduced the problem in the first place. If
something damaging was to happen, you could give Google a call. The number
was published only days ago in this NG. *smile*
>> I am fairly sure that 404 is shown in my error logs regardless of the
>> user involved.
>
> Apache would write it into your error log, right, but the custom error
> page that's shown sends 200 header just to say to your browser "OK to
> process and show on the screen". Otherwise if you return 404 header, there
> is no hope for any custom page as the browser will show the default (from
> local hard drive) 404 error page instead.
That was definitely worth learning about.
>> Why do you have 404's coming up if I may ask?
>
> That's the funniest part of the story 'cause I *DON'T* have broken links
> in that map! It seems as if they deliberately polled a bogus URL on my
> domain, got the custom error page as a response and decided they did not
> like it. All of that happened while I was waiting for the result page
> after submitting them my map. The map had not even been downloaded then
> (they got it about 8 hours later). Therefore they could not be aware of
> any broken link in the map they did not read.
Have you told them about it?
A few days ago I thought I could see a Google referrals 'rebound'... but
there has been none yet. I guess referral levels are back to the old
equilibrium and are "here to stay". Easy come, easy go, I suppose.
Roy
--
Roy S. Schestowitz | Useless fact: 21978 x 4 = 21978 backwards
http://Schestowitz.com | SuSE Linux | PGP-Key: 74572E8E
5:50am up 11 days 17:17, 3 users, load average: 0.35, 0.40, 0.51
|
|
