windows live - search problem

**mikel2k3** · 06-04-2007 08:16 AM

Wasnt sure what to put as the title, but anyay.

When viewing my website statistics with Awstats, i seem to be getting 5000+ unique hits from http://www.live.com . I dont know why this is happening, and will it make any difference if i create a robots.txt file that will dissallow everything?

Something like this:
User-agent: *
Disallow: /
Crawl-delay: 10

and has anybody else had this problem?

**Chris Z** · 06-04-2007 11:08 AM

I believe that would just suggest windows live not to crawl your website, I don't think that it would actually force it to back off.

**Zenax** · 06-04-2007 11:16 AM

The 5000+ hits from live.com are from bots that crawl websites, and obviously yours seems to be quite a popular one.

I believe the robots.txt would tell it to back off and not scan your site but I have no idea if that will stop it all together.

**Cubeform** · 06-04-2007 11:04 PM

A similar thing has occurred in the past with Yahoo's very aggressive Slurp crawler. I've heard the crawler would often create miniature denial of service attacks on some small websites with shared hosting plans.

Windows Live should be robots.txt compliant. Try this:

Code:

User-agent: msnbot
Disallow: /
Crawl-delay: 10

This should stop Windows Live from indexing your page.

Edit: Here is a useful FAQ made by Microsoft:http://search.msn.com/docs/siteowner...Indexing.htm#D Try the robots.txt examples they give you there.

**mikel2k3** · 06-05-2007 08:54 AM

The part that really confuses me is:

Robots/Spiders visitors:
Yahoo Slurp - 4+3 hits
Googlebot - 5+1 hits
MSNbot - 4+1 hits

Coonect to site from - Links from internet search engine:
Windows Live - 5618
Google - 2

Ill presume that they are not related in anyway, but ill also put the robots.txt file blocking MSNbot anyway to see if it helps.