+ Reply to Thread
Results 1 to 5 of 5

Thread: Detecting Crawlers?

  1. #1
    apoorav's Avatar
    apoorav is offline x10 Sophmore apoorav is an unknown quantity at this point
    Join Date
    Oct 2008
    Location
    INDIA
    Posts
    110

    Exclamation Detecting Crawlers?

    Can any body give me PHP code to detect web crawlers(BOTS) crawling the webpage

    For Example, it should show that GOOGLE bot was last visitor and similarly should show total list.
    If you feel my post is usefull then click on to give me Reputation

  2. #2
    Mr. DOS is offline x10 Sophmore Mr. DOS is an unknown quantity at this point
    Join Date
    Oct 2009
    Location
    Nova Scotia, Canada
    Posts
    228

    Re: Detecting Crawlers?

    I don't have code to give you, but you'll want to check the PHP internal variable $_SERVER['HTTP_USER_AGENT'] against a list of user agents used by bots (Google for them). If a match is found, you can then log it.

    --- Mr. DOS
    I've written a couple articles on automatic application of AlphaImageLoader in IE6 using nothing but IE6-specific CSS rules.

  3. #3
    descalzo's Avatar
    descalzo is offline Grim Squeaker descalzo has a brilliant futuredescalzo has a brilliant futuredescalzo has a brilliant future
    Join Date
    Jul 2009
    Location
    Ankh-Morpork
    Posts
    7,636

    Re: Detecting Crawlers?

    Quote Originally Posted by Mr. DOS View Post
    I don't have code to give you, but you'll want to check the PHP internal variable $_SERVER['HTTP_USER_AGENT'] against a list of user agents used by bots (Google for them). If a match is found, you can then log it.
    This will get you a list of the "legit" bots. The "not-so-legit" bots spoof IE or Firefox. They also ignore robots.txt.

  4. #4
    apoorav's Avatar
    apoorav is offline x10 Sophmore apoorav is an unknown quantity at this point
    Join Date
    Oct 2008
    Location
    INDIA
    Posts
    110

    Re: Detecting Crawlers?

    Ok, but i want code for it , if anybody can give??
    If you feel my post is usefull then click on to give me Reputation

  5. #5
    misson is offline x10 Spammer misson is a jewel in the rough
    Join Date
    Mar 2008
    Location
    Libertatia
    Posts
    2,506

    Re: Detecting Crawlers?

    Have you tried Google? Always try Google.
    Last edited by misson; 12-22-2009 at 04:02 PM.
    Be sure to read all pages linked in this post; they have further information that should prove useful. When asking for help, make sure you follow Eric Raymond's and Jon Skeet's guidelines for prompt, accurate responses. Please answer any questions I ask; they're not rhetorical (probably). Any posted code is intended as illustrative example, rather than a solution to your problem to be copied without alteration. Study it to learn how to write your own solution.
    Misson, not Mission.

+ Reply to Thread

Similar Threads

  1. detecting broswer
    By lordsofvine in forum Programming Help
    Replies: 2
    Last Post: 09-01-2008, 06:35 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
x10hosting free hosting for the masses
dedicated servers