网站蜘蛛访问统计

最简单的办法，就是User-Agent

百度UA举例：

如果UA信息不对，可以直接判断为非百度搜索的蜘蛛。目前UA分为移动、PC、和小程序三个应用场景，这三个渠道UA分别如下：

移动UA： 

Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko)Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html)

或

Mozilla/5.0 (iPhone;CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0;+http://www.baidu.com/search/spider.html) 

PC UA：

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

或

Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

小程序UA：

Mozilla/5.0 (iPhone;CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0;Smartapp; +http://www.baidu.com/search/spider.html)

来源：【官方说法】只需两步，正确识别百度蜘蛛（User-Agent）_搜索学堂_百度搜索资源平台 (baidu.com)

统计代码示例：

<?php

//蜘蛛统计和拦截
if(isset($_SERVER['HTTP_USER_AGENT'])){
    $ua = $_SERVER['HTTP_USER_AGENT'];
    $spider = "";  //蜘蛛识别
    //百度移动蜘蛛
    if($ua=="Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko)Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html)" || $ua=="Mozilla/5.0 (iPhone;CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0;+http://www.baidu.com/search/spider.html)"){
        $spider = "Baiduspider";
        $spiderType = "pc";
        $spiderName = "百度小pc端";
    }
    //百度pc
    elseif($ua=="Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" || $ua=="Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)"){
        $spider = "Baiduspider";
        $spiderType = "app";
        $spiderName = "百度app端";
    }
    //百度小程序ua
    elseif($ua=="Mozilla/5.0 (iPhone;CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0;Smartapp; +http://www.baidu.com/search/spider.html)"){
        $spider = "Baiduspider";
        $spiderType = "mini";
        $spiderName = "百度小程序端";
    }
    //记录蜘蛛活动
    if($spider){
        DB::sql("insert into `#system_spider`(`spider`,`spider_type`,`time`,`url`,`name`) values('$spider','$spiderType',unix_timestamp(current_timestamp),:url,'$spiderName')",array("url"=>PAGE_URL))->update();
    }
}

显示今日蜘蛛统计信息

<?php

//检测今日蜘蛛抓取信息
        $day_time = strtotime(date("Y-m-d"));
        $spider = DB::sql("select `spider`,`spider_type`,`name`,count(*)'count' from `#system_spider` where `time`>$day_time group by `spider`,`spider_type`")->selectArrList();
        assign("spider",json_encode($spider,256));

其他蜘蛛：

bing蜘蛛ua文档

google蜘蛛ua文档

手动模拟，如curl：

curl http://huaz.52dixiaowo.com -H 'User-Agent:Mozilla/5.0 (iPhone;CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko)Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0;+http://www.baidu.com/search/spider.html)'