| Browser | |
|---|---|
| User agent |
| Crawler | User Agent String | |
|---|---|---|
| • Googlebot (Desktop) |
Googlebot |
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
or (rarely used):
Googlebot/2.1 (+http://www.google.com/bot.html)
|
| • Googlebot (Smartphone) |
Googlebot | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| • Googlebot-Image • Googlebot |
Googlebot Images | Googlebot-Image/1.0 |
| • Googlebot-Video • Googlebot |
Googlebot Video | Googlebot-Video/1.0 |
| • Googlebot-News • Googlebot |
Googlebot News | Googlebot-News |
| • APIs-Google | APIs-Google | APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) |
| • Mediapartners-Google | AdSense | Mediapartners-Google |
| • Mediapartners-Google | Mobile AdSense | (Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
| • AdsBot-Google-Mobile | AdsBot Mobile Web Android (Checks Android web page ad quality) |
Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
| • AdsBot-Google-Mobile | AdsBot Mobile Web (Checks iPhone web page ad quality) |
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
| • AdsBot-Google | AdsBot (Checks desktop web page ad quality) |
AdsBot-Google (+http://www.google.com/adsbot.html) |
| • AdsBot-Google-Mobile-Apps | Mobile Apps Android | AdsBot-Google-Mobile-Apps |
| Crawler | Description | User Agent String |
|---|---|---|
| Bingbot | Bingbot is standard crawler and handles most of crawling needs each day. Bingbot uses a couple of different user agent strings which include several mobile variants with which crawl the mobile web |
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
|
| MSNBot | MSNBot used to be standard crawler before the advent of Bingbot and still handles some of crawling duties. | msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
| MSNBot-Media | MSNBot-Media is crawler for images and video. | msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
| AdIdxBot | AdIdxBot is the crawler used by Bing Ads. AdIdxBot is responsible for crawling ads and following through to websites from those ads for quality control purposes. Same as Bingbot, AdIdxBot has both desktop and mobile variants. |
Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
|
| BingPreview | BingPreview is used to generate page snapshots. Has "desktop" and "mobile" variants. |
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b
|
| Crawler | Description | User Agent String |
|---|---|---|
| Slurp | Slurp is the Yahoo Search robot for crawling and indexing web page information. |
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
|
| Crawler | User Agent Name | User Agent String |
|---|---|---|
| Baidu Web Search | Baiduspider |
One of the following
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
|
| Baidu Image Search | Baiduspider-image | |
| Baidu Mobile Search | Baiduspider-mobile | |
| Baidu Video Search | Baiduspider-video | |
| Baidu News Search | Baiduspider-news | |
| Baidu Bookmark Search | Baiduspider-favo | |
| Baidu Union Search | Baiduspider-cpro | |
| Baidu Business Search | Baiduspider-ads |
| Description | User Agent String |
|---|---|
| The main indexing robot | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) |
| Indexing robot | Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots) |
| Downloads pages to check user accessibility | Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots) |
| Determines if the page layout is suitable for mobile devices | Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots) |
| Generates dynamic banners | Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots |
| Makes a snapshot of a page | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots) |
| The indexing robot | Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) |
| The Y indexing robot | Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots) |
| The indexing robot | Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots) |
| Multimedia data indexer | Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots) |
| The robot. Indexes comments to posts | Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots) |
| The favicons indexer | Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots) |
| The indexing robot | Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots) |
| The robot that validates markup submitted through the Structured data validator form | Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots) |
| The mobile services robot | Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots) |
| Downloads the ads' landing pages to check their availability and topic | Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots) |
| The robot used for syncing with other calendars | Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots) |
| The sitelinks "fetcher" used for checking the availability of the pages detected as sitelinks | Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots) |
| The Yandex Metrika robot | Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots) |
| The Yandex.News robot | Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots) |
| Vertical search robot | Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots) |
| The robot detecting site mirrors | Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots) |
| The robot that regularly downloads product catalogues in YML files by users's requests | Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots) |
| The robot of Yandex.Verticals: Auto.ru, Yandex.Realty, Yandex.Job, Yandex.Reviews | Mozilla/5.0 (compatible; YandexVerticals/1.0; +http://yandex.com/bots) |