What are the names of crawlers used by various search engines?
Which crawlers are used by different search engines?
If you want to create a robots.txt file for your website, you need to know the names of the different crawlers, so you can give them instructions.
Here is a list of crawlers (exploration robots) used by the main search engines. For each, we will offer you the User-Agent code to use in your file to send them instructions.
1. Google's Googlebot
Google’s main crawler is simply named Googlebot. This is a generic name given to two different types of exploration robots. The first type simulates navigation on a mobile device, the second simulates a pc.
Crawler name | Full User-Agent string for robots.txt file |
|
|---|
Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
In addition to its search crawler, Google uses 9 other web crawlers with distinct functions:
Crawler name | Full User-Agent string for robots.txt file |
|---|---|
Googlebot News | Googlebot-News |
Googlebot Images | Googlebot-Image/1.0 |
Googlebot Video | Googlebot-Video/1.0 |
Google Mobile | SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
Google Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Google Adsense | Mediapartners-Google |
Google Mobile Adsense | (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
Google AdsBot | AdsBot-Google (+http://www.google.com/adsbot.html) |
Google app crawler | AdsBot-Google-Mobile-Apps |
2. Bing’s Bingbot
Bingbot is a web crawler deployed by Microsoft in 2010 to provide information to their Bing search engine. It then replaced the old MSN robot. Unlike Google, Bing uses a single crawler.
Bingbot | Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm) |
3. Yahoo’s Slurpbot
Yahoo’s search results come from Yahoo’s Slurpbot crawler and Bingbot, as much of Yahoo is powered by Bing. Sites must necessarily allow access to the Yahoo Slurpbot in order to appear in Yahoo Mobile search results.
In addition, the Yahoo! crawler offers some additional features:
- It collects content from partner sites to include in sites such as Yahoo News, Yahoo Sports and Yahoo Finance.
- It accesses website pages across the web to confirm accuracy and improve Yahoo’s custom content for his users.
Slurpbot | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
Updated on: 27/04/2023
Thank you!