Articles on: Getting Started
This article is also available in:

What are the names of crawlers used by various search engines?

Which crawlers are used by different search engines?

If you want to create a robots.txt file for your website, you need to know the names of the different crawlers, so you can give them instructions.

Here is a list of crawlers (exploration robots) used by the main search engines. For each, we will offer you the User-Agent code to use in your file to send them instructions.

1. Google's Googlebot



Google’s main crawler is simply named Googlebot. This is a generic name given to two different types of exploration robots. The first type simulates navigation on a mobile device, the second simulates a pc.

Crawler nameFull User-Agent string for robots.txt file
GooglebotMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)


In addition to its search crawler, Google uses 9 other web crawlers with distinct functions:

Crawler nameFull User-Agent string for robots.txt file
Googlebot NewsGooglebot-News
Googlebot ImagesGooglebot-Image/1.0
Googlebot VideoGooglebot-Video/1.0
Google MobileSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Google SmartphoneMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Google AdsenseMediapartners-Google
Google Mobile Adsense(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
Google AdsBotAdsBot-Google (+http://www.google.com/adsbot.html)
Google app crawlerAdsBot-Google-Mobile-Apps


2. Bing’s Bingbot



Bingbot is a web crawler deployed by Microsoft in 2010 to provide information to their Bing search engine. It then replaced the old MSN robot. Unlike Google, Bing uses a single crawler.

Crawler nameFull User-Agent string for robots.txt file
BingbotMozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)


3. Yahoo’s Slurpbot



Yahoo’s search results come from Yahoo’s Slurpbot crawler and Bingbot, as much of Yahoo is powered by Bing. Sites must necessarily allow access to the Yahoo Slurpbot in order to appear in Yahoo Mobile search results.

In addition, the Yahoo! crawler offers some additional features:

It collects content from partner sites to include in sites such as Yahoo News, Yahoo Sports and Yahoo Finance.
It accesses website pages across the web to confirm accuracy and improve Yahoo’s custom content for his users.

Crawler nameFull User-Agent string for robots.txt file
SlurpbotMozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Updated on: 27/04/2023

Was this article helpful?

Share your feedback

Cancel

Thank you!