> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://help.ex2.com/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# What are the names of crawlers used by various search engines?

Which [crawlers](https://help.ex2.com/en/article/what-is-a-web-crawler-116e25f/?bust=1784824020793) are used by different search engines?

If you want to create a [robots.txt file](https://help.ex2.com/en/article/what-is-a-robotstxt-file-1gk3nk7/?bust=1784824096579) for your website, you need to know the names of the different crawlers, so you can give them instructions.

Here is a list of crawlers (exploration robots) used by the main search engines. For each, we will offer you the **User-Agent** code to use in your file to send them instructions. 

### 1. Google's Googlebot

Google’s main crawler is simply named [Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot?hl=en). This is a generic name given to two different types of exploration robots. The first type simulates navigation on a mobile device, the second simulates a pc.

| Crawler name | Full User-Agent string for robots.txt file |  |
| ---- |
| Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |

In addition to its search crawler, Google uses 9 other web crawlers with distinct functions:

| Crawler name | Full User-Agent string for robots.txt file |
| ---- |
| Googlebot News | Googlebot-News |
| Googlebot Images | Googlebot-Image/1.0 |
| Googlebot Video | Googlebot-Video/1.0 |
| Google Mobile | SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
| Google Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Google Adsense | Mediapartners-Google |
| Google Mobile Adsense | (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
| Google AdsBot | AdsBot-Google (+http://www.google.com/adsbot.html) |
| Google app crawler | AdsBot-Google-Mobile-Apps |

### 2. Bing’s Bingbot 

Bingbot is a web crawler deployed by Microsoft in 2010 to provide information to their Bing search engine. It then replaced the old MSN robot. Unlike Google, Bing uses a single crawler.

| Crawler name | Full User-Agent string for robots.txt file|  |  

| Bingbot | Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm) |

### 3. Yahoo’s Slurpbot

Yahoo’s search results come from Yahoo’s Slurpbot crawler and Bingbot, as much of Yahoo is powered by Bing. Sites must necessarily allow access to the Yahoo Slurpbot in order to appear in Yahoo Mobile search results.

In addition, the Yahoo! crawler offers some additional features:

* It collects content from partner sites to include in sites such as **Yahoo News**, **Yahoo Sports** and **Yahoo Finance**.
* It accesses website pages across the web to confirm accuracy and improve Yahoo’s custom content for his users.

| Crawler name | Full User-Agent string for robots.txt file|  

| Slurpbot | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |