> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://help.ex2.com/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# What is a Robots.txt file?

## The Robots.txt file of a website

The optional robots.txt file is used to give [crawlers](https://help.ex2.com/en/article/what-is-a-web-crawler-116e25f/?bust=1784567747500) and bots instructions on how to browse a site. It can be used to prevent portions of your site from being indexed.

Robot.txt files are not mandatory or necessary. Without them, explorer robots will simply navigate a site based on their default guidance. However, most sites get better results using a robot.txt file to help guide bots.

The robot.txt file can be used to encourage or discourage browsing of certain pages. For example, if your site has a private section, you would not want its content to appear in search results. 

The Robot.txt files serve as much to ensure that certain pages are indexed and explored, as to avoid that poorly optimized pages are classified.