What is robots.txt?

robots.txt is a text file that webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. It specifies which parts of a website should not be processed or scanned. It's commonly used to prevent search engines from indexing certain pages, like login pages, admin areas, or duplicate content, thereby improving crawl efficiency and protecting sensitive information. Although commonly followed, it's important to note that robots.txt is advisory, not mandatory, and malicious bots may choose to ignore it.