← Glossary Term
robots.txt
A small text file on a website that tells automatic crawlers which pages they may visit.
Every website can place a file called robots.txt at its root. It lists, in simple lines, which crawlers (programs that visit pages automatically) are welcome and which parts of the site they should leave alone.
It works like a polite sign on a door, not a lock. Honest crawlers follow it, but it cannot stop someone who ignores it, and it cannot take back what was already collected.
Mentioned in