Robots exclusion standard facts for kids

The robots exclusion standard (also called the robots exclusion protocol or robots.txt protocol) is a way of telling Web crawlers and other Web robots which parts of a Web site they can see.

To give robots instructions about which pages of a Web site they can access, site owners put a text file called robots.txt in the main directory of their Web site, e.g. http://www.example.com/robots.txt. This text file tells robots which parts of the site they can and cannot access. However, robots can ignore robots.txt files, especially malicious (bad) robots. If the robots.txt file does not exist, Web robots assume that they can see all parts of the site.

Examples of robots.txt files

See also

In Spanish: Estándar de exclusión de robots para niños