robots.txt

robots.txt definition - computer

A text file placed in the root directory of a Web site that prohibits search engine spiders from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See spider.



Computer Desktop Encyclopedia THIS DEFINITION IS FOR PERSONAL USE ONLY
All other reproduction is strictly prohibited without permission from the publisher.
Copyright © 1981-2009 by Computer Language Company Inc. All rights reserved.

Comments
Improve this definition.
Do you have more to add? Share your linguistic knowledge or observation.
/Register to save your comments.