Basic terms of web robot.txt – Blogs PalaceBlogs Palace

What is a Web robot:

In basic, web robot is simply a program. Robot is used to automatically and recursively traverses a Web site to retrieve document content and information. Search engine spiders is the most common types of Web robots. These robots visit Web sites and follow the links to add more information to the search engine database.

Web robots often go by different names. You may hear them called:


Who Upvoted this Story


Comments