To avoid undesirable content within the lookup indexes, site owners can instruct spiders not to crawl selected files or directories from the normal robots.txt file in the foundation Listing on the area. On top of that, a web page may be explicitly excluded from a internet search engine's database through the use of a meta tag precise to robots (typ