Robots.txt Is a file that exists on the basis directory of each Web site and can be used to instruct search engines like google and yahoo on which directories/documents of the website they might crawl and include things like inside their index. A significant rationale that Search engine optimisation matters https://news.rafeeg.ae