WordPress Robots.txt Guide – What It Is and How to Use It

Ever heard the term robots.txt and wondered how it applies to your website? Most websites have a robots.txt file, but that doesn’t mean most webmasters understand it. In this post, we hope to change that by offering a deep dive into the WordPress robots.txt file, as well as how it can control and limit access to your site. By the end, you’ll be able to answer questions like: There’s a lot to cover so let’s get started!
What Is a WordPress Robots.txt?
Before we can talk about the WordPress robots.txt, it’s important to define what a “robot” is in this case. Robots are any type of “bot” that visits websites on the Internet. The most common example is search engine crawlers. These bots “crawl” around the web to help search engines like Google index and rank the billions of pages on the Internet.
So, bots are, in general, a good thing for the Internet…or at least a necessary thing. But that doesn’t necessarily mean that you, or other webmasters, want bots running around unfettered. The desire to control how web robots interact with websites led to the creation of the robots exclusion standard in the mid-1990s.
Source: https://managewp.org/articles/16363/wordpress-robots-txt-guide-what-it-is-and-how-to-use-it

source https://williechiu40.wordpress.com/2017/10/23/wordpress-robots-txt-guide-what-it-is-and-how-to-use-it/