By now you have learned that getting on the first page of a search engine results is not easy, you are told – you do this and that, and then the other and today we are saying create a robots.txt file in WordPress.
It gets overwhelming, right?
It is honestly not an easy feat, but, if you want that coveted traffic from search engines, you have to play along.
The fortunate part of it all is that it is always worth the effort.
In this article, I will try to help you understand what the robots.txt file is, why it is advisable to have one, and how to create one for your WordPress website.
What Is A Robots.txt File?
A robots.txt file is a file that resides in the root of your website’s public directory with instructions for search engine spiders/bots/crawlers – they all mean and do the same thing – on what part of your website to or not to index in their database.
Why Is This Important?
It is you who knows your website, you understand its workings and what areas you deem private and sensitive, you, therefore, are the one who can guide bots on what areas of your website to avoid or not.
You surely don’t want to have your admin section available in a Google search result, do you?
Search engines do love when they find that you make an effort to steer them to the right places of your website.
Though not having a robots.txt may not have any negative impact on your website, it is always recommended to have one just to keep the prying bots out of the sensitive and private areas of your website.
Now that we know what it is and why it is recommended to have it. Let us add one.
How to Create A Robots.txt
Before we create a robots.txt, did you know that WordPress actually creates one for during the installation process?
Yes it does, you can check it out at https://yourdomain/robots.txt
You notice though that this does not provide any specific instructions besides allowing all bots (user-agents) from crawling the whole WordPress site.
To manually create one, you will need to create a robots.txt file and then upload it to your host, to the same directory that your core WordPress files reside.
If you are using a host using cPanel, then that folder is public_html
Once you are logged in and can access your data, under public_html, click +File.
In the New File Name field enter robot.txt
Confirm that the new file will be created in the public_html folder.
In the public_html folder, you will see the file you just created.
Right click it and then click Edit.
What to Add in A Robots.txt File for A WordPress Website.
Generally, there are two significant instructions you add to a robots.txt file;
- Which bots do are allowed or not to crawl your website – Here is a list of search engine bots.
You specify this under User-agent: *
The asterisk * – implies that all bots have permission. You can replace the * with any bot name.
- Directories and or files bots have or do not have access to.
Block access to folders.
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Block access to single files.
Disallow: wp-login.php
You can also use the $ sign after say a folder – in the case of
Disallow: /wp-content/$
Adding a dollar sign after wp.content/$ will instruct the bots to only skip wp-content, but they can index any file or folder inside it.
Adding a $ sign in front of a single file – like in the case of folders, instructs bots to skip only that file, for instance, you can block access to the index.php, adding the dollar sign after that to form index.php/$ means that bots can only block index.php, but will index any other content with index.php ?p=123
Here is how my robots.txt file finally looks like this.
All right, that’s how you manually create a robots.txt file for a WordPress website.
Conclusion
A robot.txt file houses the instructions on how search engine bots can access and then index the content on your website.
This is good and recommended for your SEO efforts besides also protecting your private content from bots.
I would, however, want to point out, that though this file is meant to direct bots, it does not mean that bots will follow your instructions.
We have shown you the manual way. You can also explore using a plugin to create the robots.txt file. Those that come to mind are Yoast SEO, All in One SEO Pack or other WordPress SEO plugins.
At the time of writing this article, the latest version 11.4 of Yoast SEO does not have the feature, but, if you are using the older versions, you should be able to create a robots.txt file with it.
Have you tried any plugin with success? Let us know in the comments section below.
You may also want to check out our list of the best XML sitemap plugins for WordPress to submit an XML sitemap and request search engines to index the pages on your website.