It may sound counter-intuitive, but yeah, there are solid arguments someone may have to discourage search engines indexing a WordPress website.
Search engines crawl and index websites making them easily discoverable to their audience. If you, therefore, do not want your content to be available to the public, then logically, you want to block the search engines from listing your content in their databases in the first place.
Typically you want to discourage search engines from indexing your website in some of the following cases;
Why Discourage Search Engines from Indexing Your Site
On Development Websites
When you are busy putting together your website, you wouldn’t want people to see the ugly behind the scenes work.
A staging site also falls in this category, not just because it is used for testing and development purposes, but in it is a duplicate of the live site in its entirety. And we know how search engines detest duplicate content. So instead of risking your production site, it is prudent to discourage search engines from indexing the development or staging site.
Restrict Access to Premium Content
You devote time and money to create content for your website, and sometimes you want to monetize this content. However, if people can find this content from search engines they will take it without paying a dime.
To prevent that, prevent search engines from listing your content in the first place.
Restrict access to Personal Content
Before the blown-out commercialization of blogging, most people used their blogs for personal expression away from prying eyes. Search engines changed that, they snoop and divulge your personal secrets.
If you post deep heart stuff, that you prefer to keep it to yourself or a select few with access, then the best thing to do is make your content unsearchable.
With that in mind, let’s look at –
How to Discourage Search Engines from Indexing Your Site
I will show you three easy ways you can employ to stop search engines from indexing your WordPress website.
- Enabling WordPress Search Engine Visibility feature
- Password protecting a directory or the whole using cPanel
- Using a plugin to password discourage search engines.
1. Enabling WordPress Search Engine Visibility Feature
The fastest and easiest way to discourage search engines from indexing your WordPress website is by using the built-in function for that purpose.
Login to the admin section of your WordPress site.
In the left menu, look for Settings and then click the Reading sub-menu item.
On the page you are taken to, click the checkbox beside “Discourage search engines from indexing this site”
Behind the scenes, enabling this feature, WordPress modifies the robots.txt of your website and also adds metadata to your website’s header, discouraging search engines from crawling it.
It adds this piece of code to your robots.txt file.
User-agent: *
Disallow: /
And this, to your website header:
<meta name=’robots’ content=’noindex,follow’/>
Note that search engines may altogether ignore this option. So if the content you want to hide from search engines is very sensitive, try the other two options or use them alongside this one.
2. Password Protecting A Directory or The Whole Using cPanel
Our demo WordPress website runs on cPanel, but, all hosting platforms can lock your content with passwords.
By doing that, you would have locked out and discouraged search engines from crawling and indexing your website.
This is what you will need to do with a cPanel powered host.
- Log into your cPanel dashboard and go to Directory Privacy
- Select the directory where your WordPress site is hosted on
- Create a user account with a username and a password
- Save your changes
The caveat with this method is that every time someone goes to your website, they will have to enter a user name and password.
That’s may work for an in-house project or a personal site, but it may be an inconvenience for a restricted community website.
In that case, you can consider using a WordPress plugin. The website is password protected through cPanel; users have to enter both username and password when accessing the site. That’s why you may consider using a password protection plugin.
3. Using A Plugin to Password Discourage Search Engines.
If you need a better and more granular option over the cPanel option, then this is the better option.
There are a couple of options you can consider, one I recommend is WordPress Password Protect Page Plugin
The free version of Password Protect WordPress plugin offers the following features:
- Protect WordPress Pages & Posts with Multiple Passwords
- Password Protect WordPress Pages & Posts by User Roles
- Password’s Cookies Expiration
- Password Protect The Entire WordPress Website
- Partial Content Protection
Go ahead and install and activate the plugin.
Once logged in on the left menu, go to Password Protect WordPress.
Under entering sight, you can protect the entire website.
Click the toggle button to set a password.
Once password protected, your website becomes invisible to web crawlers and won’t be shown anywhere in search results. Only those with the correct passwords can unlock the private site.
Conclusion
Before you restrict or try to discourage search engines from indexing your website, you should know that this will affect your page rankings. That also implies that you will not get free organic traffic from search engines.
That’s it. Hopefully you have found this read helpful! You may also want to learn how to stop WordPress auto updates.