Select Page

The Robots Exclusion Standard was developed in 1994 so that website owners can advise search engines, how to crawl your website. Placing a WordPress robots.txt file in the root of your domain lets you stop search engines indexing sensitive files like wp-admin, plugin, password etc. 

In simple word, Let us suppose our website is like a House(home). A house consists of a large number of Room. Some Room are important for us and some are not like the storage room, private room. When someone come in our home, we focus on that thing they never go toward private area because this area is important for us. Similarly, we have to stop search engine to crawl sensitive area of our website like admin and tell, what should have to crawl and what should not. You can also use Meta Robots tag to stop  search engine to indexing particular page or post which is very important for every blogger.

wordpress robots.txt file bots

WordPress Robots

WordPress robots.txt file plays a major role in search engine ranking. It helps to block search engine bots to index and crawl important part of our blog. Major searches will follow the rules that you set. Be aware, however, that the rules you define in your robots.txt file cannot be enforced. Crawlers for malicious software and poor search engines might not comply with your rules and index whatever they want. Thankfully, major search engines follow the standard, including Google, Bing, Yandex, Ask, yahoo and Baidu.

If your website hasn’t got a Robots.txt file, learn here how to do it. If your blog or website does have a Robots.txt file but is not optimized, then follow this post and optimize your Robots.txt file.

What is Robots.txt?

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. You can also use Ping list service to make your website more searchable.

How to Create WordPress robots.txt file

google robots.txt walpaper

Google Robots.txt

Robots.txt is a general text file. So, if you don’t have this file on your website, open any text editor as you like ( as the example: Notepad, text) and make Robots.txt file made with one or more records. Every record bears important information for the search engine.

Example

Allow indexing of everything

User-agent: *
Disallow:

or

User-agent: *
Allow: /

Here ( * ) any search engine can crawl your website and ( / ) use to block search engine.

Disallow indexing of everything

User-agent: *
Disallow: /

Disallow indexing of a specific folder

User-agent: *
Disallow: /folder name/
Disallow: /foldername 1/

Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder

User-agent: Googlebot

Basic Rule for Robots.txt file

  1. Don’t keep the space at the beginning of any line and don’t make ordinary space in the file.
    Allow: /foldername 1/filename.html
  2. Don’t change the sequence of the command (first Specified the agent and then decide the allow or disallow).
  3. If you want no index, more than one directory or page don’t write along with these names (Disallow: /support /cgi-bin /images/).
  4. Use capital and small letter properly. As the example, you want no index “Download” directory but write “download” on Robots.txt file. It makes miss understand for search bot.
  5. Don’t place sitemap at the top of the robots.txt file.

What to Place in Your Robots.txt File

I have changed the contents of my blog’s robots.txt files many times. My current robots.txt file took inspiration from Jeff Starr’s robots.txt suggestionsAskApache’s suggestions, and advice from several other developers that I respect and trust.

The great thing about the robots exclusion standard is that you can view the robots.txt file of any website on the internet (as long as they have uploaded one). All you have to do is visit www.example.com/robots.txt

If you check out the robots.txt file of some WordPress websites, you will see that website owners define different rules for search engines. The general robots.txt file are;

# Disallowed and allowed directories and files
User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: /comments/feed/
Disallow: /index.php
 # Define website sitemaps  
Sitemap: http://www.exmaple.com/sitemap_index.xml 
Sitemap: http://www.example.com/post-sitemap.xml 
Sitemap: http://www.example.com/page-sitemap.xml 
Sitemap: http://www.example.com/category-sitemap.xml 

Note:- you can use ( # ) to put the comment in the field as I do?

Hint:- CreateXML Sitemap file and place at the end of the Robots.txt file so that search engine can easily crawl your website. In this way, you can increase your search efficiency.

My robots.txt file stops search engines from crawling the important directories. I encourage you to choose one of the above examples of robots.txt for your own website and then change it accordingly for your own website. Remember, it is important that you understand all the directives that you add to your robots.txt file. The Robots Exclusion Standard can be used to stop search engines crawling files and directories that you do not want to be indexed, however, if you enter the wrong code, you may end up blocking important pages from being crawled.

How To Test Your Robots.txt File And Place into WordPress

There are a number of ways in which you can test your robots.txt file. You need a google webmaster Account and use the robots.txt file feature, which can be found under the Crawl section in Webmaster Tools. 

  • Login you GWT account and Find Robots.txt file.
Google webmaster tool robots.txt file location

Robots file in webmaster

Note:-The tool will display the contents of your website’s robots.txt file. The code that is displayed comes from the last copy of robots.txt that Google retrieved from your website.

  • Place your robots.txt file into the box and then click on test. Check is there any error or not. If there is any error, fix it and click on submit. A new window will be open.

submit wordpress robots.txt file in google webmaster tool

  • Click on download to download the robots.txt file and  open it. Copy the file.
  • Login WordPress as Admin and click on tool under SEO  option. After that click on File Editor.
file editor location in wordpress

Robots.txt file option n the WordPress

  • Paste the copy file into the box. and click on save the change to robots.txt.
robots.txt file location in wordpress

Location of Robots file in the WordPress

  • Now again go to google webmaster tool and click on submit. Your WordPress robots.txt file is uploaded.

You can check your live file by simply type www.hitechwork.com/robots.txt in the browser bar. Once you are finished Robots.txt file, I suggest you add Ping list service in the WordPress so that when you update a post automatically ping all the search engine to index it.

Remember to share this post with anyone who might benefit from this information, including your Facebook friends, Twitter followers and members of your Google+ group!

If you have any problem or suggestion about WordPress robots.txt file please feel free to comment below.

Follow me

Harvinder Singh Sharry

owner at Hitechwork
I'm Professional Blogger, SEO, and Digital marketing expert. I started my blog in 2016 with the aim to share my knowledge and experiences for the people associated with my field as well as for the general public.
Follow me
Subscribe To Our Newsletter

Subscribe To Our Newsletter

Join our mailing list to receive the latest news and updates from our team.

You have Successfully Subscribed!

Pin It on Pinterest

Shares
Share This