The Robots Exclusion Standard was developed in 1994 so that website owners can advise search engines, how to crawl your website. Placing a WordPress robots.txt file in the root of your domain lets you stop search engines indexing sensitive files like wp-admin, plugin, password etc.
In simple word, Let us suppose our website is like a House(home). A house consists of a large number of Room. Some Room are important for us and some are not like the storage room, private room. When someone come in our home, we focus on that thing they never go toward private area because this area is important for us. Similarly, we have to stop search engine to crawl sensitive area of our website like admin and tell, what should have to crawl and what should not. You can also use Meta Robots tag to stop search engine to indexing particular page or post which is very important for every blogger.
WordPress robots.txt file plays a major role in search engine ranking. It helps to block search engine bots to index and crawl important part of our blog. Major searches will follow the rules that you set. Be aware, however, that the rules you define in your robots.txt file cannot be enforced. Crawlers for malicious software and poor search engines might not comply with your rules and index whatever they want. Thankfully, major search engines follow the standard, including Google, Bing, Yandex, Ask, yahoo and Baidu.
If your website hasn’t got a Robots.txt file, learn here how to do it. If your blog or website does have a Robots.txt file but is not optimized, then follow this post and optimize your Robots.txt file.
What is Robots.txt?
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. You can also use Ping list service to make your website more searchable.
How to Create WordPress robots.txt file
Robots.txt is a general text file. So, if you don’t have this file on your website, open any text editor as you like ( as the example: Notepad, text) and make Robots.txt file made with one or more records. Every record bears important information for the search engine.
Allow indexing of everything
Here ( * ) any search engine can crawl your website and ( / ) use to block search engine.
Disallow indexing of everything
Disallow indexing of a specific folder
Disallow: /folder name/
Disallow: /foldername 1/
Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder
Basic Rule for Robots.txt file
- Don’t keep the space at the beginning of any line and don’t make ordinary space in the file.
Allow: /foldername 1/filename.html
- Don’t change the sequence of the command (first Specified the agent and then decide the allow or disallow).
- If you want no index, more than one directory or page don’t write along with these names (Disallow: /support /cgi-bin /images/).
- Use capital and small letter properly. As the example, you want no index “Download” directory but write “download” on Robots.txt file. It makes miss understand for search bot.
- Don’t place sitemap at the top of the robots.txt file.
What to Place in Your Robots.txt File
I have changed the contents of my blog’s robots.txt files many times. My current robots.txt file took inspiration from Jeff Starr’s robots.txt suggestions, AskApache’s suggestions, and advice from several other developers that I respect and trust.
The great thing about the robots exclusion standard is that you can view the robots.txt file of any website on the internet (as long as they have uploaded one). All you have to do is visit www.example.com/robots.txt
If you check out the robots.txt file of some WordPress websites, you will see that website owners define different rules for search engines. The general robots.txt file are;
# Disallowed and allowed directories and files
User-agent: * Allow: /wp-admin/admin-ajax.php Allow: /wp-content/uploads/ Disallow: /wp-admin/ Disallow: /cgi-bin/ Disallow: /trackback/ Disallow: /xmlrpc.php Disallow: /comments/feed/ Disallow: /index.php
# Define website sitemaps Sitemap: http://www.exmaple.com/sitemap_index.xml Sitemap: http://www.example.com/post-sitemap.xml Sitemap: http://www.example.com/page-sitemap.xml Sitemap: http://www.example.com/category-sitemap.xml
Note:- you can use ( # ) to put the comment in the field as I do?
Hint:- CreateXML Sitemap file and place at the end of the Robots.txt file so that search engine can easily crawl your website. In this way, you can increase your search efficiency.
My robots.txt file stops search engines from crawling the important directories. I encourage you to choose one of the above examples of robots.txt for your own website and then change it accordingly for your own website. Remember, it is important that you understand all the directives that you add to your robots.txt file. The Robots Exclusion Standard can be used to stop search engines crawling files and directories that you do not want to be indexed, however, if you enter the wrong code, you may end up blocking important pages from being crawled.
How To Test Your Robots.txt File And Place into WordPress
There are a number of ways in which you can test your robots.txt file. You need a google webmaster Account and use the robots.txt file feature, which can be found under the Crawl section in Webmaster Tools.
- Login you GWT account and Find Robots.txt file.
Note:-The tool will display the contents of your website’s robots.txt file. The code that is displayed comes from the last copy of robots.txt that Google retrieved from your website.
- Place your robots.txt file into the box and then click on test. Check is there any error or not. If there is any error, fix it and click on submit. A new window will be open.
- Click on download to download the robots.txt file and open it. Copy the file.
- Login WordPress as Admin and click on tool under SEO option. After that click on File Editor.
- Paste the copy file into the box. and click on save the change to robots.txt.
- Now again go to google webmaster tool and click on submit. Your WordPress robots.txt file is uploaded.
You can check your live file by simply type www.hitechwork.com/robots.txt in the browser bar. Once you are finished Robots.txt file, I suggest you add Ping list service in the WordPress so that when you update a post automatically ping all the search engine to index it.
Remember to share this post with anyone who might benefit from this information, including your Facebook friends, Twitter followers and members of your Google+ group!
If you have any problem or suggestion about WordPress robots.txt file please feel free to comment below.