Improve Your Seo with Robot.txt

Improve Your Seo with Robot.txt

Published By: Admin

Share On:

It is nice once search engines oftentimes visit your web site and index your content however typically there square measure cases once classification components of your on-line content isn’t what you wish. As an example, if you’ve got 2 versions of a page (one for viewing within the browser and one for printing), you’d rather have the printing version excluded from travel, otherwise you risk being obligatory a replica content penalty. Also, if you happen to possess sensitive knowledge on your web site that you just don’t wish the globe to visualize, you’ll additionally like that search engines don’t index these pages (although during this case the sole certain method for not classification sensitive knowledge is to stay it offline on a separate machine). In addition, if you wish to save lots of some information measure by excluding pictures, style sheets and JavaScript from classification, you furthermore may would like the simplest way to inform spiders to stay removed from these things.


One way to inform search engines that files and folders on your computing machine to avoid is with the utilization of the Robots metatag. However since not all search engines browse Meta tags, the Robots Meta tag will merely go unnoted. A higher thanks to inform search engines concerning your can is to use a robots.txt file.

What Is Robots.txt?

Robots.txt may be a text (not html) file you set on your web site to inform search robots that pages you’d like them to not visit. Robots.txt is by no means that obligatory for search engines however usually search engines adapt what they’re asked to not do. it’s vital to clarify that robots.txt isn’t the simplest way from preventing search engines from travel your web site (i.e. it’s not a firewall, or a form of watchword protection) and therefore the indisputable fact that you set a robots.txt file are some things like swing a note “Please, don’t enter” on associate degree unsecured door – e.g. you can’t stop thieves from returning in however the great guys won’t receptive door and enter. That’s why we are saying that if you’ve got very sensitive knowledge, it’s too native to have faith in robots.txt to safeguard it from being indexed and displayed in search results.


The location of robots.txt is extremely vital. It should be within the main directory as a result of otherwise user agents (search engines) won’t be ready to notice it – they are doing not search the entire web site for a file named robots.txt. Instead, they give the impression of being 1st within the main directory and if they do not notice it there, they merely assume that this web site doesn’t have a robots.txt file and so they index everything they notice on the method. So, if you do not place robots.txt within the right place, don’t be stunned that search engines index your whole web site.

Structure of a Robots.txt File

The structure of a robots.txt is pretty easy it’s associate degree endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:






“User-agent” square measure search engines’ crawlers and disallow: lists the files and directories to be excluded from classification. Additionally to “user-agent:” and “disallow:” entries, you’ll embrace comment lines – simply place the # sign at the start of the line:


# All user agents square measure disallowed to visualize the /temp directory.


User-agent: *


Disallow: /temp/

Share On: