Tags |
|
|
Links |
|
|
Casual Articles - How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag
Accusations of spam can be so harmful to your business that every effort should be made to prevent it.However, simply requesting a surfer to enter an e-mail address into an online form on your website is not a guarantee of not being accused of spam.Here’s what could happenOne of your competitors could visit your website and enter the name of a well-known anti-spammer. When your autoresponder sends out the latest copy of your newsletter or mini-course to this person, he will become very angry.It is no good prot
That’s all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it’s probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don’t want Google to cache the page, because the site contains time sensitive information. This can be achieved simpl
China already leads the world in the number of Internet users as well as Internet usage, with over 800,000 new Internet users coming online every week. You'd think that a country with so much Internet usage would have a big effect on the web. Well, they do, but for US users, we don't often notice their presence unless we go searching for it. Here's what's out there on the Chinese information superhighway:PortalsJust like other countries, Chinese users tend to use portals to find what they're looking for. Yahoo's Chinese port
Your primary weapon of choice against duplicate content can be found within “The Robot Exclusion Protocol” which has now been adopted by all the major search engines.
There are two ways to control how the search engine spiders index your site.
1. The Robot Exclusion File or “robots.txt” and
2. The Robots < Meta > Tag
The Robots Exclusion File (Robots.txt)
This is a simple text file that can be created in Notepad. Once created you must upload the file into the root directory of your website e.g. www.yourwebsite.com/robots.txt. Before a search engine spider indexes your website they look for this file which tells them exactly how to index your site’s content.
The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.
Creating your robots.txt file
Example 1 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and make the entire site available for indexing. The robots.txt file would look like this:
User-agent: *
Disallow:
Explanation
The use of the asterisk with the “User-agent” means this robots.txt file applies to all search engine spiders. By leaving the “Disallow” blank all parts of the site are suitable for indexing.
Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
The use of the asterisk with the “User-agent” means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.
Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
By naming the particular search spider in the “User-agent” you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.
That’s all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it’s probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don’t want Google to cache the page, because the site contains time sensitive information. This can be achieved simpl
This short story, Coworkers Collaborate to Complete Successful Initiative, is part of AlphaMeasure's compilation, Tales From the Corporate Frontlines. It illustrates how important cooperation, collaboration, and communication are to achieving the common goals of a successful organization.Anonymous SubmissionThe company where I work specializes in resolving customer service issues. The results of recent customer surveys convinced management that it was time to launch a company-wide performance improvement initiative. T
The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.
Creating your robots.txt file
Example 1 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and make the entire site available for indexing. The robots.txt file would look like this:
User-agent: *
Disallow:
Explanation
The use of the asterisk with the “User-agent” means this robots.txt file applies to all search engine spiders. By leaving the “Disallow” blank all parts of the site are suitable for indexing.
Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
The use of the asterisk with the “User-agent” means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.
Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
By naming the particular search spider in the “User-agent” you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.
That’s all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it’s probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don’t want Google to cache the page, because the site contains time sensitive information. This can be achieved simpl
As an internet marketing beginner, there a number of things that you need to master just to really break into internet marketing, let alone start to make money, regardless of what all the get rich quick artists might try to tell you.These days, more and more people are starting online businesses. That is both good news and bad for you as you gear up to wade into the online business fray. The good news is, the popularity of online businesses is a testament to the way people do business these days – an increasing number of people a
Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
The use of the asterisk with the “User-agent” means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.
Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
By naming the particular search spider in the “User-agent” you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.
That’s all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it’s probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don’t want Google to cache the page, because the site contains time sensitive information. This can be achieved simpl
Keyword optimization is one of most overlooked strategies when optimizing a web site. Keywords are the words and phrases people type in a search query when they are trying to locate a service or product. Keyword research and keyword optimization is the most important aspect in search engine optimization. If you are not targeting the keywords your customers are searching with, they will never find you which results in lost profits for your business. If you plan on building a high ranking web site or are writing articles, selecting relev
Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:
User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html
Explanation
By naming the particular search spider in the “User-agent” you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.
That’s all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it’s probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don’t want Google to cache the page, because the site contains time sensitive information. This can be achieved simpl
In the wake of globalization and increased business linkages, gift-giving has been moulded to suit the demands of a growth-oriented and competitive business atmosphere. MNCs, business houses with global links and export houses are the core contributors to the growth of this culture. Gifts can play a role in awarding of contracts, finalizing joint ventures and in wooing the right kind of VC. Goal-oriented gifting is a known phenomena in the Global Corporate World.But beware. It is first important to understand the global gift cu
That’s all there is to it!
As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it’s probably necessary to use a combination of the robots.txt and the robots tag.
The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the
In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.
In this second example I don’t want Google to cache the page, because the site contains time sensitive information. This can be achieved simply by adding the “noarchive” directive.
What could be simpler!
Although there are other ways of preventing duplicate content from appearing in the Search Engines this is the simplest to implement and all websites should operate either a robots.txt file and or a Robot tag combination.
HTTP = HTML link (for blogs, profiles,phorums):
<a href="http://www.casualarticles.com/article/78435/casualarticles-How-to-Prevent-Duplicate-Content-with-Effective-Use-of-the-Robotstxt-and-Robots-Meta-Tag.html">How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag</a>
BB link (for phorums):
[url=http://www.casualarticles.com/article/78435/casualarticles-How-to-Prevent-Duplicate-Content-with-Effective-Use-of-the-Robotstxt-and-Robots-Meta-Tag.html]How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag[/url]
Related Articles:
3 Little Known Tips for Shopping for Craft Supplies Online
Discover the Keywords Secret to Making Money on eBay
Bookmark it:
|