SEO and the preventing of duplicate content.
One of the most important factors about SEO currently on the web is to prevent duplicate content. We assume you have intentionally or mistakenly created duplicate content on your website, of course not by disrespecting copyright of other resources. It is very important to prevent different URLs that link to the same content on your website. So, if you have intentionally created URLs with the same content, it is important to let crawlers know which one to index and which one not. There are several ways to do so. First, let's assume that those links are wrong.
- Make sure that those links are not displayed anywhere on your website. If wrong URLs don't exists, they cannot be indexed, it is as simple as that.
- 301 Permanent HTTP Redirect.
You can add a 301 redirect at the pages with duplicate content that should link to the correct URL. Search engines will acknowledge the HTTP Redirect and your duplicate URLs will eventually be removed from their indexes. You can speed this up with Google sitemaps which allows you to remove URLs from the index of Google.
Now, let's assume that you have intentionally created different URLs that link to roughly the same content. In this case, it is important to prevent different URLs inside search engines but the different URLs do have to be accessible.
- If you want a complete map or directory left out of the index, it is easiest to change your robot.txt file inside your webroot. Use the code Disallow: /map1/ to prevent indexing everything inside map1. Use the code User-agent: * to tell browsers that the code below should be acknowledged by all user agents, that is by all search engines that use crawlers to index a website. The code Disallow: /*? allows you to prevent all URLs that have the question mark inside their URL to be indexed. This is good practice in the case your website is well prepared for search engine friendly URLs.
- You can add a meta tag to the particular webpage in question. Add the code inside the head of your webpage as metatag.
- Another relatively new method is to assign a canonical page. Suppose you have three pages that roughly have the same content and you want all three to exists but only one inside search engines. You add a canonical link relationship in the head of the pages that shouldn't be indexed.
Would you like to comment?
Fields with * are obligatory.