Google’s Index, Follow Meta Tag: What You Need to Know
In the world of SEO, there are many elements that play a crucial role in determining the visibility and ranking of a website. One such element is the meta tag, specifically the “index, follow” meta tag. This tag has been widely used by publishers and SEOs to instruct search engine crawlers on how to handle their webpages. However, recent insights from Google’s John Mueller have shed light on the true function and impact of this meta tag. In this article, we will explore what the index, follow meta tag actually does and why Google chooses to ignore it.
Understanding the Robots Meta Tag
Before we delve into the specifics of the index, follow meta tag, let’s first gain a comprehensive understanding of the robots meta tag as a whole. The robots meta tag is an HTML element that communicates metadata to search engine crawlers, such as Googlebot. This metadata provides machine-readable instructions to the crawlers, guiding them on how to interact with and index the webpage.
There are various directives that can be communicated through the robots meta tag, but for the purpose of this article, we will focus on the two most commonly used directives: “noindex, nofollow” and “index, follow”.
The “noindex, nofollow” Meta Tag
The “noindex, nofollow” meta tag is used to instruct search engine crawlers not to index the content on the webpage and not to follow any of the links. This directive can be useful for pages that contain sensitive or duplicate content that should not appear in search engine results.
<meta name="robots" content="noindex, nofollow">
By including this meta tag in the HTML code of a webpage, publishers can effectively control whether or not certain content should be visible in search results and whether or not search engine crawlers should follow any links on the page.
The “index, follow” Meta Tag
On the other hand, the “index, follow” meta tag is commonly used to command search engines to index the content on the webpage and follow all the links. This meta tag has often been regarded as a default directive for search engine crawlers, indicating that they should perform their usual indexing and following activities.
<meta name="robots" content="index, follow">
However, recent insights from Google’s John Mueller have revealed that the “index” directive in the “index, follow” meta tag has no function in Google’s search engine. This means that Google completely ignores this directive, treating it as if it were not present in the HTML code at all.
Why Does Google Ignore the Index Directive?
The reason behind Google’s decision to ignore the “index” directive in the “index, follow” meta tag lies in the default behavior of search engine crawlers. Crawlers, such as Googlebot, are designed to automatically index content and follow links as part of their core functionality. They do not require explicit instructions to perform these actions since it is already their default behavior.
According to Google’s documentation on robots tags, the default values for search engine crawlers are “index, follow” and do not need to be explicitly specified. This means that even if the “index” directive is included in the meta tag, Google will treat it as redundant information and disregard it altogether.
The Effect of Leaving Out the “Index” Tag
A question that often arises among publishers and SEOs is the effect of leaving out the “index” directive in the meta tag. John Mueller addressed this question on Reddit, clarifying that the absence of the “index” tag has no impact on how Google crawls and indexes the webpage.
In his response, Mueller stated, “The ‘index’ robots meta tag has no function (at least in Google) – it’s completely ignored.” He further emphasized that Google’s official documentation clearly outlines the meta tags that have actual functions, and anything else included in the robots meta tag will be ignored by Googlebot.
Therefore, there is no need to worry about the omission of the “index” directive in the meta tag. Google’s crawlers will continue to index the content and follow the links on the webpage as per their default behavior.
Misunderstandings and Clarifications
It is important to address some common misunderstandings that have arisen regarding the “index, follow” meta tag. Many authoritative websites have suggested that Google supports the “index, follow” directive, leading to confusion among publishers and SEOs.
However, as John Mueller clarified, this is not the case. Google does not use the “index, follow” meta tag to determine how its crawlers should behave. Instead, it relies on the default behavior of its crawlers, which is to index content and follow links without the need for explicit directives.
The Usefulness of “Index, Follow”
Given that Google ignores the “index” directive in the “index, follow” meta tag, one might question the usefulness of including this meta tag in the HTML code. Does it serve any purpose at all?
From Google’s perspective, the “index, follow” meta tag is essentially a waste of HTML space. It provides no additional benefit or control over how Googlebot crawls and indexes a webpage. As such, it is recommended to avoid including this meta tag if you want to streamline your HTML code and eliminate any redundant information.
However, it is worth noting that other search engines, such as Bing, treat the “index” and “follow” directives differently. Bing assumes “index” and “follow” by default but provides the option to explicitly state these directives using separate meta tags. Publishers targeting Bing’s search engine may find it useful to include the “index” and “follow” directives in their meta tags for better control over how their webpages are indexed.
The Pitfall of “Noindex, Follow”
Another common practice among publishers is to use the “noindex, follow” meta tag in an attempt to compel search engines to follow the links on a page that is not indexed. However, this approach is flawed for a simple reason: search engines cannot follow a link on a page that is not indexed.
If a webpage has a “noindex” directive, it means that search engines, including Google, will not include that page in their index. Consequently, any links on that page will also not be included in the index. Therefore, the “follow” directive becomes irrelevant in this scenario, as search engines cannot follow links that are not present in their index.
It is crucial to understand this pitfall to avoid any misconceptions about the behavior of search engine crawlers. Including the “noindex” directive effectively removes the page and its links from the search engine’s index, rendering the “follow” directive useless.
See first source: Search Engine Journal
FAQ
Q1: What is the purpose of the robots meta tag in SEO?
A1: The robots meta tag is an HTML element used in SEO to communicate metadata to search engine crawlers, such as Googlebot. It provides instructions to crawlers on how to interact with and index a webpage. It helps control aspects like indexing content and following links.
Q2: What are the two most commonly used directives in the robots meta tag?
A2: The two most commonly used directives are “noindex, nofollow” and “index, follow.” “Noindex, nofollow” instructs search engine crawlers not to index the content on the webpage and not to follow any links. “Index, follow” traditionally indicated that crawlers should index content and follow links, but recent insights reveal Google ignores the “index” directive.
Q3: Why does Google ignore the “index” directive in the “index, follow” meta tag?
A3: Google’s search engine crawlers, like Googlebot, automatically index content and follow links by default. They do not require explicit instructions for these actions. Google’s documentation states that the default behavior is “index, follow,” making the “index” directive redundant, and thus Google chooses to ignore it.
Q4: What happens if the “index” directive is omitted from the “index, follow” meta tag?
A4: Omitting the “index” directive has no impact on how Google crawls and indexes the webpage. Google’s crawlers will continue to index the content and follow the links as per their default behavior, regardless of the presence or absence of the “index” directive.
Q5: Why is it recommended to avoid using the “index, follow” meta tag in HTML code?
A5: The “index, follow” meta tag is considered a waste of HTML space when targeting Google because Google’s crawlers ignore the “index” directive. Including it provides no additional control or benefit over Google’s default behavior. It is recommended to streamline HTML code by omitting this redundant tag.
Q6: Are there search engines that treat the “index” and “follow” directives differently from Google?
A6: Yes, some search engines, like Bing, treat the “index” and “follow” directives differently. Bing assumes “index” and “follow” by default but allows publishers to specify these directives using separate meta tags. Publishers targeting Bing’s search engine may find it useful to include both directives for better control.
Q7: What pitfall should publishers be aware of when using the “noindex, follow” meta tag?
A7: Publishers should be aware that using the “noindex” directive effectively removes the page and its links from the search engine’s index. In such cases, the “follow” directive becomes irrelevant because search engines cannot follow links that are not present in their index. It is essential to understand this pitfall to avoid misconceptions about crawler behavior.
Featured Image Credit: Photo by Maksym Kaharlytskyi; Unsplash – Thank you!
