Google has been updating its documentation for web administrators lately, and it added more information to one of those documents today. The search giant released an updated JavaScript SEO guide for administrators just last month and offered a time of review for web administrators and SEO experts to provide feedback. One request was for a suggestion to provide more clarity around the meta noindex tags, and Google obliged.
Google advises website administrators to use their meta robots tags carefully. Using a meta noindex tag incorrectly could keep Googlebot from crawling a page you actually it to crawl. The company provided this example:
<meta name=”robots” content=”noindex, nofollow”
That meta noindex tag will keep Googlebot from indexing the page and following links included on the page.
Google also explains that website administrators can use JavaScript to either add a meta robots tag to one of their pages or change a page’s content. Here is the example meta noindex tags code provided by Google.
fetch(‘/api/products/’ + productId)
.then(function (response) { return response.json(); })
.then(function (apiResponse) {
if (apiResponse.isError) {
// get the robots meta tag
var metaRobots = document.querySelector(‘meta[name=”robots”]’);
// if there was no robots meta tag, add one
if (!metaRobots) {
metaRobots = document.createElement(‘meta’);
metaRobots.setAttribute(‘name’, ‘robots’);
document.head.appendChild(metaRobots);
}
// tell Googlebot to exclude this page from the index
metaRobots.setAttribute(‘content’, ‘noindex’);
// display an error message to the user
errorMsg.textContent = ‘This product is no longer available’;
return;
}
// display product information
// …
});
Click here for a link to Google’s example code if you wish to easily copy and paste it for your own use.
The company explained that when Googlebot runs into a “noindex” in the robots meta tag before it sees JavaScript, it won’t render or index the page. Thus, it also doesn’t execute the JavaScript. Since Google skips the JavaScript in this example, there’s no way to remove the tag from the page.
“Using JavaScript to change or remove the robots meta tag might not work as expected,” Google warned. Googlebot skips rendering and JavaScript execution if the meta robots tag initially contains noindex.”
The company further advises website administrators who might want a particular page to be indexed not to use “noindex” in the original page code.
