.Google's John Mueller answered a concern about why Google.com indexes pages that are refused coming from crawling through robots.txt and also why the it is actually risk-free to disregard the similar Explore Console files concerning those crawls.Crawler Website Traffic To Question Specification URLs.The person talking to the inquiry chronicled that robots were making hyperlinks to non-existent inquiry criterion Links (? q= xyz) to webpages with noindex meta tags that are actually likewise blocked out in robots.txt. What caused the concern is actually that Google.com is actually creeping the links to those web pages, obtaining blocked out by robots.txt (without envisioning a noindex robots meta tag) after that receiving reported in Google Look Console as "Indexed, though blocked by robots.txt.".The individual asked the observing question:." However below's the major concern: why would Google index webpages when they can not also see the web content? What is actually the benefit in that?".Google.com's John Mueller confirmed that if they can't crawl the webpage they can not view the noindex meta tag. He likewise creates an interesting mention of the website: search operator, suggesting to neglect the results since the "average" consumers won't find those end results.He created:." Yes, you're correct: if our experts can not crawl the web page, our team can not see the noindex. That claimed, if we can't creep the web pages, then there's certainly not a great deal for us to mark. Thus while you could view several of those webpages with a targeted internet site:- query, the typical user will not observe all of them, so I wouldn't bother it. Noindex is also fine (without robots.txt disallow), it just indicates the Links will certainly find yourself being actually crept (and also wind up in the Browse Console document for crawled/not catalogued-- neither of these standings induce problems to the remainder of the web site). The vital part is that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the limits in operation the Website: search progressed search operator for analysis causes. One of those reasons is because it's certainly not attached to the normal hunt index, it is actually a distinct factor altogether.Google.com's John Mueller talked about the site search operator in 2021:." The brief answer is that a web site: concern is not suggested to become complete, nor made use of for diagnostics objectives.A web site inquiry is a particular sort of search that restricts the outcomes to a particular internet site. It is actually basically simply words site, a colon, and then the website's domain.This concern limits the end results to a certain internet site. It is actually not indicated to become a detailed compilation of all the pages from that website.".2. Noindex tag without making use of a robots.txt is actually alright for these type of situations where a crawler is actually linking to non-existent web pages that are actually receiving found out by Googlebot.3. URLs with the noindex tag are going to generate a "crawled/not recorded" item in Search Console and also those won't possess an unfavorable effect on the rest of the web site.Review the concern and also answer on LinkedIn:.Why would Google.com mark pages when they can not also view the information?Featured Image by Shutterstock/Krakenimages. com.