.Google.com's John Mueller responded to a concern about why Google.com marks webpages that are refused from crawling by robots.txt as well as why the it's risk-free to ignore the associated Search Console documents regarding those crawls.Crawler Website Traffic To Inquiry Guideline URLs.The person asking the inquiry recorded that robots were actually generating web links to non-existent concern criterion Links (? q= xyz) to webpages with noindex meta tags that are also shut out in robots.txt. What triggered the concern is actually that Google is crawling the links to those web pages, acquiring blocked out through robots.txt (without envisioning a noindex robots meta tag) then acquiring turned up in Google.com Explore Console as "Indexed, though blocked through robots.txt.".The individual talked to the following inquiry:." But listed here's the huge concern: why would Google.com index web pages when they can not even find the web content? What is actually the benefit during that?".Google.com's John Mueller confirmed that if they can't crawl the webpage they can't see the noindex meta tag. He likewise produces an interesting acknowledgment of the site: search operator, encouraging to disregard the results due to the fact that the "common" users will not observe those end results.He composed:." Yes, you are actually correct: if our company can not crawl the web page, our experts can not see the noindex. That said, if our experts can't creep the webpages, at that point there is actually not a great deal for our team to index. Thus while you may see a number of those pages along with a targeted website:- concern, the normal consumer will not find them, so I definitely would not bother it. Noindex is also alright (without robots.txt disallow), it only suggests the URLs are going to wind up being actually crawled (and also end up in the Browse Console record for crawled/not catalogued-- neither of these conditions lead to concerns to the rest of the site). The vital part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the constraints in operation the Site: hunt advanced hunt driver for diagnostic explanations. One of those explanations is considering that it is actually certainly not connected to the routine hunt mark, it is actually a separate factor altogether.Google's John Mueller discussed the internet site search operator in 2021:." The short answer is actually that a website: question is actually certainly not implied to be full, nor made use of for diagnostics objectives.A web site concern is actually a specific type of search that confines the outcomes to a specific website. It's essentially just the word internet site, a digestive tract, and after that the internet site's domain.This query confines the end results to a specific website. It's not suggested to become a complete compilation of all the web pages from that site.".2. Noindex tag without making use of a robots.txt is actually great for these sort of situations where a crawler is actually linking to non-existent web pages that are receiving discovered through Googlebot.3. URLs with the noindex tag are going to produce a "crawled/not recorded" entry in Explore Console which those won't possess a damaging impact on the remainder of the web site.Check out the inquiry and address on LinkedIn:.Why would Google.com mark webpages when they can not even view the content?Included Image through Shutterstock/Krakenimages. com.