.Google.com's John Mueller responded to a question about why Google.com marks webpages that are actually disallowed from creeping by robots.txt as well as why the it is actually secure to dismiss the relevant Browse Console files regarding those crawls.Bot Traffic To Question Specification URLs.The individual asking the question chronicled that robots were creating web links to non-existent query criterion URLs (? q= xyz) to web pages along with noindex meta tags that are actually additionally blocked in robots.txt. What triggered the question is actually that Google is crawling the web links to those pages, getting shut out through robots.txt (without seeing a noindex robotics meta tag) then obtaining shown up in Google Browse Console as "Indexed, though blocked out by robots.txt.".The person inquired the adhering to inquiry:." However listed here's the major question: why will Google.com index web pages when they can't also observe the information? What's the conveniences during that?".Google's John Mueller affirmed that if they can not crawl the page they can not observe the noindex meta tag. He additionally helps make an interesting acknowledgment of the web site: hunt operator, encouraging to disregard the results given that the "typical" individuals won't see those end results.He created:." Yes, you are actually appropriate: if we can't creep the web page, our company can not view the noindex. That mentioned, if our experts can't creep the webpages, after that there's certainly not a whole lot for our company to mark. Thus while you may view some of those web pages with a targeted site:- concern, the normal user won't find all of them, so I wouldn't fuss over it. Noindex is also great (without robots.txt disallow), it only means the URLs are going to end up being actually crawled (and end up in the Search Console report for crawled/not listed-- neither of these conditions cause issues to the remainder of the site). The important part is actually that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the limits in using the Site: search advanced search operator for analysis causes. One of those main reasons is actually due to the fact that it is actually certainly not linked to the frequent hunt mark, it is actually a separate factor altogether.Google's John Mueller discussed the internet site hunt operator in 2021:." The short answer is actually that a web site: concern is certainly not suggested to be total, neither made use of for diagnostics objectives.A site question is a specific kind of search that limits the end results to a certain web site. It's generally only the word site, a bowel, and afterwards the web site's domain.This query confines the results to a particular website. It is actually not meant to become a thorough selection of all the pages coming from that web site.".2. Noindex tag without making use of a robots.txt is actually alright for these type of scenarios where a crawler is connecting to non-existent webpages that are actually obtaining found out by Googlebot.3. Links along with the noindex tag will create a "crawled/not recorded" item in Browse Console which those will not possess a negative result on the rest of the web site.Read through the question and also address on LinkedIn:.Why would Google mark webpages when they can't even find the information?Featured Photo by Shutterstock/Krakenimages. com.