Robots and noindex diagnosis

Direct answer

robots.txt controls crawling; noindex controls whether a fetched page can appear in search results. A noindex directive must be visible to the crawler, so blocking the same page in robots.txt can stop the crawler from reading it.

What to check

1. Inspect robots.txt for the final URL. Use the final redirect destination and evaluate the relevant user-agent group. A disallow rule may prevent a crawler from fetching the page, even if the page opens normally in your browser.
2. Check both noindex locations. For HTML, inspect the robots meta tag. Also read X-Robots-Tag in the HTTP headers, including on non-HTML files. Check engine-specific directives if they matter to your site.
3. Confirm with the property owner’s tools. After changing a rule, allow recrawling and inspect the URL in the verified Search Console property. A public page check can show the current response, but only the engine can show what it previously fetched.

Example

Example: robots.txt disallows /draft/ and /draft/page also has meta noindex. Google cannot fetch the page to read that meta tag. If removal from search is the goal, permit crawling and keep noindex, or require authentication.

What this does not establish

The free RankMeOff noindex check reads the response it can fetch. It does not establish that Googlebot saw the same response or that an existing search listing has already changed.

Sources and next step

• Google: block indexing with noindex
• Google: robots.txt introduction

Read the RankMeOff measurement methodology or open the related tool page.