Robots.txt and noindex control different stages: robots.txt governs crawling, while noindex instructs a supporting search engine not to index accessible content. Google must be able to crawl a page to see its noindex instruction. Blocking the same URL in robots.txt does not strengthen that indexing instruction.
Make separate decisions about crawling and indexing
When a page should remain public but stay out of Google, use noindex and verify that Googlebot can read the instruction. HTML can use a robots meta tag; resources such as PDFs can use an X-Robots-Tag response header. Writing noindex inside robots.txt is not a supported Google method.
Robots.txt serves a different purpose when reducing crawling of unnecessary filter or sort URLs. A blocked URL discovered through links may still appear in search without its content being read. Removing an already indexed page and reducing future unnecessary crawling should therefore be planned as separate tasks.
Select the method by the outcome
Before applying a method from the table, record the URL’s current response and indexing state. Inspect the head section for HTML or the response header for a PDF, then test the same URL after the change. An administration setting alone does not verify the actual response. Test a broad robots pattern against a service URL that should remain unaffected too.
| Need | Method | Verification |
|---|---|---|
| Unnecessary filter crawling | Appropriate robots.txt pattern | Important URLs and resources excluded from the rule |
| Public HTML excluded from indexing | Crawlable noindex meta tag | Google can read the instruction |
| Public PDF excluded from indexing | X-Robots-Tag response header | Header exists on the actual file response |
| Private content | Authentication and authorisation | Unauthorised requests cannot receive content |
| Permanent URL move | Permanent redirect to a relevant destination | Final destination supplies the right content |
Do not treat search exclusion as privacy
Google's content-control guidance treats password protection as a separate approach for private material. Remaining outside search results does not stop access through a direct link. Scope actual authorisation requirements for customer documents or internal panels.
Imagine a hypothetical company storing proposal PDFs in a public folder and only disabling indexing. Someone who knows the file address may still open it. The issue requires the right access boundary rather than an additional SEO tag.
Use affected and unaffected examples to verify the change
Select an affected URL, an unaffected product page and a similarly named directory. Test the robots.txt pattern against them and check that it does not block essential CSS, JavaScript or product URLs. For noindex, inspect both page source and response headers; an index preference in one layer does not automatically override noindex in another.
After the change, verify that the instruction is accessible in a live check, then monitor a subsequent Google crawl. Temporary search removal tools do not replace access control for private files or a permanent publishing decision. Record the rule’s source, date and intended purpose for future maintenance.