Sensitive paths in robots.txt

A robots.txt lists paths for search engines, and every visitor can read it.

What it is

The file at /robots.txt names paths that crawlers should skip. People often list /admin, /backup or /internal there.

Why it matters

Naming a private path in a public file points visitors at it. The file tells search engines what to skip and carries no access control. Tenty marks the finding low and public so that every viewer can see the kind.

How Tenty checks it

Tenty reads /robots.txt and compares each path with names such as admin, backup, private, staging, config and logs.

How to fix it

Protect the paths with sign in. Remove their names from robots.txt, or use a noindex response header for pages that should leave search results.

How to reproduce

curl -sS 'https://YOUR-APP/robots.txt'

The answer lists the paths.