# pastebin.ca — robots.txt
#
# Allow major search engines to index the public surfaces. Block paths
# that hold personalized content, API responses, or admin surfaces.
#
# Per-paste URLs live at the top-level `/` form and are indexable on a
# per-page basis: public, listed pastes (public feed + profile visibility)
# emit an indexable, self-canonical page, while unlisted, signed-url,
# encrypted, single-use, and secondary (diff/rendered) views emit
# `noindex` (meta + X-Robots-Tag). Public profile pages under /u/ are
# indexable. The sitemap lists the locale-mirrored chrome routes.
#
# Signed `?s=` capability URLs are the secret themselves — never crawl or
# index them (they also emit noindex as defense-in-depth).
User-agent: *
Allow: /
Disallow: /*?s=
Disallow: /*&s=
Disallow: /account
Disallow: /account/
Disallow: /admin
Disallow: /admin/
Disallow: /api/
Disallow: /auth/
Disallow: /webhooks/
Disallow: /_server-islands/
Disallow: /_image
Disallow: /_astro/
Disallow: /bundles/
Disallow: /editor/
Disallow: /vendor/
Disallow: /pastes
Disallow: /pastes/
Disallow: /forks
Disallow: /forks/
Disallow: /drafts
Disallow: /drafts/
Disallow: /revoked
Disallow: /revoked/
Disallow: /p/
Sitemap: https://pastebin.ca/sitemap-index.xml
# Machine-readable guide for AI agents and automated clients:
# https://pastebin.ca/llms.txt