Domain Name: GETDEX.COM
Registrar: NameCheap, Inc.
Domain Status: client transfer prohibited
Creation Date: 2007-08-13T19:41:51Z
Registry Expiry Date: 2027-08-13T19:41:51Z
Updated Date: 2026-07-14T08:31:43Z
Name Server: ARA.NS.CLOUDFLARE.COM
Name Server: LEE.NS.CLOUDFLARE.COM
REGISTRAR Contact: NameCheap, Inc.
>>> Last update of RDAP database: 2026-08-04T18:30:56Z
# Dex robots.txt — AI-crawler-aware.
# Humans: see https://getdex.com/llms.txt for an AI-friendly site map.
# ----------------------------------------------------------------------
# Default tier — traditional search crawlers. Full access.
# Content-Signal (Cloudflare Content Signals Policy): public site
# content may be indexed for search and used as AI-assistant input,
# but should NOT be used to train AI models without a license.
# ----------------------------------------------------------------------
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /app
Disallow: /app/
Disallow: /appv3
Disallow: /appv3/
# ----------------------------------------------------------------------
# AI crawler tier 1 — user-agent crawlers (fetch on user request, do not
# train on content). Full access so agents can answer user questions in
# real time.
# ----------------------------------------------------------------------
User-agent: ChatGPT-User
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: OAI-SearchBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: PerplexityBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Perplexity-User
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Claude-User
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Claude-SearchBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: DuckAssistBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: YouBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Applebot-Extended
Allow: /
Disallow: /app
Disallow: /appv3
# ----------------------------------------------------------------------
# AI crawler tier 2 — training crawlers (index content for model
# training). Allowed on public marketing + documentation content.
# ----------------------------------------------------------------------
User-agent: GPTBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: ClaudeBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: anthropic-ai
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Google-Extended
Allow: /
Disallow: /app
Disallow: /appv3
# CCBot (Common Crawl) and Bytespider are pure training crawlers with no
# user-assistant surface. Hard-block them — the Content-Signal above says
# ai-train=no and these are the two UAs that most obviously violate that.
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: DeepSeekBot
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: MistralAI-User
Allow: /
Disallow: /app
Disallow: /appv3
User-agent: Cohere-AI
Allow: /
Disallow: /app
Disallow: /appv3
# ----------------------------------------------------------------------
# Sitemaps
# ----------------------------------------------------------------------
Sitemap: https://getdex.com/sitemap-index.xml
Sitemap: https://getdex.com/sitemap-0.xml
Sitemap: https://getdex.com/blog/sitemap.xml
| Položaj | Fraza | Stran | Izrezek |
|---|---|---|---|
| 5 | /blog/how-to-be-a-pe... | ||
| 7 | /blog/how-to-be-a-pe... | ||
| 13 | / | ||
| 33 | / |