Domain Name: TESTBIRDS.COM
Registrar: IONOS SE
Domain Status: client transfer prohibited
Creation Date: 2011-11-09T13:09:53Z
Registry Expiry Date: 2026-11-09T13:09:53Z
Updated Date: 2025-11-10T08:15:47Z
Name Server: NS1092.UI-DNS.BIZ
Name Server: NS1092.UI-DNS.COM
Name Server: NS1092.UI-DNS.DE
Name Server: NS1092.UI-DNS.ORG
REGISTRAR Contact: IONOS SE
>>> Last update of RDAP database: 2026-01-26T04:41:18Z
# ============================================================================ # ROBOTS.TXT - OPTIMIERT FÜR 2025 # Vollständige Abdeckung: AI Search, AI Training, Standard Crawlers # Letzte Aktualisierung: 28. November 2025 # ============================================================================ # ---------------------------------------------------------------------------- # SEKTION 1: AI SEARCH BOTS (ERLAUBT) # Diese Bots zeigen deine Inhalte in AI-Suchergebnissen und zitieren deine Seite # ---------------------------------------------------------------------------- # === OPENAI (ChatGPT Search) === User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ChatGPT-User/2.0 User-agent: ChatGPT-Agent Allow: / # === ANTHROPIC (Claude Search) === User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Anthropic-AI-Search Allow: / # === PERPLEXITY === User-agent: PerplexityBot Allow: / # === GOOGLE (Gemini & Search) === User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-News User-agent: Googlebot-Video User-agent: GoogleAgent-Mariner User-agent: Google-InspectionTool Allow: / # === BING / MICROSOFT (Copilot) === User-agent: Bingbot User-agent: BingPreview User-agent: msnbot User-agent: MSNBot-Media Allow: / # === META (Meta AI Search) === User-agent: Meta-WebIndexer Allow: / # === APPLE (Siri, Spotlight) === User-agent: Applebot User-agent: Applebot-Extended Allow: / # === ALTERNATIVE SUCHMASCHINEN === User-agent: DuckDuckBot User-agent: DuckAssistBot User-agent: Kagibot User-agent: YouBot User-agent: Mojeek-News-Bot User-agent: Yandex User-agent: YandexBot Allow: / # === AMAZON (Alexa) === User-agent: Amazonbot Allow: / # === QUORA (Poe) === User-agent: PoeBot Allow: / # === MISTRAL AI === User-agent: MistralAI-User Allow: / # === XING / LINKEDIN PREVIEW === User-agent: LinkedInBot User-agent: XINGBot Allow: / # ---------------------------------------------------------------------------- # SEKTION 2: AI TRAINING BOTS (BLOCKIERT) # Diese Bots trainieren AI-Modelle - blockiere sie, um deine Inhalte zu schuetzen # ---------------------------------------------------------------------------- # === OPENAI TRAINING === User-agent: GPTBot User-agent: OAI-SearchBot-Training Disallow: / # === GOOGLE TRAINING === User-agent: Google-Extended User-agent: GoogleOther User-agent: GoogleOther-Image User-agent: GoogleOther-Video Disallow: / # === ANTHROPIC TRAINING === User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web Disallow: / # === COMMON CRAWL (Dataset für Training) === User-agent: CCBot User-agent: cohere-ai Disallow: / # === META / FACEBOOK TRAINING === User-agent: FacebookBot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: facebookexternalhit Disallow: / # === BYTEDANCE / TIKTOK === User-agent: Bytespider User-agent: ByteDance Disallow: / # === OMGILI / WEBZ.IO (Datenverkauf) === User-agent: Omgilibot User-agent: Omgili User-agent: webzio-extended Disallow: / # === DIFFBOT (Scraping Service) === User-agent: Diffbot Disallow: / # === IMG2DATASET (Bilddaten Training) === User-agent: img2dataset User-agent: ImagesiftBot Disallow: / # === DIVERSE AI TRAINING BOTS === User-agent: PerplexityBot-Training User-agent: Timpibot User-agent: VelenPublicWebCrawler User-agent: Scrapy User-agent: peer39_crawler User-agent: peer39_crawler/1.0 Disallow: / # === ACADEMIC / RESEARCH BOTS (optional blockieren) === # User-agent: archive.org_bot # User-agent: ia_archiver # Disallow: / # ---------------------------------------------------------------------------- # SEKTION 3: SEO & MONITORING TOOLS (ERLAUBT) # ---------------------------------------------------------------------------- # === SEO TOOLS === User-agent: AhrefsBot User-agent: SemrushBot User-agent: DotBot User-agent: MJ12bot User-agent: Screaming Frog SEO Spider Allow: / # === SITE MONITORING === User-agent: Pingdom User-agent: UptimeRobot User-agent: StatusCake Allow: / # === SOCIAL MEDIA PREVIEW === User-agent: Twitterbot User-agent: facebookexternalhit User-agent: Slackbot User-agent: TelegramBot User-agent: WhatsApp User-agent: Discordbot Allow: / # ---------------------------------------------------------------------------- # SEKTION 4: BAD BOTS & SCRAPERS (BLOCKIERT) # ---------------------------------------------------------------------------- User-agent: AhrefsBot # User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: rogerbot User-agent: AhrefsSiteAudit User-agent: proximic User-agent: archive.org_bot User-agent: ia_archiver User-agent: MegaIndex.ru User-agent: SeznamBot User-agent: BLEXBot User-agent: MauiBot User-agent: DomainCrawler User-agent: Wget User-agent: curl User-agent: HTTrack Disallow: / # ---------------------------------------------------------------------------- # SEKTION 5: STANDARD CRAWLER REGELN # ---------------------------------------------------------------------------- User-agent: * Allow: / # Optional: Bestimmte Verzeichnisse für ALLE blockieren # Disallow: /admin/ # Disallow: /private/ # Disallow: /wp-admin/ # Disallow: /api/ # Disallow: /downloads/ # Disallow: /*.pdf$ # Disallow: /*.doc$ # Disallow: /*.docx$ # Disallow: /thank-you/ # Disallow: /*? # Disallow: /search # Disallow: /checkout/ # Disallow: /cart/ # ---------------------------------------------------------------------------- # SEKTION 6: CRAWL-DELAY (optional) # ---------------------------------------------------------------------------- # Crawl-Delay: 1 # ---------------------------------------------------------------------------- # SEKTION 7: SITEMAP # WICHTIG: Passe die URL an deine Domain an! # ---------------------------------------------------------------------------- Sitemap: https://www.testbirds.com/sitemap.xml # ============================================================================ # HINWEISE: # ============================================================================ # # 1. WICHTIG: robots.txt ist freiwillig - manche Bots ignorieren sie! # Ergaenzende Massnahmen: Cloudflare Bot Management, IP-Blocking, Meta-Tags # # 2. AI Search vs. AI Training: # - AI Search Bots: Zitieren deine Inhalte - ERLAUBEN fuer Sichtbarkeit # - AI Training Bots: Nutzen deine Daten fuer Modelltraining - BLOCKIEREN # # 3. Perplexity-User & ChatGPT-User: # Diese "User-Bots" reagieren auf Nutzeranfragen und ignorieren oft robots.txt # - Zusaetzliche Firewall-Regeln notwendig # # 4. Teste deine robots.txt: # - Google Search Console: Robots.txt Tester # - https://en.ryte.com/free-tools/robots-txt/ # - https://technicalseo.com/tools/robots-txt/ # # 5. Halte diese Datei aktuell: # Neue AI Bots erscheinen staendig. Ueberpruefe quartalsweise auf Updates. # Quelle: https://github.com/ai-robots-txt/ai.robots.txt # # 6. Meta-Tag Alternative (fuer selektives Blockieren): # # # ============================================================================
| Position | Udtryk | Side | Uddrag |
|---|---|---|---|
| 10 | /en/use-cases/test-objects/testing-games-vr-ar/ |