Domain Name: CRABTREEBOOKS.COM
Registrar: Tucows Domains Inc.
Domain Status: client transfer prohibited
Domain Status: client update prohibited
Creation Date: 2000-01-24T18:05:02Z
Registry Expiry Date: 2027-01-24T18:05:02Z
Updated Date: 2025-01-15T19:10:31Z
Name Server: DNS1.VAXXINE.COM
Name Server: DNS2.VAXXINE.COM
REGISTRAR Contact: Tucows Domains Inc.
>>> Last update of RDAP database: 2026-03-16T23:53:55Z
# See http://www.robotstxt.org/robotstxt.html for documentation
User-agent: *
Crawl-Delay: 5
Disallow: /account*
Disallow: /addresses*
Disallow: /admin*
Disallow: /bisac_subjects/*
Disallow: /cart*
Disallow: /downloads*
Disallow: /excerpts*
Disallow: /products/*/compare*
Disallow: /products/advanced_search
Disallow: /products/search_results*
Disallow: /links*
Disallow: /lists*
Disallow: /marc_reports*
Disallow: /quotes*
Disallow: /searches*
Disallow: /shop/search_results*
Disallow: /specs*
Disallow: /users*
Disallow: /wishlists*
# Block select search and analytics bots
User-agent: Adsbot
User-agent: AhrefsBot
User-agent: BLEXBot
User-agent: Seekport Crawler
User-agent: SemrushBot
User-agent: SenutoBot
Disallow: /
# Block all known AI crawlers and assistants
# from using content for training AI models.
# Source: https://robotstxt.com/ai
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: Claude-User
User-Agent: Claude-SearchBot
User-Agent: CCBot
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: Facebookbot
User-Agent: Meta-ExternalAgent
User-Agent: Meta-ExternalFetcher
User-Agent: diffbot
User-Agent: PerplexityBot
User-Agent: Perplexity‑User
User-Agent: Omgili
User-Agent: Omgilibot
User-Agent: webzio-extended
User-Agent: ImagesiftBot
User-Agent: Bytespider
User-Agent: TikTokSpider
User-Agent: Amazonbot
User-Agent: Youbot
User-Agent: SemrushBot-OCOB
User-Agent: Petalbot
User-Agent: VelenPublicWebCrawler
User-Agent: TurnitinBot
User-Agent: Timpibot
User-Agent: OAI-SearchBot
User-Agent: ICC-Crawler
User-Agent: AI2Bot
User-Agent: AI2Bot-Dolma
User-Agent: DataForSeoBot
User-Agent: AwarioBot
User-Agent: AwarioSmartBot
User-Agent: AwarioRssBot
User-Agent: Google-CloudVertexBot
User-Agent: PanguBot
User-Agent: Kangaroo Bot
User-Agent: Sentibot
User-Agent: img2dataset
User-Agent: Meltwater
User-Agent: Seekr
User-Agent: peer39_crawler
User-Agent: cohere-ai
User-Agent: cohere-training-data-crawler
User-Agent: DuckAssistBot
User-Agent: Scrapy
User-Agent: Cotoyogi
User-Agent: aiHitBot
User-Agent: Factset_spyderbot
User-Agent: FirecrawlAgent
Disallow: /
DisallowAITraining: /
Sitemap: https://crabtreebooks.com/sitemap.xml.gz
| Θέση | Φράση | Σελίδα | Απόσπασμα |
|---|---|---|---|
| 1 | /shop/show/16260 | ||
| 6 | /series/11297-i-read... |