Domain Name: BCCRESEARCH.COM
Registrar: GoDaddy.com, LLC
Domain Status: client delete prohibited
Domain Status: client renew prohibited
Domain Status: client transfer prohibited
Domain Status: client update prohibited
Creation Date: 2000-01-13T17:00:21Z
Registry Expiry Date: 2027-01-13T17:00:20Z
Updated Date: 2026-01-14T17:47:14Z
Name Server: NS-106.AWSDNS-13.COM
Name Server: NS-1400.AWSDNS-47.ORG
Name Server: NS-1680.AWSDNS-18.CO.UK
Name Server: NS-729.AWSDNS-27.NET
REGISTRAR Contact: GoDaddy.com, LLC
>>> Last update of RDAP database: 2026-05-23T21:01:49Z
# BCC Research robots.txt — AI-friendly with guardrails
# --- Global defaults (keep existing hygiene) ---
User-agent: *
Crawl-Delay: 2
Disallow: /index/advancedresult/
Disallow: /index/advancedsearch/
Disallow: /*?*
Disallow: /*?
Disallow: /aboutus/hubspotlead
Disallow: /statsana/piwik/
Allow: /public/
Allow: /public/images2017/
# --- AI crawlers: explicit allow (faster crawl) ---
User-agent: GPTBot
Allow: /
Crawl-Delay: 2
User-agent: OAI-SearchBot
Allow: /
Crawl-Delay: 2
User-agent: ClaudeBot
Allow: /
Crawl-Delay: 2
User-agent: Claude-SearchBot
Allow: /
Crawl-Delay: 2
User-agent: PerplexityBot
Allow: /
Crawl-Delay: 2
User-agent: CCBot
Allow: /
Crawl-Delay: 2
# Broader “infra” user-agents used for AI indexing/pipeline work
User-agent: GoogleOther
Allow: /
Crawl-Delay: 2
User-agent: Amazonbot
Allow: /
Crawl-Delay: 2
# Traditional search bots (often feed LLM features)
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /
# Training/usage policy toggles (not crawlers, but respected by some vendors)
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
# --- Machine Content Policy for AI & LLMs (already live) ---
MCP: https://www.bccresearch.com/mcp.yaml
# --- Sitemaps (already live) ---
Sitemap: https://www.bccresearch.com/public/sitemap/indexsitemap/sitemapindex.xml
| positsioon | Fraas | Lehekülg | Katkend |
|---|---|---|---|
| 1 | /company-index/s?srs... | ||
| 3 | /company-index/i?srs... | ||
| 8 | /company-index/i?srs... |