Domain Name: SENSIBO.COM
Registrar: NameCheap, Inc.
Domain Status: client transfer prohibited
Creation Date: 2013-10-31T16:09:28Z
Registry Expiry Date: 2026-10-31T16:09:28Z
Updated Date: 2025-10-01T08:52:11Z
Name Server: PDNS1.REGISTRAR-SERVERS.COM
Name Server: PDNS2.REGISTRAR-SERVERS.COM
REGISTRAR Contact: NameCheap, Inc.
>>> Last update of RDAP database: 2026-03-28T03:09:21Z
########################
# 0. XML Sitemap
########################
Sitemap: https://sensibo.com/sitemap.xml
#######################################################################
# 1. ALL MAJOR SEARCH ENGINES → ALLOW (private/dup blocks below)
#######################################################################
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: Applebot
Allow: /
#######################################################################
# 2. AI CRAWLERS → ALLOW TRAINING **AND** ANSWER ENGINES
#######################################################################
# ––– Answer / retrieval bots –––
User-agent: OAI-SearchBot # ChatGPT retrieval
User-agent: PerplexityBot
User-agent: YouBot
User-agent: PhindBot
# ––– Model-training bots –––
User-agent: GPTBot # OpenAI training
User-agent: anthropic-ai # Anthropic Claude
User-agent: ClaudeBot
User-agent: Google-Extended # Google Gemini training token
User-agent: xAIBot # xAI / Grok
User-agent: Amazonbot # Amazon Titan
Allow: /
# Optional throttle to keep bandwidth predictable
Crawl-delay: 5
#######################################################################
# 3. AGGRESSIVE SEO / LINK BOTS → THROTTLE
#######################################################################
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: MJ12bot
Crawl-delay: 10
Allow: /
###############################################################
# 4. SHARED "DO NOT CRAWL" LIST (applies to *every* group above)
###############################################################
# – private Shopify areas –
Disallow: /admin
Disallow: /account
Disallow: /cart
Disallow: /carts
Disallow: /checkout
Disallow: /checkouts/
Disallow: /orders
# – low-value internal endpoints –
Disallow: /search
Disallow: /policies/
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /recommendations/products
Disallow: /*/recommendations/products
# – duplicate / parameterised URLs –
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /*/blogs/*+*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /*/*?*ls=*&ls=*
###############################################################
# 5. FALLBACK → ANY UNLISTED BOT
###############################################################
User-agent: *
Disallow: /admin
Disallow: /account
Disallow: /cart
Disallow: /carts
Disallow: /checkout
Disallow: /checkouts/
Disallow: /orders
Disallow: /search
Crawl-delay: 10
| Položaj | Fraza | Stran | Izrezek |
|---|---|---|---|
| 9 | /blogs/articles/what... | ||
| 35 | /products/sensibo-pu... |