#
# ROBOTS.TXT - WEB CRAWLER CONTROL FILE
#
# PRIMARY FUNCTION:
# This file tells search engine crawlers (bots) which pages they can and cannot access on your website.
# It's the first file most crawlers check when visiting your site, making it crucial for SEO and security.
#
# DIRECTIVE IMPORTANCE:
#
# 1. User-agent: * (CRITICAL)
# - Specifies which bots these rules apply to
# - "*" means ALL crawlers (Google, Bing, etc.)
# - Can be specific: "User-agent: Googlebot"
#
# 2. Allow: / (HIGH IMPORTANCE)
# - Explicitly allows crawlers to access the root and all pages
# - Overrides any conflicting Disallow rules
# - Good practice for transparency
#
# 3. Sitemap: URL (HIGH IMPORTANCE)
# - Points crawlers to your sitemap location
# - Helps discovery of all your pages
# - Should be absolute URL
#
# 4. Disallow: /path/* (SECURITY & SEO)
# - Blocks crawlers from sensitive/irrelevant areas
# - Saves crawl budget for important content
# - Protects private sections
#
# BEST PRACTICES:
# - Keep file at root level (/robots.txt)
# - Use absolute URLs for Sitemap directive
# - Test with Google Search Console
# - Block admin, API, and development paths
# - Don't use for sensitive data (use authentication instead)
# - Update when site structure changes
#
# Allow all crawlers to access the website
User-agent: *
Allow: /
# Sitemap location - helps crawlers discover all pages efficiently
Sitemap: https://langflow.org/sitemap.xml
# Block crawler access to technical and private areas
# These paths don't provide value to search results and may contain sensitive data
# API endpoints - internal functionality, not for search indexing
Disallow: /api/*
# Next.js technical files - framework internals, not content
Disallow: /_next/*
# Static assets - already discoverable through page references
Disallow: /static/*
# Development and build files - technical artifacts
Disallow: /.next/*
Disallow: /node_modules/*
# Authentication & user management - privacy protection
Disallow: /auth/*
Disallow: /login/*
Disallow: /logout/*
# Download tracking and analytics - internal metrics
Disallow: /track/*
Disallow: /analytics/*
# Admin and configuration areas - sensitive operations
Disallow: /admin/*
Disallow: /config/*
# Temporary files and cache - technical storage
Disallow: /tmp/*
Disallow: /cache/*