www.striebelundjohn.com
Inscrit
Date de mise à jour de l'analyse de la page:
2026/06/17 13:52:47
Date de la dernière mise à jour whois:
2026/08/16 17:03:11
Statut du domaine
Inscrit
Payé jusqu'à
16.08.2026
Disponible à partir de
15.09.2026
×
![Capture d'écran du site Web en plein écran]()
www.striebelundjohn.com Capture d'écran du site Web
Sauvegarde de domaine
Capture d'écran de la page principale de www.striebelundjohn.com, pris en charge 14.08.2026
34.45 MB
Taille des archives
83
Fichiers en archives
14.08.2026
Date de création
HTML, CSS, JS, Images
Types de fichiers
Score SEO
63.01%
92
Score obtenu
146
Note maximale
Informations principales
Titre: Homepage | STRIEBEL & JOHN
Description: inconnu
Mots-clés: inconnu
Encodage des pages:
utf-8
Taille du fichier de page: 62 KB
Informations sur le serveur
IP: 212.227.198.229
Emplacement: Germany,DE,,,51.2993,9.491,Europe/Berlin
Serveur HTTP: nginx
Codage: utf-8
Liste des balises méta
- charset: utf-8
- name: google-site-verification content: VBtMgHLFT310tltYcahcQt44LHdlUbewjkTFdrXQ_B0
- name: Generator content: Drupal 11 (https://www.drupal.org)
- name: MobileOptimized content: width
- name: HandheldFriendly content: true
- name: viewport content: width=device-width, initial-scale=1, shrink-to-fit=no
- equiv: X-UA-Compatible content: IE=edge,chrome=1
Liens internes
Liens externes
Informations Whois
domain_name: striebelundjohn.com
update_date: 2025-08-17T07:08:43Z
update_time: 1755414523
creation_date: 2006-08-16T08:56:28Z
creation_time: 1155718588
expiration_date: 2026-08-16T08:56:28Z
Données brutes Whois
Domain Name: STRIEBELUNDJOHN.COM
Registrar: IONOS SE
Domain Status: client transfer prohibited
Creation Date: 2006-08-16T08:56:28Z
Registry Expiry Date: 2026-08-16T08:56:28Z
Updated Date: 2025-08-17T07:08:43Z
Name Server: NS1017.UI-DNS.ORG
Name Server: NS1032.UI-DNS.COM
Name Server: NS1072.UI-DNS.DE
Name Server: NS1096.UI-DNS.BIZ
REGISTRAR Contact: IONOS SE
>>> Last update of RDAP database: 2026-08-16T17:03:11Z
Robots.txt
# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/robotstxt.html User-agent: * # CSS, JS, Images Allow: /core/*.css$ Allow: /core/*.css? Allow: /core/*.js$ Allow: /core/*.js? Allow: /core/*.gif Allow: /core/*.jpg Allow: /core/*.jpeg Allow: /core/*.png Allow: /core/*.svg Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /profiles/*.svg # Directories Disallow: /core/ Disallow: /profiles/ # Files Disallow: /README.txt Disallow: /web.config # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register Disallow: /user/password Disallow: /user/login Disallow: /user/logout Disallow: /media/oembed Disallow: /*/media/oembed # Paths (no clean URLs) Disallow: /index.php/admin/ Disallow: /index.php/comment/reply/ Disallow: /index.php/filter/tips Disallow: /index.php/node/add/ Disallow: /index.php/search/ Disallow: /index.php/user/password Disallow: /index.php/user/register Disallow: /index.php/user/login Disallow: /index.php/user/logout Disallow: /index.php/media/oembed Disallow: /index.php/*/media/oembed # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf
Audit SEO
Référencement technique
✓
Code de réponse
200
Statut 200 OK - la page se charge correctement.
✓
Codage des caractères
Page: utf-8, Header: utf-8
Codage des caractères cohérent entre le HTML et les en-têtes.
✓
Taille des pages
64126 bytes
Taille de page acceptable pour un chargement rapide.
✓
Ressources
38 total
Nombre modéré de ressources. Pensez à combiner des fichiers.
✓
Hreflang balises
1 hreflang tags
Balises Hreflang trouvées pour le référencement multilingue.
✓
Robots.txt
Exists
Fichier Robots.txt trouvé.
✓
Sitemap
Declared in robots.txt
Plan du site déclaré dans robots.txt.
✓
HTTPS
Yes
Connexion HTTPS sécurisée activée.
✓
Compression
gzip
Compression Gzip ou Zstd activée pour un chargement plus rapide.
✓
Mise en cache
max-age=900, public
En-têtes de contrôle de cache correctement définis.
✓
Vitesse des pages
4.38 ms
Excellente vitesse de chargement.
Référencement sur la page
!
Titre
Homepage | STRIEBEL & JOHN
Titre trop court. Développez jusqu'à 30 à 60 caractères pour un meilleur référencement.
!
Méta-description
Lenght:0
Ajoutez une méta description (100 à 160 caractères recommandés).
!
Titre H1
0 found - ""
Ajoutez exactement un titre H1 avec des mots-clés principaux.
!
Nombre de mots
460
Contenu un peu court. Étendez-vous à plus de 500 mots pour un référencement optimal.
!
Balise canonique
Ajoutez une balise canonique pour éviter les problèmes de contenu en double.
✓
Méta en double
[]
Aucune balise méta en double trouvée.
✓
Mots-clés
Les méta-mots-clés sont facultatifs (non utilisés par les principaux moteurs de recherche).
Contenu et UX
✓
Langue
de
Attribut de langue correctement défini.
✓
Images
15 total, 0 missing ALT
Toutes les images ont le texte ALT approprié.
✓
Fenêtre
width=device-width, initial-scale=1, shrink-to-fit=no
Balise méta Viewport correctement définie pour les appareils mobiles.
!
Ouvrir le graphique
Missing: og:title, og:description, og:image, og:url
Ajoutez les balises OpenGraph manquantes pour le partage sur les réseaux sociaux :og:title, og:description, og:image, og:url
!
Données structurées
0 JSON-LD scripts
Ajoutez des données structurées (JSON-LD) pour des extraits enrichis et un meilleur référencement.