www.striebelundjohn.com
Reģistrēts
Lapas analīzes atjaunināšanas datums:
2026/06/17 13:52:47
Pēdējā whois atjaunināšanas datums:
2026/08/16 17:03:11
Domēna statuss
Reģistrēts
Maksāja līdz
16.08.2026
Pieejams no
15.09.2026
×
![Pilnekrāna vietnes ekrānuzņēmums]()
www.striebelundjohn.com Vietnes ekrānuzņēmums
Domēna dublēšana
Galvenās lapas ekrānuzņēmums www.striebelundjohn.com, uzņemts 14.08.2026
34.45 MB
Arhīva lielums
83
Faili arhīvā
14.08.2026
Izveidošanas datums
HTML, CSS, JS, Images
Failu veidi
SEO rādītājs
63.01%
92
Rezultāts sasniegts
146
Maksimālais punktu skaits
Galvenā informācija
Nosaukums: Homepage | STRIEBEL & JOHN
Apraksts: nezināms
Atslēgvārdi: nezināms
Lapas kodējums:
utf-8
Lapas faila izmērs: 62 KB
Servera informācija
IP: 212.227.198.229
Atrašanās vieta: Germany,DE,,,51.2993,9.491,Europe/Berlin
HTTP serveris: nginx
Kodēšana: utf-8
Meta tagu saraksts
- charset: utf-8
- name: google-site-verification content: VBtMgHLFT310tltYcahcQt44LHdlUbewjkTFdrXQ_B0
- name: Generator content: Drupal 11 (https://www.drupal.org)
- name: MobileOptimized content: width
- name: HandheldFriendly content: true
- name: viewport content: width=device-width, initial-scale=1, shrink-to-fit=no
- equiv: X-UA-Compatible content: IE=edge,chrome=1
Iekšējās saites
Ārējās saites
Whois informācija
domain_name: striebelundjohn.com
update_date: 2025-08-17T07:08:43Z
update_time: 1755414523
creation_date: 2006-08-16T08:56:28Z
creation_time: 1155718588
expiration_date: 2026-08-16T08:56:28Z
Whois neapstrādātie dati
Domain Name: STRIEBELUNDJOHN.COM
Registrar: IONOS SE
Domain Status: client transfer prohibited
Creation Date: 2006-08-16T08:56:28Z
Registry Expiry Date: 2026-08-16T08:56:28Z
Updated Date: 2025-08-17T07:08:43Z
Name Server: NS1017.UI-DNS.ORG
Name Server: NS1032.UI-DNS.COM
Name Server: NS1072.UI-DNS.DE
Name Server: NS1096.UI-DNS.BIZ
REGISTRAR Contact: IONOS SE
>>> Last update of RDAP database: 2026-08-16T17:03:11Z
Robots.txt
# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/robotstxt.html User-agent: * # CSS, JS, Images Allow: /core/*.css$ Allow: /core/*.css? Allow: /core/*.js$ Allow: /core/*.js? Allow: /core/*.gif Allow: /core/*.jpg Allow: /core/*.jpeg Allow: /core/*.png Allow: /core/*.svg Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /profiles/*.svg # Directories Disallow: /core/ Disallow: /profiles/ # Files Disallow: /README.txt Disallow: /web.config # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register Disallow: /user/password Disallow: /user/login Disallow: /user/logout Disallow: /media/oembed Disallow: /*/media/oembed # Paths (no clean URLs) Disallow: /index.php/admin/ Disallow: /index.php/comment/reply/ Disallow: /index.php/filter/tips Disallow: /index.php/node/add/ Disallow: /index.php/search/ Disallow: /index.php/user/password Disallow: /index.php/user/register Disallow: /index.php/user/login Disallow: /index.php/user/logout Disallow: /index.php/media/oembed Disallow: /index.php/*/media/oembed # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf # Disallow some bots User-agent: 360Spider Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Aboundex Disallow: / User-agent: acoonbot Disallow: / User-agent: Adidxbot Disallow: / User-agent: admantx Disallow: / User-agent: AhrefsBot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: aihitbot Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: antibot Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arabot Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: ASPSimply Disallow: / User-agent: asterias Disallow: / User-agent: AttributorCorporationDMCABot Disallow: / User-agent: Avant+Browser+ Disallow: / User-agent: Axonize-bot Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: backlinkcrawler Disallow: / User-agent: BDCbot Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bender Disallow: / User-agent: betaBot Disallow: / User-agent: bibnum.bnf Disallow: / User-agent: biglotron Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: binlar_2.6.3 [email protected] Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: Black Hole Disallow: / User-agent: BLEXBot Disallow: / User-agent: blexbot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: Bork-edition Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: brainobot Disallow: / User-agent: bsalsa Disallow: / User-agent: BUbiNG Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: buzzbot Disallow: / User-agent: careerbot Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CC Metadata Scaper Disallow: / User-agent: ccbot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: changedetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: citeseerxbot Disallow: / User-agent: classroombot Disallow: / User-agent: coccoc Disallow: / User-agent: Commons-HttpClient Disallow: / User-agent: content crawler spider Disallow: / User-agent: convera Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CyberPatrol Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DataparkSearch/4.47 (+http://dataparksearch.org/bot) Disallow: / User-agent: DataparkSearch/4.xx (http://www.dataparksearch.org/) Disallow: / User-agent: DBLBot Disallow: / User-agent: dcbot Disallow: / User-agent: deadlinkchecker Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: Discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: domaincrawler Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: drupact Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ec2linkfinder Disallow: / User-agent: edisterbot Disallow: / User-agent: elisabot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Embedly Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: ezooms Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: FAST Enterprise Crawler Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: Fetch API Request Disallow: / User-agent: findlink Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: findthatfile Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: fluffy Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: fr-crawlerbinlarbinlar_2.6.3 [email protected] Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: g00g1e.net Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: gigablast Disallow: / User-agent: gigabot Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: gnam gnam spider Disallow: / User-agent: Go-http-clientphpcrawl Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: grub.org Disallow: / User-agent: gsa-crawler Disallow: / User-agent: gslfbot Disallow: / User-agent: GT::WWW Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http client Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: httpunit Disallow: / User-agent: HTTrack Disallow: / User-agent: httrack Disallow: / User-agent: humanlinks Disallow: / User-agent: ichiro Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: integromedb Disallow: / User-agent: intelium_bot Disallow: / User-agent: InterfaxScanBot Disallow: / User-agent: IOI Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Isidorus Disallow: / User-agent: IstellaBot Disallow: / User-agent: it2media-domain-crawler Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Java Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_07 Disallow: / User-agent: Java/1.6.0_11 Disallow: / User-agent: Java/1.6.0_13 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: jyxobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: lb-spider Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: LexxeBot Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: libwww Disallow: / User-agent: libwww-perl Disallow: / User-agent: libwww-perl/5.65 Disallow: / User-agent: Linguee Disallow: / User-agent: Linguee Bot Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lssbot Disallow: / User-agent: lssrocketcrawler Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Missigua Locator Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: mlbot Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: moget/2.1 ([email protected]) Disallow: / User-agent: MojeekBot\\/ Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: msrbot Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: netresearchserver Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: ngbot Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: niki-bot Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: page2rss Disallow: / User-agent: panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: PerMan Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: postrank Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: proximic Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: Python-urllib Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RetrevoPageAnalyzer Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: rogerbot Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SafeSearch microdata crawler Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrapy Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy/VERSION (+http://scrapy.org) Disallow: / User-agent: scrapybot Disallow: / User-agent: scribdbot Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: searchpreview Disallow: / User-agent: seekbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: seokicks-robot Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: seznambot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: SiteBot Disallow: / User-agent: sitebot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: siteexplorer.info Disallow: / User-agent: Sitemap: http://www.jaxcarpentry.com/sitemap.xml Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slack-ImgProxy Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: Snoopy Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: Sonic Disallow: / User-agent: Sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: spbot Disallow: / User-agent: Speedy Disallow: / User-agent: speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: summify Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: suzuran Disallow: / User-agent: Sysomos Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: T-Online Browser Disallow: / User-agent: T-Rex Disallow: / User-agent: tagoobot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: TeleportProMIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: teoma Disallow: / User-agent: Teoma Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: therarestparser Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: toplistbot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Trident Disallow: / User-agent: Trove Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot /1.0turingos Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: turnitinbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: twengabot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: UnisterBot Disallow: / User-agent: URI::Fetch Disallow: / User-agent: URL Control Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: urlappendbot Disallow: / User-agent: urllib Disallow: / User-agent: URLy Warning Disallow: / User-agent: UsineNouvelleCrawler Disallow: / User-agent: uw_cse_xwc Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCIVCI Disallow: / User-agent: ViolaBot Disallow: / User-agent: voilabot Disallow: / User-agent: Voyager Disallow: / User-agent: voyager Disallow: / User-agent: voyager/1.0 Disallow: / User-agent: voyager/2.0 Disallow: / User-agent: vscooter Disallow: / User-agent: VWBot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: wbsearchbot Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: webalta Disallow: / User-agent: WebarooBot Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: Webcollage Disallow: / User-agent: webcompanycrawler Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: webcrawler Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmon Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: Wells Search II Disallow: / User-agent: WEP Search Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: wget Disallow: / User-agent: WGet Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.10.2 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: wocbot Disallow: / User-agent: worio Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: yacybot Disallow: / User-agent: yanga Disallow: / User-agent: Yanga Disallow: / User-agent: Yasni Disallow: / User-agent: yeti Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: zeus Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-Agent: Amazon CloudFront Disallow: / Allow: /*.js Allow: /*.css Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.ttf Allow: /*.ico Allow: /*.pdf
SEO audits
Tehniskais SEO
✓
Atbildes kods
200
Statuss 200 OK - lapa tiek ielādēta pareizi.
✓
Rakstzīmju kodēšana
Page: utf-8, Header: utf-8
Rakstzīmju kodējums ir konsekvents starp HTML un galvenēm.
✓
Lapas izmērs
64126 bytes
Lapas izmērs ir pieņemams ātrai ielādei.
✓
Resursi
38 total
Mērens resursu skaits. Apsveriet iespēju apvienot failus.
✓
Hreflang tagi
1 hreflang tags
Hreflang tagi atrasti daudzvalodu SEO.
✓
Robots.txt
Exists
Atrasts fails Robots.txt.
✓
Sitemap
Declared in robots.txt
Vietnes karte deklarēta failā robots.txt.
✓
HTTPS
Yes
Ir iespējots drošs HTTPS savienojums.
✓
Saspiešana
gzip
Gzip vai Zstd saspiešana ir iespējota ātrākai ielādei.
✓
Kešatmiņa
max-age=900, public
Pareizi iestatītas kešatmiņas vadības galvenes.
✓
Lapas ātrums
4.38 ms
Lielisks iekraušanas ātrums.
SEO lapā
!
Nosaukums
Homepage | STRIEBEL & JOHN
Virsraksts ir pārāk īss. Paplašiniet līdz 30–60 rakstzīmēm, lai uzlabotu SEO.
!
Meta apraksts
Lenght:0
Pievienojiet metaaprakstu (ieteicams 100–160 rakstzīmes).
!
H1 Virsraksts
0 found - ""
Pievienojiet tieši vienu H1 virsrakstu ar primārajiem atslēgvārdiem.
!
Vārdu skaits
460
Saturs nedaudz īss. Paplašiniet līdz 500+ vārdiem, lai nodrošinātu optimālu SEO.
!
Kanoniskā atzīme
Pievienojiet kanonisko tagu, lai novērstu satura dublēšanos.
✓
Meta dublikāts
[]
Nav atrasti metatagu dublikāti.
✓
Atslēgvārdi
Meta atslēgvārdi nav obligāti (lielākās meklētājprogrammas neizmanto).
Saturs un UX
✓
Valoda
de
Pareizi iestatīts valodas atribūts.
✓
Attēli
15 total, 0 missing ALT
Visiem attēliem ir atbilstošs ALT teksts.
✓
Skatu logs
width=device-width, initial-scale=1, shrink-to-fit=no
Skatvietas metatags ir pareizi iestatīts mobilajām ierīcēm.
!
Atveriet grafiku
Missing: og:title, og:description, og:image, og:url
Pievienojiet trūkstošos OpenGraph tagus sociālo mediju kopīgošanai:og:title, og:description, og:image, og:url
!
Strukturētie dati
0 JSON-LD scripts
Pievienojiet strukturētus datus (JSON-LD), lai iegūtu bagātinātus fragmentus un uzlabotu SEO.