User-Agent: Googlebot
Allow: /
Crawl-delay: 20
User-Agent: msnbot
Allow: /
Crawl-delay: 20
User-Agent: bot*
Disallow: /
Crawl-delay: 60
User-agent: Yandex # will be used by all Yandex robots
Allow: /
Crawl-delay: 240
User-agent: *
Disallow: /bot-trap/
Crawl-delay: 60
User-agent: ia_archiver
Disallow: /
| Placera | Fras | Sida | Utdrag |
|---|---|---|---|
| 5 | / | ||
| 11 | / | ||
| 35 | / | ||
| 43 | / | ||
| 90 | / |
| Placera | Fras | Sida | Utdrag |
|---|---|---|---|
| 39 | / |