Lo sapete che il vostro sito è visitato costantemente da BOT e SCRAPER?
Questi due simpatici giocherelloni accedono al nostro sito per esaminarlo e per ciucciare risorse.
Si infatti potrebbe succedere che per colpa di questi due simpatici rompini il vostro sito esaurisca le risorse della CPU, il piu delle volte crashando.
Questi BOT malevoli e gli Scraper sono i peggiori nemici del processore del nostro server.
Cosa fare?
Se siete su server LINUX e APACHE potrete utilizzare il file .htaccess
Potrete crearne uno nuovo o utilizzarne uno già esistente nella root del vostro sito web.
Copiate e incollate il codice qui sotto che contiene una lista aggiornata di bot malevoli e scraper al agosto 2018.
# Block Bad Bots & Scrapers
SetEnvIfNoCase User-Agent "Aboundex" bad_bot
SetEnvIfNoCase User-Agent "80legs" bad_bot
SetEnvIfNoCase User-Agent "360Spider" bad_bot
SetEnvIfNoCase User-Agent "^Java" bad_bot
SetEnvIfNoCase User-Agent "^Cogentbot" bad_bot
SetEnvIfNoCase User-Agent "^Alexibot" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^attach" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot" bad_bot
SetEnvIfNoCase User-Agent "^BackWeb" bad_bot
SetEnvIfNoCase User-Agent "Bandit" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot
SetEnvIfNoCase User-Agent "^BlackWidow" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "Buddy" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^ChinaClaw" bad_bot
SetEnvIfNoCase User-Agent "Collector" bad_bot
SetEnvIfNoCase User-Agent "Copier" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Custo" bad_bot
SetEnvIfNoCase User-Agent "^AIBOT" bad_bot
SetEnvIfNoCase User-Agent "^DISCo" bad_bot
SetEnvIfNoCase User-Agent "^DIIbot" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly" bad_bot
SetEnvIfNoCase User-Agent "^Drip" bad_bot
SetEnvIfNoCase User-Agent "^eCatch" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL" bad_bot
SetEnvIfNoCase User-Agent "^ebingbong" bad_bot
SetEnvIfNoCase User-Agent "^EirGrabber" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Exabot" bad_bot
SetEnvIfNoCase User-Agent "^Express\ WebPictures" bad_bot
SetEnvIfNoCase User-Agent "Extractor" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^flunky" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage" bad_bot
SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase User-Agent "^gotit" bad_bot
SetEnvIfNoCase User-Agent "^GrabNet" bad_bot
SetEnvIfNoCase User-Agent "^Grafula" bad_bot
SetEnvIfNoCase User-Agent "^Harvest" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^HMView" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Stripper" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^InfoTekies" bad_bot
SetEnvIfNoCase User-Agent "^Intelliseek" bad_bot
SetEnvIfNoCase User-Agent "^InterGET" bad_bot
SetEnvIfNoCase User-Agent "^Internet\ Ninja" bad_bot
SetEnvIfNoCase User-Agent "^Iria" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^JetCar" bad_bot
SetEnvIfNoCase User-Agent "^JOC" bad_bot
SetEnvIfNoCase User-Agent "^JustView" bad_bot
SetEnvIfNoCase User-Agent "^Jyxobot" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin.Spider" bad_bot
SetEnvIfNoCase User-Agent "^Keyword.Density" bad_bot
SetEnvIfNoCase User-Agent "^larbin" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^lftp" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^likse" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" bad_bot
SetEnvIfNoCase User-Agent "^LNSpiderguy" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^LWP::Simple" bad_bot
SetEnvIfNoCase User-Agent "^Magnet" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch" bad_bot
SetEnvIfNoCase User-Agent "^Mass\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "^Mata.Hari" bad_bot
SetEnvIfNoCase User-Agent "^Memo" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft.URL" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase User-Agent "^MIDown\ tool" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Mirror" bad_bot
SetEnvIfNoCase User-Agent "^Missigua\ Locator" bad_bot
SetEnvIfNoCase User-Agent "^Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/3.Mozilla/2.01" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla.*NEWT" bad_bot
SetEnvIfNoCase User-Agent "^NAMEPROTECT" bad_bot
SetEnvIfNoCase User-Agent "^Navroad" bad_bot
SetEnvIfNoCase User-Agent "^NearSite" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Netcraft" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^NetSpider" bad_bot
SetEnvIfNoCase User-Agent "^Net\ Vampire" bad_bot
SetEnvIfNoCase User-Agent "^NetZIP" bad_bot
SetEnvIfNoCase User-Agent "^NextGenSearchBot" bad_bot
SetEnvIfNoCase User-Agent "^NG" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^niki-bot" bad_bot
SetEnvIfNoCase User-Agent "^NimbleCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Ninja" bad_bot
SetEnvIfNoCase User-Agent "^NPbot" bad_bot
SetEnvIfNoCase User-Agent "^Octopus" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Navigator" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot" bad_bot
SetEnvIfNoCase User-Agent "^PageGrabber" bad_bot
SetEnvIfNoCase User-Agent "^Papa\ Foto" bad_bot
SetEnvIfNoCase User-Agent "^pavuk" bad_bot
SetEnvIfNoCase User-Agent "^pcBrowser" bad_bot
SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" bad_bot
SetEnvIfNoCase User-Agent "^Pockey" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^psbot" bad_bot
SetEnvIfNoCase User-Agent "^Pump" bad_bot
SetEnvIfNoCase User-Agent "^QueryN.Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload" bad_bot
SetEnvIfNoCase User-Agent "Reaper" bad_bot
SetEnvIfNoCase User-Agent "Recorder" bad_bot
SetEnvIfNoCase User-Agent "^ReGet" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "Siphon" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload" bad_bot
SetEnvIfNoCase User-Agent "^Snake" bad_bot
SetEnvIfNoCase User-Agent "^Snapbot" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy" bad_bot
SetEnvIfNoCase User-Agent "^sogou" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm" bad_bot
SetEnvIfNoCase User-Agent "Stripper" bad_bot
SetEnvIfNoCase User-Agent "Sucker" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot" bad_bot
SetEnvIfNoCase User-Agent "^SuperHTTP" bad_bot
SetEnvIfNoCase User-Agent "^Surfbot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^tAkeOut" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" bad_bot
SetEnvIfNoCase User-Agent "^The.Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^Titan" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot" bad_bot
SetEnvIfNoCase User-Agent "^URLy.Warning" bad_bot
SetEnvIfNoCase User-Agent "^Vacuum" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^VoidEYE" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch" bad_bot
SetEnvIfNoCase User-Agent "^WebGo\ IS" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector" bad_bot
SetEnvIfNoCase User-Agent "^WebLeacher" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^Website\ eXtractor" bad_bot
SetEnvIfNoCase User-Agent "^Website\ Quester" bad_bot
SetEnvIfNoCase User-Agent "^Webster" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^WebWhacker" bad_bot
SetEnvIfNoCase User-Agent "^WebZIP" bad_bot
SetEnvIfNoCase User-Agent "Whacker" bad_bot
SetEnvIfNoCase User-Agent "^Widow" bad_bot
SetEnvIfNoCase User-Agent "^WISENutbot" bad_bot
SetEnvIfNoCase User-Agent "^WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon" bad_bot
SetEnvIfNoCase User-Agent "^Xenu" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "ZmEu" bad_bot
SetEnvIfNoCase User-Agent "^Zyborg" bad_bot
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase User-Agent "SemrushBot" bad_bot
SetEnvIfNoCase User-Agent "istellabot" bad_bot
# Vulnerability Scanners
SetEnvIfNoCase User-Agent "Acunetix" bad_bot
SetEnvIfNoCase User-Agent "FHscan" bad_bot
# Aggressive Chinese Search Engine
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
# Aggressive Russian Search Engine
SetEnvIfNoCase User-Agent "Yandex" bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
# Cyveillance
deny from 38.100.19.8/29
deny from 38.100.21.0/24
deny from 38.100.41.64/26
deny from 38.105.71.0/25
deny from 38.105.83.0/27
deny from 38.112.21.140/30
deny from 38.118.42.32/29
deny from 65.213.208.128/27
deny from 65.222.176.96/27
deny from 65.222.185.72/29
Deny from env=bad_bot
</Limit>FATTO! Se volete potrete anche creare un file robots.txt e inserirlo nella root del vostro sito web con il seguente codice
User-agent: 1 2 3 Submit PRO
disallow: /
User-agent: 200PleaseBot
disallow: /
User-agent: 2ADAMbot
disallow: /
User-agent: 2ADAMbot/1.0
disallow: /
User-agent: 360Spider
disallow: /
user-agent: Abonti
disallow: /
user-agent: Abonti/0.92
disallow: /
user-agent: abot v1.0
disallow: /
user-agent: aboutthedomain
disallow: /
user-agent: Add Catalog
disallow: /
user-agent: Add Catalog/2.1
disallow: /
user-agent: AdvBot
disallow: /
user-agent: AdvBot/2.0
disallow: /
user-agent: AhrefsBot
disallow: /
user-agent: Ahrefs-Bot
disallow: /
user-agent: AhrefsBot/1.0
disallow: /
user-agent: Ahrefs-Bot/1.0
disallow: /
user-agent: Ahrefs-Bot/2.0
disallow: /
user-agent: Ahrefs-Bot/3.0
disallow: /
user-agent: Ahrefs-Bot/4.0
disallow: /
user-agent: Ahrefs-Bot/5.0
disallow: /
user-agent: aiHitBot
disallow: /
user-agent: aiHitBot/2.9
disallow: /
user-agent: Anonymous/0.0
disallow: /
user-agent: Arachnida
disallow: /
user-agent: Associative Spider
disallow: /
User-agent: Baiduspider
disallow: /
User-agent: Baidu Spider
disallow: /
User-agent: Battleztar Bazinga
disallow: /
User-agent: Battleztar Bazinga/0.01
disallow: /
User-agent: BDFetch
disallow: /
User-agent: betaBot
disallow: /
User-agent: bieshu
disallow: /
User-agent: Bigli SEO
disallow: /
User-agent: Blackboard Safeassign
disallow: /
User-agent: Blazer 1.0
disallow: /
User-agent: BLEXBot
disallow: /
User-agent: BLEXBot/1.0
disallow: /
User-agent: BLP_bbot
disallow: /
User-agent: BLP_bbot/0.1
disallow: /
User-agent: BOIA-Accessibility-Agent/PR 1.0
disallow: /
User-agent: BOT for JCE
disallow: /
User-agent: BOT/0.1 (BOT for JCE)
disallow: /
User-agent: BPImageWalker
disallow: /
User-agent: BPImageWalker/2.0
disallow: /
User-agent: BUbiNG
disallow: /
User-agent: BuiBui-Bot
disallow: /
User-agent: BuiBui-Bot/1.0
disallow: /
User-agent: ca-crawler
disallow: /
User-agent: ca-crawler/1.0
disallow: /
User-agent: CakePHP
disallow: /
User-agent: Calypso v/0.01
disallow: /
User-agent: Calypso
disallow: /
User-agent: CB/Nutch-1.7
disallow: /
User-agent: CCBot
disallow: /
User-agent: CCBot/2.0
disallow: /
User-agent: Checkbot
disallow: /
User-agent: checkgzipcompression.com
disallow: /
User-agent: chushou
disallow: /
User-agent: CloudServerMarketSpider
disallow: /
User-agent: CloudServerMarketSpider/1.0
disallow: /
User-agent: Clushbot/3.x-BinaryFury
disallow: /
User-agent: CMS Crawler
disallow: /
User-agent: CMS Crawler: http://www.cmscrawler.com
disallow: /
User-agent: coccoc
disallow: /
User-agent: CoinCornerBot
disallow: /
User-agent: CoinCornerBot/1.1
disallow: /
User-agent: Copyscape
disallow: /
User-agent: crawler4j
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.0
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.1
disallow: /
User-agent: CRAZYWEBCRAWLER 0.9.7
disallow: /
User-agent: CrazyWebCrawler
disallow: /
User-agent: CrazyWebCrawler-Spider
disallow: /
User-agent: Crowsnest
disallow: /
User-agent: Crowsnest/0.5
disallow: /
User-agent: Curious George - www.analyticsseo.com/crawler
disallow: /
User-agent: Curious George
disallow: /
User-agent: cuwhois
disallow: /
User-agent: cuwhois/1.0
disallow: /
User-agent: dahoms
disallow: /
User-agent: datagnionbot
disallow: /
User-agent: DeuSu/5.0.2
disallow: /
User-agent: Digincore
disallow: /
User-agent: Digincore bot
disallow: /
User-agent: Dispatch/0.11.0
disallow: /
User-agent: Domain Re-Animator Bot
disallow: /
User-agent: DomainAppender /1.0
disallow: /
User-agent: DomainAppender
disallow: /
User-agent: DomainCrawler/3.0
disallow: /
User-agent: DomainSigmaCrawler
disallow: /
User-agent: DomainSigmaCrawler/0.1
disallow: /
User-agent: Domnutch
disallow: /
User-agent: Domnutch-Bot
disallow: /
User-agent: Domnutch-Bot/Nutch
disallow: /
User-agent: Domnutch-Bot/Nutch-1.0
disallow: /
User-agent: dotbot
disallow: /
User-agent: ECCP/1.2.1
disallow: /
User-agent: eCommerceBot
disallow: /
User-agent: enlle punto com/Nutch-1.9
disallow: /
User-agent: EPiServer Link Checker
disallow: /
User-agent: EuripBot
disallow: /
User-agent: EuripBot/2.0
disallow: /
User-agent: evc/2.0
disallow: /
User-agent: evc-batch
disallow: /
User-agent: evc-batch/2.0
disallow: /
User-agent: Express WebPictures
disallow: /
User-agent: Faraday v0.8.8
disallow: /
User-agent: Faraday
disallow: /
User-agent: Findxbot
disallow: /
User-agent: Findxbot/1.0
disallow: /
User-agent: Flamingo_SearchEngine
disallow: /
User-agent: Flipboard Robot
disallow: /
User-agent: GetProxi.es-bot
disallow: /
User-agent: GetProxi.es-bot/1.1
disallow: /
User-agent: GigablastOpenSource
disallow: /
User-agent: GigablastOpenSource/1.0
disallow: /
User-agent: Girafabot
disallow: /
User-agent: Gluten Free Crawler
disallow: /
User-agent: Gluten Free Crawler/1.0
disallow: /
User-agent: GriffinBot
disallow: /
User-agent: GrifinBot/0.01
disallow: /
User-agent: GWPImages
disallow: /
User-agent: GWPImages/1.0
disallow: /
User-agent: Haiula
disallow: /
User-agent: Haiula/1.4
disallow: /
User-agent: HaosouSpider
disallow: /
User-agent: Hivemind
disallow: /
User-agent: HostHarvest
disallow: /
User-agent: HostHarvest/0.4.28
disallow: /
User-agent: HRCrawler
disallow: /
User-agent: HRCrawler/2.0
disallow: /
User-agent: http://git.io/tl_S2w
disallow: /
User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
disallow: /
User-agent: HubSpot Links Crawler 1.0
disallow: /
User-agent: HubSpot Webcrawler
disallow: /
User-agent: HubSpot
disallow: /
User-agent: hunchan
disallow: /
User-agent: HyperCrawl
disallow: /
User-agent: HyperCrawl/0.2
disallow: /
User-agent: ICAP-IOD
disallow: /
User-agent: ICC-Crawler
disallow: /
User-agent: ICC-Crawler/2.0
disallow: /
User-agent: Ichiro Robot
disallow: /
User-agent: image.coccoc/1.0
disallow: /
User-agent: Image2play
disallow: /
User-agent: Image2play/0.1
disallow: /
User-agent: Indy Library
disallow: /
User-agent: InsightsCollector
disallow: /
User-agent: InsightsCollector/0.1
disallow: /
User-agent: InsightsCollector/0.1beta
disallow: /
User-agent: integrity/5
disallow: /
User-agent: InterNaetBoten
disallow: /
User-agent: InterNaetBoten/0.99
disallow: /
User-agent: IRL Crawler
disallow: /
User-agent: James BOT - WebCrawler
disallow: /
User-agent: James BOT
disallow: /
User-agent: JamesBOT
disallow: /
User-agent: JetBrains 5.0
disallow: /
User-agent: JetBrains
disallow: /
User-agent: Kraken
disallow: /
User-agent: Kraken/0.1
disallow: /
User-agent: Kyoto-Tohoku-Crawler/v1
disallow: /
User-agent: larbin
disallow: /
User-agent: lechenie
disallow: /
User-agent: libwww-perl
disallow: /
User-agent: link checker
disallow: /
User-agent: Link/1.0
disallow: /
User-agent: linkCheck
disallow: /
User-agent: linkCheckV3.0
disallow: /
User-agent: Linkdex
disallow: /
User-agent: linkdex.com/v2.0
disallow: /
User-agent: linkdex.com/v2.1
disallow: /
User-agent: LinkdexBot
disallow: /
User-agent: linkdexbot/2.0
disallow: /
User-agent: linkdexbot/2.1
disallow: /
User-agent: linkdexbot-mobile/2.1
disallow: /
User-agent: LinkpadBot
disallow: /
User-agent: LinkpadBot/1.06
disallow: /
User-agent: LinqiaScrapeBot
disallow: /
User-agent: LinqiaScrapeBot/1.0
disallow: /
User-agent: Lipperhey SEO Service
disallow: /
User-agent: Lipperhey
disallow: /
User-agent: Lipperh