List of bots hitting our test servers
summercms opened this issue · comments
Just going to list them all here and then create pr's for them.
Evc-batch
UA:
Mozilla/5.0 (compatible; evc-batch/2.0)
Mozilla/5.0 (compatible; evc-batch/2.0.20170930090616)
Mozilla/5.0 (compatible; evc-batch/2.0.20170925164652)
etc.
Link: http://www.eventures.vc/
VelenPublicWebCrawler
UA:
Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
Link: https://velen.io
Datanyze
UA:
Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Link: https://www.datanyze.com/
Iodc
Old UA:
Mozilla/5.0 (iodc; odysseus 24842-138-041020155614-449; +https://iodc.co.uk)
Mozilla/5.0 (iodc; odysseus 3352-131-011119113358-349; +https://iodc.co.uk)
Latest UA:
Mozilla/5.0 (compatible; IODC-Odysseus Survey 46182-100-271115114504-101; +https://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 23490-104-250115142110-102; +https://iodc.co.uk)
Link: https://iodc.co.uk
Barkrowler
UA:
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
Link: https://beta.babbar.tech/crawler
Adsbot
UA:
Mozilla/5.0 (compatible; Adsbot/3.1)
Link: https://adsbot.co/
Petal Bot
UA:
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)
Link: https://aspiegel.com/petalbot
ZmEu
ZmEu is a computer vulnerability scanner which searches for web servers that are open to attack through the phpMyAdmin program, It also attempts to guess SSH passwords through brute-force methods, and leaves a persistent backdoor. ZmEu is a bot that tries to find vulnerabilities in phpMyAdmin (usually looks for phpmyadmin/scripts/setup.php file) and other web applications.ZmEu appears to be a security tool used for discovering security holes in in version 2.x.x of phpMyAdmin, a web based MySQL database manager.
UA:
Made by ZmEu @ WhiteHat Team – http://www.whitehat.ro
ZmEu
Link: https://en.wikipedia.org/wiki/ZmEu_%28vulnerability_scanner%29
Link: https://ensourced.wordpress.com/2011/02/25/zmeu-attacks-some-basic-forensic/
Xenu Link Sleuth
Xenu, or Xenu's Link Sleuth, is a computer program that checks websites for broken hyperlinks.
Old UA:
Xenu Link Sleuth 1.2i
Xenu Link Sleuth 1.2h
Xenu Link Sleuth 1.2g
Xenu Link Sleuth 1.2f
Xenu Link Sleuth 1.2e
Xenu Link Sleuth 1.2d
Xenu Link Sleuth 1.2c
Xenu Link Sleuth 1.2b
Current UA:
Xenu Link Sleuth/1.3.7
Xenu Link Sleuth/1.3.8
Xenu Link Sleuth/1.3.9 beta
Link: https://en.wikipedia.org/wiki/Xenu%27s_Link_Sleuth
MojeekBot
MojeekBot is the web crawler for the Mojeek search engine.
Old UA:
Mozilla/5.0 (compatible; MojeekBot/0.10; +https://www.mojeek.com/bot.html)
Latest UA:
Mozilla/5.0 (compatible; MojeekBot/0.9; +https://www.mojeek.com/bot.html)
Reverse DNS:
crawl-5-102-173-71.mojeek.com
Link: https://www.mojeek.com/bot.html
Clarabot
The Clarabot the Clarabot Company, services and products in a separate database and a search engine.
Clarabot Zrt. Is a wholly Hungarian-owned IT company founded in 2018, whose main activity is the mapping and publication of the links (so-called backlinks) of the websites present on the Internet. Clarabot Zrt. Considers the protection of the personal data of users and employees of the company to be extremely important. Clarabot Zrt. Collects all the information on Clarabot.com with uniquely coded search robots, according to its own system, which it stores on its own servers, thus avoiding the possibility of data manipulation.
The system collects data globally. Its search method differs from traditional search engines , so the result obtained is also different, which provides users with useful information for marketing, SEO or research purposes. By entering the name of the website (the name of the domain or domain ) on Clarabot.com , its occurrence on other sites will be listed as “links found on the internet”. Clarabot is well-established among search engines because it can show links between web pages.
UA:
Mozilla/5.0 (compatible; Clarabot/1.4; +http://www.clarabot.info/bots)
Link: http://www.clarabot.com/
YisouSpider
Shenma search spider YisouSpider in the past few years on the Internet, it can be said that there are complaints. Many websites are crawled too frequently and the server is paralyzed.
Shenma and Alibaba joined together to create this "mobile search engine" and YisouSpider is there crawler.
Old UA:
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e YisouSpider/5.0 Safari/602.1
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
New UA:
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
YisouSpider
Link: https://m.sm.cn/
Gofeed
The gofeed library is a robust feed parser that supports parsing both RSS, Atom and JSON feeds. The library provides a universal gofeed.Parser that will parse and convert all feed types into a hybrid gofeed.Feed model. You also have the option of utilizing the feed specific atom.Parser or rss.Parser or json.Parser parsers which generate atom.Feed, rss.Feed and json.Feed respectively.
UA:
Gofeed/1.0
Link: https://github.com/mmcdole/gofeed
British Library
The British Library and other legal deposit libraries are entitled to copy UK-published material from the internet for archiving under legal deposit.
UA:
bl.uk_lddc_bot/3.4.0-20200518 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
Link: https://www.bl.uk/legal-deposit/web-archiving
netEstate NE Crawler
German search engine.
UA:
netEstate NE Crawler (+http://www.website-datenbank.de/)
Link: https://www.website-datenbank.de/
Serp Stat
serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)
Link: https://serpstatbot.com/