Add missing and upgrade `Facebook` regex in bots list
summercms opened this issue · comments
Below are the UA being used by Facebook:
Facebook External Hit Bot
Old UA:
facebookexternalhit/1.1 ( http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
New UA:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Facebook Platform Bot
facebookplatform/1.0 (+http://developers.facebook.com)
Facebook Ad Review Bot
adreview/1.0
Facebook Cortex Bot
cortex/1.0
Facebook External UA Bot
facebookexternalua
Dead Bots
Facebook Vision Utils Bot (2014 - 2017)
visionutils/0.2
Face Bot (2014)
Facebot/1.0
Fake Bots
acebookexternalhit/1.0 ( http://www.facebook.com/externalhit_uatext.php)
acebookexternalhit/1.0 ( http://www.facebook.com/externalhit_uatext.php)
bandscraper ( facebookexternalhit/1.1 )
Facebook API PHP5 Client 1.1 (curl) 5.3.2
facebook FirePHP/0.3
Facebook share follower
facebook/danish.block.filter (sjc0.tdmsic.org - Red Hat)
FacebookConnect
facebookexternalhit
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) AppEngine-Google; (+http://code.google.com/appengine; appid: rtweetme)
facebookexternalhit/1.0 (http://www.facebook.com/externalhit_uatext.php),gzip(gfe) (via translate.google.com)
facebookexternalhit/1.0; http://www.facebook.com/externalhit_uatext.php
facebookexternalhit/1.1 (http://www.facebook.com + / Externalhit_uatext.php),gzip(gfe) (via translate.google.com)
facebookexternalhit/1.1; Mozilla/5.0 (compatible; WorksOgCrawler/1.0.0;)
facebookfriendfinder/1.1 (+http://www.facebook.com)
facebookhits
facebookscraper
facebookscraper/1.0
facebookscraper/1.0 (+http://www.facebook.com/sharescraper_help.php)
facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)
facebookscraper/1.0(+http://www.facebook.com/sharescraper_help.php)
Mozilla/4.0 (compatible; MSIE 8.0; facebook bot; Maxthon 2.0)
Mozilla/5.0 (compatible; facebook bot)
Mozilla/5.0 (compatible; Facebot 1.0; https://developers.facebook.com/docs/sharing/webmasters/crawler)
Mozilla/5.0 (compatible; The Lounge IRC Client; +https://github.com/thelounge/thelounge) facebookexternalhit/1.1 Twitterbot/1.0
Mozilla/5.0 (Facebookbot) Gecko/20100101 Firefox/5.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Ant.com Toolbar 1.5 facebookscraper/1.0(+http://www.facebook.com/sharescraper_help.php)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.6) Gecko/20091201 Ant.com Toolbar 1.5 http://www.facebook.com /bot.html
Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (compatible; facebook bot) Mozilla/20.2463; U; en) Presto/2.5.25
Samsung-SPHM320 Infraware-Polaris/5Facebook API PHP5 Client 1.1 (curl) 5.3.2.01 MMP/2.0
acebookexternalhit/1.0 ( http://www.facebook.com/externalhit_uatext.php) UNTRUSTED/1.0
acebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
acebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Facebook 4020.0 (BlackBerry; BlackBerry 9900; en_US)
Facebook 4020.0 (iPhone; iPhone OS 5.0.1; en_US)
Facebook/3330 CFNetwork/485.12.30 Darwin/10.4.0
Facebook/4100.0 CFNetwork/548.1.4 Darwin/11.0.0
Facebook/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Linux Mint/10 (Julia) Firefox/3.6.15
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) FirePHP/0.5
facebookexternalhit/1.1 (compatible;)
facebookexternalhit/1.1; kakaotalk-scrap/1.0; +https://devtalk.kakao.com/t/scrap/33984
facebookexternalhit/1.1;kakaotalk-scrap/1.0;
facebookexternalhit/1.1;kakaotalk-scrap/1.0; +https://devtalk.kakao.com/t/scrap/33984
facebookexternalhit/1.1;line-poker/1.0
http://www.cnn.com/WORLD/ http://www.cnn.com/ASIA/ http://www.paypal.com/ http://www.myspace.com/ http://www.x-casinos.com/ http://www.facebook.com/ http://www.bitcomet.com/ http://www.microsof
https://free.facebook.com/groups/500481500472426/
Mozilla/4.0 (compatible; MSIE 9.0; facebook FirePHP/0.3; Maxthon 2.0)
Mozilla/4.0 (compatible; MSIE 9.0; facebook; Maxthon 2.0)
Mozilla/4.6 (compatible; MSIE 6.4; Windows NT 4.4) http://www.cnn.com/WORLD/ http://www.cnn.com/ASIA/ https://www.paypal.com/ http://www.myspace.com/ http://x-casinos.com/ http://www.facebook.com
Mozilla/5.0 (compatible; FacebookStatistics/1; +http://www.facebook-statistics.com/)
Mozilla/5.0 (facebookexternalhit/1.0; +http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)2.0.1) Gecko/20100101 Firefox/4.0.1
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.3 facebook
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) ( .NET CLR 3.5.30729)
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.2) Gecko/20100316 facebook/3.6.2
Mozilla/5.0 Facebook says they have found the A¢a‚¬A“authentic contractA¢a‚¬A� between Paul Ceglia and Mark Zuckerberg that disproves CegliaA¢a‚¬a„¢s claim that he is entitled to ove
NokiaX2-01/5.0 (08.63) Profile/MIDP-2.1 Configuration/CLDC-1.1 Opera/9.80 (J2ME/MIDP; Opera Mini/9.80 (Windows NT 6.1; U; Edition Facebook; en) Presto/24.741; U; en) Presto/2.5.25 Version/10.54 UNTRUS
Opera/9.80 (J2ME/MIDP; Opera Mini/1.34 facebookexternalhit/23.411; U; en) Presto/2.5.25 Version/10.54
Opera/9.80 (J2ME/MIDP; Opera Mini/1.34 facebookexternalhit/24.760; U; en) Presto/2.5.25 Version/10.54
Opera/9.80 (J2ME/MIDP; Opera Mini/2.1 facebook/22.414; U; en) Presto/2.5.25 Version/10.54
Opera/9.80 (J2ME/MIDP; Opera Mini/4.0 facebookexternalhit/22.414; U; en) Presto/2.5.25 Version/10.54
Opera/9.80 (J2ME/MIDP; Opera Mini/9.80 (Windows NT 6.1; U; Edition Facebook; en) Presto/24.741; U; en) Presto/2.5.25 Version/10.54
Opera/9.80 (Windows NT 5.1; U; Edition Facebook; pl) Presto/2.9.168 Version/11.51
Opera/9.80 (Windows NT 6.1; U; Edition Facebook; en) Presto/2.6.30 Version/10.62
UCWEB/2.0 (MIDP-2.0; U; Adr 4.0.4; free.facebook.com; ST15i) U2/1.0.0 UCBrowser/9.8.0.534 U2/1.0.0 Mobile
Repo allowing hackers:
[ 'name' => 'Facebook', 'id' => 'facebook', 'regexp' => '/facebookscraper\/([0-9.]*)/u' ],
[ 'name' => 'Facebook', 'id' => 'facebook', 'regexp' => '/FacebookSecurity\/([0-9.]*)/u' ],
Example 1
facebookscraper/1.0 (+http://www.facebook.com/sharescraper_help.php)
The link http://www.facebook.com/sharescraper_help.php is not real and will give you a 404 error!
Example 2
FacebookSecurity/2.1 ([www.FacebookSecurity.com])
The link www.FacebookSecurity.com is a phishing attack! That website does NOT belong to Facebook! DO NOT GO TO THAT WEBSITE.
Testing Link: http://thadafinser.github.io/UserAgentParserComparison/v3/user-agent-detail/88/9d/889db907-52b5-4732-be85-c1fb70362a5c.html
Will remove these two from repo and add to fake bots!