Blackweb
Blackweb es un proyecto que pretende recopilar la mayor cantidad de listas negras públicas de dominios (para bloquear porno, descargas, drogas, malware, spyware, trackers, bots, redes sociales, warez, venta de armas, etc), con el objeto de unificarlas y hacerlas compatibles con Squid-Cache (Tested in v3.5.x). Para lograrlo, realizamos una depuración de urls, para evitar duplicados, dominios inválidos (validación de ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, etc), y un filtrado con listas blancas de dominios (falsos positivos, como google, hotmail, yahoo, etc), para obtener una mega ACL, optimizada para Squid-Cache, libre de "overlapping domains" (e.g: "ERROR: '.sub.example.com' is a subdomain of '.example.com'").
Blackweb is a project that aims to collect as many public domain blacklists (to block porn, downloads, drugs, malware, spyware, trackers, Bots, social networks, warez, arms sales, etc.), in order to unify them and make them compatible with Squid-Cache (Tested in v3.5.x ). To do this, we perform a debugging of urls, to avoid duplicates, invalid domains (validation, ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, etc), and filter with white lists of domains (false positives such as google , hotmail, yahoo, etc.), to get a mega ACL, optimized for Squid-Cache, free of overlapping domains (eg: "ERROR: '.sub.example.com' is a subdomain of '.example.com'").
FICHA TECNICA / DATA SHEET
| ACL | Black Domains | Size |
|---|---|---|
| blackweb.txt | 2.083.183 | 50,5 MB |
DEPENDENCIAS / DEPENDENCIES
git squid bash tar zip wget subversion python
DESCARGA / DOWNLOAD
git clone --depth=1 https://github.com/maravento/blackweb.git
MODO DE USO / HOW TO USE
La ACL blackweb.txt ya viene optimizada para Squid-Cache. Descárguela con blackweb.sh. Por defecto, la ruta de blackweb.txt es /etc/acl. Ejemplo:
The ACL blackweb.txt is already optimized for Squid-Cache. Download it with blackweb.sh. By default, blackweb.txt path is /etc/acl. Example:
wget -q -N https://github.com/maravento/blackweb/raw/master/blackweb.sh && sudo chmod +x blackweb.sh && sudo ./blackweb.sh
ACTUALIZACIÓN / UPDATE
El script bwupdate.sh actualiza la ACL blackweb.txt, realizando la captura, depuración y limpieza de dominios, sin embargo puede generar conflíctos por errores en las fuentes, por tanto deberá depurarlos manualmente. Tenga en cuenta que este script consume gran cantidad de recursos de hardware durante el procesamiento y puede tomar mucho tiempo.
The bwupdate.sh script updates blackweb.txt ACL, doing the capture, debugging and cleaning of domains, however it can generate conflicts for errors in the sources, therefore you must manually debug conflicts. Keep in mind that this script consumes a lot of hardware resources during processing and it can take a long time.
wget -q -N https://github.com/maravento/blackweb/raw/master/bwupdate/bwupdate.sh && sudo chmod +x bwupdate.sh && sudo ./bwupdate.sh
Verifique la ejecución / Check execution
path: /var/log/syslog
Blackweb for Squid: Done 06/05/2017 15:47:14
Y verifique el contenido del archivo SquidError.txt en su escritorio para corregirlos / And check the contents of the SquidError.txt file on your desktop to fix them
Importante Antes de Usar / Important Before Use
- Antes de utilizar bwupdate.sh debe activar la regla en Squid-Cache / You must activate the rule in Squid-Cache before using bwupdate.sh
- La actualización debe ejecutarse en equipos de pruebas destinados para este propósito. Nunca en servidores en producción. / The update must run on test equipment designed for this purpose. Never on servers in production.
Extra ACLs
cloudsync.txt es una ACL contiene urls de servicios appcloud, diferentes a onedrive y gdrive (dropbox, pcloud, mega, etc.) y remoteurls.txt es una ACL contiene urls de servicios remotos (Teamviewer, Anydesk, Logmein, etc). Por defecto se excluyen de blackweb.txt. Para modificarlo, edite bwupdate.sh, y elimine las líneas referentes a estas ACLs (,cloudsync,remoteurls)
cloudsync.txt is an ACL containing appcloud service urls, different from onedrive and gdrive (dropbox, pcloud, mega, etc.) and remoteurls.txt is an ACL containing remote service urls (Teamviewer, Anydesk, Logmein, etc). By default they are excluded from blackweb.txt. To modify it, edit bwupdate.sh, and delete the lines referring to these ACLs (,cloudsync, remoteurls)
REGLA Squid-Cache / Squid-Cache RULE
Edit /etc/squid/squid.conf:
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
acl blackweb dstdomain -i "/etc/acl/blackweb.txt"
http_access deny blackweb
EDICIÓN / EDITION
Blackweb contiene millones de dominios bloqueados, por tanto, editarla manualmente puede ser frustrante. Entonces, si detecta un falso positivo, utilice la ACL whitedomains.txt y reporte el incidente, para corregirlo en la próxima actualización. Lo mismo aplica para dominios no incluidos en Blackweb, que quiera bloquear, puede agregarlos en blackdomains.
Blackweb contains million domains blocked therefore manually editing can be frustrating. Then, if it detects a false positive, use the ACL whitedomains.txt and report the incident to correct it in the next update. The same applies for domains not included in Blackweb, you want to block, you can add them in blackdomains.
acl whitedomains dstdomain -i "/etc/acl/whitedomains.txt"
acl blackdomains dstdomain -i "/etc/acl/blackdomains.txt"
acl blackweb dstdomain -i "/etc/acl/blackweb.txt"
http_access allow whitedomains
http_access deny blackdomains
http_access deny blackweb
blackdomains.txt contiene dominios no incluidos en Blackweb (e.g. .youtube.com .googlevideo.com, .ytimg.com) y whitedomains.txt contiene el subdominio accounts.youtube.com desde Feb 2014, Google utiliza el subdominio accounts.youtube.com para autenticar sus servicios.
blackdomains.txt contains domains not included in Blackweb (e.g. .youtube.com .googlevideo.com, .ytimg.com) and whitedomains.txt contains subdomain accounts.youtube.com since February 2014, Google uses the accounts subdomain .youtube.com to authenticate their services.
FUENTES / SOURCES
URLs Blacklists
Capitole - Direction du Système d'Information (DSI)
Nginx Ultimate Bad Bot Blocker
The Big List of Hacked Malware Web Sites
Matomo-org referrer-spam-blacklist
URLs Blacklists (Discontinued or Replaced)
Passwall SpamAssassin (Server Down since Dec 2016). Last Update
UrlBlacklist (Server Down since July 24, 2017)
Badd-Boyz-Hosts, Someonewhocares and KADhosts. Replaced by StevenBlack Hosts
Web Miner
Ransomware
External Debugging (URLs/TLDs Whitelists, Invalid Domains, etc)
ipv6-hosts (Partial)
O365IPAddresses (Partial)
University Domains and Names Data List
Internal Debugging (URLs/TLDs Whitelists, Invalid Domains, etc)
External Tools
Internal Tools
CONTRIBUCIONES / CONTRIBUTIONS
Agradecemos a todos aquellos que han contribuido a este proyecto. Los interesados pueden contribuir, enviándonos enlaces de nuevas listas, para ser incluidas en este proyecto / We thank all those who have contributed to this project. Those interested can contribute, sending us links of new lists, to be included in this project
DONACION / DONATE
BTC: 3M84UKpz8AwwPADiYGQjT9spPKCvbqm4Bc
LICENCIA / LICENCE
© 2018 Maravento Studio
EXENCION DE RESPONSABILIDAD / DISCLAIMER
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.