T145 / black-mirror

Blacklists and whitelists built by open code, so you know what goes into them.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

logo

๐ŸŒ“ Reflection | ๐Ÿ’ฟ Redundancy | โœ… Reliability


Aiming to promote security, safety, and sanity across the internet!


humane-tech release
Made with ๐Ÿ’– by

๐Ÿšš Deliverables

The whitelists are applied to all the blacklists!

List Name File Content Unique Entries File Size Mirror 1 Mirror 2
BLOCK_DOMAIN Domain entries 8,617,729 185M ๐Ÿ”— ๐Ÿ”—
BLOCK_IPV4 IPv4 addresses 694,652 9.4M ๐Ÿ”— ๐Ÿ”—
BLOCK_IPV6 IPv6 addresses 1,204 36K ๐Ÿ”— ๐Ÿ”—
BLOCK_CIDR4 IPv4 CIDR blocks 244,702 4.1M ๐Ÿ”— ๐Ÿ”—
BLOCK_CIDR6 IPv6 CIDR blocks 178 3.0K ๐Ÿ”— ๐Ÿ”—
ALLOW_DOMAIN Domain entries 274,875 5.0M ๐Ÿ”— ๐Ÿ”—
ALLOW_IPV4 IPv4 addresses 74,659 1021K ๐Ÿ”— ๐Ÿ”—
ALLOW_IPV6 IPv6 addresses 2,595 98K ๐Ÿ”— ๐Ÿ”—
ALLOW_CIDR4 IPv4 CIDR blocks 13,740 211K ๐Ÿ”— ๐Ÿ”—
ALLOW_CIDR6 IPv6 CIDR blocks 146,655 2.4M ๐Ÿ”— ๐Ÿ”—

Source Code

๐Ÿงฎ Checksum Evaluation

sha256sum CHECKSUMS.txt --strict

The output will print OK if the check was successful. Be sure to run the command in the same directory as the lists!

๐Ÿ™ Fetching GitHub Releases

Provided below are some examples to fetch release artifacts leveraging the GitHub API.

Get all build artifacts

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[].browser_download_url'

Get a build artifact & its checksum

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[] | select(.name | startswith("BLOCK_DOMAIN")).browser_download_url'

Get a single build artifact

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[] | select(.name | startswith("BLOCK_DOMAIN")) | select(.name | endswith(".txt")).browser_download_url'

๐Ÿ–‹๏ธ Manifesto

Please report any hosts that are wrongly blocked or sources that do not wholly align in an issue.

Defines the logic behind why a host is permitted or blocked. It has been written to reflect common ideologies across the blacklisting community and those specific to this project.

Complete author bias reflection is not intended, which is why it speaks from the Black Mirror person. Free thought exchange is encouraged, so feel free to open discussions about any points!

๐Ÿ“‹ Attributes

  1. Produced in builds specific to each host's syntax.
  2. Updates at 1:27 AM UTC.
  3. No excess or trailing whitespace.
  4. No lingering webscraper garbage.
  5. Lines are terminated with lf.
  6. No blank lines.
  7. No comments.

๐Ÿ› ๏ธ List Usage

Hosts File

mawk '{print "0.0.0.0 " $0}' BLOCK_DOMAIN.txt >>hosts
# mawk '{print ":: " $0}' BLOCK_DOMAIN.txt >>hosts
mawk '{print "0.0.0.0 " $0}' BLOCK_IPV4.txt >>hosts
mawk '{print ":: " $0}' BLOCK_IPV6.txt >>hosts

Many popular platforms such as OpenWRT, DDWRT, and Pi-hole use Dnsmasq as their choice TCP powerhouse. After inspecting many domain blocklists you'll inevitably run across a list in the dnsmasq.conf format. This list doesn't support it because you can use the addn-hosts parameter to add hosts in the list. Target a file that has the hosts in a format similar to the Hosts File format.

If you're using the RADVD daemon, prepend any listed hosts with ::. Otherwise, even if you have IPv6 support set up, prepend hosts with 0.0.0.0.

This has been tested across all the mentioned platforms using dig{6} on a small sample size and had each host null-routed successfully. DNSmasq's man page discusses configuration further, and DDWRT's ad blocking wiki page provides some examples.

Amazon EC2 DNS Resolver

Follow this guide to create a DNS server on an Amazon EC2 instance.

Domain list usage is recommended.

Similar to dnsmasq, but requires more manual configuration. Name any products as a *.conf file. Then follow Steffinstanly's instructions on how to apply blocklists.

Leverage the ADGUARD_SOURCES.txt list or the domain list.

๐Ÿ‹ Docker Usage

To provide a temporary container to experiment with Black Mirror scripts and the CLI utilities it uses, run the following:

docker container run -it --rm -h black-mirror ghcr.io/t145/black-mirror

For a persistant container, use something like what's given below:

docker container run -it --name black-mirror -h black-mirror ghcr.io/t145/black-mirror

Then to access the container at a later date, run:

docker start black-mirror
docker exec -it black-mirror /bin/bash

To use the root user just append -u 0.

NOTE: Before using rkhunter, be sure to run:

rkhunter --update

๐Ÿ‘จโ€๐Ÿ’ป Development

Workspace Setup

Install Docker, PNPM, and NodeJS. These can usually be installed with a popular package manager. The specific Node version used in this project is provided in .node-version, but any version over 16 should work. With Docker running in the background, execute pnpm lint to debug any changes.

๐Ÿ›ก๏ธ Privacy Protectors

Good causes that help secure your digital life.

๐ŸŽถ Notes

Need for speed

Some List References

Lamers Unwelcome

Big Data Lists

Typically used by other blacklist projects as whitelists.

List Name URL
Alexa https://s3.amazonaws.com/alexa-static/top-1m.csv.zip
Umbrella https://downloads.majestic.com/majestic_million.csv
Statvoo https://statvoo.com/dl/top-1million-sites.csv.zip
OpenPageRank https://www.domcop.com/files/top/top10milliondomains.csv.zip
Quantcast https://toplists.net.in.tum.de/archive/quantcast/quantcast-top-sites-2021-06-25_0900_UTC.txt.xz
Tranco list https://tranco-list.eu/

IP Block Providers

Simply provide IP blocks for entire geographic regions.

About

Blacklists and whitelists built by open code, so you know what goes into them.

License:GNU Affero General Public License v3.0


Languages

Language:Shell 62.6%Language:Dockerfile 27.0%Language:Perl 10.3%