caffeinewriter / FilterLists

:shield: The independent, comprehensive directory of filter and host lists for advertisements, trackers, malware, and annoyances.

FilterLists is the independent, comprehensive directory of filter and host lists for advertisements, trackers, malware, and annoyances.

Website:
A React & TypeScript UI built with Ant Design.

API:
An ASP.NET Core API CRUD-ing with a MariaDB instance via EF Core.

Agent:
A .NET Core console application performing crawling, archiving, and URL validation.

Background

FilterLists helps to protect your privacy and security when using the internet. It provides a comprehensive directory of subscription lists to block advertisements, malware, trackers, and other general annoyances. First, install one of the software tools that supports these lists. Then, you can subscribe to a set of these lists inside of that software. If you need help or have a comment/question, open an Issue.

Using a software firewall or adblocker has become a key tool in protecting one's privacy online. While the current business model of the "free" internet relies upon advertising revenue, publishers have become too flippant about tracking users without their consent. Advertising networks track users' browsing habits around the internet building lucrative profiles to exploit for targeted marketing or big data research. Until internet publishers develop more transparency or a new business model, we must continue to take the protection of our privacy into our hands.

Typical adblockers run as an extension in popular web browsers. As we browse the internet, they compare HTTP requests to their list of hosts and filters to selectively block advertisements, trackers, and malware. This filtering helps to protect the surfer's privacy, prevents malvertising attacks, and reduces bandwidth requirements. Thousands of contributors work hard to maintain these FilterLists for anyone to use. Locating and selecting these lists, however, has not historically been very easy. This project aims to solve that problem.

Contributing

Community

Check out the FilterLists Hub, a place for discussing how to write rules, maintain FilterLists, and chat about the state of the adblocking community.

Adding or Updating Lists

To submit a new list or update data about an existing list, please submit a pull request to data in conjunction with the data model described here. Alternatively, you can open a new issue providing information for all of the fields described in the data model.

Adding or Updating Rules

FilterLists does not maintain any of these lists. It serves only as a discovery tool to direct users to lists that they may want to use. If you want to request addition, modification, or removal of a rule from a list, you will need to contact the maintainers of that list directly. FilterLists provides a variety of ways you can get in contact with the maintainers to do so.

Building and Running Locally

We have containerized FilterLists to make it as easy as possible for contributers to get the project up and running locally.

Up and Running

1. Install Docker CE for your computer's operating system. Docs
2. Clone the FilterLists git repository to your computer. Docs
3. Navigate to the root directory of your locally cloned FilterLists git repository in a command-line interface.
4. Execute docker-compose up. (Optionally, include the -d flag to launch in detached mode.)
5. After a minute or so, visit the locally running version of FilterLists in a web browser at http://localhost/.

Configuration

The appsettings*.json files are only used when debugging in Visual Studio outside of Docker (they are not copied into the built runtime images). When running any of the projects in containers via Docker/Docker Compose, the configuration is passed in as environment variables configured in .env and docker-compose*.yml files.

Testing changes to the data (.json files)

Automated

Execute:

docker volume create test-data-results && docker-compose -p test-data -f docker-compose.data.tests.yml down -v && docker-compose -f docker-compose.data.tests.yml build api && docker-compose -p test-data -f docker-compose.data.tests.yml run --rm api

Manual

1. Execute docker-compose up -d --build api.
2. Verify your changes are properly reflected at http://localhost/.

Testing changes to the Api, Services, or Data projects

Automated

• To run FilterLists.Services.Tests:

  docker volume create test-services-results && docker-compose -f docker-compose.services.tests.yml build api && docker-compose -p test-services -f docker-compose.services.tests.yml run --rm api

• To run FilterLists.Data.Tests:

  docker volume create test-data-results && docker-compose -p test-data -f docker-compose.data.tests.yml down -v && docker-compose -f docker-compose.data.tests.yml build api && docker-compose -p test-data -f docker-compose.data.tests.yml run --rm api

Manual

1. Execute docker-compose up -d --build api.
2. Verify your changes are properly reflected at http://localhost/api.

Testing changes to the Web project

1. Execute docker-compose up -d --build web.
2. Verify your changes are properly reflected at http://localhost/.

Testing changes to the Agent project

The Agent takes the following command line arguments:

• -a Archive copies of all lists in a git repository.
• -c Validate all URLs in the FilterLists database.

Automated

• To run FilterLists.Agent.Tests:

  docker volume create test-agent-results && docker-compose -f docker-compose.agent.tests.yml build agent && docker-compose -p test-agent -f docker-compose.agent.tests.yml run --rm agent

Manual

1. Execute docker-compose build agent && docker-compose run agent [-c] [-v] (don't include the square brackets, they indicate optional command line arguments).
2. Verify your changes are properly reflected in the console logger.

Debugging

1. Execute docker container ls -a to find the CONTAINER ID of the container in question.
2. Execute docker logs [CONTAINER ID] replacing [CONTAINER ID] with the hash from step 1 to view the logs from that container.

Acknowledgements

• Imre Kristoffer Eilertsen @DandelionSprout : major contributor
• Raymond Hill @gorhill : uBlock Origin owner, advised on some technical aspects of FilterLists
• Andrey Meshkov @ameshkov : AdGuard CTO, advised on some technical aspects of FilterLists
• More contributors

Miscellany

Disclaimer

FilterLists does not condone subscribing to any particular FilterList or blocking any particular advertisement network. Make sure to financially support your favorite sites in some way if you choose to block their advertisements. FilterLists is also in no way officially affiliated with any of the software vendors or list maintainers referenced on this site.

Privacy

We respect your privacy. That is the whole point of this project. The only minor data we collect about visitors is what our infrastructure providers collect. These include:

• GitHub
• Azure DevOps
• Docker Hub
• DigitalOcean
• Application Insights
• Cloudflare

Internet Explorer

FilterLists does not support Internet Explorer. To view a smaller, TPL-only archive with subscribable links, right-click on this link, choose to save it as an HTML file, and open it in Internet Explorer ≥9.