salihbyk / seo

WebAce is a java HTTP client for web scraping (mirror, issues & PR on gitlab)

WebAce java HTTP client

WebAce is a java HTTP client for web scraping with great proxy support based on Apache HttpComponents.

Homepage : https://gitlab.com/serphacker/webace

Issues and bug report : https://gitlab.com/serphacker/webace/issues

Features :

• Designed for web scraping, form submission and web automation (bots, link building, etc.)
• Support for HTTP and Socks proxy with authentication
• Support multiple sources IP (BindProxy)
• Proxy rotation
• Cookie management
• Easy bypass of SSL/TLS verification (works well with debugging proxy like burp and ZAP)
• Great form submission and charsets handling
• HTTP/2
• Java 11

Install

(require java minimal version 11 )

<dependency>
 <groupId>com.serphacker.webace</groupId>
 <artifactId>webace</artifactId>
 <version>LATEST</version>
</dependency>

Usage

TODO

Build

Building jar

mvn clean package

Running integration tests

1. Copy docker/webace.env.template to docker/webace.env
2. In docker directory, run ./docker-compose.sh --env webace.env up. It will starts all the required container (httpbin server, squid proxies, socks proxies, etc.).
3. Back in base directory, run the integration tests with the following command : mvn -P integration-test -DhttpBinDomain=172.29.1.1 -Dtest.service-backend=docker-compose clean verify

License

The MIT License (MIT)