Giters

truongbo17 / proxy-rotator

Proxy Rotator with Load Balancing Strategies

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

guzzleload-balancingproxy-rotationround-robinstrategyproxy-rotatorcrawlproxy

Proxy Rotation with Laravel Guzzle

Proxy Rotation Logo

Description

  • Implement Nginx simple load balancing strategies to build a Package Proxy Rotation to use with Guzzle or anymore...
  • Use only effective, consistent and simple strategies to rotate proxies

Optional Features

Strategy

Multiple Endpoint

Automatically retry connect and switch between endpoint

Next feature

  • Use backup hosts (if the main host is not accessible, Guzzle will automatically connect to the backup hosts to get data...) ✔️
  • Configure strategies to run separately with hosts, smarter ✔️
  • Automatically retry connecting hosts when it fails to connect (configure number of retries, response code types) ✔️
  • Load balancing, apply separate strategies to each Cluster ❌
  • Anymore...

Installation

Install the package:

composer require truongbo/proxy-rotation

Run test:

./vendor/bin/phpunit

Publish package files (config):

php artisan vendor:publish --provider="TruongBo\ProxyRotation\ProxyRotationServiceProvider"

Quick Use

You need to choose a rotation strategy and configure it with Rotation:

use TruongBo\ProxyRotation\Rotation;
use TruongBo\ProxyRotation\Strategy\RoundRobin;

$rotation = new Rotation(new RoundRobin(counter: 0));

Initialize a cluster consisting of proxy nodes:

use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1'),
            new ProxyNode(name: 'proxy-node2'),
            new ProxyNode(name: 'proxy-node3'),
            new ProxyNode(name: 'proxy-node4'),
        ]);

Then, we should connect ProxyMiddleware to Guzzle for the proxy balancing to work:

use GuzzleHttp\HandlerStack;
use GuzzleHttp\Client;
use TruongBo\ProxyRotation\Middleware\ProxyMiddleware;

$stack = HandlerStack::create();
$stack->push(new ProxyMiddleware(rotation: $rotation,proxy_cluster: $proxy_cluster));

$client = new Client([
    'handler' => $stack,
]);

Finished, check it out now:

while (true) {
    /** @var ResponseInterface $response */
    $response = $client->get('https://httpbin.org/ip');

    // ...
}

Result: (With proxy-node as your proxy address)

+-------------+
| proxy-node1 |
| proxy-node2 |
| proxy-node3 |
| proxy-node4 |
| proxy-node1 |
| proxy-node2 |
| proxy-node3 |
| proxy-node4 |
| etc...      |
+-------------+

Sort Nodes

Sort to adjust the order of nodes in ascending or descending order:

use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1',weight: 1000),
            new ProxyNode(name: 'proxy-node2',weight: 20),
            new ProxyNode(name: 'proxy-node3',weight: 200),
            new ProxyNode(name: 'proxy-node4',weight: 10),
        ]);
        
$proxy_cluster->sort("desc");

Top proxy nodes will be used more and vice versa, for example:

+-------------+-------------+
| name        | weight      |
+-------------+-------------+
| proxy-node1 | 1000        |
| proxy-node3 | 200         |
| proxy-node2 | 20          |
| proxy-node4 | 10          |
+-------------+-------------+

Sorting nodes can help you use the Frequency strategy better. Use asc for reverse sort.

Max Use

  • Store the proxy usage count in the cache and count it. If it is used more than allowed, the proxy will be temporarily idle for the configured time period.
  • Use only with Random , Frequency and RoundRobin strategies

Example:

$cluster = new ProxyCluster('cluster1', [
            new ProxyNode(name: 'node1',weight: 3,max_use: 4,max_use_wait: 10),
            new ProxyNode(name: 'node2',weight: 1,max_use: 2,max_use_wait: 20),
        ]);

Everything else runs automatically

Random

  • $input_random : How are proxies random?
- both : All proxy nodes are random
- has_weight : Only the weighted proxy node will be random
- no_weight : Only proxy nodes without weights can be random

Config:

use TruongBo\ProxyRotation\Rotation;
use TruongBo\ProxyRotation\Strategy\Random;
use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;

$rotation = new Rotation(new Random(input_random: "both"));

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1',weight: 20),
            new ProxyNode(name: 'proxy-node2'),
            new ProxyNode(name: 'proxy-node3'),
            new ProxyNode(name: 'proxy-node4',weight: 100),
        ]);

Output(both):

+-------------+-------------+
| name        | weight      |
+-------------+-------------+
| proxy-node3 | 0           |
| proxy-node1 | 20          |
| proxy-node2 | 0           |
| proxy-node4 | 100         |
+-------------+-------------+

Frequency

More efficient using sort node

use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;
use TruongBo\ProxyRotation\Strategy\Frequency;

$rotation = new Rotation(new Frequency(frequency: 0.8, depth: 0.2));

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1',weight: 2048),
            new ProxyNode(name: 'proxy-node2',weight: 1024),
            new ProxyNode(name: 'proxy-node3',weight: 512),
            new ProxyNode(name: 'proxy-node4',weight: 256),
            new ProxyNode(name: 'proxy-node5',weight: 256),
            new ProxyNode(name: 'proxy-node6',weight: 64),
            new ProxyNode(name: 'proxy-node7',weight: 32),
            new ProxyNode(name: 'proxy-node8',weight: 16),
            new ProxyNode(name: 'proxy-node9',weight: 8),
            new ProxyNode(name: 'proxy-node10',weight: 4),
        ]);
        
$cluster->sort("desc");

The probability of choosing nodes for Frequency can be visualized as follows::

+--------------+--------+
| nodes        | chance |
+--------------+--------+
| proxy-node1  | 40%    |
| proxy-node2  | 40%    |
+--------------+--------+
| proxy-node3  | 2.5%   |
| proxy-node8  | 2.5%   |
| proxy-node5  | 2.5%   |
| proxy-node6  | 2.5%   |
| proxy-node4  | 2.5%   |
| proxy-node7  | 2.5%   |
| proxy-node9  | 2.5%   |
| proxy-node10 | 2.5%   |
+-----------+-----------+

Round Robin

The proxies will be rotated in turn ($counter : start counting from somewhere)

use TruongBo\ProxyRotation\Rotation;
use TruongBo\ProxyRotation\Strategy\RoundRobin;
use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;

$rotation = new Rotation(new RoundRobin(counter: 0));

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1'),
            new ProxyNode(name: 'proxy-node2'),
            new ProxyNode(name: 'proxy-node3'),
            new ProxyNode(name: 'proxy-node4'),
        ]);

Output:

+-------------+
| proxy-node1 |
| proxy-node2 |
| proxy-node3 |
| proxy-node4 |
| proxy-node1 |
| proxy-node2 |
| proxy-node3 |
| proxy-node4 |
| etc...      |
+-------------+
  • You can interfere with proxy usage for a certain period of time if the proxy is restricted from use.Using Max Use

Weighted Round Robin

The number of times this proxy node is called is the weight parameter passed in the initialization of the ProxyNode ($counter : start counting from somewhere)

use TruongBo\ProxyRotation\Rotation;
use TruongBo\ProxyRotation\Strategy\WeightedRoundRobin;
use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;

$rotation = new Rotation(new WeightedRoundRobin(counter: 0));

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1', weight: 3),
            new ProxyNode(name: 'proxy-node2'),
            new ProxyNode(name: 'proxy-node3', weight: 1),
            new ProxyNode(name: 'proxy-node4', weight: 1),
        ]);

Output:

+-------------+
| proxy-node1 |
| proxy-node1 |
| proxy-node1 |
| proxy-node3 |
| proxy-node4 |
| etc...      |
+-------------+
  • Proxy Node without weight will not be used

Multiple Dynamic

Dynamically change strategies according to the passed callable condition (Absolutely do not use if you do not know about it)

use TruongBo\ProxyRotation\Rotation;
use TruongBo\ProxyRotation\Strategy\WeightedRoundRobin;
use TruongBo\ProxyRotation\ProxyServer\ProxyCluster;
use TruongBo\ProxyRotation\ProxyServer\ProxyNode;

$rotation = new Rotation(new MultipleDynamic(
            new RoundRobin(counter: 0),
            new Random(input_random: "has_weight"),
            new WeightedRoundRobin(counter: 0),
        ));

$proxy_cluster = new ProxyCluster(
        cluster_name: 'cluster1', 
        array_proxy_node: [
            new ProxyNode(name: 'proxy-node1', weight: 3),
            new ProxyNode(name: 'proxy-node2'),
            new ProxyNode(name: 'proxy-node3', weight: 1),
            new ProxyNode(name: 'proxy-node4', weight: 1),
        ]);

        while (true) {
            $node = $rotation->pick($cluster, function (ProxyNode $proxy_node){
                //condition switch between strategies in here.
            });

            echo $node?->name;
        }

About

Proxy Rotator with Load Balancing Strategies

guzzleload-balancingproxy-rotationround-robinstrategyproxy-rotatorcrawlproxy

Languages

Language:PHP 100.0%