express-rate-limit-spider
express-rate-limit-spider is a middleware for rate limiting web crawling or spidering activities in Express applications. It allows you to restrict the number of requests made by web crawlers within a specified time interval, helping to prevent excessive crawling and protect your server resources.
Benchmarks:
- express-rate-limit x 222,950 ops/sec ±29.79% (63 runs sampled)
- express-rate-limit-spider x 504,694,760 ops/sec ±0.41% (90 runs sampled)
Fastest is express-rate-limit-spider
Installation
You can install express-rate-limit-spider using npm:
npm install express-rate-limit-spider
Usage
To use express-rate-limit-spider in your Express application, follow these steps:
1.Import the middleware into your project:
const rateLimitSpider = require('express-rate-limit-spider');
2.Define the rate limit configuration:
const limit = 10; // Maximum number of requests allowed by the spider within the interval
const interval = 60000; // Interval duration in milliseconds (e.g., 60 seconds)
const timeout = 0; // Optional timeout duration in milliseconds (0 for no timeout)
3.Apply the middleware to the desired routes:
app.use(rateLimitSpider(limit, interval, timeout));
The middleware will automatically track and limit the number of requests made by web crawlers based on the specified configuration.
Configuration Options
The express-rate-limit-spider middleware accepts the following configuration options:
- limit: Maximum number of requests allowed by the spider within the interval.
- interval: Interval duration in milliseconds. The middleware resets the request count for each spider after this interval.
- timeout (optional): Timeout duration in milliseconds. If set to a non-zero value, the request count for each spider will be reset to 0 after this timeout.
Error Handling
When the rate limit is exceeded by a spider, the middleware responds with a 429 Too Many Requests status code and a JSON object containing an error message:
{
"error": "Rate limit exceeded"
}
If an internal server error occurs during rate limiting, the middleware responds with a 500 Internal Server Error status code and a JSON object containing an error message:
{
"error": "Internal server error"
}
Usage Considerations
It's important to note that express-rate-limit-spider identifies spiders based on the User-Agent header in the incoming requests. Make sure that the web crawlers you want to limit have unique and identifiable user agent strings.
License
MIT