tomnomnom / fff

The Fairly Fast Fetcher. Requests a bunch of URLs provided on stdin fairly quickly.

fff

The Fairly Fast Fetcher. Requests a bunch of URLs provided on stdin fairly quickly.

The main idea is to launch a new request every n milliseconds, without waiting for the last request to finish first. This makes for consistently fast fetching, but can be hard on system resources (e.g. you might run out of file descriptors). The advantage though, is that hitting a bunch of very slow URLs or URLs that result in timeouts doesn't slow the overall progress very much.

Install

▶ go get -u github.com/tomnomnom/fff

Usage

Basic usage:

▶ cat urls.txt | fff

Options:

▶ fff --help
Request URLs provided on stdin fairly frickin' fast

Options:
  -b, --body <data>         Request body
  -d, --delay <delay>       Delay between issuing requests (ms)
  -H, --header <header>     Add a header to the request (can be specified multiple times)
  -k, --keep-alive          Use HTTP Keep-Alive
  -m, --method              HTTP method to use (default: GET, or POST if body is specified)
  -o, --output <dir>        Directory to save responses in (will be created)
  -s, --save-status <code>  Save responses with given status code (can be specified multiple times)
  -S, --save                Save all responses
  -x, --proxy <proxyURL>    Use the provided HTTP proxy

Tuning

You might want to increase your open file descriptor limit before doing anything crazy:

▶ ulimit -n 16384

TODO

• Create an index file in the output directory