Make several requests with common options for all, or specific options for each of them.
aiohttp library will be used only for default behavior (text retrieving, no callback) and if not explicitly disabled via {'allow_aio' = False} in common_options. requests will be used otherwise
Options dictionary has shape (for example):
{
"method": "get",
"data": b"Hola Mundo!!!",
"params": {"q": "frase a buscar"},
"json": [{ "si": "no" }],
"proxy_cfg": {"http": "http://miproxy"},
"headers": {"Content-Type": "application/pdf"},
"allow_aio": True,
"aio_client": some_aiohttp_client,
"timeout": 15,
"logger": some_logger,
"callback": some_function
}- method: 'get' by default
- data: used only in non-get requests
- params: used only in 'get' requests
- proxy: could have also the "http://user:pass@host:port" format
- allow_aio: enables or disables the use of aiohttp. If False, it ensures the use of requests always and excludes aiohttp. Default: True
- aio_client: aiohttp.ClientSession. It's the only option that, if present, must be global, not specific for a single url
- logger: a logging.logger instance
- callback: function that receives a requests.Response as argument
If callback exists, the URL will be fetched using 'requests' only for standardization
Args:
- urloptions_list: tuple (url, opts), where 'url' is the url of the request, and opts is a dictionary with specific options
- common_options: dictionary with common options for every request, if that request does not specify its own options
Returns: A list of responses defined by each specific callback or the general callback, or each response text by default
from web_requester import request_all
urls = ['https://google.com',
('https://actualidad.rt.com', {'json': {'num': 2}}),
'http://www.cubadebate.cu',
('https://www.filehorse.com/es', {'timeout': 12})]
response_list = request_all(urls, {'method': 'get', 'timeout': '10'})