Reuse external HTTP connections from workers

[Ángel M]

Discussed in #268

^{Originally posted by dlozina March 29, 2024}
Hi all,

First of all thank you for this project. I am really glad that we can build workers in such a quick manner and test them out.

A key feature of our Workers is their ability to perform HTTP requests externally. This capability allows our Worker to accept data from a client, modify it, and then forward it to another service or worker.

Currently, I'm exploring a use case that involves retrieving data, processing it, and then saving it to Data Storage. This requires one HTTP request to fetch data from Data Storage and another to save the processed data back.

However, I've observed that while it's possible to initiate multiple HTTP requests within the workers, these requests don't reuse connections when directed to the same server. For HTTPS requests, this necessitates a new handshake for each request, leading to slower execution times. This could be due to the fact that each request initiated by a worker potentially launches a new thread and a separate reqwest client, making connection pooling unfeasible.

Is there a workaround or solution for this issue that would allow us to reuse connections, especially since the requests are being made from the host machine? (We don't need any modifications to wasam interface)
Can we reuse the same instance of the reqwest client and enable connection pooling.

Thank you!

Dino