WARNING: Connection pool is full
lightiverson opened this issue · comments
Issue Summary
We're using the twilio-python library to send out thousands of SMS messages concurrently using Python's concurrent features library. During usage the Logging module gives us a warning saying: urllib3.connectionpool:_put_conn:304 - Connection pool is full, discarding connection: api.twilio.com
.
Steps to Reproduce
- Get an array of valid phonenumbers
- Map the client.message.create function using concurrent.futures
- Run the program with Logging
Code Snippet
def send_session_message():
message = client.messages.create(
body=body,
from_=from_,
to=[recipient],
persistent_action=coordinates,
)
with concurrent.futures.ThreadPoolExecutor() as executor:
results = executor.map(
send_session_message,
recipients,
repeat(body),
repeat(from_),
repeat(messaging_service_sid),
)
Exception/Log
# WARNING | urllib3.connectionpool:_put_conn:304 - Connection pool is full, discarding connection: api.twilio.com
Technical details:
- twilio-python version: 7.1.0
- python version: 3.8.12
hello @lightiverson The issue is with your implementation of connection pool logic nothing to do with twilio-python. you can try blocking new connection when connection pool is full. you can look into python/urllib documentation and find the solution to handle connection pool full issue. This link might be helpful to you.
@lightiverson can you please update to latest version of twilio-python. We are not supporting 7.X version.
hi @charan678 thanks for the link! We came across this information and handled the connection pool full issue in the TwilioHttpClient like so:
![afbeelding](https://private-user-images.githubusercontent.com/33266555/243652093-1bda5016-6487-4c79-9b99-aae224641ef6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY3NjcyODcsIm5iZiI6MTcxNjc2Njk4NywicGF0aCI6Ii8zMzI2NjU1NS8yNDM2NTIwOTMtMWJkYTUwMTYtNjQ4Ny00Yzc5LTliOTktYWFlMjI0NjQxZWY2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI2VDIzNDMwN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc5YzM5NTdiYzUxZGY1MmI1ODk0YTRkMDk5ZjgyMzI4Y2I4N2E2MmRmMWQ2MTgwNTk0M2IxNWM5MjdjYjJiYTAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FGmVdSNh3rCsBs-8ukP8V3XpH9ghcpU4i9asTvcBjdI)
This seems to have solved the problem. However, you mentioned that the issue has nothing to do with twilio-python. Is there a better solution which does not involve editing the twilio-python package?
@charan678 as for the package version, you're right we will update to the latest version asap. Thank you.
@lightiverson We can't remove this snippet, It limits maximum pool size. you can set max_worker parameter in concurrent.futures.ThreadPoolExecutor.
@charan678 it seems there's a misunderstanding. The highlighted snippet, is what we have added locally to the twilio-python source code to solve this issue. It currently does not exist in your repository.
Would you like us to make a pull request to have this snippet added?
@lightiverson Sorry for misunderstanding, you can raise the pull request. It will take next 10-11 days to be visible in next release. Meanwhile, you can change max_workers to equal or less than default pool size of urllib which 10.
@charan678 No worries! Got it, we'll raise the pull request shortly ,with the max_workers set to equal or less than 10. Thanks for your help.
Edit: Realized the max workers bit was about ThreadPoolExecutor() and not about HTTPAdapter().