botocore: how to close or clean up a session or client

Question

While doing some automation around AWS-EC2 with the botocore library in Python, I noticed a lot of HTTPS connections remained established that were no longer needed by processes that were busy doing other things (so killing them or recoding them to exit is not an option). I think the botocore session and/or client object is leaving the connections to AWS endpoints established. The botocore documentation shows how to start or create them, but not how to close them or clean things up. I tried a .close method but it did not exist. How can I get these connections to gracefully close without killing the processes?

Answer 1

I had the same issue, but from a slightly different angle: When closing worker threads, my log files would get cluttered with these warnings - also due to open connections:

Exception ignored in: <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('[internal ip]', 49266), raddr=('[external ip]', 443)>

ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('[internal ip]', 49266), raddr=('[external ip]', 443)>

After some time down the rabbit hole, I have figured out how to close the connections correctly, before closing the threads.

I'm using SQS with boto3, so you might need to modify the call a bit for it to work with botocore.

My example to produces the warning above is:

import boto3
import boto3.session
import warnings

warnings.simplefilter('error', ResourceWarning)  # Display warnings
session = boto3.session.Session()
sqs = session.resource('sqs', region_name=AWSregion)
sqs_q = sqs.Queue(url=SQSQueueUrl)
sqs_msg = sqs_q.receive_messages(MaxNumberOfMessages=1)

The SQS connection can be closed using:

sqs.meta.client._endpoint.http_session.close()  # closing a boto3 resource
sqs._endpoint.http_session.close()  # closing a boto3 client