AWS Java TransferManager issue, JVM could not obtain any more native threads

Question

I've been fighting with a TransferManager issue for a few weeks now and I am hoping to get some feedback,

I have a server application that waits for tasks to be sent to it. Some of these tasks require an upload to S3. The task contains all of the upload information I need including the access keys.

In my actual upload code, I need to create a new S3Client and TransferManager object every time I need to do an upload because I do not know my access keys ahead of time.

After the system processes approximately 1200 uploads I received an error indicating that the JVM could not obtain any more native threads. I attached a profiler to the application and noticed that the TransferManager object was not getting cleaned up properly, there were thousands of "s3-transfer-manager-worker-1" threads sitting idle.

I attempted to add a call to TransferManager.shutdownNow() after the upload was complete. That did clean up the threads. However, I started receiving a RejectedExecutionException whenever a new TransferManager was created and an upload was attempted.

TransferManager contains an UploadMonitor and UploadMonitor has a static ScheduledExecutorService. TransferManager.shutdownNow() calls the static method UploadMonitor.shutdownNow() which calls shutdownNow() on the executor service. This makes it so that I can no longer use any TransferManager objects, even if I try to create a new one.

How do I go about using multiple transfer objects without running out of threads? This seems like it is a bug.

Answer 1

This was confirmed by Amazon as a bug. See details here https://forums.aws.amazon.com/thread.jspa?threadID=83520&tstart=0