NTLM Authentication failing in MultiThreaded application

Question

I have been trying to put together some code that will- among other things - upload files to a Sharepoint site that uses NTLM authentication. Earlier versions of the code were single threaded, and worked perfectly. They uploaded the file exactly as expected without the slightest issue. However, I eventually tried to multi-thread this application, so that it could upload many files at once, while still going about the rest of its business.

However when I tried to multithread the code, it fails every single time, throwing an IndexOutOfBoundsException. This is singularly unhelpful to me in diagnosing the actual cause of the problem.

In case you are wondering, if I change out the CachedThreadExecutor for a SingleThreadExecutor - forcing the code bask to a single-threaded state - it once again works fine.

Creating the executor and connection manager, and constructing threads:

class OrderProcessor implements Runnable {

    //Other variables for object

    private final ExecutorService executorService = Executors
            .newCachedThreadPool();
    //      .newSingleThreadExecutor();

    private HttpClientConnectionManager conManager;

    private void setup() {
        //always called before execution of anything else in object
        conManager = new PoolingHttpClientConnectionManager();
    }
    //lots of other code
}

The actual code for submitting the threads is complicated, so this version is somewhat simplified, but gets the point across.

for(Request request : requests){
    //Do other stuff
    simpleSubmitFile(request);
    //Do other stuff
}

Here is the simplified file submission method

public Future<Boolean> simpleSubmitFile(Request request){

        transferer = new SharePointTransferer(extractionRequest, conManager);

    Future<Boolean> future = executorService.submit(transferer);
    return future;
}

SharePointTransferer code

//actual values scrubbed
private final String USERNAME = "";
private final String PASSWORD = "";
private final String DOMAIN = "";

private final File sourceFile;
private final String destinationAddress;
private final CloseableHttpClient client;


public SharePointTransferer(final Request extractionRequest, HttpClientConnectionManager conManager) {
    super(extractionRequest);
    this.sourceFile = this.extractionRequest.getFile();
    this.destinationAddress = this.extractionRequest.getDestinationAddress();
    this.client = HttpClients.custom()
            .setConnectionManager(conManager).build();
}

public Boolean call() throws Exception {

    String httpAddress = correctSharePointAddress(destinationAddress);
    HttpPut put = new HttpPut(httpAddress + sourceFile.getName());

    // construct basic request
    put.setEntity(new FileEntity(sourceFile));
    HttpClientContext context = HttpClientContext.create();

    // set credentials for the SharePoint login
    CredentialsProvider credProvider = new BasicCredentialsProvider();
    credProvider.setCredentials(AuthScope.ANY, new NTCredentials(USERNAME,
            PASSWORD, "", DOMAIN));
    context.setCredentialsProvider(credProvider); 
    // execute request
    try {
        HttpResponse response = client.execute(put, context);
        logger.info("response code was: "
                + response.getStatusLine().getStatusCode());
        if (response.getStatusLine().getStatusCode() != 201) {
            throw new FileTransferException(
                    "Could not upload file. Http response code 201 expected."
                            + "\nActual status code: "
                            + response.getStatusLine().getStatusCode());
        }
    } catch (ClientProtocolException e) {
        throw new FileTransferException(
                "Exception Occurred while Transferring file "
                        + sourceFile.getName(), e);
    } catch (IOException e) {
        throw new FileTransferException(
                "Exception Occurred while Transferring file "
                        + sourceFile.getName(), e);
    }finally{
        logger.info("deleting source file: " + sourceFile.getName());
        sourceFile.delete();
        client.close();
    }

    logger.info("successfully transfered file: "+sourceFile.getName());
    return true;
}

If I submit multiple files it throws essentially the exact same exception for all of the files. The trace is below Exception Stack Trace

2015-04-16 11:49:26 ERROR OrderProcessor:224 - error processing file: FILE_NAME_SCRUBBED
PACKAGE_SCRUBBED.FileProcessingException: Could not process file: FILE_NAME_SCRUBBED
        at PACKAGE_SCRUBBED.OrderProcessor.finishProcessingOrder(OrderProcessor.java:223)
        at PACKAGE_SCRUBBED.OrderProcessor.run(OrderProcessor.java:124)
        at PACKAGE_SCRUBBED.FileTransferDaemon.process(FileTransferDaemon.java:48)
        at PACKAGE_SCRUBBED.FileTransferDaemon.start(FileTransferDaemon.java:83)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 41
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at PACKAGE_SCRUBBED.OrderProcessor.finishProcessingOrder(OrderProcessor.java:208)
        ... 8 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 41
        at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addByte(NTLMEngineImpl.java:924)
        at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addUShort(NTLMEngineImpl.java:946)
        at org.apache.http.impl.auth.NTLMEngineImpl$Type1Message.getResponse(NTLMEngineImpl.java:1052)
        at org.apache.http.impl.auth.NTLMEngineImpl.getType1Message(NTLMEngineImpl.java:148)
        at org.apache.http.impl.auth.NTLMEngineImpl.generateType1Msg(NTLMEngineImpl.java:1641)
        at org.apache.http.impl.auth.NTLMScheme.authenticate(NTLMScheme.java:139)
        at org.apache.http.impl.auth.AuthSchemeBase.authenticate(AuthSchemeBase.java:138)
        at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthenticator.java:239)
        at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResponse(HttpAuthenticator.java:202)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:262)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at PACKAGE_SCRUBBED.SharePointTransferer.call(SharePointTransferer.java:74)
        at PACKAGE_SCRUBBED.SharePointTransferer.call(SharePointTransferer.java:1)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

If anyone can figure out what is causing this problem, I would greatly appreciate it.

EDIT: I managed to find a workaround that fixes the issue for me, but would still appreciate an explanation of exactly what is going on.

Answer 1

this is a bug, solved in httpclient version 4.5.2

http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.5.x.txt

Release 4.5.2

Changelog:

• [HTTPCLIENT-1715] NTLMEngineImpl#Type1Message not thread safe but declared as a constant. Contributed by Olivier Lafontaine , Gary Gregory

Answer 2

You can't reuse nor HttpClientContext neither NTLMScheme in a concurrent environment because they are both marked as @NotThreadSafe (see javadoc). In my environment I got the same error, solved with something like:

synchronized(context) {
    HttpResponse response = client.execute(put, context);
}

The authenticated context is reused, but one thread at time.