RestTemplate not working with S3 pre-signed put URLs

Question

I have a server generate AWS S3 pre-signed PUT URLs and then I'm trying to uploading a byte[] into that URL using RestTemplate with this code:

RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.ALL));
HttpEntity<byte[]> entity = new HttpEntity<>("Testing testing testing".getBytes(), headers);
System.out.println(restTemplate.exchange(putUrl, HttpMethod.PUT, entity, String.class));

When I run that code, I get this error:

Exception in thread "JavaFX Application Thread" org.springframework.web.client.HttpClientErrorException: 400 Bad Request
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531)
    at tech.dashman.dashman.controllers.RendererAppController.lambda$null$2(RendererAppController.java:95)

Unfortunately, there's nothing in the AWS S3 logs, so, I'm not sure what's going on. If I take that exact same URL and put it in the REST Client of IntelliJ IDEA, it just works (it creates an empty file in S3).

Any ideas what's wrong with my Java code?

Here's a full example that does the signing and tries to uploading a small payload to S3:

import com.amazonaws.HttpMethod;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import org.joda.time.DateTime;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.web.client.RestTemplate;
import java.util.Date;

public class S3PutIssue {
    static public void main(String[] args) {
        String awsAccessKeyId = "";
        String awsSecretKey = "";
        String awsRegion = "";
        String path = "";
        String awsBucketName = "";
        BasicAWSCredentials awsCredentials = new BasicAWSCredentials(awsAccessKeyId, awsSecretKey);
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(awsRegion).
                withCredentials(new AWSStaticCredentialsProvider(awsCredentials)).build();
        Date expiration = new DateTime().plusDays(1).toDate();
        GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(awsBucketName, path);
        urlRequest.setMethod(HttpMethod.PUT);
        urlRequest.setExpiration(expiration);
        String putUrl = s3Client.generatePresignedUrl(urlRequest).toString();

        RestTemplate restTemplate = new RestTemplate();
        HttpHeaders headers = new HttpHeaders();
        HttpEntity<byte[]> entity = new HttpEntity<>("Testing testing testing".getBytes(), headers);
        restTemplate.exchange(putUrl, org.springframework.http.HttpMethod.PUT, entity, Void.class);
    }
}

Answer 1

The source of issue is a double encoding of url characters. There are / in extended secret key which are encoded as %2 by s3Client.generatePresignedUrl. When already encoded string is passed to restTemplate.exchange it's internally converted to URI and encoded for the second time as %252 by UriTemplateHandler in RestTemplate source code.

@Override
@Nullable
public <T> T execute(String url, HttpMethod method, @Nullable RequestCallback requestCallback,
        @Nullable ResponseExtractor<T> responseExtractor, Object... uriVariables) throws RestClientException {

    URI expanded = getUriTemplateHandler().expand(url, uriVariables);
    return doExecute(expanded, method, requestCallback, responseExtractor);
}

So the easiest solution is to convert URL to URI using URL.toURI(). If you don't have URI and have String when RestTemplate is invoked then two options are possible.

Pass URI instead for string to exchange method.

restTemplate.exchange(new URI(putUrl.toString()), HttpMethod.PUT, entity, Void.class);

Create default UriTemplateHandler with NONE encoding mode and pass it to RestTemplate.

DefaultUriBuilderFactory defaultUriBuilderFactory = new DefaultUriBuilderFactory();
defaultUriBuilderFactory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.NONE);
restTemplate.setUriTemplateHandler(defaultUriBuilderFactory);
restTemplate.exchange(putUrl.toString(), org.springframework.http.HttpMethod.PUT, entity, Void.class);

Answer 2

Do not convert your URL to String. Instead convert it to URI. I think there are some encoding issues when you converted to String. For example the URL in String format had %252F, where it should have just been %2F. Looks like some sort of double encoding issue.

Leave as URL...

     URL putUrl = amazonS3Client.generatePresignedUrl(urlRequest);

Convert to URI...

    ResponseEntity<String> re = restTemplate.exchange(putUrl.toURI(), org.springframework.http.HttpMethod.PUT, entity, String.class);

EDIT: More info to clarify what is going on.

The problem that occurred here is that when you call URL.toString() in this instance you are given back an encoded String representation of the URL. But the RestTemplate is expecting a String url that is not yet encoded. RestTemplate will do the encoding for you.

For example look at the code below...

public static void main(String[] args) {
    RestTemplate rt = new RestTemplate();
    rt.exchange("http://foo.com/?var=<val>", HttpMethod.GET, HttpEntity.EMPTY, String.class);
}

When you run this you get the following debug message from Spring, notice how the url in the debug msg is encoded.

[main] DEBUG org.springframework.web.client.RestTemplate - Created GET request for "http://foo.com/?var=%3Cval%3E" 

So you can see the RestTemplate will encode for you any String url's passed to it. But the URL provided by AmazonS3Client is already encoded. See the code below.

URL putUrl = amazonS3Client.generatePresignedUrl(urlRequest);
System.out.println("putUrl.toString = " + putUrl.toString());

This prints out a String that is already encoded.

https://private.s3.amazonaws.com/testing/?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20171114T191829Z&X-Amz-SignedHeaders=host&X-Amz-Expires=0&X-Amz-Credential=AKIAIJ7ZSL22IJTM6NTQ%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=eca611ea33d9ad5710207568dcf181e4318ce39271fd0f1ce05bd99ebbf4097

So when I stick that into the exchange method of the RestTemplate I get the following debug message.

[main] DEBUG org.springframework.web.client.RestTemplate - PUT request for "https://turretmaster.s3.amazonaws.com/testing/?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20171114T191829Z&X-Amz-SignedHeaders=host&X-Amz-Expires=0&X-Amz-Credential=AKIAIJ7ZSL22IJTM6NTQ%252F20171114%252Fus-east-1%252Fs3%252Faws4_request&X-Amz-Signature=eca611ea33d9ad5710207568dcf181e4318ce39271fd0f1ce05bd99ebbf40975"

Notice how every %2F from the url String turned into %252F. %2F is the encoded representation of /. But %25 is %. So it encoded a url that was already encoded. The solution was to pass a URI object to RestTemplate.exchange, instead of an encoded String url.