Livy REST API: GET requests work but POST requests fail with '401 Authentication required'

Question

I’ve written a Java client for parts of Livy’s REST API at https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md. The client uses Spring’s RestTemplate.getForObject() and postForObject() to make GET and POST requests respectively. The Livy server is secured with Kerberos.

GET /sessions and GET /batches requests work fine: I get the expected responses from Livy. But both POST /sessions and POST /batches requests fail with:

org.springframework.web.client.HttpClientErrorException: 401 Authentication required

Does anyone know why the POST requests fail when the GET requests succeed? My code does nothing explicit with authentication.

I've tried authenticating as several different users via Kerberos but I always get this problem. Does Livy need extra configuration to allow POST requests from particular users (since POST requests effectively create interactive sessions or submit jobs to Spark)?

Answer 1

It turns out that whilst the regular org.springframework.web.client.RestTemplate class is sufficient for GET requests, you need to use org.springframework.security.kerberos.client.KerberosRestTemplate for POST requests. You may also need to add an extra header to POST requests if the Livy server has CSRF (cross-site request forgery) protection enabled as described here.

GET /batches example

RestTemplate restTemplate = new RestTemplate();
GetBatchesResponse response2 = restTemplate.getForObject("http://your_livy_server:8998" + "/batches", GetBatchesResponse.class);

where GetBatchesResponse is a simple POJO I've written that represents the response body to GET /batches.

POST /batches example

PostBatchesRequest postRequestBody = new PostBatchesRequest();
postRequestBody.setFile("/path/to/your/application"); // In HDFS

KerberosRestTemplate kerberosRestTemplate = new KerberosRestTemplate("path_to_your_key_tab_file", "your_user@your_realm");

// Add CSRF header if required:
HttpHeaders headers = new HttpHeaders();
headers.set("X-Requested-By", "your_user@your_realm");
headers.setContentType(MediaType.APPLICATION_JSON);

HttpEntity<PostBatchesRequest> postRequest = new HttpEntity<PostBatchesRequest>(postRequestBody, headers);

Batch batch = kerberosRestTemplate.postForObject("http://your_livy_server:8998" + "/batches", postRequest, Batch.class);

where PostBatchesRequest and Batch are POJOs I've written to represent the request body and response respectively.