How to handle JSONP response through Retrofit and GsonConverter?

Question

I need to parse a response from the Flickr API.
http://api.flickr.com/services/feeds/photos_public.gne?tagmode=any&format=json

which returns a response in jsonFlickrFeed jQuery call back function (which is not a valid JSON response).

I know we can remove JSON callback method for Flickr API using nojsoncallback=1 query.

But is there any better approach to handle JSONP response if it is mandatory to use JSON with Padding (JSONP)?

Instead of getting the response as a String, then trimming of the JSON padding and then parse the remaining JSON data.

Sample Flickr API response-

jsonFlickrFeed({
"title": "Recent Uploads tagged mountrainier",
"link": "http:\/\/www.flickr.com\/photos\/tags\/mountrainier\/",
"description": "",
"modified": "2016-12-15T16:56:42Z",
"generator": "http:\/\/www.flickr.com",
"items": [ {
    "title": "Gateway Arts District Open Studio Tour, December 10, 2016",
    "link": "http:\/\/www.flickr.com\/photos\/kimsworldofart\/31274762970\/",
    "media": {
        "m": "http:\/\/farm1.staticflickr.com\/381\/31274762970_c40599d623_m.jpg"
    },
    "date_taken": "2016-12-10T15:49:03-08:00",
    "description": " <p><a href=\"http:\/\/www.flickr.com\/people\/kimsworldofart\/\">kimsworldofart<\/a> posted a photo:<\/p> <p><a href=\"http:\/\/www.flickr.com\/photos\/kimsworldofart\/31274762970\/\" title=\"Gateway Arts District Open Studio Tour, December 10, 2016\"><img src=\"http:\/\/farm1.staticflickr.com\/381\/31274762970_c40599d623_m.jpg\" width=\"240\" height=\"135\" alt=\"Gateway Arts District Open Studio Tour, December 10, 2016\" \/><\/a><\/p> <p>This photo was taken at the Otis Street Art Project in Mount Rainier, Maryland.<\/p>",
    "published": "2016-12-14T20:25:11Z",
    "author": "[email protected] (\"kimsworldofart\")",
    "author_id": "8508061@N02",
    "tags": "otisstreetartsproject gatewayartsdistrict mountrainier princegeorgescounty maryland"
}]})

How to override GSON Converter to trim of these extra function syntax and then parse the remaining valid JSON?

Answer 1

For parsing JSON response, use GsonConverterFactory.

For parsing JSONP or String or invalid JSON response, use ScalarConverterFactory.

If you use say flatMap to call JSON API followed by JSONP API then use both GsonConverterFactory(needed for JSON) and ScalarConverterFactory(needed for JSONP).

Make sure you have below dependencies in your gradle :

implementation 'com.squareup.retrofit2:converter-gson:2.5.0'
//For serialising JSONP add converter-scalars
implementation 'com.squareup.retrofit2:converter-scalars:2.1.0'
//An Adapter for adapting RxJava 2.x types.
implementation 'com.squareup.retrofit2:adapter-rxjava2:2.2.0'

Add converterFactories to retrofit and use setLenient() while building Gson to get rid of error JSON document was not fully consumed.

val gson = GsonBuilder()
            .setLenient()
            .create()

val retrofit = Retrofit.Builder()
            .baseUrl("http://api.flickr.com/")
            .client(builder.build())
            .addConverterFactory(ScalarsConverterFactory.create()) //important
            .addConverterFactory(GsonConverterFactory.create(gson))
            .addCallAdapterFactory(RxJava2CallAdapterFactory.create())
            .build()

@GET("end-point/to/some/jsonp/url")
fun getJsonpData() : Observable<String>

Use converters to convert JSONP to get JSON by removing prefix and suffix present in JSONP. And then convert the string to your data model via

SomeDataModel model = Gson().fromJson<SomeDataModel>(jsonResponse,
            SomeDataModel::class.java)

Answer 2

Using the standard GsonConverterFactory as guide, we can construct one that removes the JSONP from the front of the stream, thus avoid having to read the whole thing and trim --

public final class GsonPConverterFactory extends Converter.Factory {

  Gson gson;

  public GsonPConverterFactory(Gson gson) {
    if (gson == null) throw new NullPointerException("gson == null");
    this.gson = gson;
  }

  @Override
  public Converter<ResponseBody, ?> responseBodyConverter(Type type, Annotation[] annotations,
                                                          Retrofit retrofit) {
    TypeAdapter<?> adapter = gson.getAdapter(TypeToken.get(type));
    return new GsonPResponseBodyConverter<>(gson, adapter);
  }

  @Override
  public Converter<?, RequestBody> requestBodyConverter(Type type,
                                                        Annotation[] parameterAnnotations,
                                                        Annotation[] methodAnnotations,
                                                        Retrofit retrofit) {
    return null;
  }
}

and the converter body. By creating our own json reader, we avoid the assertion that the stream was fully consumed. This allows us to leave the closing JSONP elements in the stream when we close it.

final public class GsonPResponseBodyConverter<T> implements Converter<ResponseBody, T> {
  private final Gson gson;
  private final TypeAdapter<T> adapter;

  GsonPResponseBodyConverter(Gson gson, TypeAdapter<T> adapter) {
    this.gson = gson;
    this.adapter = adapter;
  }

  @Override public T convert(ResponseBody value) throws IOException {
    Reader reader = value.charStream();
    int item = reader.read();
    while(item != '(' && item != -1) {
      item = reader.read();
    }
    JsonReader jsonReader = gson.newJsonReader(reader);
    try {
      return adapter.read(jsonReader);
    } finally {
      reader.close();
    }
  }
}

add to your retrofit like you would the regular Gson factory --

Retrofit retrofit = new Retrofit.Builder()
    .baseUrl(/* you base url */)
    .addConverterFactory(new GsonPConverterFactory(new Gson()))
    .build();

Note: using this converter will require all responses to be in JSONP. It will fail on regular JSON responses, and you cannot use the Gson and GsonP converters at the same time.