How to ignore duplicate keys while parsing json using gson?

Question

I am getting a a duplicate key exception while parsing JSON response containing timestamps as keys using GSON. It gives the following error:

com.google.gson.JsonSyntaxException: duplicate key: 1463048935 at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:186) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:141)

How do I make it ignore the duplicate entries, and just parse it to a map with any one from the duplicate entries?

Answer 1

Hackerman solution, tested and working using GSON v2.8.5, but use at your own risk! Whenever you update GSON to a new version, make sure to check whether this is still working!

Basically, you can use the fact that the generic ObjectTypeAdapter ignores duplicates as seen here:

Looks like MapTypeAdapterFactory checks for duplicate

  V replaced = map.put(key, value);
  if (replaced != null) {
    throw new JsonSyntaxException("duplicate key: " + key);
  }

however ObjectTypeAdapter does not

case BEGIN_OBJECT:
  Map<String, Object> map = new LinkedTreeMap<String, Object>();
  in.beginObject();
  while (in.hasNext()) {
    map.put(in.nextName(), read(in));
  }
  in.endObject();
  return map;

What you can do now is trying to deserialize using fromJson as usual, but catch the "duplicate key" exception, deserialize as a generic Object, which will ignore duplicates, serialize it again, which will result in a JSON string without duplicate keys, and finally deserialize using the correct type it's actually meant to be.

Here is a Kotlin code example:

fun <T> String.deserialize(gson: Gson, typeToken: TypeToken<T>): T {
    val type = typeToken.type
    return try {
        gson.fromJson<T>(this, type)
    } catch (e: JsonSyntaxException) {
        if (e.message?.contains("duplicate key") == true) {
            gson.toJson(deserialize(gson, object : TypeToken<Any>() {})).deserialize(gson, typeToken)
        } else {
            throw e
        }
    }
}

Obviously, this adds (potentially heavy) overhead by requiring 2 deserializations and an additional serialization, but currently I don't see any other way to do this. If Google decides to add an option for a built-in way to ignore duplicates, as suggested here, better switch to that.