How to convert arbitrary JSON into a usable structure in Java

Question

I'm trying to use gson to convert this returned JSON into some kind of data structure such that I can extract useful data.

For Example:

http://search.twitter.com/search.json?q=test&rpp=1

Returns:

{
    "completed_in":0.028,
    "max_id":196386333906837504,
    "max_id_str":"196386333906837504",
    "next_page":"?page=2&max_id=196386333906837504&q=test&rpp=1",
    "page":1,
    "query":"test",
    "refresh_url":"?since_id=196386333906837504&q=test",
       "results":[
          {
             "created_at":"Sat, 28 Apr 2012 23:52:05 +0000",
             "from_user":"della_ky",
             "from_user_id":525641596,
             "from_user_id_str":"525641596",
             "from_user_name":"kydella modeste",
             "geo":null,
             "id":196386333906837504,
             "id_str":"196386333906837504",
             "iso_language_code":"en",
             "metadata":{
                "result_type":"recent"
             },
             "profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2159990525\/webcam-toy-photo3_20_2__normal.jpg",
             "profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2159990525\/webcam-toy-photo3_20_2__normal.jpg",
             "source":"<a href="http:\/\/mobile.twitter.com" rel="nofollow">Mobile Web<\/a>",
             "text":"RT @Y__U__NOOO: #SongsIKnowOffByHeart ALL SONGS I LISTEN TO. BRAIN, Y U NO REMEMBER TEST ANSWERS LIKE THAT?!?",
             "to_user":null,
             "to_user_id":null,
             "to_user_id_str":null,
             "to_user_name":null
          }
       ],
       "results_per_page":1,
       "since_id":0,
       "since_id_str":"0"
    }

Ultimately, I would like to be able to output a list of tweets with the name of the sender and the date/time of the tweet.

I have read through the gson documentation but it's going over my head to be honest - lots of new concepts there for me.

Do I need to define a class which maps exactly to the structure of the JSON in order to then populate an instance of that class? If so this seems very inflexible/laborious. Ideally I'm looking for something which will handle JSON in any form and give me a structure I can use automatically...

Is anyone able to give me some pointers? Being new to this - the more detailed and in words of the fewest syllables the better!

Update - Thanks to the responses I've already had on this I've had a go at putting a class together to capture the twitter JSON. However, since the JSON has an embedded ArrayList of Objects I'm struggling a bit... So far I have

public class tweetData {
    private double completed_in;
    private long max_id;
    private long max_id_str;
    private String next_page;
    private int page;
    private String query;
    private String refresh_url;
    private List<tweetDetails> tweets = new ArrayList<tweetDetails>();
}

and

public class tweetDetails {
    private String created_at;
    private String from_user;
    private long from_user_id;
    private long from_user_id_str;
    private String from_user_name;
    private String geo;
    private long id;
    private long id_str;
    private String iso_language_code;
//  "metadata":
//  {
//  "result_type":"recent"
//  },
    private String profile_image_url;
    private String profile_image_url_https;
    private String source;
    private String text;
    private String to_user;
    private String to_user_id;
    private String to_user_id_str;
    private String to_user_name;
}

Which I'm instantiating with

URI uri = new URI("http", "search.twitter.com", "/search.json", "q="+ searchTerms + "&rrp=" + RRP, null);
URL twitterSearch = uri.toURL();
URLConnection yc = twitterSearch.openConnection();
JsonReader reader = new JsonReader(new InputStreamReader(yc.getInputStream()));
Gson gson = new Gson();
tweetData data = gson.fromJson(reader, tweetData.class);
System.out.println(data);

The basic name:values are being populated correctly but the ArrayList is not.

tweetData : 0.17196614959919140865196614959919140865?page=2&max_id=196614959919140865&q=test1test?since_id=196614959919140865&q=testSIZE 0[]

So, I'm still struggling a bit - any more tips hugely appreciated!

Tia, Tom

Answer 1

Do I need to define a class which maps exactly to the structure of the JSON in order to then populate an instance of that class? If so this seems very inflexible/laborious.

Yes. GSON is a library that can be used to convert Java Objects into their JSON representation. It can also be used to convert a JSON string to an equivalent Java object. This is really powerful because you can automagically instantiate your Java objects from the JSON representation. Assuming your JSON doesn't change its structure, you only have to define the appropriate Java object representation once.

Ideally I'm looking for something which will handle JSON in any form and give me a structure I can use automatically...

However, if you don't want automagical serialisation/deserialisation, then try looking at a simpler library such as java.net/projects/jsonp.

You can extract stuff from it just by querying the keys:

final JSONObject json = new JSONObject(theJsonString);
final String id = json.getString("max_id");
final JSONArray results = json.getJSONArray("results");
final String user = results.getJSONObject(2).getString("from_user");