How can REST API pass large JSON?

Question

I am building a REST API and facing this issue: How can REST API pass very large JSON?

Basically, I want to connect to Database and return the training data. The problem is in Database I have 400,000 data. If I wrap them into a JSON file and pass through GET method, the server would throw Heap overflow exception.

What methods we can use to solve this problem?

DBTraining trainingdata = new DBTraining();
@GET
@Produces("application/json")
@Path("/{cat_id}")
public Response getAllDataById(@PathParam("cat_id") String cat_id) {
    List<TrainingData> list = new ArrayList<TrainingData>();
    try {
        list = trainingdata.getAllDataById(cat_id);
        Gson gson = new Gson();
        Type dataListType = new TypeToken<List<TrainingData>>() {
        }.getType();
        String jsonString = gson.toJson(list, dataListType);
        return Response.ok().entity(jsonString).header("Access-Control-Allow-Origin", "*").header("Access-Control-Allow-Methods", "GET").build();

    } catch (SQLException e) {
        logger.warn(e.getMessage());
    }
    return null;
}

Answer 1

The RESTful way of doing this is to create a paginated API. First, add query parameters to set page size, page number, and maximum number of items per page. Use sensible defaults if any of these are not provided or unrealistic values are provided. Second, modify the database query to retrieve only a subset of the data. Convert that to JSON and use that as the payload of your response. Finally, in following HATEOAS principles, provide links to the next page (provided you're not on the last page) and previous page (provided you're not on the first page). For bonus points, provide links to the first page and last page as well.

By designing your endpoint this way, you get very consistent performance characteristics and can handle data sets that continue to grow.

The GitHub API provides a good example of this.