java.lang.OutOfMemoryError: Java heap space when try to convert Java Object to Json String

Question

I try to convert csv file to Json file 200K of objects where object represents 1 row in csv.

I have Java installed on 32 bit and Project configuration VM arguments: -Xmx1024m

However I get:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuffer.append(Unknown Source)
    at java.io.StringWriter.write(Unknown Source)
    at com.google.gson.stream.JsonWriter.string(JsonWriter.java:478)
    at com.google.gson.stream.JsonWriter.value(JsonWriter.java:328)
    at com.google.gson.Streams.write(Streams.java:113)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Streams.write(Streams.java:124)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Gson.toJson(Gson.java:362)
    at com.google.gson.Gson.toJson(Gson.java:346)
    at com.google.gson.Gson.toJson(Gson.java:260)
    at com.google.gson.Gson.toJson(Gson.java:240)
    at ConvertFromCsv2JsonTWC.init(ConvertFromCsv2JsonTWC.java:186)
    at ConvertFromCsv2JsonTWC.main(ConvertFromCsv2JsonTWC.java:48)

In row:

Gson gson = new Gson();

String output = gson.toJson(container);// <---- crash

for 50k rows it works fine.

This is a template of Json I build:

{
    "crs": {
        "type": "none"
    },
    "type": "FeatureCollection",
    "features": [{
        "geometry": {
            "type": "Point"
        },
        "properties": {
            "ap mac": "00:11:22:33:44:55",
            "ssid": "WiFi",
            "lat": "35.111111",
            "long": "-118.11111",
            "address": "370 xxxxxx",
            "city": "xxxxxxx",
            "state": "CA",
            "zip code": "11111",
            "country": "US",
            "business n": "",
            "location c": "Health Club/Gym",
            "location q": "",
            "indoor fla": "yes"
        },
        "point": [35.390284,
        -118.9929],
        "id": 0,
        "type": "Feature"
    },
          {...},
          ... 
          200000...

So I have 200K objects in properties list

The workaround is to create separate files 20k per each but it not good way.

How can I solve this?

Thanks,

Answer 1

I suggest you use streaming instead of trying to copy it all into a String.

Answer 2

You can use the Gson streaming API to instead walk (stream) the data, instead of trying to load it all at once.