Twitter4j TwitterStream or BufferReading causing java.lang.OutOfMemoryError: Java Heap Space

Question

This is my first Stackoverflow post, sorry if it's not great. Feedback will definitely help!

I'm currently running into a java.lang.OutOfMemoryError: Java Heap space problem with a project I'm working on using the Twitter Streaming API.

The error comes up after streaming in around 500 - 1000 tweets and I can't pinpoint what is causing it to occur.

In the onStatus method for the StatusListener I have the following code:

public void onStatus(Status status) {

        tweetCount++;
        System.out.println("Tweet #" + tweetCount);

        String statusInfo = status.getText().replaceAll("\n", "").replaceAll("\r", "");

        String usersCountry = getTweetUserLocation(status);
        status = null;

        if(!usersCountry.equals("INVALID_LOCATION")){
           countryList.updateWhoTalkedAboutWho(usersCountry, statusInfo);
        }

        try {
           Thread.sleep(1000);
        } catch (InterruptedException e) {
           // TODO Auto-generated catch block
           System.out.println("Exception in onStatus() catch block");
           e.printStackTrace();
        }
     }

Could the error be caused by the speed at which the status are coming in? I can see why it would take up more memory if status are coming in much faster than they are being processed.

The other suspect is the BufferReader, here is the code:

URL url = new URL(urlStr);
URLConnection conn = url.openConnection();

BufferedReader rd = new BufferedReader(new InputStreamReader(
       conn.getInputStream()));
StringBuffer sb = new StringBuffer();
String line;

while ((line = rd.readLine()) != null) {
    sb.append(line);
}

rd.close();
rd=null;
result = sb.toString();

Please let me know if you have an idea why the OutOfMemoryError is occurring. If you want to see the entire code, please check out my GitHub repository

Here is the stack trace after hitting the Error:

Exception in thread "Twitter Stream consumer-1[Receiving stream]" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
at java.lang.StringBuffer.append(StringBuffer.java:306)
at java.io.BufferedReader.readLine(BufferedReader.java:333)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at twitter4j.StatusStreamBase.handleNextElement(StatusStreamBase.java:85)
at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:57)
at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:478)
Exception in thread "Twitter4J Async Dispatcher[0]" java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedReader.<init>(BufferedReader.java:80)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at TweetCountry.sendGetRequest(TweetCountry.java:75)
at TweetCountry.findCountryName(TweetCountry.java:28)
at StreamTweets.getTweetUserLocation(StreamTweets.java:135)
at StreamTweets.access$4(StreamTweets.java:115)
at StreamTweets$1.onStatus(StreamTweets.java:45)
at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75)
at twitter4j.StatusStreamBase$1.run(StatusStreamBase.java:114)
at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:116)

Answer 1

I now know what I was doing wrong! The onStatus() method in the status listener should be (at most) storing the incoming status information instead of processing. I'll be using two programs this time, one for populating a DB or file with the status information, and another to process that data.