Generate a large stream for testing

Question

We have a web service where we upload files and want to write an integration test for uploading a somewhat large file. The testing process needs to generate the file (I don't want to add some larger file to source control).

I'm looking to generate a stream of about 50 MB to upload. The data itself does not much matter. I tried this with an in-memory object and that was fairly easy, but I was running out of memory.

The integration tests are written in Groovy, so we can use Groovy or Java APIs to generate the data. How can we generate a random stream for uploading without keeping it in memory the whole time?

Answer 1

Here is a simple program which generates a 50 MB text file with random content.

import java.io.PrintWriter;
import java.util.Random;


public class Test004 {

    public static void main(String[] args) throws Exception {
        PrintWriter pw = new PrintWriter("c:/test123.txt");
        Random rnd = new Random();
        for (int i=0; i<50*1024*1024; i++){
            pw.write('a' + rnd.nextInt(10));
        }
        pw.flush();
        pw.close();
    }

}

Answer 2

You could construct a mock/dummy implementation of InputStream to supply random data, and then pass that in wherever your class/library/whatever is expecting an InputStream.

Something like this (untested):

class MyDummyInputStream extends InputStream {
    private Random rn = new Random(0);

    @Override
    public byte read() { return (byte)rn.nextInt(); }
}

Of course, if you need to know the data (for test comparison purposes), you'll either need to save this data somewhere, or you'll need to generate algorithmic data (i.e. a known pattern) rather than random data.

(Of course, I'm sure you'll find existing frameworks that do all this kind of thing for you...)