Can someone tell me how to clone an inputstream, taking as little creation time as possible? I need to clone an inputstream multiple times for multiple methods to process the IS. I've tried three ways and things don't work for one reason or another.
Method #1: Thanks to the stackoverflow community, I found the following link helpful and have incorporated the code snippet in my program.
How to clone an InputStream?
However, using this code can take up to one minute (for a 10MB file) to create the cloned inputstreams and my program needs to be as fast as possible.
int read = 0;
byte[] bytes = new byte[1024*1024*2];
ByteArrayOutputStream bos = new ByteArrayOutputStream();
while ((read = is.read(bytes)) != -1)
bos.write(bytes,0,read);
byte[] ba = bos.toByteArray();
InputStream is1 = new ByteArrayInputStream(ba);
InputStream is2 = new ByteArrayInputStream(ba);
InputStream is3 = new ByteArrayInputStream(ba);
Method #2: I also tried using BufferedInputStream to clone the IS. This was fast (slowest creation time == 1ms. fastest == 0ms). However, after I sent is1 to be processed, the methods processing is2 and is3 threw an error saying there was nothing to process, almost like all 3 variables below referenced the same IS.
is = getFileFromBucket(path,filename);
...
...
InputStream is1 = new BufferedInputStream(is);
InputStream is2 = new BufferedInputStream(is);
InputStream is3 = new BufferedInputStream(is);
Method #3: I think the compiler is lying to me. I checked markSupported() for is1 for the two examples above. It returned true so I thought I could run
is1.mark()
is1.reset()
or just
is1.reset();
before passing the IS to my respective methods. In both of the above examples, I get an error saying it's an invalid mark.
I'm out of ideas now so thanks in advance for any help you can give me.
P.S. From the comments I've received from people, I need to clarify a couple things regarding my situation: 1) This program is running on a VM 2) The inputstream is being passed into me from another method. I'm not reading from a local file 3) The size of the inputstream is not known
If all you want to do is read the same information more than once, and the input data is small enough to fit into memory, you can copy the data from your InputStream to a ByteArrayOutputStream. Then you can obtain the associated array of bytes and open as many "cloned" ByteArrayInputStreams as you like.
Another solution would be to convert InputStream to byte array, then iterate over the array as many time as you need. You can find several solutions in this post Convert InputStream to byte array in Java using 3rd party libs or not. Caution, if the read content is too big you might experience some memory troubles.
The operating system will only allow a single process to open a certain number of files, and if you don't close your input streams, it might forbid the JVM from opening any more. I am talking about an InputStream in general.
InputStream is = new URL(someUrl). openStream(); long length = is. skip(Long. MAX_VALUE);
how to clone an inputstream, taking as little creation time as possible? I need to clone an inputstream multiple times for multiple methods to process the IS
You could just create some kind of a custom ReusableInputStream
class wherein you immediately also write to an internal ByteArrayOutputStream
on the 1st full read, then wrap it in a ByteBuffer
when the last byte is read and finally reuse the very same ByteBuffer
on the subsequent full reads which get automatically flipped when limit is reached. This saves you from one full read as in your 1st attempt.
Here's a basic kickoff example:
public class ReusableInputStream extends InputStream {
private InputStream input;
private ByteArrayOutputStream output;
private ByteBuffer buffer;
public ReusableInputStream(InputStream input) throws IOException {
this.input = input;
this.output = new ByteArrayOutputStream(input.available()); // Note: it's resizable anyway.
}
@Override
public int read() throws IOException {
byte[] b = new byte[1];
read(b, 0, 1);
return b[0];
}
@Override
public int read(byte[] bytes) throws IOException {
return read(bytes, 0, bytes.length);
}
@Override
public int read(byte[] bytes, int offset, int length) throws IOException {
if (buffer == null) {
int read = input.read(bytes, offset, length);
if (read <= 0) {
input.close();
input = null;
buffer = ByteBuffer.wrap(output.toByteArray());
output = null;
return -1;
} else {
output.write(bytes, offset, read);
return read;
}
} else {
int read = Math.min(length, buffer.remaining());
if (read <= 0) {
buffer.flip();
return -1;
} else {
buffer.get(bytes, offset, read);
return read;
}
}
}
// You might want to @Override flush(), close(), etc to delegate to input.
}
(note that the actual job is performed in int read(byte[], int, int)
instead of in int read()
and thus it's expected to be faster when the caller itself is also streaming using a byte[]
buffer)
You could use it as follows:
InputStream input = new ReusableInputStream(getFileFromBucket(path,filename));
IOUtils.copy(input, new FileOutputStream("/copy1.ext"));
IOUtils.copy(input, new FileOutputStream("/copy2.ext"));
IOUtils.copy(input, new FileOutputStream("/copy3.ext"));
As to the performance, 1 minute per 10MB is more likely a hardware problem, not a software problem. My 7200rpm laptop harddisk does it in less than 1 second.
However, using this code can take up to one minute (for a 10MB file) to create the cloned inputstreams and my program needs to be as fast as possible.
Well copying a stream takes time, and (in general) that is the only way to clone a stream. Unless you tighten the scope of the problem, there is little chance that the performance can be significantly improved.
Here are a couple of circumstances where improvement is possible:
If you knew beforehand the number of bytes in the stream then you can read directly into the final byte array.
If you knew that the data is coming from a file, you could create a memory mapped buffer for the file.
But the fundamental problem is that moving lots of bytes around takes time. And the fact that it is taking 1 minute for a 10Mb file (using the code in your Question) suggests that the real bottleneck is not in Java at all.
Regarding your first approach, the one consisting in putting all your bytes in an ByteArrayOutputStream:
You second approach is bogus, you cannot decorate the same input stream within different other streams and expect the things to work. As the bytes are consumed by one stream, the inner stream is exhausted as well, and cannot provide the other streams with accurate data.
Before I extend my answer let me ask, are your other methods expecting to receive copies of the input stream running on a separate thread? Because if so, this sounds like the work for the PipedOutputStream and PipedInputStream?
Do you intend the separate methods to run in parallel or sequentially? If sequentially, I see no reason to clone the input stream, so I have to assume you're planning to spin off threads to manage each stream.
I'm not near a computer right now to test this, but I'm thinking you'd be better off reading the input in chunks, of say 1024 bytes, and then pushing those chunks (or array copies of the chunks) onto your output streams with input streams attached to their thread ends. Have your readers block if there's no data available, etc.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With