Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What is the most efficient (fastest) way to concatenate two large (over 1.5GB) files in java?

Tags:

java

file

io

nio

I have utilized the techniques here and concatenated two 1.5GB files in 70 seconds.

http://nadeausoftware.com/articles/2008/02/java_tip_how_read_files_quickly

My code involved using FileChannels with memory mapping and ByteBuffers with an 8KB buffer size.

How could I improve this speed?

File file = new File(binDirectory + "/donjon.avi");
File oFile = new File(binDirectory + "/donjon2.avi");

FileInputStream is = new FileInputStream(file);
FileOutputStream fos = new FileOutputStream(oFile);
FileChannel f1 = is.getChannel();
FileChannel f2 = fos.getChannel();

f2.transferFrom(f1, 0, f1.size());
f2.transferFrom(f1, f1.size(), f1.size());

f2.close();
f1.close();
like image 568
Danny Rancher Avatar asked Jan 03 '14 05:01

Danny Rancher


People also ask

What is the best method to write large amount of data to a file?

The best solution would be implement own Writer which directly uses write(byte[]) method of FileOutputStream which used underlying native writeBytes method . like @DavidMoles said source format of data is also very important in this scenario. If data is already available in bytes write directly to FileOutputSteam.

How do you process large files?

Process Large File In Chunks (BufferdInputStream) We will use BufferedInputStream stream with the same size buffer as we used for FileChannels, and analyse the results. Next is an example of Reading and Writing Large Files in Chunks using Java BufferedInputStream. And, the performance we see is similar to the Scanner.

How do you concatenate files?

To choose the merge option, click the arrow next to the Merge button and select the desired merge option. Once complete, the files are merged. If there are multiple files you want to merge at once, you can select multiple files by holding down the Ctrl and selecting each file you want to merge.


1 Answers

try this

    FileChannel c1 = new FileInputStream("1").getChannel();
    FileChannel c2 = new FileOutputStream("2", true).getChannel();
    c2.transferFrom(c1, c2.size(), c1.size());

javadoc says that FileChannel.transferFrom is potentially much more efficient than a simple loop that reads from this channel and writes to the target channel. Many operating systems can transfer bytes directly from the filesystem cache to the target channel without actually copying them.

like image 114
Evgeniy Dorofeev Avatar answered Oct 24 '22 02:10

Evgeniy Dorofeev