How do I concatenate sequential files in order with Java?

Question

I have a directory that contains sequentially numbered log files and some Excel spreadsheets used for analysis. The log file are ALWAYS sequentially numbered beginning at zero, but the number of them can vary. I am trying to concatenate the log files, in the order they were created into a single text file which will be a concatenation of all the log files.

For instance, with log files foo0.log, foo1.log, foo2.log would be output to concatenatedfoo.log by appending foo1 after foo0, and foo2 after foo1.

I need to count all the files in the given directory with the extension of *.log, using the count to drive a for-loop that also generates the file name for concatenation. I'm having a hard time finding a way to count the files using a filter...none of the Java Turtorials on file operations seem to fit the situation, but I'm sure I'm missing something. Does this approach make sense? or is there an easier way?

int numDocs = [number of *.log docs in directory];
 //
for (int i = 0; i <= numberOfFiles; i++) {
     fileNumber = Integer.toString(i);
     try
     {
          FileInputStream inputStream = new FileInputStream("\\\\Path\\to\\file\\foo" + fileNumber + ".log");
          BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
          try
          {
               BufferedWriter metadataOutputData = new BufferedWriter(new FileWriter("\\\\Path\\to\\file\\fooconcat.log").append());
               metadataOutputData.close();
          }
           //
          catch (IOException e)  // catch IO exception writing final output
          {
               System.err.println("Exception: ");
               System.out.println("Exception: "+ e.getMessage().getClass().getName());
               e.printStackTrace();
          }
     catch (Exception e)    // catch IO exception reading input file
     {
          System.err.println("Exception: ");
          System.out.println("Exception: "+ e.getMessage().getClass().getName());
          e.printStackTrace();
     }
}

Answer 1

how about

public static void main(String[] args){

    final int BUFFERSIZE = 1024 << 8;
    File baseDir = new File("C:\\path\\logs\\");

    // Get the simple names of the files ("foo.log" not "/path/logs/foo.log")
    String[] fileNames = baseDir.list(new FilenameFilter() {
        @Override
        public boolean accept(File dir, String name) {
            return name.endsWith(".log");
        }
    });

    // Sort the names
    Arrays.sort(fileNames);

    // Create the output file
    File output = new File(baseDir.getAbsolutePath() + File.separatorChar + "MERGED.log");
    try{
        BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(output), BUFFERSIZE);
        byte[] bytes = new byte[BUFFERSIZE];
        int bytesRead;
        final byte[] newLine = "\n".getBytes(); // use to separate contents

        for(String s : fileNames){
            // get the full path to read from
            String fullName = baseDir.getAbsolutePath() + File.separatorChar + s;
            BufferedInputStream in = new BufferedInputStream(new FileInputStream(fullName),BUFFERSIZE);
            while((bytesRead = in.read(bytes,0,bytes.length)) != -1){
                out.write(bytes, 0, bytesRead);
            }
            // close input file and ignore any issue with closing it
            try{in.close();}catch(IOException e){}
            out.write(newLine); // seperation
        }

        out.close();
    }catch(Exception e){
        throw new RuntimeException(e);
    }
}

This code DOES assume that the "sequential naming" would be zero padded such that they will lexigraphically (?? sp) sort correctly. i.e. The files would be

• 0001.log (or blah0001.log, or 0001blah.log etc)
• 0002.log
• ....
• 0010.log

and not

• 1.log
• 2.log
• ...
• 10.log

The latter pattern will not sort correctly with the code I have given.

Answer 2

Here's some code for you.

File dir = new File("C:/My Documents/logs");
File outputFile = new File("C:/My Documents/concatenated.log");

Find the ".log" files:

File[] files = dir.listFiles(new FilenameFilter() {

    @Override
    public boolean accept(File file, String name) {
        return name.endsWith(".log") && file.isFile();
    }
});

Sort them into the appropriate order:

Arrays.sort(files, new Comparator<File>() {

    @Override
    public int compare(File file1, File file2) {
        return numberOf(file1).compareTo(numberOf(file2));
    }

    private Integer numberOf(File file) {
        return Integer.parseInt(file.getName().replaceAll("[^0-9]", ""));
    }
});

Concatenate them:

byte[] buffer = new byte[8192];

OutputStream out = new BufferedOutputStream(new FileOutputStream(outputFile));
try {
    for (File file : files) {
        InputStream in = new FileInputStream(file);
        try {
            int charCount;
            while ((charCount = in.read(buffer)) >= 0) {
                out.write(buffer, 0, charCount);
            }
        } finally {
            in.close();
        }
    }
} finally {
    out.flush();
    out.close();
}