Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Java : Merging multiple log files by date

Tags:

java

I have 2 log files. I need to merge those 2 files based on the timestamp. I am struggling to find the correct algorithm to do the same.

I have the file 'File1' with content:

2016-07-18 09:58:19,243 : INFO: My Test File1 - 1
2016-07-18 09:58:19,244 : INFO: My Test File1 - 2
2016-07-18 09:58:19,255 : INFO: My Test File1 - 3
2016-07-18 09:58:19,255 : INFO: My Test File1 - 4
2016-07-18 09:58:19,258 : INFO: My Test File1 - 5

The 'File2' with content:

2016-07-18 09:57:09,674 : INFO: My Test File2 - 1
2016-07-18 09:57:09,674 : INFO: My Test File2 - 2
2016-07-18 09:57:09,679 : INFO: My Test File2 - 3
2016-07-18 09:57:09,679 : INFO: My Test File2 - 4
2016-07-18 09:57:09,680 : INFO: My Test File2 - 5
2016-07-18 09:58:49,685 : INFO: My Test File2 - 6
2016-07-18 09:58:49,686 : INFO: My Test File2 - 7
2016-07-18 09:58:49,686 : INFO: My Test File2 - 8

Expected Output File after merge:

2016-07-18 09:57:09,674 : INFO: My Test File2 - 1
2016-07-18 09:57:09,674 : INFO: My Test File2 - 2
2016-07-18 09:57:09,679 : INFO: My Test File2 - 3
2016-07-18 09:57:09,679 : INFO: My Test File2 - 4
2016-07-18 09:57:09,680 : INFO: My Test File2 - 5
2016-07-18 09:58:19,243 : INFO: My Test File1 - 1
2016-07-18 09:58:19,244 : INFO: My Test File1 - 2
2016-07-18 09:58:19,255 : INFO: My Test File1 - 3
2016-07-18 09:58:19,255 : INFO: My Test File1 - 4
2016-07-18 09:58:19,258 : INFO: My Test File1 - 5
2016-07-18 09:58:49,685 : INFO: My Test File2 - 6
2016-07-18 09:58:49,686 : INFO: My Test File2 - 7
2016-07-18 09:58:49,686 : INFO: My Test File2 - 8

The below is the code which I have wrote to merge the 2 file.

private static final String DATE_FORMAT_REGEX = "(\\d{4})-(\\d{2})-(\\d{2}) (\\d{2}):(\\d{2}):(\\d{2}).*";
private static final Pattern DATE_FORMAT_PATTERN = Pattern.compile(DATE_FORMAT_REGEX);
private static final String COLON = " : ";
// 2016-07-18 09:57:09,674
private static final String DATE_PATTERN = "yyyy-MM-dd HH:mm:ss,SSS";
File file1 = new File("C:\\Users\\Admin\\Downloads\\log.log");
File file2 = new File("C:\\Users\\Admin\\Downloads\\log2.log");

LineIterator it = FileUtils.lineIterator(file1,"UTF-8");
LineIterator it1 = FileUtils.lineIterator(file2,"UTF-8");
PrintWriter writer = new PrintWriter("C:\\logMerge.txt", "UTF-8");
SimpleDateFormat sdf = new SimpleDateFormat(DATE_PATTERN);
      try {
            while (it.hasNext()) {
            String line = it.nextLine();
                    String date = line.split(COLON)[0].trim();
                    Matcher matcher = DATE_FORMAT_PATTERN.matcher(date);
                    if (matcher.matches()) {
                        System.out.println(line);
                        while (it1.hasNext()) {
                            String line1 = it1.nextLine();
                            String date1 = line1.split(COLON)[0].trim();
                            Matcher matcher1 = DATE_FORMAT_PATTERN.matcher(date1);
                            if (matcher1.matches()) {
                                Date convertedDate = sdf.parse(date);
                                Date convertedDate1 = sdf.parse(date1);
                                if (convertedDate.before(convertedDate1)) {

                                    writer.println(line);
                                    break;
                                }
                                else if (convertedDate1.before(convertedDate)) {
                                    writer.println(line1);
                                }
                                else {
                                    writer.println(line1);
                                }
                            }
                            else {
                                writer.println(line1);
                            }
                        }   
                    }
                    else {
                        writer.println(line);
                    }
                }
            }
            catch (ParseException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
            finally {
                it.close();
            }

But with the above code I am not able to merge both the record. Any help is appreciated.

Edit:

Actual result

2016-07-18 09:57:09,674 : INFO: My Test File2 - 1
2016-07-18 09:57:09,674 : INFO: My Test File2 - 2
2016-07-18 09:57:09,679 : INFO: My Test File2 - 3
2016-07-18 09:57:09,679 : INFO: My Test File2 - 4
2016-07-18 09:57:09,680 : INFO: My Test File2 - 5
2016-07-18 09:58:49,685 : INFO: My Test File2 - 6
2016-07-18 09:58:49,686 : INFO: My Test File2 - 7
2016-07-18 09:58:49,686 : INFO: My Test File2 - 8
like image 926
MMA Avatar asked Aug 22 '16 07:08

MMA

2 Answers

IMHO the basic algorithm is the following:

  • Get the FileList.
  • Create BufferedReader list with respect to the file.
  • Read the 1st line of each line and store it in another list.
  • Store the date from the 1st line of each file.
  • While BufferedReader's size is not zero then, (repeat the process from 6 - 12 if 5 is true)
  • Take the index of minimum date from dateList
  • Get the Line with the index you got in the previous step (lineToWrite) and then write the lineToWrite.
  • Get the buffered reader with the index.
  • If the BR is not null then read the line.
  • If line is not equal to null then remove the lineList from the index and add the line to index. If line is null then remove the min indexed line from lineList,dateList,BufferedReader list. Do BufferedReader--.
  • If the line length is greater than 23 (yyyy-MM-dd HH:mm:ss,SSS) then take the first 23 String from the line. Check if the date matches to the pattern, if matches then add the date to dateList
  • Repeat the process from Step5

Here is the code which does combine the log files.

    public static void main(String[] args) throws Exception {
    final String DATE_FORMAT_REGEX = "(\\d{4})-(\\d{2})-(\\d{2}) (\\d{2}):(\\d{2}):(\\d{2}).*";
    final Pattern DATE_FORMAT_PATTERN = Pattern.compile(DATE_FORMAT_REGEX);
    final String DATE_PATTERN = "yyyy-MM-dd HH:mm:ss,SSS";
    SimpleDateFormat sdf = new SimpleDateFormat(DATE_PATTERN);
    String file1 = "C:\\templog\\myLog1.log";
    String file2 = "C:\\templog\\myLog2.log";
    PrintWriter writer = new PrintWriter("C:\\templog\\expectedLog.txt", "UTF-8");
    // 1) Get the file list
    List<String> fileFile = Arrays.asList(file1, file2);
    ArrayList<BufferedReader> readerIndex = new ArrayList<BufferedReader>();
    ArrayList<String> lineList = new ArrayList<String>();
    ArrayList<Date> dateList = new ArrayList<Date>();
    for (String file : fileFile) {
        // 2) Create BufferedReader list with respect to the file.
        BufferedReader br1 = new BufferedReader(new FileReader(file));
        readerIndex.add(br1);
        // 3) Read the 1st line of each line and store it in another list.
        String line = br1.readLine();
        lineList.add(line);
        // 4) Store the date from the 1st line of each file.
        String date = line.substring(0, 23);
        Date convertedDate = sdf.parse(date);
        dateList.add(convertedDate);
    }
    int index = readerIndex.size();
    // 5) While BufferedReader's size is not zero then,
    while (index > 0) {
        // 6) Take the index of minimum date from dateList
        int indexMin = minDateIndex(dateList);
        // 7) Get the Line with the index you got in the previous step (lineToWrite).
        String lineToWrite = lineList.get(indexMin);
        writer.println(lineToWrite);
        // 8) Get the buffered reader with the index.
        BufferedReader br1 = readerIndex.get(indexMin);
        if (br1 != null) {
            //  9) If the BR is not null then read the line.
            String line = br1.readLine();
            if (line != null) {
                // 10)If line is not equal to null then remove the lineList from the index and add the line to index.
                lineList.remove(indexMin);
                lineList.add(indexMin, line);
                if (line.length() > 23) {
                    // 11)If the line length is greater than 23 (yyyy-MM-dd HH:mm:ss,SSS) then take the first 23 String from the line.
                    String date = line.substring(0, 23);
                    Matcher matcher = DATE_FORMAT_PATTERN.matcher(date);
                    if (matcher.matches()) {
                        // 12)Check if the date matches to the pattern, if matches then add the date to dateList
                        Date convertedDate = sdf.parse(date);
                        dateList.remove(indexMin);
                        dateList.add(indexMin, convertedDate);
                    }
                }
            } else {
                //If line is null then remove the min indexed line from lineList,dateList,BufferedReader list. Do BufferedReader--.
                lineList.remove(indexMin);
                dateList.remove(indexMin);
                readerIndex.remove(indexMin);
                br1.close();
                index--;
            }
        }
    }
    writer.close();
}

private static int minDateIndex(ArrayList<Date> dateList) {
    // return index of min date
    Date minDate = dateList.get(0);
    int minIndex = 0;
    for (int i = 1; i < dateList.size(); i++) {
        Date currentDate = dateList.get(i);
        if (minDate != null) {
            if (currentDate != null) {
                if (minDate.after(currentDate)) {
                    // We have a new min
                    minDate = dateList.get(i);
                    minIndex = i;
                } else {
                    // we keep current min
                }
            } else {
                // we keep current min
            }
        } else {
            minDate = currentDate;
            minIndex = i;
        }
    }
    return minIndex;
}

Test:

myLog1.log

2016-08-24 16:32:13,888[main] INFO  org.hibernate.cfg.Environment - HHH000206: hibernate.properties not found
2016-08-24 16:40:13,888 [main] INFO  o.h.tool.hbm2ddl.SchemaUpdate - HHH000396: Updating schema

myLog2.log

2016-08-24 16:33:13,888 : SEVERE : http-bio-8983-exec-3 : StandardWrapperValve.invoke :    Servlet.service() for servlet [Faces Servlet] in context with path [/TestApp] threw exception [Cannot create a session after the response has been committed] with root cause
java.lang.IllegalStateException: Cannot create a session after the response has been committed
    at org.apache.catalina.connector.Request.doGetSession(Request.java:2882)
    at org.apache.catalina.connector.Request.getSession(Request.java:2316)
    at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:898)
    at com.sun.faces.context.ExternalContextImpl.getSession(ExternalContextImpl.java:155)
    at com.sun.faces.mgbean.BeanManager$ScopeManager$SessionScopeHandler.handle(BeanManager.java:575)
    at com.sun.faces.mgbean.BeanManager$ScopeManager.pushToScope(BeanManager.java:458)
    at com.sun.faces.mgbean.BeanManager.createAndPush(BeanManager.java:410)
    at com.sun.faces.mgbean.BeanManager.create(BeanManager.java:269)
    at com.sun.faces.el.ManagedBeanELResolver.resolveBean(ManagedBeanELResolver.java:244)
    at com.sun.faces.el.ManagedBeanELResolver.getValue(ManagedBeanELResolver.java:116)
    at com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:176)
    at com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:203)
    at org.apache.el.parser.AstIdentifier.getValue(AstIdentifier.java:72)
    at org.apache.el.parser.AstValue.getTarget(AstValue.java:94)
    at org.apache.el.parser.AstValue.setValue(AstValue.java:210)
    at org.apache.el.ValueExpressionImpl.setValue(ValueExpressionImpl.java:253)
    at org.apache.jasper.el.JspValueExpression.setValue(JspValueExpression.java:89)
    at javax.faces.component.UIComponent.processEvent(UIComponent.java:2185)
    at com.sun.faces.lifecycle.RestoreViewPhase$1.visit(RestoreViewPhase.java:276)
    at com.sun.faces.component.visit.FullVisitContext.invokeVisitCallback(FullVisitContext.java:151)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1446)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at javax.faces.component.UIForm.visitTree(UIForm.java:333)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at com.sun.faces.component.visit.VisitUtils.doFullNonIteratingVisit(VisitUtils.java:75)
    at com.sun.faces.lifecycle.RestoreViewPhase.deliverPostRestoreStateEvent(RestoreViewPhase.java:271)
    at com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:251)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
    at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:111)
    at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:312)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:349)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.ow2.jonas.web.tomcat7.CheckOpenResourcesValve.invoke(CheckOpenResourcesValve.java:69)
    at org.ow2.jonas.web.tomcat7.tx.TransactionValve.invoke(TransactionValve.java:89)
    at org.ow2.jonas.web.tomcat7.ResetAuthenticationValve.invoke(ResetAuthenticationValve.java:95)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.ow2.jonas.web.tomcat7.versioning.VersioningValve.invoke(VersioningValve.java:105)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)

After running the program in expectedLog.txt

2016-08-24 16:32:13,888[main] INFO  org.hibernate.cfg.Environment - HHH000206: hibernate.properties not found
2016-08-24 16:33:13,888 : SEVERE : http-bio-8983-exec-3 : StandardWrapperValve.invoke :    Servlet.service() for servlet [Faces Servlet] in context with path [/TestApp] threw exception [Cannot create a session after the response has been committed] with root cause
java.lang.IllegalStateException: Cannot create a session after the response has been committed
    at org.apache.catalina.connector.Request.doGetSession(Request.java:2882)
    at org.apache.catalina.connector.Request.getSession(Request.java:2316)
    at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:898)
    at com.sun.faces.context.ExternalContextImpl.getSession(ExternalContextImpl.java:155)
    at com.sun.faces.mgbean.BeanManager$ScopeManager$SessionScopeHandler.handle(BeanManager.java:575)
    at com.sun.faces.mgbean.BeanManager$ScopeManager.pushToScope(BeanManager.java:458)
    at com.sun.faces.mgbean.BeanManager.createAndPush(BeanManager.java:410)
    at com.sun.faces.mgbean.BeanManager.create(BeanManager.java:269)
    at com.sun.faces.el.ManagedBeanELResolver.resolveBean(ManagedBeanELResolver.java:244)
    at com.sun.faces.el.ManagedBeanELResolver.getValue(ManagedBeanELResolver.java:116)
    at com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:176)
    at com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:203)
    at org.apache.el.parser.AstIdentifier.getValue(AstIdentifier.java:72)
    at org.apache.el.parser.AstValue.getTarget(AstValue.java:94)
    at org.apache.el.parser.AstValue.setValue(AstValue.java:210)
    at org.apache.el.ValueExpressionImpl.setValue(ValueExpressionImpl.java:253)
    at org.apache.jasper.el.JspValueExpression.setValue(JspValueExpression.java:89)
    at javax.faces.component.UIComponent.processEvent(UIComponent.java:2185)
    at com.sun.faces.lifecycle.RestoreViewPhase$1.visit(RestoreViewPhase.java:276)
    at com.sun.faces.component.visit.FullVisitContext.invokeVisitCallback(FullVisitContext.java:151)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1446)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at javax.faces.component.UIForm.visitTree(UIForm.java:333)
    at javax.faces.component.UIComponent.visitTree(UIComponent.java:1457)
    at com.sun.faces.component.visit.VisitUtils.doFullNonIteratingVisit(VisitUtils.java:75)
    at com.sun.faces.lifecycle.RestoreViewPhase.deliverPostRestoreStateEvent(RestoreViewPhase.java:271)
    at com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:251)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
    at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:111)
    at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:312)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:349)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.ow2.jonas.web.tomcat7.CheckOpenResourcesValve.invoke(CheckOpenResourcesValve.java:69)
    at org.ow2.jonas.web.tomcat7.tx.TransactionValve.invoke(TransactionValve.java:89)
    at org.ow2.jonas.web.tomcat7.ResetAuthenticationValve.invoke(ResetAuthenticationValve.java:95)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.ow2.jonas.web.tomcat7.versioning.VersioningValve.invoke(VersioningValve.java:105)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
2016-08-24 16:40:13,888 [main] INFO  o.h.tool.hbm2ddl.SchemaUpdate - HHH000396: Updating schema

Hope it helps !

like image 134
Unknown Avatar answered Oct 03 '22 04:10

Unknown

I wouldn't even bother parsing the dates. A simple alphanumerical sort using sort would provide the correct result since the log writer has the nice idea of writing year-month-day format which matches alphanum format (otherwise a date parsing would have been necessary)

MSYS/linux (using GNU sort)

sort File1 File2 > Sorted.txt

Windows CMD-line (using Windows sort):

(type File1 & type File2) | sort > Sorted.txt

unless you absolutely want to bother Java for that, but since you are writing a main program, I suspect that it is equivalent to the cmdline solution.

like image 28
Jean-François Fabre Avatar answered Oct 03 '22 06:10

Jean-François Fabre
Sign in to Comment

Related questions

Fire and forget for HTTP in Java
Selenium Webdriver - Element not visible
Unit test for togglz feature on and off
Scala converters convert Java collections to Wrapper objects
JSON Deserialization from ID into Object
Joda Time get Duration between two Instants
Integration Test Coverage for a webapp deployed on tomcat?
intelliJ code-inspection standalone using Java
Android -Retained headless fragment
java.lang.ClassCastException: com.google.gson.internal.LinkedTreeMap cannot be cast to model
SQL Server Nvarchar and Java prepared statement
In Java Priority Queue implementation remove at method, why it does a sift up after a sift down?
Currency Symbol given by DecimalFormat looks invalid
Eclipse shows warning for checkstyle SuppresWarnings that exists
How does google-services.json replace default_web_client_id?
How to change my method to a generic method?
Connecting to remote Windows machine with JSch
org.apache.thrift.transport.TTransportException: Read a negative frame size (-2080374784)!
JUnit rollback transaction with @Async method
An efficient way to get and store the shortest paths

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!