Secure / Encrypt log4j files

Question

I have a problem ; security requirement i have java swing app that have logging files generated with log4j for support issues in case of tracking a bug.

I have to ecrypt/cypher/secure the files so the client cant open them and see them (at least not as human readable way) and at the same time when support tech team take these files they will know how to read (decrypt) them .

I did a lot of searches and i tried my best option i found which is build custom appender by extending SkeletonAppender .

Now know that i have log4j working great as below configuration, but i created new class to encrypt it but i cant get it work even with simple setup it dose not create the file , so i can continue in the ecnryption part.

Any help , links are good.

Working...version

<appender name="cache" class="com.MyAppender">  
            <param name="Threshold" value="ALL" />
            <param name="ImmediateFlush" value="true" />  
            <param name="File" value="${home}/logs/cache.log"/> 
            <param name="Append" value="true"/>
            <param name="Threshold" value="ALL" />
            <param name="Encoding" value="UTF-8" />

            <layout class="org.apache.log4j.EnhancedPatternLayout">
            <param name="ConversionPattern" value="%-5p %d{MMM-dd-yyyy HH:mm:ss,SSS} %c{1} - %m%n" />
        </layout>
    </appender>

Not Working...version

   <appender name="cache" class="com.MyAppender">   
            <param name="Threshold" value="ALL" />
            <param name="ImmediateFlush" value="true" />  
            <param name="File" value="${home}/logs/cache.log"/> 
            <param name="Append" value="true"/>
            <param name="Threshold" value="ALL" />
            <param name="Encoding" value="UTF-8" />

            <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
                 <param name="FileNamePattern"
            value="${home}/logs/cache.%d{yyyy-MM-dd-HH}.gz" />
                 <param name="ActiveFileName" value="${home}/logs/cache.log" />
             </rollingPolicy> 

            <layout class="org.apache.log4j.EnhancedPatternLayout">
            <param name="ConversionPattern"
                value="%-5p %d{MMM-dd-yyyy HH:mm:ss,SSS} %c{1} - %m%n" />
        </layout>
    </appender>

The simple class test

http://www.javaworld.com/article/2075817/core-java/customized-appender-extending-org-apache-log4j-fileappender.html

package com.MyAppender;

import org.apache.log4j.spi.LoggingEvent;

public class MyAppender extends org.apache.log4j.RollingFileAppender {

    private String file;
    private boolean initialized = false;
    private String baseFileName = null;

    // private static final Log log = LogFactory.getLog(MyAppender.class);

    /**
     * 
     * write to ActivityLog
     * 
     * @param event
     *            logging event invoked.
     * 
     */
    @Override
    protected void subAppend(LoggingEvent event) {
        if (!initialized) {
            createNewFile();
        }
        synchronized (this) {
            super.subAppend(event);
        }
    }

    /**
     * 
     * create a new ActivityLog File
     * 
     */
    public void createNewFile() {
        try {
            baseFileName = file;
            super.setFile(baseFileName);
            super.activateOptions();
            initialized = true;
        } catch (Exception e) {
            // log.error("*#*Error in configuration of log4j params,unable to create ActivityLog file");
        }
    }

    /**
     * 
     * invokes File Appender's activateOptions() which controls the creation of
     * log files.
     * 
     */
    @Override
    public void activateOptions() {
        super.setFile(file);
        super.activateOptions();
    }

    /**
     * 
     * Close and rename the current ActivityLog file and reset counter and
     * timestamp.
     * 
     */
    public void rollOver() {
        closeFile();
        initialized = false;
    }

    @Override
    public void setFile(String file) {
        this.file = file;
    }

}

Then i plan to implement the code in Cipher OutputStream

Answer 1

A possible workaround to the problem is to write the logs to an embedded database that supports encryption, e.g. H2 natively supports encryption and SQLite has open source encryption extensions - this way you can just use the JDBCAppender and let the database take care of encryption without having to worry about a custom appender.

---

From this question, SQLite config would look something like

<appender name="jdbcAppender" class="org.apache.log4j.jdbc.JDBCAppender">
    <param name="URL" value="jdbc:sqlite:D:/download/mapLogic/sf_log.db" />
    <param name="user" value="" />
    <param name="password" value="" />
    <param name="driver" value="org.sqlite.JDBC" />
    <param name="sql"
        value="INSERT INTO Log(Message,Priority,Logger,Date) VALUES ('%m','%p','%c','%d{ABSOLUTE}')" />
</appender>

where your log table looks like

CREATE TABLE Log (
    LogId        INTEGER PRIMARY KEY,
    Date         DATETIME NOT NULL,
    Level        VARCHAR(50) NOT NULL,
    Logger       VARCHAR(255) NOT NULL,
    Message      TEXT DEFAULT NULL
);

Documentation on the JDBCAppender can be found here

---

There's an official encryption extension for SQLite as well as at least one third party open source extension; I've never had to encrypt SQLite, but if I had to do so then I'd go with the official extension unless I ran into problems with it.

---

If you're running this on the client, then ideally you'll be able to have the program phone home at boot time to get the database encryption key so that the key never exists on the client's disk drive (ignoring the possibility that it goes to the swap file) - the client could still use a debugger or whatever to try to get the key out of memory, but presumably they're not interested enough in decrypting the logs to go to that amount of trouble. If you've got to store the key on the client side then you can at a minimum obfuscate it by hashing it several times before using it, e.g. hard-code the base_key in the program, then at boot time you create actual_key by running base_key through SHA512 (or whatever) several times; the client could still figure out what you're doing by using a debugger, but again they hopefully won't want to go to the trouble.

Answer 2

Option 1: Use a Custom SocketAppender

As an alternative to Zim-Zam's answer about using a JDBC-capable appender (remember to enable secure transport as well, by the way, if you go down this route), you could also look into using a SocketAppender and roll out your own encryption method.

Option 2: Use Flume and a FlumeAppender

Refer to the log4j documentation on appenders and have a look at using a FlumeAppender, which supports event encryption:

A sample FlumeAppender configuration that is configured with a primary and a secondary agent, compresses the body, formats the body using the RFC5424Layout, and persists encrypted events to disk. This sample "compresses the body, formats the body using the RFC5424Layout, and persists encrypted events to disk:"

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
  <Appenders>
    <Flume name="eventLogger" compress="true" type="persistent" dataDir="./logData">
      <Agent host="192.168.10.101" port="8800"/>
      <RFC5424Layout enterpriseNumber="18060" includeMDC="true" appName="MyApp"/>
      <Property name="keyProvider">MySecretProvider</Property>
    </Flume>
  </Appenders>
  <Loggers>
    <Root level="error">
      <AppenderRef ref="eventLogger"/>
    </Root>
  </Loggers>
</Configuration>

---

Interesting Reads

This does not answer your question directly, but is fairly interesting as well: Creating an encrypted log file