Detect an executable file in java

Question

I have a use case where we are allowing users to upload files. Now in the back end java(controller which extracts file from http request and checks), i want to detect if user uploads any executable file. If he uploads, i have to discard that file. I have googled it but couldn't find a fine solution. Some people suggested verifying extension(.exe). But i am not sure how far it will filter exe files. I want to completely block executable files from uploading.

If any of you have come across this scenario or have solution about this, please let me know. I would be grateful to you.

I would be more happy if you can point me to any JAVA implementation or Java API or algorithm which cam do this job.

Answer 1

I suspect that, apart from the extension-checking method you have already mentioned, there will be no way to catch every possible case. Executable files are ultimately sequences of machine instructions which make them largely indistinguishable from any other data.

Despite this however, there are things you could look for in certain types of executable. For example:

• Windows uses the Portable Executable format, which should always start with the magic number 4d5a (ASCII characters MZ)
• ELF format executable used by Linux start with 7f454c46
• Java class files always begin with cafebabe (that's hex, not ASCII!).
• As far as I can see, Mach-O files used by Mac-OSX have the magic number feedface (hex again)

I suggest you create a FileInputStream or similar and read the first few bytes of the file, checking for these magic numbers. It doesn't detect any file which contains executable code, but it should stop files in these standard executable formats from being allowed, which I think is what you hoped for.

So for example:

public static boolean isExecutable(File file) {
  byte[] firstBytes = new byte[4];
  try {
    FileInputStream input = new FileInputStream(file);
    input.read(firstBytes);

    // Check for Windows executable
    if (firstBytes[0] == 0x4d && firstBytes[1] == 0x5a) {
      return true;
    }
    return false;
  }
  catch (Exception e) {
    e.printStackTrace();
  }
}

Also beware that it is possible to get a false positive, where you reject a file which was not executable. I don't know what type of file you are intending to have uploaded so you should consider how likely it is for this to happen.

Answer 2

To complete the response of devrobf: Each executable file (by which I mean that the file contains machine instructions) can be identified by the magic number contained in the file's metadata. The magic number is identified by it's size (in byte) and by it's offset (which can be different depending on the type of file). You can find a database that contains this information HERE.

For example EXE file :

Extension :     EXE     
Signature :     4D 5A
Description :   Windows|DOS executable file
MZ (ASCII)  
Sizet :         2 Bytes
Offset:         0 Bytes

As you will surely understand, doing a check only on the extension does not make it possible to determine with certainty what kind of executable file. As the proposed Cratylus. Why? because the following example:

touch notAnExecutableWithExtensionExe.exe

This command just create file with as extension "exe", but it's just file data.

Implementation in Java to make a correct check of any kind of file :

public enum ExecutableSignatures{
    WINDOWS_EXE("Windows|DOS executable file",  (byte) 0x00, (byte) 0x02, 
                new byte[]{(byte)0x4d, (byte)0x5a}),
    JAVA_BYTECODE("Java Bytecode",              (byte) 0x00, (byte) 0x04, 
                new byte[]{(byte)0xca, (byte)0xfe, (byte)0xba, (byte)0xbe});

     /* Here more enumeration */
private String description;
private byte offset;
private byte size;
private byte[] magicNumber;

private ExecutableSignatures(String description, byte offset, byte size, byte [] magicNumber){

    this.description = description;
    this.offset = offset;
    this.size = size;
    this.magicNumber = magicNumber;

}

public String getDescription(){
    return this.description;
}

public byte getOffset(){
    return this.offset;
}

public byte getSize(){
    return this.size;
}

public byte[] getMagicNumbers(){
    return this.magicNumber;
}

After you can create a method to make this check by using apache librairies see HERE see @Filters - MagicNumberFilter. This constructor can take 2 paramaters; the magicNumbers(byte array) and the offset(byte).

 /**
     * Perform a check of what kind of executable is by checking the signature 
     * of file.
     * If it's an executable that is enumerate then the attributes 
     * magicNumber and executableDescription are updated with their corresponding 
     * values.
     * @return true if is an executable supported by the program otherwise false
     */
    public boolean isExecutableFile(){
        MagicNumberFileFilter mnff = null;

        for(ExecutableSignatures es : EnumSet.allOf(ExecutableSignatures.class)){
            mnff = new 
                    MagicNumberFileFilter(es.getMagicNumbers(), es.getOffset());

            if(mnff.accept(this.file)){
                this.magicNumber = es.getMagicNumbers();
                this.executableDescription = es.getDescription();
                return true;
            }
        }
        return false;
    }