Issues with SHA1 hash implementation in Android

Question

I have two small snippets for calculating SHA1.

One is very fast but it seems that it isn't correct and the other is very slow but correct.
I think the FileInputStream conversion to ByteArrayInputStream is the problem.

Fast version:

MessageDigest md = MessageDigest.getInstance("SHA1");
FileInputStream fis = new FileInputStream("path/to/file.exe");
ByteArrayInputStream byteArrayInputStream =
    new ByteArrayInputStream(fis.toString().getBytes());
DigestInputStream dis = new DigestInputStream(byteArrayInputStream, md);
BufferedInputStream bis = new BufferedInputStream(fis);
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

int ch;
while ((ch = dis.read()) != -1) {
    byteArrayOutputStream.write(ch);
}

byte[] newInput = byteArrayOutputStream.toByteArray();
System.out.println("in digest : " +
    byteArray2Hex(dis.getMessageDigest().digest()));

byteArrayOutputStream = new ByteArrayOutputStream();
DigestOutputStream digestOutputStream =
    new DigestOutputStream(byteArrayOutputStream, md);
digestOutputStream.write(newInput);

System.out.println("out digest: " +
    byteArray2Hex(digestOutputStream.getMessageDigest().digest()));
System.out.println("length: " + 
    new String(
        byteArray2Hex(digestOutputStream.getMessageDigest().digest())).length());

digestOutputStream.close();
byteArrayOutputStream.close();
dis.close();

Slow version:

MessageDigest algorithm = MessageDigest.getInstance("SHA1");
FileInputStream fis = new FileInputStream("path/to/file.exe");
BufferedInputStream bis = new BufferedInputStream(fis);
DigestInputStream   dis = new DigestInputStream(bis, algorithm);

// read the file and update the hash calculation
while (dis.read() != -1);

 // get the hash value as byte array
byte[] hash = algorithm.digest();

Conversion method:

private static String byteArray2Hex(byte[] hash) {
    Formatter formatter = new Formatter();
    for (byte b : hash) {
        formatter.format("%02x", b);
    }
    return formatter.toString();
}

I hope there is another possibility to get it running because I need the performance.

Answer 1

I used a high performance c++ implementation which I load with JNI.
For more details write a comment, please.

EDIT:
Requirements for JNI is the Android NDK. For Windows is needed in addition cygwin or something similar.
If you decided for cygwin, I give you some little instructions how to get it working with the NDK:

1. Download the setup.exe from cygwin and execute it.
2. Click on Next and choice Install from Internet confirm with Next.
3. The next two steps adjust the settings as desired and as always click Next.
4. Select your internet connection and the same procedure as in the final stages.
5. A download page will catch the eye select it or take just a download page, which is in your country. There is nothing more to say.
6. We need the packages make and gcc-g++. You can find them using the search in the left upper corner, click on the Skip til a version is displayed and the first field is selected. Do that what we have always done after a selection.
7. You will get the information, that there are dependencies, which must be resolved. It is usually not necessary to do it yourself and confirm it.
8. The download and installation started.
9. If you need you can create shortcuts otherwise click on exceptional Finish.
10. Download the zip file and extract the NDK to a non space containing path.
11. You can start now cygwin.
12. Navigate to the NDK. The path /cydrive gives you all available drives f.e. cd /cygdrive/d navigates to the drive with the letter D.
13. In the root folder of the NDK you can execute the file ndk-build with ./ndk-build. There should be an error occurs like Android NDK: Could not find application project directory !.
    You have to navigate in an Android project to execute the command. So let's start with a project.

Before we can start with the project search for a C/C++ implementation of the hash algorithm. I took the code from this site CSHA1.
You should edit the source code for your requirements.

Now we can start with JNI.
You create a folder called jni in your Android project. It contains all native source files and the Android.mk (more about that file later), too.
Copy your downloaded (and edited) source files in that folder.

My java package is called de.dhbw.file.sha1, so I named my source files similar to find them easily.

Android.mk:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_LDLIBS := -llog

# How the lib is called?
LOCAL_MODULE    := SHA1Calc
# Which is your main SOURCE(!) file?
LOCAL_SRC_FILES := de_dhbw_file_sha1_SHA1Calc.cpp

include $(BUILD_SHARED_LIBRARY)

Java code:
I used the AsyncTask with a ProgressDialog to give the user some feedback about the action.

package de.dhbw.file.sha1;

// TODO: Add imports

public class SHA1HashFileAsyncTask extends AsyncTask<String, Integer, String> {
    // [...]

    static {
        // loads a native library
        System.loadLibrary("SHA1Calc");
    }

    // [...]

    // native is the indicator for native written methods
    protected native void calcFileSha1(String filePath);

    protected native int getProgress();

    protected native void unlockMutex();

    protected native String getHash();

    // [...]
}

Native code (C++):

Remember accessing variables inside native code or other way around using threads needs synchronizing or you will get a segmentation fault soon!

For JNI usage you have to add #include <jni.h>.

For logging insert following include #include <android/log.h>.
Now you can log with __android_log_print(ANDROID_LOG_DEBUG, DEBUG_TAG, "Version [%s]", "19");.
The first argument is the type of message and the second the causing library.
You can see I had a version number in my code. It is very helpful because sometimes the apk builder doesn't use the new native libraries. Troubleshooting can be extremely shortened, if the wrong version is online.

The naming conventions in the native code are a little bit crasier: Java_[package name]_[class name]_[method name].

The first to arguments are always given, but depending on the application you should distinguish:

• func(JNIEnv * env, jobject jobj) -> JNI call is an instance method
• func(JNIEnv * env, jclass jclazz) -> JNI call is a static method

The header for the method calcFileSha1(...):
JNIEXPORT void JNICALL Java_de_dhbw_file_sha1_SHA1HashFileAsyncTask_calcFileSha1(JNIEnv * env, jobject jobj, jstring file)

The JDK delivers the binary javah.exe, which generates the header file for the native code. The usage is very simple, simply call it with the full qualified class:
javah de.dhbw.file.sha1.SHA1HashFileAsyncTask

In my case I have to give the bootclasspath additionally, because I use Android classes: javah -bootclasspath <path_to_the_used_android_api> de.dhbw.file.sha1.SHA1HashFileAsyncTask

That would be the generated file:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class de_dhbw_file_sha1_SHA1HashFileAsyncTask */

#ifndef _Included_de_dhbw_file_sha1_SHA1HashFileAsyncTask
#define _Included_de_dhbw_file_sha1_SHA1HashFileAsyncTask
#ifdef __cplusplus
extern "C" {
#endif
#undef de_dhbw_file_sha1_SHA1HashFileAsyncTask_ERROR_CODE
#define de_dhbw_file_sha1_SHA1HashFileAsyncTask_ERROR_CODE -1L
#undef de_dhbw_file_sha1_SHA1HashFileAsyncTask_PROGRESS_CODE
#define de_dhbw_file_sha1_SHA1HashFileAsyncTask_PROGRESS_CODE 1L
/*
 * Class:     de_dhbw_file_sha1_SHA1HashFileAsyncTask
 * Method:    calcFileSha1
 * Signature: (Ljava/lang/String;)V
 */
JNIEXPORT void JNICALL Java_de_dhbw_file_sha1_SHA1HashFileAsyncTask_calcFileSha1
  (JNIEnv *, jobject, jstring);

/*
 * Class:     de_dhbw_file_sha1_SHA1HashFileAsyncTask
 * Method:    getProgress
 * Signature: ()I
 */
JNIEXPORT jint JNICALL Java_de_dhbw_file_sha1_SHA1HashFileAsyncTask_getProgress
  (JNIEnv *, jobject);

/*
 * Class:     de_dhbw_file_sha1_SHA1HashFileAsyncTask
 * Method:    unlockMutex
 * Signature: ()V
 */
JNIEXPORT void JNICALL Java_de_dhbw_file_sha1_SHA1HashFileAsyncTask_unlockMutex
  (JNIEnv *, jobject);

/*
 * Class:     de_dhbw_file_sha1_SHA1HashFileAsyncTask
 * Method:    getHash
 * Signature: ()Ljava/lang/String;
 */
JNIEXPORT jstring JNICALL Java_de_dhbw_file_sha1_SHA1HashFileAsyncTask_getHash
  (JNIEnv *, jobject);

#ifdef __cplusplus
}
#endif
#endif

You can change the file without further notice. But do not use javah again!

Class and methods
To get a class instance you can use jclass clz = callEnv->FindClass(CALL_CLASS);. In this case is CALL_CLASS the full qualified path to the class de/dhbw/file/sha1/SHA1HashFileAsyncTask.

To find a method you need the JNIEnv and an instance of the class:
jmethodID midSet = callEnv->GetMethodID(callClass, "setFileSize", "(J)V"); The first argument is the instance of the class, the second the name of the method and the third is the signature of the method.
The signature you can get with the from JDK given binary javap.exe. Simply call it with the full qualified path of the class f.e. javap -s de.dhbw.file.sha1.SHA1HashFileAsyncTask.
You will get an result like:

Compiled from "SHA1HashFileAsyncTask.java"
public class de.dhbw.file.sha1.SHA1HashFileAsyncTask extends android.os.AsyncTas
k<java.lang.String, java.lang.Integer, java.lang.String> {
  [...]
  static {};
    Signature: ()V

  public de.dhbw.file.sha1.SHA1HashFileAsyncTask(android.content.Context, de.dhb
w.file.sha1.SHA1HashFileAsyncTask$SHA1AsyncTaskListener);
    Signature: (Landroid/content/Context;Lde/dhbw/file/sha1/SHA1HashFileAsyncTas
k$SHA1AsyncTaskListener;)V

  protected native void calcFileSha1(java.lang.String);
    Signature: (Ljava/lang/String;)V

  protected native int getProgress();
    Signature: ()I

  protected native void unlockMutex();
    Signature: ()V

  protected native java.lang.String getHash();
    Signature: ()Ljava/lang/String;

  [...]

  public void setFileSize(long);
    Signature: (J)V

  [...]
}

If the method is found the variable is not equal 0.
Calling the method is very easy:

callEnv->CallVoidMethod(callObj, midSet, size);

The first argument is the given jobject from the "main" method and I think the others are clear.

Remember that you can call from native code although private methods of the class, because the native code is part of it!

Strings
The given string would be converted with following code:

jboolean jbol;
const char *fileName = env->GetStringUTFChars(file, &jbol);

And the other way:

TCHAR* szReport = new TCHAR;
jstring result = callEnv->NewStringUTF(szReport);

It can be every char* variable.

Exceptions
Can be thrown with the JNIEnv:

callEnv->ThrowNew(callEnv->FindClass("java/lang/Exception"), 
    "Hash generation failed");

You can also check if there is an exception occurred also with JNIEnv:

if (callEnv->ExceptionOccurred()) {
    callEnv->ExceptionDescribe();
    callEnv->ExceptionClear();
}

Specifications

• Java Native Interface Specifications

Build/Clean

Build
After we have created all files and filled them with content, we can build it.
Open cygwin, navigate to the project root and execute from there the ndk-build, which is in the NDK root.
This start the compile, if it is success you will get an output like that:

$ /cygdrive/d/android-ndk-r5c/ndk-build
Compile++ thumb  : SHA1Calc <= SHA1Calc.cpp
SharedLibrary  : libSHA1Calc.so
Install        : libSHA1Calc.so => libs/armeabi/libSHA1Calc.so

If there is any error, you will get the typical output from the compiler.

Clean
Open cygwin, switch in your Android project and execute the command /cygdrive/d/android-ndk-r5c/ndk-build clean.

Build apk
After you have build the native libraries you can build your project. I've found clean, it is advantageous to use the eclipse feature clean project.

Debugging
Debugging of java code isn't different as before.
The debugging of c++ code will follow in the next time.

Answer 2

Do this:

MessageDigest md = MessageDigest.getInstance("SHA1");
InputStream in = new FileInputStream("hereyourinputfilename");
byte[] buf = new byte[8192];
for (;;) {
    int len = in.read(buf);
    if (len < 0)
        break;
    md.update(buf, 0, len);
}
in.close();
byte[] hash = md.digest();

Performance comes from handling data by blocks. An 8 kB buffer, as here, ought to be blocky enough. You do not have to use a BufferedInputStream since the 8 kB buffer also serves as I/O buffer.

Answer 3

The reason the fast one is fast and incorrect is (I think) that it is not hashing the file contents!

FileInputStream fis = new FileInputStream("C:/Users/Ich/Downloads/srware_iron.exe");
ByteArrayInputStream byteArrayInputStream = 
        new ByteArrayInputStream(fis.toString().getBytes());

The fis.toString() call does not read the contents of the file. Rather it gives you a string that (I suspect) looks something like this:

"java.io.FileInputStream@xxxxxxxx"

which you are then proceeding to calculate the SHA1 hash for. FileInputStream and its superclasses do not override Object::toString ...

---

The simple way to read the entire contents of an InputStream to a byte[] is to use an Apache Commons I/O helper method - IOUtils.toByteArray(InputStream).

Answer 4

    public void computeSHAHash(String path)// path to your file
    {
            String SHAHash = null;
    try 
    {
        MessageDigest md = MessageDigest.getInstance("SHA1");
        InputStream in = new FileInputStream(path);
        byte[] buf = new byte[8192];
        int len = -1;
        while((len = in.read(buf)) > 0) 
        {
            md.update(buf, 0, len);
        }
        in.close();
        byte[] data = md.digest();
        try 
        {
           SHAHash = convertToHex(data);
        } 
        catch (IOException e) 
        {
           // TODO Auto-generated catch block
           e.printStackTrace();
        }
    } catch (NoSuchAlgorithmException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (FileNotFoundException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
      Toast.makeToast(getApplicationContext(),"Generated Hash ="+SHAHash,Toast.LENGTH_SHORT).show();  

   }
 private static String convertToHex(byte[] data) throws java.io.IOException
{
    StringBuffer sb = new StringBuffer();
    String hex = null;

    hex = Base64.encodeToString(data, 0, data.length, NO_OPTIONS);

    sb.append(hex);

    return sb.toString();
}