Verifying SHA256 signature with OpenSSL in Delphi fails

Question

I'm trying to implement SHA256 signing and verification in Delphi using OpenSSL libeay32.dll. Therefor in a first step I created a RSA 2048-bit key-pair using the the following OpenSSL commands:

openssl genrsa -out private.pem 2048
openssl rsa -in private.pem -outform PEM -pubout -out public.pem

That far that easy. The next step I did was creating a function that was able to read the public and private keys from the PEM files:

function TSignSHA256.ReadKeyFile( aFileName : String; aType : TKeyFileType ) : pEVP_PKEY;
var locFile : RawByteString;
    locBIO  : pBIO;
begin
  locFile := UTF8Encode( aFileName );

  locBIO := BIO_new( BIO_s_file() );

  try
    BIO_read_filename( locBIO, PAnsiChar(locFile) );

    result := NIL;
    case aType of
      kfPrivate : result := PEM_read_bio_PrivateKey( locBIO, result, nil, nil );
      kfPublic  : result := PEM_read_bio_PUBKEY( locBIO, result, nil, nil );
    end;
  finally
    BIO_free( locBIO );
  end;
end;

That seemed to work as well. So I implemented some sign procedure:

procedure TSignSHA256.Sign;
var locData   : RawByteString;
    locKey    : pEVP_PKEY;
    locCtx    : pEVP_MD_CTX;
    locSHA256 : pEVP_MD;
    locSize   : Cardinal;
    locStream : TBytesStream;
begin
  locKey  := ReadKeyFile( 'private.pem', kfPrivate );
  locData := ReadMessage( 'message.txt' );

  locCtx := EVP_MD_CTX_create;
  try
    locSHA256 := EVP_sha256();

    EVP_DigestSignInit( locCtx, NIL, locSHA256, NIL, locKey );
    EVP_DigestSignUpdate( locCtx, PAnsiChar(locData), Length(locData) );
    EVP_DigestSignFinal( locCtx, NIL, locSize );

    locStream := TBytesStream.Create;
    try
      locStream.SetSize( locSize );
      EVP_DigestSignFinal( locCtx, PAnsiChar( locStream.Memory ), locSize );
      WriteSignature( 'message.sig', locStream.Bytes, locSize );
    finally
      FreeAndNIL(locStream);
    end;
  finally
    EVP_MD_CTX_destroy( locCtx );
  end;
end;

As you can see the procedure is reading a file called message.txt, calculating the signature and storing that sig to message.sig. If I run the following OpenSSL command the result is Verified OK:

openssl dgst -sha256 -verify public.pem -signature message.sig message.txt

So it seems like my signing procedure is also working correct. So I finally implemented a verification procedure:

function TSignSHA256.Verify : Boolean;
var locData   : RawByteString;
    locSig    : TArray<Byte>;
    locKey    : pEVP_PKEY;
    locCtx    : pEVP_MD_CTX;
    locSHA256 : pEVP_MD;
    locSize   : Cardinal;
    locStream : TBytesStream;
begin
  locKey  := ReadKeyFile( 'public.pem', kfPublic );
  locData := ReadMessage( 'message.txt' );
  locSig  := ReadSignature( 'message.sig' );
  locSize := Length(locSig);

  locCtx := EVP_MD_CTX_create;
  try
    locSHA256 := EVP_sha256();

    EVP_DigestVerifyInit( locCtx, NIL, EVP_sha256(), NIL, locKey ); //Returns 1
    EVP_DigestVerifyUpdate( locCtx, PAnsiChar(locData), Length(locData) ); //Returns 1

    locStream := TBytesStream.Create( locSig );
    try
      result := ( EVP_DigestVerifyFinal( locCtx, PAnsiChar(locStream.Memory), locSize ) = 1 ); //Returns false! WHY???
    finally
      FreeAndNIL(locStream);
    end;
  finally
    EVP_MD_CTX_destroy( locCtx );
  end;
end;

As you can see I implemented this procedure exactly the same way as I did implement the signing procedure. Unfortunately the result of this is false. The error code returned by OpenSSL is

error04091077:lib(4):func(145):reason:(119)

That translates to an error in lib RSA, function int_rsa_verify, reason Wrong signature length. I searched Google but I didn't find any useful information about that error. I also tried to understand the OpenSSL sources, but I'm not that deep into C and it seems it can take ages till I'm able to figure it out.

My personal feeling is that I did something wrong reading the public key. But that is only a feeling and I have no idea how I could do it in a different way. My second guess would be that I did something wrong initalizing the context in the verification procedure. But I have no clue what that might be.

Why is signature verification failing?

Answer 1

OK, I found the solution. In fact I had to deal with two errors. The first error was that I was passing the signature into EVP_DigestVerifyFinal in a wrong way. That was what Maarten Bodewes stated in his answer and I'll accept that as an answer to my question.

The second problem was within my definition of the entry point into the DLL. I had declared the third parameter of EVP_DigistVerifyFinal as a var param. Probably a copy&past error as the third parameter of EVP_DigistSignFinal IS a var param.

For everybody who will ever have to do the same, I post my solution here. It was inspired by reading EVP Signing and Verifying, DelphiOpenSSL and the OpenSSL sources (mainly dgst.c). The code was implemented and tested with Delphi XE2.

Be aware that my code does not do any error handling nor did I care too much about freeing memory. That means the code is not production ready and you should use it with care!

The import unit:

unit uOpenSSLCrypt;

interface

type
  pBIO = Pointer;
  pBIO_METHOD = Pointer;

  pEVP_MD_CTX = Pointer;
  pEVP_MD = Pointer;

  pEVP_PKEY_CTX = Pointer;
  pEVP_PKEY = Pointer;

  ENGINE = Pointer;

  TPWCallbackFunction = function( buffer : PAnsiChar; length : Integer; verify : Integer; data : Pointer ) : Integer; cdecl;

  //Error functions
  function ERR_get_error : Cardinal; cdecl;
  function ERR_error_string( e : Cardinal; buf : PAnsiChar ) : PAnsiChar; cdecl;

  function ERR_GetErrorMessage : String;

  //BIO functions
  function BIO_new( _type : pBIO_METHOD ) : pBIO; cdecl;
  function BIO_new_file( const aFileName : PAnsiChar; const aMode : PAnsiChar ) : pBIO; cdecl;
  function BIO_free(a: pBIO): integer; cdecl;
  function BIO_s_file : pBIO_METHOD; cdecl;
  function BIO_f_md : pBIO_METHOD; cdecl;
  function BIO_ctrl( bp : pBIO; cmd : Integer; larg : Longint; parg : Pointer ) : Longint; cdecl;
  function BIO_read( b : pBIO; buf : Pointer; len : Integer ) : integer; cdecl;
  function BIO_get_md_ctx(bp: pBIO; mdcp: Pointer): Longint;
  function BIO_read_filename( bp : pBIO; filename : PAnsiChar ) : Integer;

  function PEM_read_bio_PrivateKey( bp : pBIO; x : pEVP_PKEY; cb : TPWCallbackFunction; u : pointer ) : pEVP_PKEY; cdecl;
  function PEM_read_bio_PUBKEY( bp : pBIO; x : pEVP_PKEY; cb : TPWCallbackFunction; u : Pointer ) : pEVP_PKEY; cdecl;

  //EVP functions
  function EVP_MD_CTX_create() : pEVP_MD_CTX; cdecl;
  procedure EVP_MD_CTX_destroy( ctx : pEVP_MD_CTX ); cdecl;
  function EVP_sha256() : pEVP_MD; cdecl;

  function EVP_PKEY_size(key: pEVP_PKEY): integer; cdecl;
  function EVP_DigestSignInit( aCtx : pEVP_MD_CTX; aPCtx : pEVP_PKEY_CTX; aType : pEVP_MD; aEngine : ENGINE; aKey : pEVP_PKEY  ) : Integer; cdecl;
  function EVP_DigestSignUpdate( ctx : pEVP_MD_CTX; const d : Pointer; cnt : Cardinal ) : Integer; cdecl;
  function EVP_DigestSignFinal( ctx : pEVP_MD_CTX; const d : PByte; var cnt : Cardinal ) : Integer; cdecl;

  function EVP_DigestVerifyInit( aCtx : pEVP_MD_CTX; aPCtx : pEVP_PKEY_CTX; aType : pEVP_MD; aEngine : ENGINE; aKey : pEVP_PKEY  ) : Integer; cdecl;
  function EVP_DigestVerifyUpdate( ctx : pEVP_MD_CTX; const d : Pointer; cnt : Cardinal ) : Integer; cdecl;
  function EVP_DigestVerifyFinal( ctx : pEVP_MD_CTX; const d : PByte; cnt : Cardinal ) : Integer; cdecl;

  function CRYPTO_malloc( aLength : LongInt; const f : PAnsiChar; aLine : Integer ) : Pointer; cdecl;
  procedure CRYPTO_free( str : Pointer ); cdecl;

const BIO_C_SET_FILENAME = 108;
      BIO_C_GET_MD_CTX   = 120;

      BIO_CLOSE   = $01;
      BIO_FP_READ = $02;

implementation

uses System.SysUtils, Windows;

const LIBEAY_DLL_NAME = 'libeay32.dll';

function ERR_get_error : Cardinal; external LIBEAY_DLL_NAME;
function ERR_error_string;         external LIBEAY_DLL_NAME;

function ERR_GetErrorMessage : String;
var locErrMsg: array [0..160] of Char;
begin
  ERR_error_string( ERR_get_error, @locErrMsg );
  result := String( StrPas( PAnsiChar(@locErrMsg) ) );
end;

function BIO_new;                 external LIBEAY_DLL_NAME;
function BIO_new_file;            external LIBEAY_DLL_NAME;
function BIO_free;                external LIBEAY_DLL_NAME;
function BIO_ctrl;                external LIBEAY_DLL_NAME;
function BIO_s_file;              external LIBEAY_DLL_NAME;
function BIO_f_md;                external LIBEAY_DLL_NAME;
function BIO_read;                external LIBEAY_DLL_NAME;

function BIO_get_md_ctx( bp : pBIO; mdcp : Pointer ) : Longint;
begin
  result := BIO_ctrl( bp, BIO_C_GET_MD_CTX, 0, mdcp );
end;

function BIO_read_filename( bp : pBIO; filename : PAnsiChar ) : Integer;
begin
  result := BIO_ctrl( bp, BIO_C_SET_FILENAME, BIO_CLOSE or BIO_FP_READ, filename );
end;

function PEM_read_bio_PrivateKey;   external LIBEAY_DLL_NAME;
function PEM_read_bio_PUBKEY;       external LIBEAY_DLL_NAME;

function EVP_MD_CTX_create;   external LIBEAY_DLL_NAME;
procedure EVP_MD_CTX_destroy; external LIBEAY_DLL_NAME;
function EVP_sha256;          external LIBEAY_DLL_NAME;

function EVP_PKEY_size;         external LIBEAY_DLL_NAME;
function EVP_DigestSignInit;    external LIBEAY_DLL_NAME;
function EVP_DigestSignUpdate;  external LIBEAY_DLL_NAME name 'EVP_DigestUpdate';
function EVP_DigestSignFinal;   external LIBEAY_DLL_NAME;

function EVP_DigestVerifyInit;   external LIBEAY_DLL_NAME;
function EVP_DigestVerifyUpdate; external LIBEAY_DLL_NAME name 'EVP_DigestUpdate';
function EVP_DigestVerifyFinal;  external LIBEAY_DLL_NAME;

function CRYPTO_malloc; external LIBEAY_DLL_NAME;
procedure CRYPTO_free;  external LIBEAY_DLL_NAME;

end.

The implementation:

unit uSignSHA256;

interface

uses uOpenSSLCrypt;

type
  TKeyFileType = ( kfPrivate, kfPublic );

  TSignSHA256 = class(TObject)
  private
    function ReadKeyFile( aFileName : String; aType : TKeyFileType ) : pEVP_PKEY;
    function ReadMessage( aName : String ) : RawByteString;
    function ReadSignature( aName : String; var aLength : Cardinal ) : Pointer;
    procedure FreeSignature( aSig : Pointer );

    procedure WriteSignature( aName : String; aSignature : TArray<Byte>; aLength : Integer );

  public
    constructor Create;
    destructor Destroy; override;

    procedure Sign( aKeyFile : String; aMsgFile : String; aSigFile : String );
    function Verify( aKeyFile : String; aMsgFile : String; aSigFile : String ) : Boolean;
  end;

implementation

uses System.Classes, System.SysUtils;

{ TSignSHA256 }

constructor TSignSHA256.Create;
begin

end;

destructor TSignSHA256.Destroy;
begin

  inherited;
end;

procedure TSignSHA256.FreeSignature( aSig : Pointer );
begin
  CRYPTO_free( aSig );
end;

function TSignSHA256.ReadKeyFile( aFileName : String; aType : TKeyFileType ) : pEVP_PKEY;
var locFile : RawByteString;
    locBIO  : pBIO;
begin
  locFile := UTF8Encode( aFileName );

  locBIO := BIO_new( BIO_s_file() );

  try
    BIO_read_filename( locBIO, PAnsiChar(locFile) );

    result := NIL;
    case aType of
      kfPrivate : result := PEM_read_bio_PrivateKey( locBIO, nil, nil, nil );
      kfPublic  : result := PEM_read_bio_PUBKEY( locBIO, nil, nil, nil );
    end;
  finally
    BIO_free( locBIO );
  end;
end;

function TSignSHA256.ReadMessage( aName : String ) : RawByteString;
var locFileStream : TFileStream;
    locSize       : Cardinal;
    locBytes      : TArray<Byte>;
    locText       : String;
begin
  locFileStream := TFileStream.Create( aName, fmOpenRead );
  try
    locSize := locFileStream.Size;

    SetLength(locBytes, locSize);
    locFileStream.Read( locBytes[0], locSize );
  finally
    FreeAndNIL(locFileStream);
  end;

  SetString( locText, PAnsiChar(locBytes), locSize );
  result := UTF8Encode( locText );
end;

function TSignSHA256.ReadSignature( aName : String; var aLength : Cardinal ) : Pointer;
var locSigBio : pBIO;
    locFile   : RawByteString;
    locMode   : RawByteString;
begin
  locFile := UTF8Encode( aName );
  locMode := UTF8Encode('rb');

  locSigBio := BIO_new_file( PAnsiChar(locFile), PAnsiChar(locMode) );
  try
    result := CRYPTO_malloc( aLength, NIL, 0 );
    aLength := BIO_read( locSigBio, result, aLength );
  finally
    BIO_free( locSigBio );
  end;
end;

procedure TSignSHA256.Sign( aKeyFile : String; aMsgFile : String; aSigFile : String );
var locData   : RawByteString;
    locKey    : pEVP_PKEY;
    locCtx    : pEVP_MD_CTX;
    locSHA256 : pEVP_MD;
    locSize   : Cardinal;
    locStream : TBytesStream;
begin
  locKey  := ReadKeyFile( aKeyFile, kfPrivate );
  locData := ReadMessage( aMsgFile );

  locCtx := EVP_MD_CTX_create;
  try
    locSHA256 := EVP_sha256();

    EVP_DigestSignInit( locCtx, NIL, locSHA256, NIL, locKey );
    EVP_DigestSignUpdate( locCtx, PAnsiChar(locData), Length(locData) );
    EVP_DigestSignFinal( locCtx, NIL, locSize );

    locStream := TBytesStream.Create;
    try
      locStream.SetSize( locSize );
      EVP_DigestSignFinal( locCtx, PByte( locStream.Memory ), locSize );
      WriteSignature( aSigFile, locStream.Bytes, locSize );
    finally
      FreeAndNIL(locStream);
    end;
  finally
    EVP_MD_CTX_destroy( locCtx );
  end;
end;

function TSignSHA256.Verify( aKeyFile : String; aMsgFile : String; aSigFile : String ) : Boolean;
var locData   : RawByteString;
    locSig    : Pointer;
    locKey    : pEVP_PKEY;
    locBio    : pBIO;
    locCtx    : pEVP_MD_CTX;
    locKeyCtx : pEVP_PKEY_CTX;
    locSHA256 : pEVP_MD;
    locSize   : Cardinal;
    locStream : TBytesStream;
begin
  locKey    := ReadKeyFile( aKeyFile, kfPublic );
  locData   := ReadMessage( aMsgFile );
  locSize   := EVP_PKEY_size( locKey );

  locBio := BIO_new( BIO_f_md );
  try
    BIO_get_md_ctx( locBio, @locCtx );
    locSHA256 := EVP_sha256();

    EVP_DigestVerifyInit( locCtx, NIL, locSHA256, NIL, locKey );
    EVP_DigestVerifyUpdate( locCtx, PAnsiChar(locData), Length(locData) );

    try
      locSig := ReadSignature( aSigFile, locSize );
      result := ( EVP_DigestVerifyFinal( locCtx, PByte(locSig), locSize ) = 1 );
    finally
      FreeSignature( locSig );
    end;
  finally
    BIO_free( locBio );
  end;
end;

procedure TSignSHA256.WriteSignature( aName : String; aSignature : TArray<Byte>; aLength : Integer );
var locFileStream : TFileStream;
begin
  locFileStream := TFileStream.Create( aName, fmCreate );
  try
    locFileStream.Write( aSignature[0], aLength );
  finally
    FreeAndNIL(locFileStream);
  end;
end;

end.

Answer 2

A signature is not a textual signature. It consists of a byte array for which the bytes may have any value. You are converting that byte array directly to and from ANSI strings. That will fail if the array contains values outside the ANSI range (whatever that may be, I'd presume ASCII).

You need to treat the signature as binary data. You can use a base 64 codec if you need to treat it as a string (containing text).