Problems to verify a file signed with python

Question

I'm trying to create a signed file using OpenSSL and Python, and I'm not receiving any error mesage, but the proccess is not working properly and I can't find the reason.

Below is my step-by-step to sign the file and check the signature:

1. First I create the crt in command line

   openssl req -nodes -x509 -sha256 -newkey rsa:4096 -keyout "cert.key" -out "cert.crt" -subj "/C=BR/ST=SP/L=SP/O=Company/OU=IT Dept/CN=cert"

At this point, I have two files: cert.key and cert.crt

2. Sign the file use a Python Script like below:

   import os.path
   from Crypto.PublicKey import RSA
   from Crypto.Signature import PKCS1_v1_5
   from Crypto.Hash import SHA256
   from base64 import b64encode, b64decode
   
   def __init__(self):
       folder = os.path.dirname(os.path.realpath(__file__))
       file_path = os.path.join(folder, '../static/cert.key')
       self.key = open(file_path, "r").read()
   
   def sign_data(self, my_file):
       rsakey = RSA.importKey(self.key) # I opened the cert.key in __init__
       signer = PKCS1_v1_5.new(rsakey)
       digest = SHA256.new()
   
       digest.update(my_file)
       sign = signer.sign(digest)
   
       return sign, b64encode(sign)
   

All works fine and after save the files, I have other three files: my_file.csv (the original one), my_file.txt.sha256 and my_file.txt.sha256.base64. At this point, I can decode the base64 file and compare with the signed one and both are fine.

The problem is when I try to verify the signature using the following command:

`openssl dgst -sha256 -verify  <(openssl x509 -in "cert.crt"  -pubkey -noout) -signature my_file.txt.sha256 my_file.csv`

At this point I always receive the "Verification Failure" and don't understand why.

Maybe the problem is my lack of Python's Knowledge, because when I sign the file using the following command (after step 1 and before use the Python script described in 2), the same verification works fine.

openssl dgst -sha256 -sign "cert.key" -out my_file.txt.sha256 my_file.csv

Am I doing anything wrong?

UPDATE

Based on the comments, I tried the script in a local virtualnv with python 2.7 and it worked, so the problem must be in the read/write operations.

I'm updating this quetion with the complete script, including the read/write operations 'cause I can run it locally, but I still don't get any error in the GAE environment and can't understand why.

The first step is the CSV creation and storage in the Google Storage (Bucket) with the script below

import logging
import string
import cloudstorage as gcs
from google.appengine.api import app_identity

def create_csv_file(self, filename, cursor=None):    
    filename = '/' + self.bucket_name + filename
    try:
        write_retry_params = gcs.RetryParams(backoff_factor=1.1)
        # the cursor stores a MySQL result set
        if cursor is not None:
            gcs_file = gcs.open(filename,
                                'w',
                                content_type='text/csv',
                                retry_params=write_retry_params)
            for row in cursor:
                gcs_file.write(','.join(map(str, row)) + '\n')
            gcs_file.close()
    except Exception as ex:
        logging.critical("Problem to write in th GC Storage with the exception:{}".format(ex))
        raise ex

It works fine and store a CSV in the correct path inside Google storage. After that part, the next read/write operation is the signature of the file.

def cert_file(self, original_filename):
    filename = '/' + self.bucket_name + original_filename
    cert = Cert() # This class just has one method, that is that described in my original question and is used to sign the file.

    with gcs.open(filename) as cloudstorage_file:
        cloudstorage_file.seek(-1024, os.SEEK_END)
        signed_file, encoded_signed_file = cert.sign_data(cloudstorage_file.read()) #the method to sign the file
    signature_content = encoded_signed_file

    signed_file_name = string.replace(original_filename, '.csv', '.txt.sha256')
    encoded_signed_file_name = string.replace(signed_file_name, '.txt.sha256', '.txt.sha256.base64')

    self.inner_upload_file(signed_file, signed_file_name)
    self.inner_upload_file(encoded_signed_file, encoded_signed_file_name)

    return signed_file_name, encoded_signed_file_name, signature_content

The inner_upload_file, just save the new files in the same bucket:

def inner_upload_file(self, file_data, filename):
    filename = '/' + self.bucket_name + filename
    try:
        write_retry_params = gcs.RetryParams(backoff_factor=1.1)
        gcs_file = gcs.open(filename,
                            'w',
                            content_type='application/octet-stream',
                            retry_params=write_retry_params)
        gcs_file.write(file_data)
        gcs_file.close()
    except Exception as ex:
        logging.critical("Problem to write in th GC Storage with the exception:{}".format(ex))
        raise ex

Here is the app.yaml for reference. The cert.key and cert.crt generated by command line are stored in a static folder inside the app folder (the same directory where is my app.yaml).

UPDATE 2

Following the comments, I tried to run the signature proccess locally and then compare the files. Below is the step-by-setp and results.

First, I adapted the signature process to run as python sign.py file_name.

#!/usr/bin/python

import sys
import os
from Crypto.PublicKey import RSA
from Crypto.Signature import PKCS1_v1_5
from Crypto.Hash import SHA256
from base64 import b64encode, b64decode

file_path = 'static/cert.key'
key = open(file_path, "rb").read()

rsakey = RSA.importKey(key)
signer = PKCS1_v1_5.new(rsakey)
digest = SHA256.new()
file_object  = open(sys.argv[1], "r")
digest.update(file_object.read())
sign = signer.sign(digest)
signed_path = "signed"
f = open(signed_path + '.txt.sha256', 'w')
f.write(sign)
f.close()
f2 = open(signed_path + '.txt.sha256.base64', 'w')
f2.write(b64encode(sign))
f2.close()

I ran the automatic proccess that saved the signed file in GCS's bucket (along with the original CSV file). After it I download both files through Google web panel for GCS.

I ran the command python sign.py gcs_file_original.csv in a virtualenv with python 2.7.10 using the CSV file I just downloaded.

After it, I compared the two signed files with cmp -b gcs_signed.txt.sha256 locally_signed.txt.sha256 resulting in:

gcs_signed.txt.sha256 locally_signed.txt.sha256 differ: byte 1, line 1 is 24 ^T 164 t

Using the VisualBinaryDiff, the result looks like two totally different files.

Now, I know the problem, but have no idea on how to fix it. This problem is beeing very trick.

Answer 1

I finally found the problem. I was so focused in find a problem in the openssl signature proccess and didn't pay attention to old Ctrl+C/Ctrl+V problem.

For test purposes, I copied the 'Read from GCS' example from this tutorial.

When I moved the test to the real world application, I didn't read the page again and didn't note the gcs_file.seek(-1024, os.SEEK_END).

As I said in the original question, I'm not Python specialist, but this line was reading just part of the GCS file, so the signature was indeed different from the original one.

Just cut that line of my reading methods and now all works fine.