Include nonce and block count in PyCrypto AES MODE_CTR

Question

Some background information, you can skip this part for the actual question

this is my third question about this topic here at stackoverflow. To be complete, these are the other questions AES with crypt-js and PyCrypto and Match AES de/encryption in python and javascript. Unfortunately my last attempt got two downvots for the original question. The problem was, even I did not know what my real question was. I just dug around to find the real question I was looking for. With the feedback in the comments, and reading some additional information, I updated my question. I excavate the right question, I think. But my problem didn't get any more views after my updates. So I really hope, that this question is now more clear and understandable - even I know what my Problem is now :D
Thank you all for making stackoverflow to this cool community - I often found here solutions for my problems. Please keep giving feedback to bad questions, so they can be improved and updated, which increases this huge knowledge and solutions database. And feel free to correct my english grammar and spelling.

The Problem

AES in Javascript

I have an encrypted String which I can decrypt with this this Javascript Implementation of AES 256 CTR Mode

password = "myPassphrase"
ciphertext = "bQJdJ1F2Y0+uILADqEv+/SCDV1jAb7jwUBWk"
origtext = Aes.Ctr.decrypt(ciphertext, password, 256);
alert(origtext)

This decrypts my string and the alert box with This is a test Text pops up.

AES with PyCrypto

Now I want to decrypt this string with python and PyCrypto

password = 'myPassphrase'
ciphertext = "bQJdJ1F2Y0+uILADqEv+/SCDV1jAb7jwUBWk"
ctr = Counter.new(nbits=128)
encryptor = AES.new(key, AES.MODE_CTR, counter=ctr)
origtext = encryptor.decrypt(base64.b64decode(ciphertext))
print origtext

This code does not run. I get an ValueError: AES key must be either 16, 24, or 32 bytes long. When I recognized, that I have to do more in PyCrypto then just call a decrypt method, I started to investigate end try to figure out what I have to do.

Investigation

The basic things I figured out first, were:

• AES 256 Bit (?). But AES standard is 128 bit. Does increasing the passphrase to 32 Byte is enough?
• Counter Mode. Easily to set in PyCrypto with AES.MODE_CTR. But I have to specify a counter() Method. So I used the basic binary Counter provided by PyCrypto. Is this compatible with the Javascript Implementation? I can't figure out what they are doing.
• The string is base64 encoded. Not a big problem.
• Padding in general. Both passphrase and the encrypted string.

For the passphrase they do this:

for (var i=0; i<nBytes; i++) {
    pwBytes[i] = isNaN(password.charCodeAt(i)) ? 0 : password.charCodeAt(i);
}

Then I did this in python

l = 32
key = key + (chr(0)*(l-len(key)%l))

But this did not help. I still get a weird string ? A???B??d9= ,?h????' with the following code

l = 32
key = 'myPassphrase'
key = key + (chr(0)*(l-len(key)%l))
ciphertext = "bQJdJ1F2Y0+uILADqEv+/SCDV1jAb7jwUBWk"
ctr = Counter.new(nbits=128)
encryptor = AES.new(key, AES.MODE_CTR, counter=ctr)
origtext = encryptor.decrypt(base64.b64decode(ciphertext))
print origtext

Then I read more about the Javascript Implementation and it says

[...] In this implementation, the initial block holds the nonce in the first 8 bytes, and the block count in the second 8 bytes. [...]

I think this could be the key to the solution. So I tested what happens when I encrypt an empty string in Javascript:

origtext = ""
var ciphertext =Aes.Ctr.encrypt(origtext, password, 256);
alert(ciphertext)

The alert box shows /gEKb+N3Y08= (12 characters). But why 12? Shouldn't it be 8+8 = 16Bytes? Well anyway, I tried a bruteforce method on the python decryption by testing the decryption with for i in xrange(0,20): and ciphertext[i:] or base64.b64decode(ciphertext)[i:]. I know this is a very embarrassing try, but I got more and more desperate. And it didn't work either.

The future prospects are also to implement the encryption in the same way.

additional information

The encrypted string was not originally encrypted with this Javascript implementation, It's from another source. I just recognized, that the Javascript code does the right thing. So I affirm that this kind of implementation is something like a "standard".

The question

What can I do, that the encryption and decryption from a string with PyCrypto is the same like in the Javascript implementation, so that I can exchange data between Javascript and Python? I also would switch to another crypto library in python, if you can suggest another one. Furthermore I'm happy with any kind of tips and feedback.

And I think, all comes down to How can I include the nonce and block count to the encrypted string? and How can I extract this information for decryption?

Answer 1

We are still dealing with a bunch of questions here.

How can I extract the nonce and the counter for decryption?

This is easy. In the Javascript implementation (which does not follow a particular standard in this respect) the 8-byte nonce is prepended to the encrypted result. In Python, you extract it with:

import base64
from_js_bin = base64.decode(from_js)
nonce = from_js_bin[:8]
ciphertext = from_js_bin[8:]

Where from_js is a binary string you received.

The counter cannot be extracted because the JS implementation does not transmit it. However, the initial value is (as typically happens) 0.

How can I use the nonce and counter to decrypt the string in Python?

First, it must be established how nonce and counter are combined to get the counter block. It seems that the JS implementation follows the NIST 800-38A standard, where the left half is the nonce, and the right half is the counter. More precisely, the counter is in big endian format (LSB is the rightmost byte). This is also what Wikipedia shows: .

Unfortunately, CTR mode is poorly documented in PyCrypto (a well-known problem). Basically, the counter parameter must be a callable object that returns the correct 16-byte (for AES) counter block, for each subsequent call. Crypto.Util.Counter does that, but in an obscure way.

It is there only for performance purposes though. You can easily implement it yourself like this:

from Crypto.Cipher import AES
import struct

class MyCounter:

  def __init__(self, nonce):
    """Initialize the counter object.

    @nonce      An 8 byte binary string.
    """
    assert(len(nonce)==8)
    self.nonce = nonce
    self.cnt = 0

  def __call__(self):
    """Return the next 16 byte counter, as binary string."""
    righthalf = struct.pack('>Q',self.cnt)
    self.cnt += 1
    return self.nonce + righthalf

cipher_ctr = AES.new(key, mode=AES.MODE_CTR, counter=MyCounter(nonce))
plaintext = cipher_ctr.decrypt(ciphertext)

How long is the key for AES?

The key length for AES-128 is 16 bytes. The key length for AES-192 is 24 bytes. The key length for AES-256 is 32 bytes. Each algorithm is different, but much of the implementation is shared. In all cases, the algorithm operate on 16 byte blocks of data. For simplicity, stick to AES-128 (nBits=128).

Will your code work?

I have the feeling it won't, because the way you compute the AES key seems incorrect. The JS code encodes the password in UTF-8 and encrypts it with itself. The result is the actual key material. It is 16 byte long, so for AES-192 and -256, the implementation copies part of it at the back. Additionally, the plaintext is also UTF-8 encoded before encryption.

In general, I suggest you follow this approach:

1. Make your JS implementation reproducible (right now encryption depends on the current time, which changes quite often ;-) ).
2. Print the value of the keys and data at each step (or use a debugger).
3. Try to reproduce the same algorithm in Python, and print the values too.
4. Investigate where they start to differ.

Once you have duplicated the encryption algorithm in Python, the decryption one should be easy.