How do I decrypt data encrypted by Ruby's `symmetric-encryption` gem in another language?

Question

I want to access data in a database created by Rails for use by non-Ruby code. Some fields use attr_encrypted accessors, and the library in use is the symmetric-encryption gem. I consistently get a "wrong final block length" error if I try to decrypt the data with, e.g., the NodeJS crypto library.

I suspect this has to do either with character encoding or with padding, but I can't figure it out based on the docs.

As an experiment, I tried decrypting data from symmetric-encryption in Ruby's own OpenSSL library, and I get either a "bad decrypt" error or the same problem:

SymmetricEncryption.cipher = SymmetricEncryption::Cipher.new(
  key: "1234567890ABCDEF",
  iv:  "1234567890ABCDEF",
  cipher_name: "aes-128-cbc"
)

ciphertext = SymmetricEncryption.encrypt("Hello world")

c = OpenSSL::Cipher.new("aes-128-cbc")
c.iv = c.key = "1234567890ABCDEF"
c.update(ciphertext) + c.final

That gives me a "bad decrypt" error.

Interestingly, the encrypted data in the database can be decrypted by the symmetric-encryption gem, but isn't the same as the output of SymmetricEncryption.encrypt (and OpenSSL doesn't successfully decrypt it, either).

---

Edit:

psql=# SELECT "encrypted_firstName" FROM people LIMIT 1;
                   encrypted_firstName                    
----------------------------------------------------------
 QEVuQwBAEAAuR5vRj/iFbaEsXKtpjubrWgyEhK5Pji2EWPDPoT4CyQ==
(1 row)

Then

irb> SymmetricEncryption.decrypt "QEVuQwBAEAAuR5vRj/iFbaEsXKtpjubrWgyEhK5Pji2EWPDPoT4CyQ=="
=> "Lurline"
irb> SymmetricEncryption.encrypt "Lurline"
=> "QEVuQwAAlRBeYptjK0Fg76jFQkjLtA=="

Answer 1

Looking at the source for the symmetric-encryption gem, by default it adds a header to the output and base64 encodes it, although both of these are configurable.

To decrypt using Ruby’s OpenSSL directly, you will need to decode it and strip off this header, which is 6 bytes long in this simple case:

ciphertext = Base64.decode64(ciphertext)
ciphertext = ciphertext[6..-1]

c = OpenSSL::Cipher.new("aes-128-cbc")
c.decrypt
c.iv = "1234567890ABCDEF"
c.key = "1234567890ABCDEF"

result = c.update(ciphertext) + c.final

Of course, you may need to alter this depending on what settings you are using in symmetric-encryption, e.g. the header length may vary. In order to decrypt the result from the database you will need to parse the header. Have a look at the source.

Answer 2

Based on the Rust implementation done by @Shepmaster in my other question (and the source code for the symmetric-encryption gem), I have a working version in TypeScript. @matt is close with his answer, but the header can actually have additional bytes containing metadata about the encrypted data. Note that this doesn't handle (1) compressed encrypted data, or (2) setting the encryption algorithm from the header itself; neither situation is relevant to my use case.

import { createDecipher, createDecipheriv, Decipher } from "crypto";

// We use two types of encoding with SymmetricEncryption: Base64 and UTF-8. We
// define them in an `enum` for type safety.
const enum Encoding {
    Base64 = "base64",
    Utf8 = "utf8",
}

// Symmetric encryption's header contains the following data:
interface IHeader {
    version: number, // The version of the encryption algo
    isCompressed: boolean, // Whether the data is compressed (TODO: Implement)
    hasIv: boolean, // Whether the header itself has the IV
    hasKey: boolean, // Whether the header itself has the Key
    hasCipherName: boolean, // Whether the header contains the cipher name
    hasAuthTag: boolean, // Whether the header has an authorization tag
    offset: number, // How many bytes into the encoded ciphertext the actual encrypted data starts
    iv?: Buffer, // The IV, present only if `hasIv` is true
    key?: Buffer, // The key, present only if `hasKey` is true
    // The cipher name, present only if `hasCipherName` is true. Currently ignored.
    cipherName?: string,
    authTag?: string, // The authorization tag, present only if // `hasAuthTag` is true
}

// Byte 6 of the header contain bit flags
interface IFlags {
    isCompressed: boolean,
    hasIv: boolean,
    hasKey: boolean,
    hasCipherName: boolean,
    hasAuthTag: boolean
}

// The 7th byte until the end of the header have the actual values. If all
// of the flags are false, the header ends at the 6th byte.
interface IValues {
    iv?: Buffer,
    key?: Buffer,
    cipherName?: string,
    authTag?: string,
    size: number,
}

/**
 * Represent the encoded ciphertext, complete with the SymmetricEncryption header.
 */
class Ciphertext {
    // Bit flags corresponding to the data encoded in byte 6 of the
    // header.
    readonly FLAG_COMPRESSED = 0b1000_0000;
    readonly FLAG_IV = 0b0100_0000;
    readonly FLAG_KEY = 0b0010_0000;
    readonly FLAG_CIPHER_NAME = 0b0001_0000;
    readonly FLAG_AUTH_TAG = 0b0000_1000;

    // The literal data encoded in bytes 1 - 4 of the header
    readonly MAGIC_HEADER = "@EnC";

    // If any of the values represented by the bit flags is present, the first 2
    // bytes of the data tells us how long the actual value is. In other words,
    // the first 2 bytes aren't the value itself, but rather give the info about
    // the length of the rest of the value.
    readonly LENGTH_INFO_SIZE = 2;

    public header: IHeader | null;
    public data: Buffer;

    private cipherBuffer: Buffer;

    constructor(private input: string) {
        this.cipherBuffer = new Buffer(input, Encoding.Base64);
        this.header = this.getHeader();
        const offset = this.header ? this.header.offset : 0; // If no header, then no offset
        this.data = this.cipherBuffer.slice(offset);
    }

    /**
     * Extract the header from the data
     */
    private getHeader(): IHeader | null {
        let offset = 0;

        // Bytes 1 - 4 are the literal `@EnC`. If that's absent, there's no
        // SymmetricEncryption header.
        if (this.cipherBuffer.toString(Encoding.Utf8, offset, offset += 4) != this.MAGIC_HEADER) {
            return null;
        }

        // Byte 5 is the version
        const version = this.cipherBuffer.readInt8(offset++); // Post increment

        // Byte 6 is the flags
        const rawFlags = this.cipherBuffer.readInt8(offset++);
        const flags = this.readFlags(rawFlags);

        // Bytes 7 - end are the values.
        const values = this.getValues(offset, flags);

        offset += values.size;

        return Object.assign({ version, offset }, flags, values);
    }

    /**
     * Get the values for `iv`, `key`, `cipherName`, and `authTag`, if any are
     * set, based on the bitflags. Return that data, plus how many bytes in the
     * header those values represent.
     * 
     * @param offset - What byte we're on when we get to the values. Should be 7
     * @param flags - The flags we've extracted, showing us which values to expect
     */
    private getValues(offset: number, flags: IFlags): IValues {
        let iv: Buffer | undefined = undefined;
        let key: Buffer | undefined = undefined;
        let cipherName: string | undefined = undefined;
        let authTag: string | undefined = undefined;

        let size = 0; // If all of the bit flags are false, there is no additional data.

        // For each value, see if the flag is set to true. If it is, we need to
        // read the value. Keys and IVs need to be `Buffer` types; other values
        // should be strings.
        [iv, size] = flags.hasIv ? this.readBuffer(offset) : [undefined, size];
        [key, size] = flags.hasKey ? this.readBuffer(offset + size) : [undefined, size];
        [cipherName, size] = flags.hasCipherName ? this.readString(offset + size) : [undefined, size];
        [authTag, size] = flags.hasAuthTag ? this.readString(offset + size) : [undefined, size];

        return { iv, key, cipherName, authTag, size };
    }

    /**
     * Parse the 16-bit integer representing the bit flags into an object for
     * easier handling
     * 
     * @param flags - The 16-bit integer that contains the bit flags
     */
    private readFlags(flags: number): IFlags {
        return {
            isCompressed: (flags & this.FLAG_COMPRESSED) != 0,
            hasIv: (flags & this.FLAG_IV) != 0,
            hasKey: (flags & this.FLAG_KEY) != 0,
            hasCipherName: (flags & this.FLAG_CIPHER_NAME) != 0,
            hasAuthTag: (flags & this.FLAG_AUTH_TAG) != 0
        }
    }

    /**
     * Read a string out of the value at the specified offset. Return the value
     * itself, plus the number of bytes consumed by the value (including the
     * 2-byte encoding of the length of the actual value).
     * 
     * @param offset - The offset (bytes from the beginning of the encoded,
     * encrypted Buffer) at which the value in question begins
     */
    private readString(offset: number): [string, number] {
        // The length is the first 2 bytes, encoded as a little-endian 16-bit integer
        const length = this.cipherBuffer.readInt16LE(offset);
        // The total size occupied in the header is the 2 bytes encoding length plus the length itself
        const size = this.LENGTH_INFO_SIZE + length;

        const value = this.cipherBuffer.toString(Encoding.Base64, offset + this.LENGTH_INFO_SIZE, offset + size);
        return [value, size];
    }

    /**
     * Read a Buffer out of the value at the specified offset. Return the value
     * itself, plus the number of bytes consumed by the value (including the
     * 2-byte encoding of the length of the actual value).
     * 
     * @param offset - The offset (bytes from the beginning of the encoded,
     * encrypted Buffer) at which the value in question begins
     */
    private readBuffer(offset: number): [Buffer, number] {
        // The length is the first 2 bytes, encoded as a little-endian 16-bit integer
        const length = this.cipherBuffer.readInt16LE(offset);
        // The total size occupied in the header is the 2 bytes encoding length plus the length itself
        const size = this.LENGTH_INFO_SIZE + length;

        const value = this.cipherBuffer.slice(offset + this.LENGTH_INFO_SIZE, offset + size);
        return [value, size];
    }
}

/**
 * Allow decryption of data encrypted by Ruby's `symmetric-encryption` gem
 */
class SymmetricEncryption {
    private key: Buffer;
    private iv?: Buffer;

    constructor(key: string, private algo: string, iv?: string) {
        this.key = new Buffer(key);
        this.iv = iv ? new Buffer(iv) : undefined;
    }

    public decrypt(input: string): string {
        const ciphertext = new Ciphertext(input);

        // IV can be specified by the user. But if it's encoded in the header
        // itself, go with that instead.
        const iv = (ciphertext.header && ciphertext.header.iv) ? ciphertext.header.iv : this.iv;

        // Key can be specified by the user. but if it's encoded in the header,
        // go with that instead.
        const key = (ciphertext.header && ciphertext.header.key) ? ciphertext.header.key : this.key;

        const decipher: Decipher = iv ?
            createDecipheriv(this.algo, key, iv) :
            createDecipher(this.algo, key);

        // Terse version of `update()` + `final()` that passes type checking
        return Buffer.concat([decipher.update(ciphertext.data), decipher.final()]).toString();
    }
}

const s = new SymmetricEncryption("1234567890ABCDEF", "aes-128-cbc", "1234567890ABCDEF");

console.log(s.decrypt("QEVuQwAADWK0cKzgFIovdIThq9Scrg==")); // => "Hello world"