BouncyCastle PGP and McAfee eBusiness Server 8.6 Incompatibility

Question

I have been banging my head against the wall now for a couple of weeks trying to figure out why our bank cannot decrypt a message that has been single-pass signed and encrypted using BouncyCastle PGP. The bank is using McAfee E-Business Server 8.6 for decryption.

The data is encrypted with the bank's public key, and is signed with our private key.

Using our own public key for encryption, I am successfully able to decrypt and verify the signature on a file generated with the code below. Gnupg can decrypt and verify the file just fine.

However, the bank is unable to decrypt the file. I have tried first turning compression off, and then turning ASCII armoring off. Neither option seems to work, and they always get the same error message no matter what options I try:

event 1: initial
event 13: BeginLex
event 8: Analyze
File is encrypted.  event 9: Recipients
Secret key is required to read it.
Key for user ID "XXXXXXXXX <XXXXXX@XXXX>"
event 6: Passphrase
event 23: Decryption

symmetric cipher used: CAST5
event 3: error -11391

event 2: final

Error decrypting file '/somepath/FILENAME'.
Corrupt data.
Bad packet

exitcode = 32

Here is the code I am using to do the single pass sign and encrypt:

public class PGPService {

    private static final Logger log = Logger.getLogger(PGPService.class);


    static {
        Security.addProvider(new BouncyCastleProvider());
    }


    /**
     * A simple routine that opens a key ring file and loads the first available key
     * suitable for signature generation.
     * 
     * @param input stream to read the secret key ring collection from.
     * @return a secret key.
     * @throws IOException on a problem with using the input stream.
     * @throws PGPException if there is an issue parsing the input stream.
     */
    @SuppressWarnings("rawtypes")
    private static PGPSecretKey readSecretKey(InputStream input) throws IOException, PGPException {
        PGPSecretKeyRingCollection pgpSec = new PGPSecretKeyRingCollection(
        PGPUtil.getDecoderStream(input));

        // We just loop through the collection till we find a key suitable for encryption, in the real
        // world you would probably want to be a bit smarter about this.

        Iterator keyRingIter = pgpSec.getKeyRings();
        while (keyRingIter.hasNext()) {
            PGPSecretKeyRing keyRing = (PGPSecretKeyRing)keyRingIter.next();
            Iterator keyIter = keyRing.getSecretKeys();
            while (keyIter.hasNext()) {
                PGPSecretKey key = (PGPSecretKey)keyIter.next();
                if (key.isSigningKey()) {
                    return key;
                }
            }
        }

        throw new IllegalArgumentException("Can't find signing key in key ring.");
    }

    /**
     * Single pass signs and encrypts the given file to the given output file using the provided keys.
     * 
     * @param fileNameIn - The name and location of the source input file.
     * @param fileNameOut - The name and location of the destination output file.
     * @param privateKeyIn - The input stream for the private key file.
     * @param privateKeyPassword - The password for the private key file.
     * @param publicEncryptionKey - The public encryption key.
     * @param armoredOutput - Whether or not to ASCII armor the output.
     */
    @SuppressWarnings("rawtypes")
    public static void signAndEncrypt(String fileNameIn, String fileNameOut, InputStream privateKeyIn, String privateKeyPassword, PGPPublicKey publicEncryptionKey, boolean armoredOutput, boolean compress) {

        int bufferSize = 1<<16;
        InputStream input = null;
        OutputStream finalOut = null;
        OutputStream encOut = null;
        OutputStream compressedOut = null;
        OutputStream literalOut = null;
        PGPEncryptedDataGenerator encryptedDataGenerator = null;
        PGPCompressedDataGenerator compressedDataGenerator = null;
        PGPSignatureGenerator signatureGenerator = null;
        PGPLiteralDataGenerator literalDataGenerator = null;

        try {

            File output = new File(fileNameOut);
            OutputStream out = new FileOutputStream(output);
            if (armoredOutput) out = new ArmoredOutputStream(out);

            // ? Use BCPGOutputStreams ?

            // Init encrypted data generator
            encryptedDataGenerator = new PGPEncryptedDataGenerator(PGPEncryptedDataGenerator.CAST5, true, new SecureRandom(), "BC");
            encryptedDataGenerator.addMethod(publicEncryptionKey);
            finalOut = new BufferedOutputStream(out, bufferSize);
            encOut = encryptedDataGenerator.open(finalOut, new byte[bufferSize]);

            // Init compression
            if (compress) {
                compressedDataGenerator = new PGPCompressedDataGenerator(PGPCompressedData.ZLIB);
                compressedOut = new BufferedOutputStream(compressedDataGenerator.open(encOut));
            }

            // Init signature
            PGPSecretKey pgpSec = readSecretKey(privateKeyIn);
            PGPPrivateKey pgpPrivKey = pgpSec.extractPrivateKey(privateKeyPassword.toCharArray(), "BC");        
            signatureGenerator = new PGPSignatureGenerator(pgpSec.getPublicKey().getAlgorithm(), PGPUtil.SHA1, "BC");
            signatureGenerator.initSign(PGPSignature.CANONICAL_TEXT_DOCUMENT, pgpPrivKey);
            Iterator it = pgpSec.getPublicKey().getUserIDs();
            if (it.hasNext()) {
                PGPSignatureSubpacketGenerator spGen = new PGPSignatureSubpacketGenerator();
                spGen.setSignerUserID(false, (String)it.next());
                signatureGenerator.setHashedSubpackets(spGen.generate());
            }
            PGPOnePassSignature onePassSignature = signatureGenerator.generateOnePassVersion(false);
            if (compress) onePassSignature.encode(compressedOut);
            else onePassSignature.encode(encOut);

            // Create the Literal Data generator Output stream which writes to the compression stream
            literalDataGenerator = new PGPLiteralDataGenerator(true);
            if (compress) literalOut = literalDataGenerator.open(compressedOut, PGPLiteralData.BINARY, output.getName(), new Date(), new byte[bufferSize]);
            else literalOut = literalDataGenerator.open(encOut, PGPLiteralData.TEXT, fileNameIn, new Date(), new byte[bufferSize]);

            // Update sign and encrypt
            byte[] buffer = new byte[bufferSize];

            int bytesRead = 0;
            input = new FileInputStream(fileNameIn);
            while((bytesRead = input.read(buffer)) != -1) {
                literalOut.write(buffer,0,bytesRead);
                signatureGenerator.update(buffer,0,bytesRead);
                literalOut.flush();
            }

            // Close Literal data stream and add signature
            literalOut.close();
            literalDataGenerator.close();
            if (compress) signatureGenerator.generate().encode(compressedOut);
            else signatureGenerator.generate().encode(encOut);

        } catch (Exception e) {
            log.error(e);
            throw new RuntimeException(e);
        } finally {
            // Close all streams
            if (literalOut != null) try { literalOut.close(); } catch (IOException e) {}
            if (literalDataGenerator != null) try { literalDataGenerator.close(); } catch (IOException e) {}
            if (compressedOut != null) try { compressedOut.close(); } catch (IOException e) {}
            if (compressedDataGenerator != null) try {  compressedDataGenerator.close(); } catch (IOException e) {}
            if (encOut != null) try {  encOut.close(); } catch (IOException e) {}
            if (encryptedDataGenerator != null) try {  encryptedDataGenerator.close(); } catch (IOException e) {}
            if (finalOut != null) try {  finalOut.close(); } catch (IOException e) {}
            if (input != null) try {  input.close(); } catch (IOException e) {}
        }
    }

    @SuppressWarnings("rawtypes")
    private static PGPPublicKey readPublicKeyFromCol(InputStream in) throws Exception {
        PGPPublicKeyRing pkRing = null;
        PGPPublicKeyRingCollection pkCol = new PGPPublicKeyRingCollection(in);
        log.info("Key ring size = " + pkCol.size());
        Iterator it = pkCol.getKeyRings();
        while (it.hasNext()) {
            pkRing = (PGPPublicKeyRing) it.next();
            Iterator pkIt = pkRing.getPublicKeys();
            while (pkIt.hasNext()) {
                PGPPublicKey key = (PGPPublicKey) pkIt.next();
                log.info("Encryption key = " + key.isEncryptionKey() + ", Master key = " + 
                key.isMasterKey());
                if (key.isEncryptionKey()) {
                    // Find out a little about the keys in the public key ring.
                    log.info("Key Strength = " + key.getBitStrength());
                    log.info("Algorithm = " + key.getAlgorithm());
                    log.info("Bit strength = " + key.getBitStrength());
                    log.info("Version = " + key.getVersion());
                    return key;
                }
            }
        }
        return null;
    }

    private static PGPPrivateKey findSecretKey(InputStream keyIn, long keyID, char[] pass) 
            throws IOException, PGPException, NoSuchProviderException {
        PGPSecretKeyRingCollection pgpSec = new PGPSecretKeyRingCollection(PGPUtil.getDecoderStream(keyIn));
        PGPSecretKey pgpSecKey = pgpSec.getSecretKey(keyID);
        if (pgpSecKey == null) {
            return null;
        }
        return pgpSecKey.extractPrivateKey(pass, "BC");
    }
}

Any idea what could be causing this? I've found numerous reports using Google, but no resolutions or follow-ups.

Answer 1

I'm answering my own question in hopes it will help someone else down the line since the answer to this question was VERY hard to find anywhere online.

McAfee E-Business Server versions prior to 8.6 had a compatibility issue with BouncyCastle PGP, and it seems most people were unable to get it to work. So if your vendor/client/bank is using a version of E-Business Server prior to 8.6, you may very well be SOL and might need to find a different encryption package.

Source: https://kc.mcafee.com/corporate/index?page=content&id=KB60816&cat=CORP_EBUSINESS_SERVER&actp=LIST

"Decrypting a file that was encrypted with Bouncy Castle v1.37 may result in Access Violation Error (or SIGSEG on UNIX platforms). This issue has been addressed in this release."

Luckily, our bank is using McAfee E-Business Server 8.6. However, this is only part of the equation. In order to solve the incompatibility issue, we had to turn BOTH compression and ASCII armoring off before they could successfully decrypt and verify our file. So using the original code I posted, this is how you would call it for clients using E-Business Server 8.6:

PGPService.signAndEncrypt(clearTextFileName, secureFileName, privKeyIn, privateKeyFilePassword, pubKeyIn, true, false, false);

Of course this means you cannot use ASCII armoring which may or may not be an issue with you. If it is, David on the BouncyCastle dev mailing list, suggested you use BouncyCastle in non-packet mode. IE: Do not pass a byte buffer into the open commands in the stream. Or sign and encrypt the file in two passes.

Such as by calling:

public static void signFile(String fileNameIn, String fileNameOut, InputStream privKeyIn, String password, boolean armoredOutput) {

        OutputStream out = null;
        BCPGOutputStream bOut = null;
        OutputStream lOut = null;
        InputStream fIn = null;

        try {
            out = new FileOutputStream(fileNameOut);
            if (armoredOutput) {
                out = new ArmoredOutputStream(out);
            }
            PGPSecretKey pgpSec = readSecretKey(privKeyIn);
            PGPPrivateKey pgpPrivKey = pgpSec.extractPrivateKey(password.toCharArray(), "BC");        
            PGPSignatureGenerator sGen = new PGPSignatureGenerator(pgpSec.getPublicKey().getAlgorithm(), PGPUtil.SHA1, "BC");

            sGen.initSign(PGPSignature.BINARY_DOCUMENT, pgpPrivKey);

            Iterator it = pgpSec.getPublicKey().getUserIDs();
            if (it.hasNext()) {
                PGPSignatureSubpacketGenerator spGen = new PGPSignatureSubpacketGenerator();
                spGen.setSignerUserID(false, (String)it.next());
                sGen.setHashedSubpackets(spGen.generate());
            }

            PGPCompressedDataGenerator cGen = new PGPCompressedDataGenerator(PGPCompressedData.ZLIB);
            bOut = new BCPGOutputStream(cGen.open(out));

            sGen.generateOnePassVersion(false).encode(bOut);

            File file = new File(fileNameIn);
            PGPLiteralDataGenerator lGen = new PGPLiteralDataGenerator();
            lOut = lGen.open(bOut, PGPLiteralData.BINARY, file);
            fIn = new FileInputStream(file);
            int ch = 0;

            while ((ch = fIn.read()) >= 0) {
                lOut.write(ch);
                sGen.update((byte) ch);
            }

            lGen.close();
            sGen.generate().encode(bOut);
            cGen.close();

        } catch (Exception e) {
            log.error(e);
            throw new RuntimeException(e);
        } finally {
            if (lOut != null) try { lOut.close(); } catch (IOException e) {}
            if (bOut != null) try { bOut.close(); } catch (IOException e) {}
            if (out != null) try { out.close(); } catch (IOException e) {}
            if (fIn != null) try { fIn.close(); } catch (IOException e) {}
        }
    }

Followed by calling:

public static byte[] encrypt(byte[] data, InputStream pubKeyIn, boolean isPublicKeyArmored) {

        FileOutputStream fos = null;
        BufferedReader isr = null;

        try {
            if (isPublicKeyArmored) pubKeyIn = new ArmoredInputStream(pubKeyIn);
            PGPPublicKey key = readPublicKeyFromCol(pubKeyIn);
            log.info("Creating a temp file...");
            // Create a file and write the string to it.
            File tempfile = File.createTempFile("pgp", null);
            fos = new FileOutputStream(tempfile);
            fos.write(data);
            fos.close();
            log.info("Temp file created at: " + tempfile.getAbsolutePath());
            log.info("Reading the temp file to make sure that the bits were written...\n");
            isr = new BufferedReader(new FileReader(tempfile));
            String line = "";
            while ((line = isr.readLine()) != null ) {
                log.info(line + "\n");
            }
            int count = 0;
            for (java.util.Iterator iterator = key.getUserIDs(); iterator.hasNext();) {
                count++;
                log.info(iterator.next());
            }
            log.info("Key Count = " + count);
            // Encrypt the data.
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            _encrypt(tempfile.getAbsolutePath(), baos, key);
            log.info("Encrypted text length = " + baos.size());         
            tempfile.delete();
            return baos.toByteArray();
        } catch (PGPException e) {
            log.error(e);
            throw new RuntimeException(e);
        } catch (Exception e) {
            log.error(e);
            throw new RuntimeException(e);
        } finally {
            if (fos != null) try { fos.close(); } catch (IOException e) {}
            if (isr != null) try { isr.close(); } catch (IOException e) {}
        }
    }

Caveat emptor, as I was not able to test this method to see if this will even solve the incompatibility issue. But it is a path you can try if you are out of options and must use ASCII armoring for an E-Business Server destination.

Some more information may be found on the BouncyCastle dev mailing list archive that may help if you decide to go down this path. Specifically this thread: http://www.bouncycastle.org/devmailarchive/msg12080.html