XML Signature not validating when adding an element explicitly to the document (JAVA)?

Question

I am creating the following XML document, using JAXB of a given XSD that is provided by third party. The third party request to sign the document and add to it an extra element that holds the signature. Using JDK 1.7.

Below is the code sample for marshalling:

JAXBContext jaxbContext = JAXBContext.newInstance(DataPDU.class);
DataPDU myDataPDU = new DataPDU();
myDataPDU.setRevision("2.0.6");

// marshall the file
Marshaller marshaller = jaxbContext.createMarshaller();

DOMResult domResult = new DOMResult();
marshaller.marshal(myDataPDU, domResult);

// get the document list
Document document = (Document) domResult.getNode();

Then I create the element (LAU) as below and sign the document using HMAC-SHA256 algorithm and the JSR105 JAVA API (I am not going to include the whole signature code to reduce verbosity, I am using using the standard behavior of the XMLSignature class and then transforming the document to a file output stream from a DOMSource using XML transformers):

Element LAUElement = document.createElementNS("urn:swift:saa:xsd:saa.2.0", "Saa:LAU");
Element rootElement = document.getDocumentElement();
rootElement.appendChild(LAUElement);

// sign the document
XMLSignatureUtil.sign(document, secret, LAUElement, "ds");
// create the output file
TransformerUtil.transformDocumentToFile(document, "resultingFile.xml");

The XML is signing properly but when validating, the calculated digest value is different than the digest value.

I have noticed that when changing the namespace value when creating the LAU element, the digest is never changing, as if the document is getting signed and neglecting the namespace of the LAU element, and I guess this is the reason why its failing. Any other change in the document as a whole, or change in the prefix of LAU element directly affects the calculated digest of the payload.

If I append the signature to the root element directly rather than creating the LAU element, the validation works properly.

The LAU element exists within the XSD and can be created using JAXB but the problem is that I cannot find a way to assign a prefix (only for it in the document) of the same namespace as the root element.

Questions:

Is the namespace actually being omitted from the payload digest calculation when adding the element to the document using createElementNS and appendChild?

Is there a way to provide through JAXB a prefix for the same root namespace to a single element only?

How can I find the actual XML string being signed by the API, I tried reading the reference input stream after enabling javax.xml.crypto.dsig.cacheReference but this did not work when signing, it only worked when validating?

Below is a sample of the XML:

<DataPDU xmlns="urn:swift:saa:xsd:saa.2.0">
  <Revision>2.0.6</Revision>
  <Saa:LAU xmlns:Saa="urn:swift:saa:xsd:saa.2.0"> Signature lies here </Saa:LAU>
</DataPDU>

Update - Full XML signature process

JAXBContext jaxbContext = JAXBContext.newInstance(DataPDU.class);
DataPDU myDataPDU = new DataPDU();
myDataPDU.setRevision("2.0.6");

// marshall the file
Marshaller marshaller = jaxbContext.createMarshaller();

DOMResult domResult = new DOMResult();
marshaller.marshal(myDataPDU, domResult);

// get the document list
Document document = (Document) domResult.getNode();

// signing process
XMLSignatureFactory factory = XMLSignatureFactory.getInstance("DOM");

SignatureMethod signatureMethod =
    factory.newSignatureMethod("http://www.w3.org/2001/04/xmldsig-more#hmac-sha256", null);

CanonicalizationMethod canonicalizationMethod =
    factory.newCanonicalizationMethod(CanonicalizationMethod.EXCLUSIVE, (XMLStructure) null);

List<Transform> transforms = new ArrayList<Transform>();
transforms.add(factory.newTransform(Transform.ENVELOPED, (XMLStructure) null));
transforms.add(factory.newTransform("http://www.w3.org/2001/10/xml-exc-c14n#", (XMLStructure) null));

DigestMethod digestMethod = factory.newDigestMethod("http://www.w3.org/2001/04/xmlenc#sha256", null);

Reference reference = factory.newReference("", digestMethod, transforms, null, null);

SignedInfo signedInfo =
    factory.newSignedInfo(canonicalizationMethod, signatureMethod, Collections.singletonList(reference));

String secretKey = "Abcd1234abcd1234Abcd1234abcd1234";
SecretKeySpec secret_key = new SecretKeySpec(secretKey.getBytes(), "HmacSHA256");

Element LAUElement = document.createElementNS("urn:swift:saa:xsd:saa.2.0", "Saa:LAU");
Element rootElement = document.getDocumentElement();
rootElement.appendChild(LAUElement);

DOMSignContext domSignContext = new DOMSignContext(secret_key, LAUElement);
domSignContext.setDefaultNamespacePrefix("ds");

XMLSignature signature = factory.newXMLSignature(signedInfo, null);
signature.sign(domSignContext);

Answer 1

How can I find the actual XML string being signed by the API, I tried reading the reference input stream after enabling javax.xml.crypto.dsig.cacheReference but this did not work when signing, it only worked when validating?

Fortunately the default implementation, assuming that's being used, offers some debug capabilities. Here's an example of settings to log to the console using the FINEST granularity, although FINE seems to be enough.

handlers= java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = FINER
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
org.jcp.xml.dsig.internal.level = FINER
com.sun.org.apache.xml.internal.security.level = FINER

You can save this as a logging.properties file and provide its path via command line option -Djava.util.logging.config.file, although it can probably also be done programmatically.

Doing so will actually show the canonicalized XML being signed, which answers this:

Is the namespace actually being omitted from the payload digest calculation when adding the element to the document using createElementNS and appendChild?

Indeed, it is. Running the code I saw this in the logging as the canonicalized XML for the signing.

<DataPDU xmlns="urn:swift:saa:xsd:saa.2.0"><Revision>2.0.6</Revision><Saa:LAU></Saa:LAU></DataPDU>

Notice how the Saa prefix is there but the namespace declaration is not. Yet, the info is in there because after transformation the declaration with the prefix shows up in the result. That explains why changes to the namespace URI itself don't result in a different digest but changing the prefix does. No idea why this is happening. It feels like it shouldn't, but the rules for what is included in the canonicalized version are so confusing in the specs that either I'm missing something, or it's a bug in the implementation. A work-around for this is adding this line after creating the LAUElement:

 LAUElement.setAttribute("xmlns:Saa", "urn:swift:saa:xsd:saa.2.0");

I thought maybe the DOMResult used a document builder that wasn't set to be namespace-aware, but I tried this with no difference:

DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance();
domFactory.setNamespaceAware(true);
DocumentBuilder docBuilder = domFactory.newDocumentBuilder();
Document doc = docBuilder.newDocument();
DOMResult domResult = new DOMResult(doc);
marshaller.marshal(myDataPDU, domResult);

It might be a bug of DOMResult itself. The workaround is a bit messed up but it does the job. This does in fact change the digest. I've tried this by providing both the LAUElement and the document root as the input for DOMSignContext, which in both cases results in the same signature (digest and signature value) although in the latter case the signature is added to the root, not the element.

This also teaches us that the entire document is used as input for the signing, even when you supply the element and use the exclusive canonicalization method. It only changes where the signature is placed. The URI resolving finds the ancestors up to the root element and everything below it. This surprised me a bit.

Doing the above, I managed to correctly verify the signature using the below code:

XMLSignatureFactory factory = XMLSignatureFactory.getInstance("DOM");

NodeList sigList = doc.getElementsByTagNameNS(XMLSignature.XMLNS, "Signature");
if (sigList.getLength() == 0) {
    throw new Exception("Cannot find Signature element");
}
String secretKey = "Abcd1234abcd1234Abcd1234abcd1234";
SecretKeySpec secret_key = new SecretKeySpec(secretKey.getBytes(), "HmacSHA256");
DOMValidateContext valContext = new DOMValidateContext(secret_key, sigList.item(0));
XMLSignature signature = factory.unmarshalXMLSignature(valContext);

System.out.println("Core validity: " + signature.validate(valContext));

It does indeed seem that the issue was that when the namespace declaration on was dropped, the digest generated differed from the one generated on validation.

Is there a way to provide through JAXB a prefix for the same root namespace to a single element only?

The short answer is, there's no trivial way to do this. The long answer can be found here: https://stackoverflow.com/a/42301479/630136

EDIT:

An additional note. Be careful with the transformer for outputting the signed document to a file. If it's set to indent the output, the resulting file would actually have a different digest. There's whitespace that can be ignored (like in <empty /> vs <empty/>) but within elements it's normally regarded as text nodes and part of the canonicalized version over which the digest is calculated.