WSO2 ESB Unable to convert complete JSON data to XML

Question

I am constructing a POC. And I created a pass through proxy service for Google Plus. Without using any proxy service I get this is my output :

 {
   "kind":"plus#person",
   "etag":"\"ExituU7aUpmkkfyD52VulzptThw/4J1clegrhxYC2fsJOu2XWCs1Ewg\"",
   "id":"117488614303967062311",
   "displayName":"Abhi NeoN",
   "name":{
      "familyName":"NeoN",
      "givenName":"Abhi"
   },
   "tagline":"hey guys ! ssup!! check out ma recnt videos... uploaded",
   "gender":"male",
   "aboutMe":"\u003cb\u003ehie, abhishek - ma full name \u003c/b\u003e\u003cdiv\u003e\u003cb\u003em a DANCER ,\u003c/b\u003e\u003c/div\u003e\u003cdiv\u003e\u003cb\u003ei luv ma dancing .\u003c/b\u003e\u003c/div\u003e\u003cdiv\u003e\u003cb\u003ei care ma dancing ,\u003c/b\u003e\u003c/div\u003e\u003cdiv\u003e\u003cb\u003ei jus hv a gr8 thng in me dats ma dancing.\u003c/b\u003e\u003c/div\u003e",
   "relationshipStatus":"single",
   "url":"https://plus.google.com/117488614303967062311",
   "image":{
      "url":"https://lh6.googleusercontent.com/-tF-ip0tUxD4/AAAAAAAAAAI/AAAAAAAAAAA/WKI3USUh_DA/photo.jpg?sz=50"
   },
   "urls":[
      {
         "value":"https://plus.google.com/117488614303967062311",
         "type":"profile"
      },
      {
         "value":"https://www.googleapis.com/plus/v1/people/117488614303967062311",
         "type":"json"
      }
   ],
   "organizations":[
      {
         "name":"our lady of nazareth high school",
         "title":"science",
         "type":"school"
      },
      {
         "name":"",
         "title":"BLUEBYTES",
         "type":"work"
      }
   ]
}

But when I try to do the same using a simple pass through service I get only :

{
   "kind":"plus#person"
}

I read on the wso2esb site that they had a bug and the explanation given to resolve the bug was that json data received was not in the proper format. But now how do I resolve the problem. I mean is their any way I can manipulate the json data before the esb converts it into json data.

Answer 1

We have solved this issue in the latest release of ESB (version 4.5.0). By default it comes with JSONMessageFormatter/JSONBuilder that can handle JSON payloads with multiple keys.

We also came up with another solution for handling message flows that involve different types of JSON <--> XML (or JSON <--> JSON) conversions. JSONStreamBuilder and JSONStreamFormatter can be used to implement such scenarios with the 'script' mediator. Have a look at sample #441 in ESB 4.5.0.

To run sample #441;

• Add JSONStreamBuilder and JSONStreamFormatter as the builder and formatter for JSON in repository/conf/axis2/axis2.xml file
• Deploy SimpleStockQuoteService
• Start the sample axis2server
• Run the JSON client with 'ant newjsonclient'

Answer 2

The only way to reliably convert json to xml and back again is via the use of type hints in the xml. the default converter does not do this. it 1. drops everything after the first property 2. confuses single element lists with properties when going from xml to json

i have reimplemented the transconversion classes using the json-util library, which converts the json to xml containing type hints as element attributes, to ensure no ambiguity.

in this way we can smart proxy (ie content route and mediate on transport and payload) for ALL json based rest services through WSO2 with no issues

This solves the problem (I think camel does it this way by default).

Here is the pom file and code:

place the jar into /repository/components/lib

you must update the messageformatter and messagebuilder mappings for content type "application/json" in axis2.xml

---

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <name>wso2 json/xml converter</name>
    <groupId>x.y.z</groupId>
    <artifactId>wso2converter</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <jdk.version>1.6</jdk.version>
    </properties>

    <build>
        <finalName>wso2converter</finalName>
        <resources>
            <resource>
                <filtering>false</filtering>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${jdk.version}</source>
                    <target>${jdk.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>1.0.1</version>
                <executions>
                    <execution>
                        <id>enforce-jdk</id>
                        <phase>validate</phase>
                        <goals>
                            <goal>display-info</goal>
                            <goal>enforce</goal>
                        </goals>
                        <configuration>
                            <rules>
                                <requireJavaVersion>
                                    <version>[${jdk.version},)</version>
                                </requireJavaVersion>
                            </rules>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>net.sf.json-lib</groupId>
            <artifactId>json-lib</artifactId>
            <version>2.3</version>
            <classifier>jdk15</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.2</version>
            <type>jar</type>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.ws.commons.axiom</groupId>
            <artifactId>axiom-api</artifactId>
            <version>1.2.13</version>
        </dependency>

        <dependency>
            <groupId>org.apache.axis2</groupId>
            <artifactId>axis2-kernel</artifactId>
            <version>1.6.2</version>
        </dependency>

        <dependency>
            <groupId>xom</groupId>
            <artifactId>xom</artifactId>
            <version>1.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.synapse</groupId>
            <artifactId>synapse-core</artifactId>
            <version>2.1.0</version>
        </dependency>
        <dependency>
            <groupId>com.sun.jersey</groupId>
            <artifactId>jersey-json</artifactId>
            <version>1.1.5</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.8.2</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.13</version>
            <!--scope>provided</scope-->
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.4</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

</project>

---

package a.b.wso2;

import java.io.InputStream;
import net.sf.json.JSON;
import net.sf.json.JSONSerializer;
import net.sf.json.xml.XMLSerializer;
import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.util.AXIOMUtil;
import org.apache.axis2.AxisFault;
import org.apache.axis2.builder.Builder;
import org.apache.axis2.context.MessageContext;
import org.apache.commons.io.IOUtils;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;


public class WsoJtoX implements Builder {

    Logger logger = Logger.getLogger("a.b.wso2converter");

    @Override
    public OMElement processDocument(InputStream is, String contentType,
            MessageContext messageContext) throws AxisFault {
        String jsonData = "";
        try {

            jsonData = IOUtils.toString(is,"UTF-8");


            String output = process(jsonData);

            OMElement e = AXIOMUtil.stringToOM(output);
            return e;


        } catch (Exception e) {
            logger.error("error converting json string " + jsonData, e);
            if (e instanceof AxisFault) {
                throw (AxisFault) e;
            }
            throw new AxisFault("(B"+counter+") error converting json to xml", e);
        }

    }

    static int counter=0;

    public String process(String jsonData) throws AxisFault {

        try {
            String tran = "__ns__";

            jsonData=jsonData.replace("\r", "").trim();
            //jsonData=jsonData.replace("\n", "");

            String decoded = (jsonData.replaceAll("\"([a-zA-Z0-9_]*)\\:([a-zA-Z0-9]*)\"(\\s*)(:)", "\"$1" + tran + "$2\"$3:"));

            counter++;

            if (logger.isDebugEnabled()) {
                logger.debug("\n>>>>> (B"+counter+") converting json\n " + jsonData + "\n====");
            }

            XMLSerializer serializer = new XMLSerializer();
            JSON json = JSONSerializer.toJSON(decoded);

            String xml = serializer.write(json);

            //add in the soap stuff
            StringBuilder sb = new StringBuilder();
            sb.append("<soap:Envelope xmlns:soap=\"http://www.w3.org/2001/12/soap-envelope\" soap:encodingStyle=\"http://www.w3.org/2001/12/soap-encoding\"> <soap:Body>");
            sb.append(xml.replace("<?xml version=\"1.0\" encoding=\"UTF-8\"?>", ""));
            sb.append("</soap:Body></soap:Envelope>");

            if (logger.isDebugEnabled()) {
                logger.debug("\n==== (B"+counter+") to xml\n" + sb.toString()+"\n<<<<<");
            }

            return sb.toString();


        } catch (Exception e) {
            throw new AxisFault("(B"+counter+") error transforming json to xml", e);
        }

    }

}

---

package a.b.wso2;

import java.io.IOException;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.URL;
import net.sf.json.JSON;

import net.sf.json.xml.XMLSerializer;
import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.OMOutputFormat;
import org.apache.axiom.om.util.AXIOMUtil;
import org.apache.axis2.AxisFault;
import org.apache.axis2.Constants;
import org.apache.axis2.context.MessageContext;
import org.apache.axis2.transport.MessageFormatter;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;

public class WsoXtoJ implements MessageFormatter {

    Logger logger = Logger.getLogger("a.b.wso2converter");

    private static int counter=0;

    public String convert(String xData) {

        counter++;

            if (logger.isDebugEnabled()) {
                logger.debug("\n]]]]] (A"+counter+") converting xml\n " + xData + "\n-----");
            }


        try {
            String tran = "__ns__";
            XMLSerializer serializer = new XMLSerializer();
            OMElement e = AXIOMUtil.stringToOM(xData);
            OMElement b = (OMElement) e.getChildrenWithLocalName("Body").next();
            b = (OMElement) b.getChildElements().next();
            String xfrag = b.toStringWithConsume();
            String str = "";
            JSON j = serializer.read(xfrag);
            str = j.toString();
            String nstr = str.replaceAll("\"([a-zA-Z0-9_]+)" + tran + "([a-zA-Z0-9]+)\"(\\s*)(:)", "\"$1:$2\"$3:");  //", "\"$1:$2\"");

            if (logger.isDebugEnabled()) {
                logger.debug("\n----- (A"+counter+") to json\n" + nstr+"\n[[[[[");
            }

            return nstr;

        } catch (Exception e) {
            throw new RuntimeException(e);
        }

    }

    @Override
    public String formatSOAPAction(MessageContext msgCtxt, OMOutputFormat format,
            String soapActionString) {
        return null;
    }

    @Override
    public byte[] getBytes(MessageContext ctx, OMOutputFormat format)
            throws AxisFault {
        String env="";
        try {
            OMElement element = ctx.getEnvelope().getBody().getFirstElement();
            String payload = this.convert(element.toString());
            return payload.getBytes(format.getCharSetEncoding());
        } catch (UnsupportedEncodingException e) {
            logger.error("(A"+counter+") error converting xml to json "+ctx.getEnvelope().toString());
            throw AxisFault.makeFault(e);
        }
    }

    @Override
    public String getContentType(MessageContext msgCtxt, OMOutputFormat format,
            String soapActionString) {
        String contentType = (String) msgCtxt.getProperty(Constants.Configuration.CONTENT_TYPE);
        String encoding = format.getCharSetEncoding();
        if (contentType == null) {
            contentType = (String) msgCtxt.getProperty(Constants.Configuration.MESSAGE_TYPE);
        }
        if (encoding != null) {
            contentType += "; charset=" + encoding;
        }
        return contentType;
    }

    @Override
    public URL getTargetAddress(MessageContext msgCtxt, OMOutputFormat format,
            URL targetURL) throws AxisFault {
        return targetURL;
    }

    @Override
    public void writeTo(MessageContext msgCtxt, OMOutputFormat format,
            OutputStream out, boolean preserve) throws AxisFault {
        try {
            out.write(this.getBytes(msgCtxt, format));
            out.flush();
        } catch (IOException e) {
            throw AxisFault.makeFault(e);
        }
    }

}

Answer 3

I had the same problem.

In my experience, the JSON parser for WSO2 ESB (based on Axis2-json) supports only a subset of JSON:

1. The JSON has to start with "{", i.e. there can't be a JSONArray at the root.

2. Only the first key-value pair will be considered. This is because JSON is mapped to XML-like datastructure, and XML must have a root, so the first key-value pair is considered as root.

3. The value of the first key-value pair must not be an array. This because the converter has to know which XML tag should be used for each value:

   e.g.: ... { "key": ["val1", "val2", ...]} -> <key>val1</key><key>val2</key>....

I have the same problem here and want to find a fix for this. My thoughts are to create a new JSONBuilder (the parser which builds the internal SOAP message construct) and the JSONFormatter (the serializer) to use a virtual root (e.g. { "root" : ... } ) to fake the parser.