Can generate .thrift files from existing java/scala interfaces and data types?

Question

Is there an easy way to take existing Java/scala datatypes and API interfaces and produce corresponding .thrift files? Having Thrift generate server data structures is excessively invasive as it has the consequences:

• I cannot annotate my data structures (e.g. for, XML, JSON, hibernate persistence, ...)
• this pattern conflicts with other serialization frameworks that want to own, or require modification of my source files.

As a result, its seems like thrift forces itself into being the exclusive persistence format for my server -- unless, that is, I create a data-marshalling wrapper around Thrift or the other my persistence formats that deal with these data structures (hibernate, Jackson, scala BeanProperty, ...). However, this defeats the purpose of an automated data-marshalling tool such as thrift and leads straight to the error-prone world of having to maintain identical-but-separate interfaces and data-structures (= waste of talented engineer time and energy).

I'm totally happy with Thrift auto-generating client code. However, I (strongly) feel that I need the freedom to edit the data structures my server deals with in the APIs.

Answer 1

You can use Swift.

To make a long story short; annotate your classes and interfaces (structs and services in Thrift parlance). Then you can either run Swift's client/server code or you can use the swift2thrift generator to produce equivalent IDL and use the Thrift compiler to generate clients (the latter is what I recommend for what you're describing).

Once that is done to create a TProcessor that you can use in a TServlet with normal TProtocol/TTransport objects, do something like this in your servlet's init():

protected void addProcessor(String name, Object svc) {
    ThriftCodecManager codecManager = new ThriftCodecManager(
        new CompilerThriftCodecFactory(false)
    );
    List<ThriftEventHandler> eventList = Collections.emptyList();
    ThriftServiceProcessor proc = new ThriftServiceProcessor(codecManager, eventList, svc);
    this.processors.put(name, proc);
    this.multiplex.registerProcessor(name, NiftyProcessorAdapters.processorToTProcessor(proc));
}

The multiplex instance variable in this example is an instance of TMultiplexedProcessor from libthrift.jar.

Then just do this in your doPost():

@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
        throws ServletException, IOException {
    getServletContext().log("entering doPost()");
    TTransport inTransport = null;
    TTransport outTransport = null;
    try {

        InputStream in = request.getInputStream();
        OutputStream out = response.getOutputStream();

        TTransport transport = new TIOStreamTransport(in, out);
        inTransport = transport;
        outTransport = transport;

        TProtocol inProtocol = getInProtocolFactory().getProtocol(inTransport);
        TProtocol outProtocol = getOutProtocolFactory().getProtocol(outTransport);

        if (multiplex.process(inProtocol, outProtocol)) {
            out.flush();
        } else {
            throw new ServletException("multiplex.process() returned false");
        }
    } catch (TException te) {
        throw new ServletException(te);
    } finally {
        if (inTransport != null) {
            inTransport.close();
        }
        if (outTransport != null) {
            outTransport.close();
        }
    }
}

FYI - TJSONProtocol doesn't work with the version of Swift prior to version 0.14 so at this time you'll need to build from source if you need to use that.

Also... Swift forces your structs to be marked final... JPA spec says entities can't be final... seems to work ok with Eclipselink anyhow but YMMV

Answer 2

Since you mention Java: For some project of ours I did implement an Xtext based solution which generates source code and Thrift IDL files from a project specific DSL. Since Xtext/Xtend is based on Java, it may at least be worth a look if that solution could fit your needs. However, I have a slight feeling that it may be overkill in your situation.

As a result, its seems like thrift forces itself into being the exclusive persistence format for my server [...]

That's not Thrifts fault, and it could be any format.

However, I (strongly) feel that I need the freedom to edit the data structures my server deals with in the APIs.

I fully agree. Especially in such cases it is recommended to separate serialization from internal data structures, and look at serialization as what it really is: just one way to manipulate data¹⁾. If you have more than one serialization format, you will always end up with a lot of stuff being implemented multiple times in a similar way. This is effect is more or less inavoidable, providing us with the opportunity to do it in whatever is the best way for the project.

The simple serialization strategy works fine with one format. It becomes cumbersome with two/three formats, and finally turns into a real PITA with more than three formats. ²⁾ As always, the maintainable and extendable solution comes at the cost of some added complexity. ³⁾

You see, altough you may pick one of the formats as the "favourite master data format", technically you don't have to.

---

_{(1) A surprising percentage of programming beginners tutorials and books fail to bring that point across properly. (2) This was exactly the use case I solved by means of a DSL as mentioned at the beginning. (3) Yes, I know what YAGNI means. And I know how to estimate the expectation value of risks.}