Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Trying to get a Python client talk to a Java Server using thrift's TFileTransport and TFileProcessor

I am trying to wrap a Java class to be called from Python using thrift's TFileTransport. I tried using two protocols TJSON and TBinary but I keep getting this exception

org.apache.thrift.transport.TTransportException: FileTransport error: bad event size
    at org.apache.thrift.transport.TFileTransport.readEvent(TFileTransport.java:327)
    at org.apache.thrift.transport.TFileTransport.read(TFileTransport.java:468)
    at org.apache.thrift.transport.TFileTransport.readAll(TFileTransport.java:439)
    at org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:263)
    at org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:320)
    at org.apache.thrift.protocol.TJSONProtocol.readJSONArrayStart(TJSONProtocol.java:784)
    at org.apache.thrift.protocol.TJSONProtocol.readMessageBegin(TJSONProtocol.java:795)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.thrift.transport.TFileProcessor.processUntil(TFileProcessor.java:69)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:102)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:111)
    at org.apache.thrift.transport.TFileProcessor.processChunk(TFileProcessor.java:118)
    at com.netflix.suro.client.SendToPyServer.startThriftServer(SendToPyServer.java:51)
    at com.netflix.suro.client.SendToPyServer.main(SendToPyServer.java:67)

This is how my Python client looks:

 def __init__(self):
        self.outFile=open("../../ThriftFile.in","a")
        self.transport = TTransport.TFileObjectTransport(self.outFile)
        self.protocol = TJSONProtocol.TJSONProtocol(self.transport)
        self.client = sendPyInterface.Client(self.protocol)
        self.transport.open()

    def send(self,routingKey, message):
        self.transport.write(pickle.dumps(self.client.send_send(routingKey, message))) 


    def configClient(self,configurationDict):       
        self.transport.write(pickle.dumps(self.client.send_ClientConfig(configurationDict)))    

if __name__ == "__main__":
    SuroClient=SuroPyClient()
    configurationDict={"ClientConfig.LB_TYPE":"static","ClientConfig.LB_SERVER":"localhost:7101"}
    SuroClient.configClient(configurationDict)
    SuroClient.send("routingKey", "testMessage")

and this is my server:

public static void startThriftServer(SendPyInterface.Processor processor) {
            try {
                File input = new File("src/main/java/com/netflix/suro/client/ThriftFile.in");
                if(!input.exists()){
                    input.createNewFile();
                }

                File output = new File("src/main/java/com/netflix/suro/client/ThriftFile.out");
                if(!output.exists()){
                    output.createNewFile();
                }

                TFileTransport inputFileTransport = new TFileTransport(input.getAbsolutePath(), true);
                TFileTransport outputFileTransport = new TFileTransport(output.getAbsolutePath(), false);
                System.out.println(input.getAbsolutePath());
                System.out.println(input.length());

                inputFileTransport.open();
                outputFileTransport.open();
                System.out.println(inputFileTransport.getBytesRemainingInBuffer());
                inputFileTransport.setTailPolicy(tailPolicy.WAIT_FOREVER);
                System.out.println("Wait ...");
                System.out.println(inputFileTransport.getBuffer());
                TFileProcessor fProcessor = 
                        new TFileProcessor(processor, new TJSONProtocol.Factory(), inputFileTransport, outputFileTransport);
                try {
                    fProcessor.processChunk();
                } catch (TTransportException e) {
                    e.printStackTrace();
                }


                System.out.println("File Thrift service started ...");
            } catch (Exception e) {
                e.printStackTrace();
            }

I have followed the example from this thread: Example on how to use TFileTransport in Thrift (Client/Server)

and from this post: http://theprogrammersguideto.com/thrift/blog/code/chapter-3-moving-bytes-with-transports/

like image 830
user2294950 Avatar asked Jun 17 '14 22:06

user2294950


1 Answers

According to the docs in the old Thrift wiki, the Java and C++ TFileTransport is actually a framed transport, which writes an additional 4-byte header preceding the data. Seems the names are a bit misleading here.

In contrast, the Python TFileObjectTransport used is just a simple wrapper. This means, that the transports are not compatible, hence the error message.

So instead of TFileTransport, the TSimpleFileTransport should be used with Java.

like image 181
JensG Avatar answered Sep 26 '22 05:09

JensG