Tensorflow Serving - Stateful LSTM

Question

Is there a canonical way to maintain a stateful LSTM, etc. with Tensorflow Serving?

Using the Tensorflow API directly this is straightforward - but I'm not certain how best to accomplish persisting LSTM state between calls after exporting the model to Serving.

Are there any examples out there which accomplish the above? The samples within the repo are very basic.

Answer 1

From Martin Wicke on the TF mailing list:

"There's no good integration for stateful models in the model server yet. As you noted, it basically assumes models are pure functions. We're working on this, and you should see this functionality appear eventually, but it's too far out to promise a time. So in the meantime, you can write a simple wrapper which keeps state on the server (and assigns some sort of session ID which is passed around in requests), or you can write your own server which maintains the TensorFlow session state (and similarly returns some session ID). The latter is more performant. Both will require some sort of garbage collection/session timeout logic."