TensorFlow in production for real time predictions in high traffic app - how to use?

Question

What is the right way to use TensorFlow for real time predictions in a high traffic application.

Ideally I would have a server/cluster running tensorflow listening on a port(s) where I can connect from app servers and get predictions similar to the way databases are used. Training should be done by cron jobs feeding the training data through the network to the same server/cluster.

How does one actually use tensorflow in production? Should I build a setup where the python is running as a server and use the python scripts to get predictions? I'm still new to this but I feel that such script will need to open sessions etc.. which is not scalable. (I'm talking about 100s of predictions/sec).

Any pointer to relevant information will be highly appreciated. I could not find any.

Answer 1

This morning, our colleagues released TensorFlow Serving on GitHub, which addresses some of the use cases that you mentioned. It is a distributed wrapper for TensorFlow that is designed to support high-performance serving of multiple models. It supports both bulk processing and interactive requests from app servers.

For more information, see the basic and advanced tutorials.