Can we run tensorflow lite on linux ? Or it is for android and ios only

Question

Hi is there any possibility to run tensorflow lite on linux platform? If yes, then how we can write code in java/C++/python to load and run models on linux platform? I am familiar with bazel and successfully made Android and ios application using tensorflow lite.

Answer 1

I think the other answers are quite wrong.

Look, I'll tell you my experience... I've been working with Django for many years, and I've been using normal tensorflow, but there was a problem with having 4 or 5 or more models in the same project. I don't know if you know Gunicorn + Nginx. This generates workers, so if you have 4 machine learning models, for every worker it multiplies, if you have 3 workers you will have 12 models preloaded in RAM. This is not efficient at all, because if the RAM overflows your project will fall or in fact the service responses are slower.

So this is where Tensorflow lite comes in. Switching from a tensorflow model to tensorflow lite improves and makes things much more efficient. Times are reduced absurdly. Also, Django and Gunicorn can be configured so that the model is pre-loaded and compiled at the same time. So every time the API is used up, it only generates the prediction, which helps you make each API call a fraction of a second long. Currently I have a project in production with 14 models and 9 workers, you can understand the magnitude of that in terms of RAM. And besides doing thousands of extra calculations, outside of machine learning, the API call does not take more than 2 seconds. Now, if I used normal tensorflow, it would take at least 4 or 5 seconds.

In summary, if you can use tensorflow lite, I use it daily in Windows, MacOS, and Linux, it is not necessary to use Docker at all. Just a python file and that's it. If you have any doubt you can ask me without any problem.

Here a example project Django + Tensorflow Lite

Answer 2

I agree with Nouvellie. It is possible and worth the time implementing. I developed a model on my Ubuntu 18.04 32 processor server and exported the model to tflite. The model ran in 178 secs on my ubuntu server. On my raspberry pi4 with 4GB memory, the tflite implementation ran in 85 secs, less than half the time of my server. When I installed tflite on my server the run time went down to 22 secs, an 8 fold increase in performance and now almost 4 times faster than the rpi4.

To install for python, I did not have to build the package but was able to use one of the prebuilt interpreters here:

https://www.tensorflow.org/lite/guide/python

I have Ubuntu 18.04 with python 3.7.7. So I ran pip install with the Linux python 3.7 package:

pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_x86_64.whl

Then import the package with:

from tflite_runtime.interpreter import Interpreter

Previous posts show how to use tflite.