how to deploy pytorch neural network for production code in Unity

Question

I have a neural network trained in pytorch that I'd like to deploy into a Unity app. What's the best way to do it? I'm also interested in allowing the user to further train the neural network in the Unity app, which I guess would require to integrate some part of pytorch into Unity (maybe there's a way to integrate pytorch's C++ / torchscript API with Unity?). If anybody has experience with this, I'd like to know what the best alternatives are.

Answer 1

Check out the new features in Unity ML Agents. There is an inference engine within Unity ML Agents (called Barracuda) that allows you to use pretrained models within your app. AFAIK, you can convert Tensorflow and ONNX models into Barracuda. It should not be a problem as Pytorch models can be converted to the ONNX format. You may need to retrain your model if it is directly affected by the app (for example, if it is an RL agent).

EDIT: To answer your second question, you can continue to train the model but not in real time. What you may be able to do is collect data from the user, and use that to further train the model (that is how TensorFlow Serving works). You can do that by converting the PyTorch model into a TensorFlow model via ONNX.

EDIT 2: Barracuda is now a standalone and production ready inference engine that runs exclusively on the ONNX format. Any framework that can be converted into the format (e.g. Keras, Pytorch, MXNet) will work as long as they contain the supported operators.

Answer 2

This is not a complete answer. But it should be able to help you progress further.

Essentially you'd as I see it just need to be able to run Python code within C# code. In this case in collaboration with Unity's framework.

I did some searching and came across 4 partial solutions:

1. Unity python interpreter: https://forum.unity.com/threads/python-interpreter-in-unity.86461/

2. An example of running code with IronPython in Unity: https://gamedev.stackexchange.com/questions/123526/a-python-script-controlling-a-unity-game

3. Communication example between Unity3D C# and Python, using ZeroMQ: https://unitylist.com/p/hc8/Unity3D-Python-Communication

4. Implementing language support for other languages yourself via an XML file: https://forum.unity.com/threads/add-multiple-language-support-to-your-unity-projects.206271/

Once you're able to run the code you'd then need to refer to the location of your python files. This way you should be able to run it if the environment you run it in also has Python installed and set up correctly.

You must ensure that the files are in your project on deployment. And that Unity can access them.

Hope this helps you.