How to host cython web app on heroku?

Question

At the current moment I'm playing with Cython and trying to figure out how I can host a Cython Flask app (for example) on heroku.

Let's say my project looks like this (after cython compile):

_/cythonheroku
 |-- requirements.txt
 |-- run.py
 |-- Procfile
 |__/app
    |-- __init__.py
    |-- app.c
    |-- app.cpython-36m-darwin.so
    |-- app.pyx

Now, app.pyx has a standard Flask app in it with some cython adjustments, like so:

#cython: infer_types=True
from flask import Flask

app = Flask(__name__)

@app.route('/', methods=['GET'])
def index():
    cdef long x = 10000000
    cdef long long y = 0
    cdef int i
    for i in range(x):
        y += i
    return str(y)

Then, with the command cythonize -i app/app.pyx I compile my app.pyx code.

In run.py file I have:

from app.app import app

app.run()

And starting this from my command line python run.py will start a server on a localhost when I see a returned value from my for loop.

The problem: After I push this to heroku I get the error on the first line of run.py:

no module named app

As far as I understand – heroku just cant see my compiled app file.

UPD: Command in Procfile:

web: gunicorn run:app --log-file=-

UPD2: After some tests I figured out that Heroku can't recognize app.cpython-36m-darwin.so as a module. That's why I got that error.

Now the question is – how can I make heroku recognize .so file as python module?

Answer 1

Ok, I managed to find a solution.

Let's say our project looks like this

_/cythonheroku
 |-- requirements.txt
 |-- Procfile
 |__/app
    |-- __init__.py
    |-- app.pyx

Steps are as follows:

1. Create a project and push it to heroku without compiling .pyx file with cython
2. You need to set your stack to container and install some linux packages. So, basically, you need:

3. Remove your Procfile and install manifest plugin with:

   heroku update beta
   heroku plugins:install @heroku-cli/plugin-manifest
   heroku manifest:create
   

4. In created heroku.yml you need to specify packages and run command. Minimal viable manifest file will look like this:

   setup:
     config: {}
   build:
     languages:
       - python
     packages:
       - build-essential
   run:
     web: 'gunicorn app.app:app'
   

   We want to install build-essential so linux machine on heroku side will be able to compile our cython code. run command is a bit tricky – as far as I understand you can't run your cython app properly in heroku with something like:

   from app.app import app
   app.run()
   

   Heroku will put an error, that address already in use (for some reason it will run your app locally, and then try to start your app for web and you'll get an error. I don't know, I'll try to fix this somehow, but atm it's not that critical).

5. Change you stack to container and push:

   heroku stack:set container
   git push heroku master
   

6. You'll see a lot of stuffs in terminal. That's ok. Heroku will try to run your app with command specified in heroku.yml and you'll get an error. That's because we don't have our cython file yet. Now, the problem is that compiled file is unique for every OS and, I guess, every machine (depends on CPU chipset I think). That's why we want to compile our file on heroku side not locally. To do it, you need to connect to heroku terminal, to do it just type:

   heroku run bash
   

   Then you need to compile your code with simple:

   cythonize -i app/app.pyx
   

   Now, each heroku dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code, so your compiled file won't save at all. You need to download it on your machine, add to project and push to heroku again. You can do it with transfer.sh like so:

   curl --upload-file ./app/<your-file> https://transfer.sh/<name-of-file>
   

   Then you can download it with the link generated in the command line. Note: that you have to copy 2 files — <file>.c and <file>.cpython-36m-x86_64-linux-gnu.so. .so file can be named differently but you got the idea.

7. Add these two files to your project, commit and push to heroku.

8. That's it. Your app will be started and be available in web.

http://cython.herokuapp.com/