Write a recipe in yocto for a python application

Question

I have a simple python application which does:

1. acquire information from a GPS
2. parse information
3. store it in InfluxDB

The package requirements:

certifi==2018.4.16
chardet==3.0.4
idna==2.6 
influxdb==5.0.0
pynmea2==1.12.0 
pyserial==3.4
python-dateutil==2.7.3
pytz==2018.4
requests==2.18.4
six==1.11.0
urllib3==1.22          

The above is generated by using:

pip3 install pynmea2 pyserial influxdb

In the OpenEmbedded Layers Index I have already found pyserial package for Python3. Which implies on the board I just might need to do pip3 install pynmea2 influxdb.

How do you go ahead writing my application's recipe with all the above mentioned pip dependencies in mind?

There aren't any tutorials I have found for writing recipes for python applications. (On the contrary Node applications do have some guidance on the wiki page for yocto.

Upon checking some recipes in meta-python layer I found some .inc files but not sure how to go about it

Answer 1

Creating Recipes for non-available python apps

Since influxdb-python and pynmea2 are not available as standard python recipes, I began by creating recipes for them using devtool.

Steps

1. use devtool to add the influxdb-python

   devtool add influxdb-python https://github.com/influxdata/influxdb-python/archive/v5.2.0.tar.gz

2. use devtool to add the pynmea2

   devtool add pynmea2 https://github.com/Knio/pynmea2/archive/1.7.1.tar.gz

The above mentioned steps creates a folder workspace in your $BUILD_DIR and created auto-generated recipes for the repos.

3. Edit the recipes

   devtool edit-recipe influxdb-python

4. add or check DEPEND_${PN} and RDEPENDS_${PN} to your recipes accordingly. I added all the requirements.txt for influxdb-python to RDEPENDS_${PN} viz.

   RDEPEND_${PN} += "${PYTHON_PN}-modules ${PYTHON_PN}-requests ${PYTHON_PN}-dateutil ${PYTHON_PN}-pytz ${PYTHON_PN}-six"

   NOTE: I have not added pandas or numpy as they aren't relevant for my application.

5. I added DEPENDS_${PN} = "${PYTHON_PN}-modules also.

NOTE: Perform the same for pynmea2 but since it does not have any requirements.txt I added RDEPENDS_${PN} = "${PYTHON_PN}-modules" so all major things are available on the target.

Recipe Structure

GitHub Gist for Recipes

I followed the meta-python folder's structure where each of the recipes consists of :

• recipe.inc
• recipe_version_number.bb

In the influxdb_python.inc keep all the stuff generated from devtool viz.

# Recipe created by recipetool
# This is the basis of a recipe and may need further editing in order to be fully functional.
# (Feel free to remove these comments when editing.)
#
# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
# your responsibility to verify that the values are complete and correct.
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=046523829184aac3703a4c60c0ae2104"

HOMEPAGE = "https://github.com/influxdb/influxdb-python"
SUMMARY = "InfluxDB client"

SRC_URI = "https://github.com/influxdata/influxdb-python/archive/v${PV}.tar.gz"
SRC_URI[md5sum] = "105d88695151e241523b31dd1375096e"
SRC_URI[sha256sum] = "620de85bcca5207b06ec1565884b6d10b4be01d579a78e08b1e922f453fdac05"

DEPENDS_${PN} = "${PYTHON_PN}-modules"
RDEPENDS_${PN} = "${PYTHON_PN}-modules ${PYTHON_PN}-requests ${PYTHON_PN}-dateutil ${PYTHON_PN}-pytz ${PYTHON_PN}-six"

In the influxdb_python_5.2.0.bb I added the following lines:

inherit setuptools3 pypi                              
require influxdb-python.inc

NOTE: I added setuptools3 since I want my app to be run on python3.5. For python2.7 use setuptools.

Similarly, I did the same for pynmea2.inc:

# Recipe created by recipetool
# This is the basis of a recipe and may need further editing in order to be fully functional.
# (Feel free to remove these comments when editing.)
#
# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
# your responsibility to verify that the values are complete and correct.
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=bb5e173bc54080cb25079199959ba6b6"

HOMEPAGE = "https://github.com/Knio/pynmea2"
SUMMARY = "Python library for the NMEA 0183 protcol"

SRC_URI = "https://github.com/Knio/pynmea2/archive/${PV}.tar.gz"
SRC_URI[md5sum] = "a90baf61f4e676bef76099e4bd7c0581"
SRC_URI[sha256sum] = "8f8f68623bd2d5dab7f04a9c31813a3f4aa15467db0373cbce6b9b0ae44ca48e"

#DEPENDS_${PN} = "${PYTHON_PN}-datetime ${PYTHON_PN}-threading ${PYTHON_PN}-io"
DEPENDS_${PN} = "${PYTHON_PN}-modules"
# WARNING: the following rdepends are determined through basic analysis of the
# python sources, and might not be 100% accurate.
RDEPENDS_${PN} = "${PYTHON_PN}-modules"

For pynmea2_1.7.1.bb:

inherit setuptools3 pypi
require pynmea2.inc

Baking the recipes

You could test them with bitbake -k influxdb-python and bitbake -k pynmea2 or with devtool build influxdb-python and devtool build pynmea2

If you have no errors then you can deploy it on target using:

devtool deploy-target influxdb-python user@machineIP:dest_folder

Checks

You can check by firing the python shell

# python3 

 >> import influxdb-python
 >> import pyserial

if the import is throws no missing modules error then it is success!!

Final steps

• You can undeploy the modules: devtool undeploy-target recipe_name [address of target]

• send the recipes to you custom meta layer devtool finish recipe_name ../meta-custom

NOTE: If you are using krogoth or lower the you will have to move your recipes to you meta layer manually

• Now include these recipes in your conf/local.conf with IMAGE_INSTALL_append = " influxdb-python pynmea2" and bitbake -k your-image-name

Custom Application

Not tested yet.

But I think I will simple add my app like mentioned in YoctoCookBook Repository for hello-world with my meta layer.

NUGGETS

• ${PYTHON_PN}-modules is a saviour really. I tried manually added runtime deps and everytime i deployed it on the board there were always some dependencies missing. But adding the modules solved all the missing deps issue in an instance.

• I am not sure when to use DEPENDS_${PN} but I assume most python applications depend on the basic python-modules hence I added them.

• NOT A YOCTO EXPERT but this is just my finding in the last 2 weeks. There is a lack of proper examples for Python in Yocto. hope this helps someone.