ruamel.yaml requires python-dev in Ubuntu container build

Question

I recently switched from pyyaml to ruamel.yaml at the request of my OSS project's maintainer. Everything worked fine until I submitted the PR for the project's Buildkite Verification build which uses tox to automate multiple python version testing using a Ubuntu 18 container. In the second test (after a Python 3.8 test succeeded) using Python 3.9, ruamel suddenly decided to try to build the c extension which failed because we don't have Python-dev in the container. Here's an example of this failure:

https://buildkite.com/uavcan/nunavut-pr/builds/417#80433505-7b0c-4ea5-bdc6-ecf5938fc30d/111-1231

running build_ext
  building '_ruamel_yaml' extension
  creating build/temp.linux-x86_64-3.9
  creating build/temp.linux-x86_64-3.9/ext
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/repo/.tox/py39-rstdoctest/include -I/usr/include/python3.9 -c ext/_ruamel_yaml.c -o build/temp.linux-x86_64-3.9/ext/_ruamel_yaml.o
  ext/_ruamel_yaml.c:4:10: fatal error: Python.h: No such file or directory
      4 | #include "Python.h"
        |          ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

How do we force ruamel to use an all-python implementation and how do we prevent it from trying to build the native extension?

Answer 1

You are installing 'ruamel.yaml<16.0' and for that old version the wheels were never generated for Python 3.9 (or Python 3.8) for that matter.

The latest ruamel.yaml 0.15 version (0.15.100) was released July 2019, and the first version of Python 3.8 in October 2019, and the first release of Python 3.9 in October 2020 (and it takes some time before manylinux, used for the linux wheels, to be updated as well). So wheels were never generated.

The easiest solution is to use a newer version of ruamel.yaml, there is no reason to stick with that old version.

In newer versions of ruamel.yaml ( starting 0.16 IIRC) the C code is factored out into ruamel.yaml.clib (so I don't have to generate the wheels for every change in Python that warrents a new version of ruamel.yaml). If such a version tries to compile from source (e.g for Python 3.10 because those wheels are not available), you can install ruamel.yaml using pip install --no-deps ruamel.yaml and then it will not try to retrieve and compile ruamel.yaml.clib.