How to work with Python applications and modules in Yocto Project

Introduction

This is a high-level tutorial and the intention is not to cover the Yocto Project in depth. If you are interested in detailed information, we recommend that you read the Yocto Project Mega-Manual.

Python is a commonly used language developing applications for the IoT and especially in the early stages of product development while one might still be evaluating an idea and maybe move on to a more performance oriented framework/languague once the idea has been validated

This tutorial will cover how to work with Python within the Yocto Project, and will cover how to:

  • include python3 in your image
  • inspect what Python modules are provided in the core layers
  • add a Python module that is not already provided
  • creating a recipe for a Python application

This tutorial will only cover Python3 as Python2 is reaching EOF by the end of 2019, but he workflows in Yocto Project are similar regardless if you are using Python2 or Python3.

Prerequisites

To follow this tutorial, you will need:

Step 1 - Installing Python

Ensuring that python3 is installed in your image is relatively simple and can be done by adding the following to local.conf:

IMAGE_INSTALL_append = " python3"

NOTE You can add this to your custom image as well. Please take a look at the How to create custom images using Yocto Project tutorial.

With this addition we can now re-build our intended image and python3 will be included.

bitbake core-image-base

You should see something similar to below during your build process which hints that python3 package is being prepared:

0: python3-3.7.2-r0 do_configure - 6s (pid 19653)

But this will only install the python3 runtime on to our target, which in it self is not very useful unless we install some of the available Python modules and this is where it can get complicated to figure out what is available if you are not familiar with the Yocto Project environment.

Step 2 - Listing and installing Python modules

First of all there many modules that are provided in the standard Python standard library. Modules such as:

  • datetime
  • io
  • core
  • multiprocessing
  • gzip

The Yocto Project environment has a special handling of these modules, and as a beginner this can lead to confusion and most people wonder why the standard library modules are not installed on the device when you installed python3.

In the Yocto Project environment, the most common Python modules are split out in to individual packages so that you can select which parts of the standard library you are interested in. This is an optimization if you are building a minimalistic system where you would typically would not want to include the complete standard library if the only module you are using is datetime or io.

You can get a list of these individual packages by reading this file:

cat ../sources/poky/meta/recipes-devtools/python/python3/python3-manifest.json

If the standard library module is not listed in the above file, it is part of the python3-misc package.

In the Yocto Project environment all module names are prefixed with python3-, and it looks like this (following our above example):

  • python3-datetime
  • python3-io
  • python3-core
  • python3-multiprocessing

NOTE! There is no python3-gzip package and that module is part of python3-misc package.

There is also a package called python3-modules which will install all standard library modules.

Third-party Python modules are provided as individual recipes, and one can run the following to get a list:

bitbake -s | grep ^python3

This list might be sparse and this is because only a small number of modules are provided by openembedded-core/poky.

To get a more complete list of modules we need to add the meta-openembedded/meta-python layer to our environment.

We can do this by cloning meta-openembedded:

NOTE You might already have the meta-openembedded layer in ../sources if you followed the tutorials on how to setup the Yocto Project environment. In which case you can skip this step.

git clone -b warrior https://github.com/openembedded/meta-openembedded ../sources/meta-openembedded

and then adding meta-openembedded/meta-python to our environment:

bitbake-layers add-layer ../sources/meta-openembedded/meta-python

Now you can list available Python modules again by running:

bitbake -s | grep ^python3

You should see a significant increase in available modules.

Installing Python modules can be done in a similar fashion to how we installed python3, which was covered in Step 1 - Installing Python, e.g to install python3-requests we can add the following to local.conf:

IMAGE_INSTALL_append = " python3-requests"

It is also always a good idea to search the OpenEmbedded Layer Index
if you are looking for missing modules. In many cases you will find that someone else has created a recipe for it, and you can either add that layer to your environment or make a copy of the recipe.

Step 3 - Creating a recipe for a Python module

There is a high probably that you will run in to the situation where you application depends on a specific Python module which is not provided by the methods that we covered in the previous sections. This is simply because there is a wast number of third party Python modules and it has to provide everything.

In this situation you are forced to add a recipe for that specific module that you might require, but luck ally it is fairly simple due to the β€œframeworks” that the Yocto Project provides.

For the sake of this tutorial we will use the memory-profiler Python module as a example, which is not provided out-of-the-box.

Step 3.1 - Create a custom layer

To be able to proceed with this tutorial we first must create a custom layer that will contain any custom modifications or additions. Please note that the tutorials on this site will re-use this layer and if you have already created this structure by following another tutorial you can skip this step.

Create a new layer called meta-stargazer using the bitbake-layers helper application:

bitbake-layers create-layer ../sources/meta-stargazer

This will create a basic structure in meta-stargazer directory:

../sources/meta-stargazer/
β”œβ”€β”€ conf
β”‚   └── layer.conf
β”œβ”€β”€ COPYING.MIT
β”œβ”€β”€ README
└── recipes-example
    └── example
        └── example_0.1.bb

3 directories, 4 files

Include the layer in our Yocto Project environment:

bitbake-layers add-layer ../sources/meta-stargazer

Step 3.2 - Create memory-profiler recipe

Create directory structure to house our recipe:

mkdir -p ../sources/meta-stargazer/recipes-devtools/python

Create the recipe:

cat <<- 'EOF' > ../sources/meta-stargazer/recipes-devtools/python/python3-memory-profiler_0.55.0.bb
SUMMARY = "This is a python module for monitoring memory consumption of a \
process as well as line-by-line analysis of memory consumption for python programs"
HOMEPAGE = "https://github.com/conda-forge/memory_profiler-feedstock"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://COPYING;md5=cde4ca348bb7feea790673481be0d980"

SRC_URI[md5sum] = "53ecb3d4be95a36fc1da339dce26d57f"
SRC_URI[sha256sum] = "5fa47b274c929dd2cbcd9190afb62fec110701251d2ac2d301caaf545c81afc1"

PYPI_PACKAGE = "memory_profiler"

inherit pypi setuptools3

RDEPENDS_${PN} += " \
    python3-psutil \
"
EOF

As you can see the recipe is fairly small and this largely due to the pypi and setuptools3 classes which do all the work for us, and what we essentially need to provide is the name and version of the module (license type + checksums) that we want to install and then it will download the sources from https://pypi.org/ and create a package which we can include to our image.

Majority of the information that I provided in above recipe I got from reading the memory-profiler page on https://pypi.org/. The most important to get right are:

  • module name
  • module version
  • dependencies
  • license type and md5sum
  • source md5sum and sha256

Now we can build the created Python module recipe:

bitbake python3-memory-profiler

Step 4 - Create recipe for a Python application

The workflow to create a recipe for a Python application is very similar to what we covered in Step 3 - Creating a recipe for a Python module. The difference might be that if you write a custom application it is probably not hosted https://pypi.org/ but other then that the steps should be pretty much the same.

The most painful thing is usually figuring out which dependencies the application has and ensuring that the recipe specifies these to avoid using the python3-modules package which is usually the easy way out , but it does waste a lot of space.

Step 5 - Python pip

You can add Python pip to your image by adding the following to local.conf:

IMAGE_INSTALL_append = " python3-pip"

But this is only useful for experimenting and testing and the pip flow does not really apply in a Yocto Project setting where you at build time want to define the content of your distribution

Conclusion

In this tutorial how to work with Python when using the Yocto Project in the hopes that it will help you get started quickly without need of digging to deep in Yocto Project internals. The layer that we created (meta-stargazer) an populated during this tutorial can be used a starting point for your own custom application/distribution layer.


If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!

3 Likes

Hi @mirzak fix

IMAGE_INSTALL_append = "python3"

in to

IMAGE_INSTALL_append = " python3"

Done, feel free to edit as well :slight_smile: