Introduction
This is a high-level tutorial and the intention is not to cover the Yocto Project in depth. If you are interested in detailed information, we recommend that you read the Yocto Project Mega-Manual.
Python is a commonly used language developing applications for the IoT and especially in the early stages of product development while one might still be evaluating an idea and maybe move on to a more performance oriented framework/languague once the idea has been validated
This tutorial will cover how to work with Python within the Yocto Project, and will cover how to:
- include
python3in your image - inspect what Python modules are provided in the core layers
- add a Python module that is not already provided
- creating a recipe for a Python application
This tutorial will only cover Python3 as Python2 has reached EOL in 2019, but the workflow in Yocto Project is similar regardless if you are using Python2 or Python3.
Version notes
This tutorial targets the following Yocto Project versions:
| Yocto Project | Tutorial applies | Maintenance |
|---|---|---|
| nanbield (4.3) |  |
development |
| mickledore (4.2) | |
current stable |
| langdale (4.1) | |
stable |
| kirkstone (4.0) | |
LTS |
| honister (3.4) | |
 EOL |
| hardknott (3.3) |  1 |
EOL |
| gatesgarth (3.2) | 2 |
EOL |
| dunfell (3.1) | 3 |
LTS |
| zeus (3.0) | 4 |
EOL |
| warrior (2.7) | 4 |
EOL |
| thud (2.6) | 4 |
EOL |
| sumo (2.5) | 4 |
EOL |
| rocko (2.4) | 4 |
EOL |
1. Support for the :-override syntax was added to hardknott in the 3.3.3 patch release.
2. Support for the :-override syntax was added to gatesgarth master only, no patch release.
3. Support for the :-override syntax was added to dunfell in the 3.1.11 patch release.
Prerequisites
To follow this tutorial, you will need:
- A configured Yocto environment. Please take a look at one of the following if you do not have this prepared:
- Enough Free Disk: Your development host should have at least 50 GBytes of free disk space for building images.
Step 1 - Installing Python
Ensuring that python3 is installed in your image is relatively simple and can be done by adding the following to local.conf:
IMAGE_INSTALL:append = " python3"
NOTE You can add this to your custom image as well. Please take a look at the How to create custom images using Yocto Project tutorial.
With this addition we can now re-build our intended image and python3 will be included.
bitbake core-image-base
You should see something similar to below during your build process which hints that python3 package is being prepared:
0: python3-3.10.7-r0 do_configure - 9s (pid 577367)
But this will only install the python3 runtime on to our target, which in it self is not very useful unless we install some of the available Python modules and this is where it can get complicated to figure out what is available if you are not familiar with the Yocto Project environment.
Step 2 - Listing and installing Python modules
First of all there many modules that are provided in the standard Python standard library. Modules such as:
datetimeiocoremultiprocessinggzip
The Yocto Project environment handles the standard library modules slightly different then one is used to on a desktop distribution, and as a beginner this can lead to confusion and most people wonder why the standard library modules are not installed on the device when you installed python3.
In the Yocto Project environment, the most common Python modules are split out in to individual packages so that you can select which parts of the standard library you are interested in. This is an optimization if you are building a minimalist operating system where you would typically would not want to include the complete standard library if the only module you are using is datetime or io.
You can get a list of these individual packages by reading this file:
cat ../sources/poky/meta/recipes-devtools/python/python3/python3-manifest.json
If the standard library module is not listed in the above file, it is part of the python3-misc package.
In the Yocto Project environment all module names are prefixed with python3-, and it looks like this (following our above example):
python3-datetimepython3-iopython3-corepython3-multiprocessing
NOTE! There is no python3-gzip package and that module is part of python3-misc package.
There is also a package called python3-modules which will install all standard library modules.
Third party Python modules are provided as individual recipes, and one can run the following to get a list:
bitbake -s | grep ^python3
This list might be sparse and this is because only a small number of modules are provided by openembedded-core/poky.
To get a more complete list of modules we need to add the meta-openembedded/meta-python layer to our environment.
We can do this by cloning meta-openembedded:
NOTE You might already have the meta-openembedded layer in ../sources if you followed the tutorials on how to setup the Yocto Project environment. In which case you can skip this step.
git clone -b kirkstone https://github.com/openembedded/meta-openembedded ../sources/meta-openembedded
and then adding meta-openembedded/meta-python to our environment:
bitbake-layers add-layer ../sources/meta-openembedded/meta-python
As meta-python depends on meta-openembedded you may have to include that as well if you haven’t done so following a previous tutorial:
bitbake-layers add-layer ../sources/meta-openembedded/meta-oe
Now you can list available Python modules again by running:
bitbake -s | grep ^python3
You should see a significant increase in available modules.
Installing Python modules can be done in a similar fashion to how we installed python3, which was covered in Step 1 - Installing Python, e.g to install python3-requests we can add the following to local.conf:
IMAGE_INSTALL:append = " python3-requests"
It is also always a good idea to search the OpenEmbedded Layer Index
if you are looking for missing modules. In many cases you will find that someone else has created a recipe for it, and you can either add that layer to your environment or make a copy of the recipe.
Step 3 - Creating a recipe for a Python module
There is a high probably that you will run in to the situation where you application depends on a specific Python module which is not provided by the methods that we covered in the previous sections. This is simply because there is a vast number of third party Python modules and it has to provide everything.
In this situation you are forced to add a recipe for that specific module that you might require, but luck ally it is fairly simple due to the “frameworks” that the Yocto Project provides.
For the sake of this tutorial we will use the memory-profiler Python module as a example, which is not provided out-of-the-box.
Step 3.1 - Create a custom layer
To be able to proceed with this tutorial we first must create a custom layer that will contain any custom modifications or additions. Please note that the tutorials on this site will re-use this layer and if you have already created this structure by following another tutorial you can skip this step.
Create a new layer called meta-stargazer using the bitbake-layers helper application:
bitbake-layers create-layer ../sources/meta-stargazer
This will create a basic structure in meta-stargazer directory:
../sources/meta-stargazer/
├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-example
└── example
└── example_0.1.bb
3 directories, 4 files
Include the layer in our Yocto Project environment:
bitbake-layers add-layer ../sources/meta-stargazer
Step 3.2 - Create memory-profiler recipe
Create directory structure to house our recipe:
mkdir -p ../sources/meta-stargazer/recipes-devtools/python
Create the recipe:
cat <<- 'EOF' > ../sources/meta-stargazer/recipes-devtools/python/python3-memory-profiler_0.61.0.bb
SUMMARY = "This is a python module for monitoring memory consumption of a \
process as well as line-by-line analysis of memory consumption for python programs"
HOMEPAGE = "https://github.com/conda-forge/memory_profiler-feedstock"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://COPYING;md5=cde4ca348bb7feea790673481be0d980"
SRC_URI[md5sum] = "5fe93d5035288095c4f86ef69ee19f37"
SRC_URI[sha256sum] = "4e5b73d7864a1d1292fb76a03e82a3e78ef934d06828a698d9dada76da2067b0"
PYPI_PACKAGE = "memory_profiler"
inherit pypi setuptools3
RDEPENDS_${PN} += " \
python3-psutil \
"
EOF
As you can see the recipe is fairly small and this due to the pypi and setuptools3 classes which do all the work for us, and what we essentially need to provide is the name and version of the module (license type + checksums) that we want to install and then it will download the sources from https://pypi.org/ and create a package which we can include to our image.
The majority of the information that I provided in above recipe I got from reading the memory-profiler page on https://pypi.org/. The most important to get right are:
- module name
- module version
- dependencies
- license type and md5sum
- source md5sum and sha256
Now we can build the created Python module recipe:
bitbake python3-memory-profiler
Step 4 - Create recipe for a Python application
The workflow to create a recipe for a Python application is very similar to what we covered in Step 3 - Creating a recipe for a Python module. The difference might be that if you write a custom application it is probably not hosted https://pypi.org/ but other then that the steps should be pretty much the same.
The most painful thing is usually figuring out which dependencies the application has and ensuring that the recipe specifies these to avoid using the python3-modules package which is usually the easy way out , but it does waste a lot of space.
Step 5 - Python pip
You can add Python pip to your image by adding the following to local.conf:
IMAGE_INSTALL:append = " python3-pip"
Note: this is only useful for experimenting and testing. The pip flow does not really apply in a Yocto Project setting where you want to define the content of your distribution at build time.
Conclusion
In this tutorial we covered how to work with Python when using the Yocto Project in the hopes that it will help you get started quickly without need of digging to deep in Yocto Project internals. The layer that we created (meta-stargazer) and populated during this tutorial can be used a starting point for your own custom application layer.
Appendix - EOL versions
override syntax
The syntax for override operations is currently using the :-character as a separator. Older versions of Yocto respectively BitBake used the _-character to separate variable name from operator. In order to apply this tutorial to older setups you generally have to replace the : separators with _, but be sure to cross check with other operations in your setup.
If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!
