# How to work with Python applications and modules in Yocto Project
**URL:** https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135
**Category:** Yocto Project
**Created:** [October 10, 2019, 12:39pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135 "2019-10-10T12:39:11Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [October 10, 2019, 12:39pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/1 "2019-10-10T12:39:11Z")
## Introduction
This is a high-level tutorial and the intention is not to cover the Yocto Project in depth. If you are interested in detailed information, we recommend that you read the [Yocto Project Mega-Manual](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html).
Python is a commonly used language developing applications for the IoT and especially in the early stages of product development while one might still be evaluating an idea and maybe move on to a more performance oriented framework/languague once the idea has been validated
This tutorial will cover how to work with Python within the Yocto Project, and will cover how to:
- include `python3` in your image
- inspect what Python modules are provided in the core layers
- add a Python module that is not already provided
- creating a recipe for a Python application
**This tutorial will only cover Python3 as Python2 has reached [EOL in 2019](https://devguide.python.org/versions/), but the workflow in Yocto Project is similar regardless if you are using Python2 or Python3.**
## Version notes
This tutorial targets the following Yocto Project versions:
| Yocto Project | Tutorial applies | Maintenance |
| --- | --- | --- |
| nanbield (4.3) | :test_works: | :test_works: development |
| mickledore (4.2) | :test_works: | :test_works: current stable |
| langdale (4.1) | :test_works: | :test_works: stable |
| kirkstone (4.0) | :test_works: | :test_works: LTS |
| honister (3.4) | :test_works: | :test_fails: EOL |
| hardknott (3.3) | :test_workaround: [1](#footnote1) | :test_fails: EOL |
| gatesgarth (3.2) | :test_workaround: [2](#footnote2) | :test_fails: EOL |
| dunfell (3.1) | :test_workaround: [3](#footnote3) | :test_works: LTS |
| zeus (3.0) | :test_fails: [4](#footnote4) | :test_fails: EOL |
| warrior (2.7) | :test_fails: [4](#footnote4) | :test_fails: EOL |
| thud (2.6) | :test_fails: [4](#footnote4) | :test_fails: EOL |
| sumo (2.5) | :test_fails: [4](#footnote4) | :test_fails: EOL |
| rocko (2.4) | :test_fails: [4](#footnote4) | :test_fails: EOL |
1. Support for the `:`-override syntax was added to `hardknott` in the 3.3.3 patch release.
2. Support for the `:`-override syntax was added to `gatesgarth` master only, no patch release.
3. Support for the `:`-override syntax was added to `dunfell` in the 3.1.11 patch release.
## Prerequisites
To follow this tutorial, you will need:
- **A configured Yocto environment**. Please take a look at one of the following if you do not have this prepared:
- [Preparing the Yocto Project environment from scratch](https://hub.mender.io/t/preparing-the-yocto-project-environment-from-scratch/801)
- [Preparing the Yocto Project environment based on Mender demo environment](https://hub.mender.io/t/preparing-the-yocto-project-environment-based-on-mender-demo-environment)
- **Enough Free Disk:** Your development host should have at least 50 GBytes of free disk space for building images.
## Step 1 - Installing Python
Ensuring that `python3` is installed in your image is relatively simple and can be done by adding the following to `local.conf`:
```auto
IMAGE_INSTALL:append = " python3"
```
**NOTE** You can add this to your custom image as well. Please take a look at the [How to create custom images using Yocto Project](https://hub.mender.io/t/how-to-create-custom-images-using-yocto-project/902) tutorial.
With this addition we can now re-build our intended image and `python3` will be included.
```auto
bitbake core-image-base
```
You should see something similar to below during your build process which hints that `python3` package is being prepared:
> 0: python3-3.10.7-r0 do\_configure - 9s (pid 577367)
But this will only install the `python3` runtime on to our target, which in it self is not very useful unless we install some of the available Python modules and this is where it can get complicated to figure out what is available if you are not familiar with the Yocto Project environment.
## Step 2 - Listing and installing Python modules
First of all there many modules that are provided in the standard [Python standard library](https://github.com/python/cpython/tree/master/Lib). Modules such as:
- `datetime`
- `io`
- `core`
- `multiprocessing`
- `gzip`
The Yocto Project environment handles the standard library modules slightly different then one is used to on a desktop distribution, and as a beginner this can lead to confusion and most people wonder why the standard library modules are not installed on the device when you installed `python3`.
In the Yocto Project environment, the most common Python modules are split out in to individual packages so that you can select which parts of the standard library you are interested in. This is an optimization if you are building a minimalist operating system where you would typically would not want to include the complete standard library if the only module you are using is `datetime` or `io`.
You can get a list of these individual packages by reading this file:
```auto
cat ../sources/poky/meta/recipes-devtools/python/python3/python3-manifest.json
```
If the standard library module is not listed in the above file, it is part of the `python3-misc` package.
In the Yocto Project environment all module names are prefixed with `python3-`, and it looks like this (following our above example):
- `python3-datetime`
- `python3-io`
- `python3-core`
- `python3-multiprocessing`
**NOTE!** There is no `python3-gzip` package and that module is part of `python3-misc` package.
There is also a package called `python3-modules` which will install **all standard library modules**.
Third party Python modules are provided as individual recipes, and one can run the following to get a list:
```auto
bitbake -s | grep ^python3
```
This list might be sparse and this is because only a small number of modules are provided by [openembedded-core](https://github.com/openembedded/openembedded-core)/[poky](https://git.yoctoproject.org/cgit/cgit.cgi/poky/).
To get a more complete list of modules we need to add the [meta-openembedded/meta-python](https://github.com/openembedded/meta-openembedded/tree/master/meta-python) layer to our environment.
We can do this by cloning [meta-openembedded](https://github.com/openembedded/meta-openembedded.git):
**NOTE** You might already have the meta-openembedded layer in `../sources` if you followed the tutorials on how to setup the Yocto Project environment. In which case you can skip this step.
```auto
git clone -b kirkstone https://github.com/openembedded/meta-openembedded ../sources/meta-openembedded
```
and then adding [meta-openembedded/meta-python](https://github.com/openembedded/meta-openembedded/tree/master/meta-python) to our environment:
```auto
bitbake-layers add-layer ../sources/meta-openembedded/meta-python
```
As `meta-python` depends on `meta-openembedded` you may have to include that as well if you haven’t done so following a previous tutorial:
```auto
bitbake-layers add-layer ../sources/meta-openembedded/meta-oe
```
Now you can list available Python modules again by running:
```auto
bitbake -s | grep ^python3
```
You should see a significant increase in available modules.
Installing Python modules can be done in a similar fashion to how we installed `python3`, which was covered in _Step 1 - Installing Python_, e.g to install `python3-requests` we can add the following to `local.conf`:
```auto
IMAGE_INSTALL:append = " python3-requests"
```
It is also always a good idea to search the [OpenEmbedded Layer Index
](https://layers.openembedded.org/layerindex/branch/master/layers/) if you are looking for missing modules. In many cases you will find that someone else has created a recipe for it, and you can either add that layer to your environment or make a copy of the recipe.
## Step 3 - Creating a recipe for a Python module
There is a high probably that you will run in to the situation where you application depends on a specific Python module which is not provided by the methods that we covered in the previous sections. This is simply because there is a vast number of third party Python modules and it has to provide everything.
In this situation you are forced to add a recipe for that specific module that you might require, but luck ally it is fairly simple due to the “frameworks” that the Yocto Project provides.
For the sake of this tutorial we will use the [memory-profiler](https://pypi.org/project/memory-profiler/) Python module as a example, which is not provided out-of-the-box.
### Step 3.1 - Create a custom layer
To be able to proceed with this tutorial we first must create a custom layer that will contain any custom modifications or additions. Please note that the tutorials on this site will re-use this layer and if you have already created this structure by following another tutorial you can skip this step.
Create a new layer called `meta-stargazer` using the `bitbake-layers` helper application:
```auto
bitbake-layers create-layer ../sources/meta-stargazer
```
This will create a basic structure in `meta-stargazer` directory:
```auto
../sources/meta-stargazer/
├── conf
│ └── layer.conf
├── COPYING.MIT
├── README
└── recipes-example
└── example
└── example_0.1.bb
3 directories, 4 files
```
Include the layer in our Yocto Project environment:
```auto
bitbake-layers add-layer ../sources/meta-stargazer
```
### Step 3.2 - Create memory-profiler recipe
Create directory structure to house our recipe:
```auto
mkdir -p ../sources/meta-stargazer/recipes-devtools/python
```
Create the recipe:
```auto
cat <<- 'EOF' > ../sources/meta-stargazer/recipes-devtools/python/python3-memory-profiler_0.61.0.bb
SUMMARY = "This is a python module for monitoring memory consumption of a \
process as well as line-by-line analysis of memory consumption for python programs"
HOMEPAGE = "https://github.com/conda-forge/memory_profiler-feedstock"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://COPYING;md5=cde4ca348bb7feea790673481be0d980"
SRC_URI[md5sum] = "5fe93d5035288095c4f86ef69ee19f37"
SRC_URI[sha256sum] = "4e5b73d7864a1d1292fb76a03e82a3e78ef934d06828a698d9dada76da2067b0"
PYPI_PACKAGE = "memory_profiler"
inherit pypi setuptools3
RDEPENDS_${PN} += " \
python3-psutil \
"
EOF
```
As you can see the recipe is fairly small and this due to the `pypi` and `setuptools3` classes which do all the work for us, and what we essentially need to provide is the name and version of the module (license type + checksums) that we want to install and then it will download the sources from [https://pypi.org/](https://pypi.org/) and create a package which we can include to our image.
The majority of the information that I provided in above recipe I got from reading the [memory-profiler](https://pypi.org/project/memory-profiler/#modal-close) page on [https://pypi.org/](https://pypi.org/). The most important to get right are:
- module name
- module version
- dependencies
- license type and md5sum
- source md5sum and sha256
Now we can build the created Python module recipe:
```auto
bitbake python3-memory-profiler
```
## Step 4 - Create recipe for a Python application
The workflow to create a recipe for a Python application is very similar to what we covered in _Step 3 - Creating a recipe for a Python module_. The difference might be that if you write a custom application it is probably not hosted [https://pypi.org/](https://pypi.org/) but other then that the steps should be pretty much the same.
The most painful thing is usually figuring out which dependencies the application has and ensuring that the recipe specifies these to avoid using the `python3-modules` package which is usually the easy way out , but it does waste a lot of space.
## Step 5 - Python pip
You can add Python `pip` to your image by adding the following to `local.conf`:
```auto
IMAGE_INSTALL:append = " python3-pip"
```
Note: this is only useful for experimenting and testing. The `pip` flow does not really apply in a Yocto Project setting where you want to define the content of your distribution at build time.
## Conclusion
In this tutorial we covered how to work with Python when using the Yocto Project in the hopes that it will help you get started quickly without need of digging to deep in Yocto Project internals. The layer that we created (`meta-stargazer`) and populated during this tutorial can be used a starting point for your own custom application layer.
## Appendix - EOL versions
### override syntax
The syntax for override operations is currently using the `:`-character as a separator. Older versions of Yocto respectively BitBake used the `_`-character to separate variable name from operator. In order to apply this tutorial to older setups you generally have to replace the `:` separators with `_`, but be sure to cross check with other operations in your setup.
* * *
_If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!_
---
### Author:  [@jorisoffouga](https://hub.mender.io/u/jorisoffouga)
#### Post date: [October 10, 2019, 12:52pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/2 "2019-10-10T12:52:35Z")
Hi @mirzak fix
```auto
IMAGE_INSTALL_append = "python3"
```
in to
```auto
IMAGE_INSTALL_append = " python3"
```
---
### Author:  [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [October 10, 2019, 12:53pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/3 "2019-10-10T12:53:43Z")
Done, feel free to edit as well 🙂
---
### Author:  [@HerrMuellerluedensch](https://hub.mender.io/u/HerrMuellerluedensch)
#### Post date: [February 28, 2020, 3:48pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/4 "2020-02-28T15:48:50Z")
Hey,
I remember doing this tutorial a few days ago and it worked. However, now, I worked on the stargazer image a little (basically following [the systemd tutorial](https://hub.mender.io/t/how-to-create-your-first-recipe-and-enable-auto-start-using-systemd/1195/8). When I add `python3` again, start qemu and try to start python it fails:
```
root@qemux86-64:~# python3
[56.914963] traps: python3[315] trap invalid opcode ip:7f7f73325686 sp:7ffe8025d930 error:0 in libpython3.7m.so.1.0[7f7f731e2000+1bc000]
Illegal instruction
```
Do you have a hint on how to debug this?
---
### Author:  [@HerrMuellerluedensch](https://hub.mender.io/u/HerrMuellerluedensch)
#### Post date: [February 29, 2020, 7:48pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/5 "2020-02-29T19:48:41Z")
Is it possible that this issue is related to the host system? Python3 works perfectly fine when building for qemu on `Ubuntu 18.04.3 LTS` but if fails on when building on another machine running `Ubuntu 18.04.4 LTS`. Could you check if anyone is running `18.04.4 LTS` to exclude this possible error source? Would be very appreciated.
EDIT: The most likely reason was most obvious: A mismatch between poky (warrior) and meta-openembedded (zeus). I stashed everything and just started a clean, fresh rebuild. If the error persists, I’ll keep you updated.
EDIT II: I was wrong. The error persists on a clean warrior build, with and without meta-oe/meta-python from the same branch.
---
### Author:  [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [March 3, 2020, 10:16am UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/6 "2020-03-03T10:16:08Z")
Hi @HerrMuellerluedensch, I think this error depends on how you start the QEMU machine.
Can you try adding `-m core2duo` to your qemu arguments on startup?
---
### Author:  [@bearrito](https://hub.mender.io/u/bearrito)
#### Post date: [April 7, 2020, 1:52am UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/7 "2020-04-07T01:52:04Z")
This advice did not work for me,it was close however. Instead I used
> -cpu core2duo
On my host the -m flag is for memory of the virtual host
---
### Author:  [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [December 1, 2020, 7:48am UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/8 "2020-12-01T07:48:39Z")
2 posts were split to a new topic: [Bitbake parse error with python recipe](https://hub.mender.io/t/bitbake-parse-error-with-python-recipe/2915)
---
### Author:  [@VinayakSingh](https://hub.mender.io/u/VinayakSingh)
#### Post date: [May 7, 2021, 3:50pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/9 "2021-05-07T15:50:36Z")
I want to install python 3.7 in yocto image
what is the syntax?
---
### Author:  [@drewmoseley](https://hub.mender.io/u/drewmoseley)
#### Post date: [May 7, 2021, 5:44pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/10 "2021-05-07T17:44:25Z")
Well the basic syntax is:
> IMAGE\_INSTALL\_append = " python3 "
But depending on the branch you are on that may not get you python 3.7. For instance, currently, Dunfell has Python 3.8.
Drew
---
### Author:  [@Yasmine](https://hub.mender.io/u/Yasmine)
#### Post date: [June 21, 2021, 9:40am UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/11 "2021-06-21T09:40:55Z")
Hello, So while m new to the whole yocto things, I dont know really where to run those command, I added a new document in the python directory I named it with the name of the recipe but It doesnt work after when I use this ’ ```
bitbake python3-memory-profiler
```auto
THank you
```
---
### Author:  [@drewmoseley](https://hub.mender.io/u/drewmoseley)
#### Post date: [June 21, 2021, 2:09pm UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/12 "2021-06-21T14:09:33Z")
That bitbake command will only build the python3-memory-profiler and not a full image. You need to include that package in your image. The following links may help:
[https://wiki.yoctoproject.org/wiki/Building\_your\_own\_recipes\_from\_first\_principles#Testing\_the\_example\_on\_an\_emulated\_target](https://wiki.yoctoproject.org/wiki/Building_your_own_recipes_from_first_principles#Testing_the_example_on_an_emulated_target)
> **[Beginners Guide Yocto OpenEmbedded Recipe | wolfSSL Embedded SSL](https://www.wolfssl.com/docs/yocto-openembedded-recipe-guide/)**
>
> This tutorial comprises simple instructions for writing a recipe for the Yocto Project/OpenEmbedded and then installing it on a linux image.
Drew
---
### Author:  [@Yasmine](https://hub.mender.io/u/Yasmine)
#### Post date: [July 19, 2021, 10:11am UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/13 "2021-07-19T10:11:07Z")
Hello, m having big trouble finding the LIC\_FILES\_CHKSUM , SRC\_URI[md5sum], SRC\_URI[sha256sum], because m building the recipe for persist-queue package , i really dont know how to find them, pleaaase help
thank youu
---
### Author:  [@kemaangu77](https://hub.mender.io/u/kemaangu77)
#### Post date: [August 24, 2021, 7:06am UTC](https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135/14 "2021-08-24T07:06:48Z")
Hi
I want to back port python 3.9.5 (python3\_3.9.5.bb) from hardknott yocto 3.3 to my build environment morty (yocto 2.2). I don’t have any other option for upgrading my yocto version from morty (yocto 2.2) to latest. Please let me know, how big my porting effort?
What are key points I need to look for while porting?
Thanks
Manjunatha Srinivasan N