How to add multiple containers in mender layer

Hi Team,

As per the article, How to update a Docker container with Mender using Raspberry Pi | Mender the docker is used to update modules. In this scenario, whether only one container is used or multiple containers are used? If so, how it can be done. Please provide procedure if any.

How to perform this updation with multiple containers using Yocto (iMx6 setup)? Please provide procedure if any.

Thanking you
Best regards,
Aysha

Hello @Aysha,

The tutorial for using the Docker update module is here. You can use it for one or more container images to be managed.

Greetz,
Josef

Hi Josef,
Thanks for your response.
The link shared is of Ubuntu, can I know how to perform in Yocto? What changes to be made? In meta-mender layer, which file to be used to add custom container information?

Regards,
Aysha

Hello @Aysha,

EDIT: correction.

Unless you have modified the metadata for the Mender Client in your layers, the docker Update Module is installed by default: https://github.com/mendersoftware/meta-mender/blob/c0f539ffedba6070963abe64c67c4c5d8b1ec872/meta-mender-core/recipes-mender/mender-client/mender-client.inc#L48

Greetz,
Josef

Dear Team,

As I was trying to update the container image in i.MX device using mender. I had created artifact file using below command.
./mender-artifact write rootfs-image -t imx -n container -f container.tar.gz -o artifact.mender

But while deployment process, I am unable to deploy with below failure messages.

2023-08-04 09:48:06 +0000 UTC info: Running Mender client version: 3.5.0
2023-08-04 09:48:08 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2023-08-04 09:48:08 +0000 UTC info: No public key was provided for authenticating the artifact
2023-08-04 09:48:11 +0000 UTC info: Opening device “/dev/mmcblk1p2” for writing
2023-08-04 09:48:11 +0000 UTC info: Native sector size of block device /dev/mmcblk1p2 is 512 bytes. Mender will write in chunks of 1048576 bytes
2023-08-04 09:51:18 +0000 UTC info: All bytes were successfully written to the new partition
2023-08-04 09:51:18 +0000 UTC info: The optimized block-device writer wrote a total of 308 frames, where 0 frames did need to be rewritten (i.e., skipped)
2023-08-04 09:51:18 +0000 UTC info: Wrote 322796469/322796469 bytes to the inactive partition
2023-08-04 09:51:19 +0000 UTC info: State transition: update-store [Download_Enter] → update-after-store [Download_Leave]
2023-08-04 09:51:19 +0000 UTC info: State transition: update-after-store [Download_Leave] → mender-update-control-refresh-maps [none]
2023-08-04 09:51:19 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] → mender-update-control [none]
2023-08-04 09:51:19 +0000 UTC info: State transition: mender-update-control [none] → update-install [ArtifactInstall]
2023-08-04 09:51:19 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2023-08-04 09:51:20 +0000 UTC info: State transition: update-install [ArtifactInstall] → mender-update-control-refresh-maps [none]
2023-08-04 09:51:20 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] → mender-update-control [none]
2023-08-04 09:51:20 +0000 UTC info: State transition: mender-update-control [none] → reboot [ArtifactReboot_Enter]
2023-08-04 09:51:21 +0000 UTC info: Rebooting device(s)
2023-08-04 09:51:21 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk1p1
2023-08-04 09:51:21 +0000 UTC error: error forwarding from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF
2023-08-04 09:51:22 +0000 UTC warning: error while sending close message: write tcp 127.0.0.1:44645->127.0.0.1:52362: use of closed network connection
2023-08-04 09:51:23 +0000 UTC info: Daemon terminated with SIGTERM
2023-08-04 09:54:22 +0000 UTC info: Running Mender client version: 3.5.0
2023-08-04 09:54:22 +0000 UTC info: State transition: init [none] → after-reboot [ArtifactReboot_Leave]
2023-08-04 09:54:22 +0000 UTC error: transient error: Reboot to the new update failed. Expected “upgrade_available” flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back
2023-08-04 09:54:22 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] → rollback [ArtifactRollback]
2023-08-04 09:54:22 +0000 UTC info: Performing rollback
2023-08-04 09:54:22 +0000 UTC info: No update available, so no rollback needed.
2023-08-04 09:54:22 +0000 UTC info: State transition: rollback [ArtifactRollback] → rollback-reboot [ArtifactRollbackReboot_Enter]
2023-08-04 09:54:22 +0000 UTC info: Rebooting device(s) after rollback
2023-08-04 09:54:22 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk1p1
2023-08-04 09:54:24 +0000 UTC info: Daemon terminated with SIGTERM
2023-08-04 09:57:19 +0000 UTC info: Running Mender client version: 3.5.0
2023-08-04 09:57:19 +0000 UTC info: Mender shut down in state: rollback-reboot
2023-08-04 09:57:19 +0000 UTC info: State transition: init [none] → verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-08-04 09:57:19 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] → after-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-08-04 09:57:19 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] → update-error [ArtifactFailure]
2023-08-04 09:57:19 +0000 UTC info: State transition: update-error [ArtifactFailure] → cleanup [Error]
2023-08-04 09:57:19 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]
2023-08-04 09:57:19 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-08-04 09:57:19 +0000 UTC info: Output (stderr) from command “/usr/share/mender/identity/mender-device-identity”: using interface /usr/share/
2023-08-04 09:57:21 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2023-08-04 09:57:21 +0000 UTC info: Local proxy started
2023-08-04 09:57:21 +0000 UTC info: Reauthorization successful

Please let me know what changes to be made to deploy container successfully to i.Mx device.

Best regards,
Ayshathul Thuhara

Hi @Aysha,

without exactly knowing what you put into the Artifact, the only guess I can take is based on this line in the log

2023-08-04 09:54:22 +0000 UTC error: transient error: Reboot to the new update failed. Expected “upgrade_available” flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back

So it seems that whatever is in that container.tar.gz that you tried to install is not properly handled. Even more guessing: it is not a full root filesystem, as you are talking about containers.

mender-artifact write rootfs-image

is meant for full system deployments. To create an Artifact for the docker Update Module, use the matching generator script, as mentioned in the already linked tutorial.

Greetz,
Josef

Hi @TheYoctoJester ,
Thanks for your response.
As I wanted to update a custom image using docker, the custom-image (docker-container.tar) contains a custom application.
I used below command to generate it,

$ mender-artifact write rootfs-image -t imx -n release-2.5 --software-version rootfs-v1 -f docker-container.tar -o artifact.mender
Writing Artifact…
Version ✓
Manifest ✓
Manifest signature ✓
Header ✓
Payload
… - 100 %

I had even tried to generate artifact.mender using docker-artifact-gen, which resulted in below error.

$docker-artifact-gen -n container-update-1.0 -t imx -o /home/root/ docker-container.tar
Using default tag: latest
Error response from daemon: pull access denied for docker-container.tar, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

Please let me know how to generate artifact.mender for a docker custom image. Since I am new to mender.

Best regards,
Aysha

Hi @Aysha ,

I think you are randomly mixing up things, and therefore they do not work. By custom image, we understand a full linux root filesystem including your modifications. So, the command style mender-artifact write rootfs-image only applies if you have this. The filename docker-container.tar sounds very different. So, it does not apply here!

The docker-artifact-gen script is suitable for packing docker containers into a Mender artifact. Please use it, and please, read the tutorial. It expects the name (and optionally tag) of the container, not an archive. The corresponding section explains the required arguments in detail.

Greetz,
Josef

Hi @TheYoctoJester,

Sorry for the confusion, right now I am in state to update a single docker container image. For that I was using below command, which resulted in update failure.

mender-artifact write rootfs-image -t imx -n release-2.5 --software-version rootfs-v1 -f docker-container.tar -o artifact.mender

By considering your suggestions, I will try to use docker-artifact-gen. I have seen in docker-artifact-gen uses mender-artifact to generate the .mender file,

mender-artifact write module-image -t imx6 -o docker-2.mender -T docker -n docker-2.0 -f docker-container.tar

Here docker-container.tar is my custom container image, which should get update during updation.
After performing this, I am able to update the image over the target device with few static changes in /usr/share/mender/modules/v3/docker.

ArtifactInstall)
    cp "$FILES"/files/* /home/root/
    docker load -i /home/root/docker-container.tar
    docker stop hello-world || true
    docker rm hello-world || true
    docker run -d --name hello-world hello-world-image:latest

    ;;

But I found there are few containers were not listed in /var/lib/mender/modules/v3/payloads/0000/tree/tmp/prev_containers.list file, even though they were listed in “docker container list -a”.

:# ls /var/lib/mender/modules/v3/payloads/0000/
:#

Can I know how this prev_containers.list file is created and is there any changes to be made to get containers listed in prev_container.list.

Thanking you,
Regards,
Aysha

Hi Team,

As I had tried to generate mender.artifact using docker-artifact-gen. I had observed that the docker images should be available in docker repo. And in my case I wanted to have the docker container in tar format, which contains compressed docker image.

I had even tried one of the image to upload to target device from mender. But the docker update was not shown success even mender shown as success. While checking on this, I found /var/lib/mender/modules/v3/payloads/0000/tree/tmp/prev_containers.list empty and not available.
This file wasa read by docker script i.e., /usr/share/mender/modules/v3/docker.

Can I know which script creates this file “/var/lib/mender/modules/v3/payloads/0000/tree/tmp/prev_containers.list”, and when this file is created and updated.

Hoping for your reply
Regards,
Aysha

Hi @Aysha,

Okay, so you are effectively asking for “how can I update multiple containers on a device, and have them transported in the Artifact”.
This is not a use case that we have a ready to run solution, you need to create a custom Update Module. Please find the documentation here, and inspiration in the Update Module repository.

The snippet that you listed is part of the File API that Update Modules can use, as documented here

Greetz,
Josef