Directory Overlay

Description

The Directory Overlay Update Module installs a user defined file tree structure into a given destination directory in the target.

Before the deploy into the destination folder on the device, the Update Module will take a backup copy of the current contents, allowing restore of it using the rollback mechanism of the Mender client if something goes wrong. The Update Module will also delete the current installed content that was previously installed using the same module, this means that each deployment is self contained and there is no residues left on the device storage from the previous deployment.

Example use-cases:

  • Deploy root filesystem overlays

Specification

Module name dir-overlay
Supports rollback yes
Requires restart no
Artifact generation script yes
Full operating system updater no
Source code Update Module, Artifact Generator
Maintainer Community

Please be aware that Mender Update modules are meant to update parts of the operating system and if not configured properly they could potentially delete parts or the complete operating system. Always inspect the code carefully and only test modules on a devices that you can recover easily.

It is not recommended to install a Mender Artifact on your workstation

Prepare the device

This section describes how to setup your target device, i.e. the device to be updated. This will also be referred to as the device environment.

All commands outlined in this section should be run in the device environment.

Prerequisites

This update module has the following prerequisites for the device environment:

The dir-overlay module will store state information in /var/lib/mender, and that is:

  • Manifest files of current and previous installation
    • list of files
  • Backup of current installed files based on previous manifest, which is used in case of roll-back

Sample directory layout:

$ ls /var/lib/mender/dir-overlay/
backup.tar     manifest       manifest.prev

The dir-overlay directory is created by the module.

Install the Update Module

Download the latest version of this Update Module by running:

mkdir -p /usr/share/mender/modules/v3 && wget -P /usr/share/mender/modules/v3 https://raw.githubusercontent.com/mendersoftware/mender-update-modules/master/dir-overlay/module/dir-overlay && chmod +x /usr/share/mender/modules/v3/dir-overlay

Prepare the development environment on your workstation

This section describes how to set up your development environment on your workstation.

All commands outlined in this section should be run in the development environment.

Prerequisites

This Update Modules has the following prerequisites for the development environment:

Create Mender Artifacts

For convenience, an Artifact generator tool dir-overlay-artifact-gen is provided along the module. This tool will generate Mender Artifacts in the same format that the Update Module expects them.

To download dir-overlay-artifact-gen, run the following:

wget https://raw.githubusercontent.com/mendersoftware/mender-update-modules/master/dir-overlay/module-artifact-gen/dir-overlay-artifact-gen

Make it executable by running,

chmod +x dir-overlay-artifact-gen

Create example root filesystem content to deploy,

# The permissions must match DEST_DIR on the device, otherwise this module will
# overwrite the set permissions.
#
# You can check this by running the following command on the device (if DEST_DIR="/"):
#
#    stat -c %a /
#
sudo install -d -m 755 -g root -o root rootfs_overlay

# Create directory structure
sudo install -d -m 755 -g root -o root rootfs_overlay/etc
sudo install -d -m 755 -g root -o root rootfs_overlay/usr/bin
sudo install -d -m 755 -g root -o root rootfs_overlay/usr/share/app

# Create sample files
sudo touch rootfs_overlay/etc/app.conf
sudo touch rootfs_overlay/usr/bin/app
sudo touch rootfs_overlay/usr/share/app/data.txt

# You would typically chmod above files to have the appropriate mode in a production environment

The result should look like this:

$ tree rootfs_overlay/
rootfs_overlay/
├── etc
│   └── app.conf
└── usr
    ├── bin
    │   └── app
    └── share
        └── app
            └── data.txt

5 directories, 3 files

Now generate your Artifact using the following command:

ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="my-device-type"
OUTPUT_PATH="my-update-1.0.mender"
DEST_DIR="/"
OVERLAY_TREE="rootfs_overlay"
./dir-overlay-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${OVERLAY_TREE}
  • ARTIFACT_NAME - The name of the Mender Artifact
  • DEVICE_TYPE - The compatible device type of this Mender Artifact
  • OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
  • DEST_DIR - The path on target device where content of OVERLAY_TREE will be installed.
  • OVERLAY_TREE - The path a folder containing the contents to be installed on top of the DEST_DIR on the device in the update.

You can either deploy this Artifact in managed mode with the Mender server (upload it under Releases in the server UI) or by using the Mender client only in Standalone deployments.

Artifact technical details

The Mender Artifact used by this Update Module has three payload files:

  • update.tar - a tarball containing all the files to be deployed
  • dest_dir - a regular file with the DEST_DIR in plain text
  • manifest - a regular file with a list of files installed in plain text

The Mender Artifact contents will look like:

Artifact my-update-1.0.mender generated successfully:
Mender artifact:
  Name: my-update-1.0
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[my-device-type]'
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   dir-overlay
    Provides: Nothing
    Depends: Nothing
    Metadata: Nothing
    Files:
      name:     update.tar
      size:     10240
      modified: 2019-04-01 12:01:41 +0000 UTC
      checksum: fabfd7f6739d35130dd05dbab42ff7fe8baa76fea22f134b254671bd52604807
    Files:
      name:     dest_dir
      size:     2
      modified: 2019-04-01 12:01:41 +0000 UTC
      checksum: f465c3739385890c221dff1a05e578c6cae0d0430e46996d319db7439f884336
    Files:
      name:     manifest
      size:     48
      modified: 2019-04-01 12:01:41 +0000 UTC
      checksum: fc582174946ec48e2782ed9357e9d4389cc0e2a3bfbc261b418d4fd5a326d472
1 Like

hmm, what is the difference between Directory and Directory Overlay modules?

Directory - Here you can choose one path and all the files inside the selected path will be updated.
Ex: /etc/conf where all files inside the conf directory will update.
Please note that, this is selecting only a single path.

Directory Overlay - Here you can select multiple directories and all files from multiple directories can be updated.
Ex: /etc/conf and /etc/bin where all files of these 2 directories can be updated.
Please note that, here the user can select and update multiple locations at once.

The advantage of each method depends on the usage of the update :slight_smile:

3 Likes

4 posts were split to a new topic: State-script support in dir-overlay Update Module

10 posts were split to a new topic: Issues with dir-overlay Update Module rollback

Hi guys,

I am getting this error on trying out this very sample directory overlay update. Can anyone help me? I am testing it on Raspberry Pi 3 module. Please do let me know if more information is required to debug the error.

2020-11-11 16:05:29 +0000 UTC info: Running Mender client version: 2.4.0
2020-11-11 16:05:36 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2020-11-11 16:05:36 +0000 UTC info: No public key was provided for authenticating the artifact
2020-11-11 16:05:36 +0000 UTC info: Update module output: + STATE=Download
2020-11-11 16:05:36 +0000 UTC info: Update module output: +
2020-11-11 16:05:36 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:36 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:36 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:36 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:36 +0000 UTC info: Update module output: + STATE=SupportsRollback
2020-11-11 16:05:36 +0000 UTC info: Update module output: +
2020-11-11 16:05:36 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:36 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:36 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:36 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + echo Yes
2020-11-11 16:05:36 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:37 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2020-11-11 16:05:37 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2020-11-11 16:05:37 +0000 UTC info: Update module output: + STATE=ArtifactInstall
2020-11-11 16:05:37 +0000 UTC info: Update module output: +
2020-11-11 16:05:37 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:37 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:37 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + [ -ne 0 ]
2020-11-11 16:05:37 +0000 UTC info: Update module output: /usr/share/mender/modules/v3/dir-overlay: 65: [: -ne: unexpected operator
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir=/home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + mkdir -p /home/pi/dir_overlay /var/lib/mender/dir-overlay-install
2020-11-11 16:05:37 +0000 UTC info: Update module output: + [ -e /var/lib/mender/dir-overlay-install/manifest ]
2020-11-11 16:05:37 +0000 UTC info: Update module output: + mkdir -p /var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + stat -c %a /home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + chmod 755 /var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + stat -c %U:%G /home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + chown root:root /var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -cpf - /home/pi/dir_overlay/etc/app.conf
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -xpf -
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Removing leading /' from member names 2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/etc/app.conf: Cannot stat: No such file or directory 2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors 2020-11-11 16:05:37 +0000 UTC info: Update module output: + + cdtar -cpf /var/lib/mender/modules/v3/payloads/0000/tree/tmp/ - 2020-11-11 16:05:37 +0000 UTC info: Update module output: /home/pi/dir_overlay/usr/bin/app 2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -xpf - 2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Removing leading /’ from member names
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/usr/bin/app: Cannot stat: No such file or directory
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -cpf - /home/pi/dir_overlay/usr/share/app/data.txt
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -xpf -
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Removing leading `/’ from member names
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/usr/share/app/data.txt: Cannot stat: No such file or directory
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -cpf /var/lib/mender/dir-overlay-install/backup.tar -C /var/lib/mender/modules/v3/payloads/0000/tree/tmp .
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + rm /home/pi/dir_overlay/etc/app.conf
2020-11-11 16:05:37 +0000 UTC info: Update module output: rm: cannot remove ‘/home/pi/dir_overlay/etc/app.conf’: No such file or directory
2020-11-11 16:05:37 +0000 UTC error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:37 +0000 UTC error: transient error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:37 +0000 UTC info: State transition: update-install [ArtifactInstall] -> rollback [ArtifactRollback]
2020-11-11 16:05:37 +0000 UTC info: Performing rollback
2020-11-11 16:05:37 +0000 UTC info: Update module output: + STATE=ArtifactRollback
2020-11-11 16:05:37 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:37 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:37 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + [ -ne 0 ]
2020-11-11 16:05:37 +0000 UTC info: Update module output: /usr/share/mender/modules/v3/dir-overlay: 106: [: -ne: unexpected operator
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir=/home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + test -f /var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cp /var/lib/mender/dir-overlay-install/manifest.prev /var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: cp: cannot stat ‘/var/lib/mender/dir-overlay-install/manifest.prev’: No such file or directory
2020-11-11 16:05:37 +0000 UTC error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:37 +0000 UTC error: Rollback failed: Update module terminated abnormally: exit status 1
2020-11-11 16:05:38 +0000 UTC error: fatal error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:38 +0000 UTC info: State transition: rollback [ArtifactRollback] -> update-error [ArtifactFailure]
2020-11-11 16:05:38 +0000 UTC info: Update module output: + STATE=ArtifactFailure
2020-11-11 16:05:38 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:38 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:38 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:38 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:38 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2020-11-11 16:05:38 +0000 UTC info: Update module output: + STATE=Cleanup
2020-11-11 16:05:38 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:38 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:38 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:38 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:38 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]

Thanks in advance.

Saman

Hi @SamanRatna

It is unclear to me, but the script is failing to create the backup of the previous deploy. Might it be that some files have been removed manually?

The first error is:

16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/etc/app.conf: Cannot stat: No such file or directory

Which comes from the source code line:

            for file in $(cat $manifest_file); do
                tar -cpf - $dest_dir/$file | (cd $backup_dir/ && tar -xpf -)
            done

So, as I said, the module knows that there was a previous deployment with this module (fiel list at $manifest_file) and tries do to a backup of the files. However file /home/pi/dir_overlay/etc/app.conf does not exist any more.

Hi @lluiscampos,

Are you sure that you have set appropriate permissions? And also, can you share all the commands you had used to create the artifact?

Regards,
Saman

@SamanRatna I am not running any commands on my devices, sorry for the miss understanding. I only analyzed the output that you shared with us here.

Hi @lluiscampos,

I’m so sorry. I had actually solved the issue with file permissions and I thought you were trying to ask for solution. I didn’t realize you were trying to help me. :sweat_smile:

Thanks @lluiscampos. Good day.

Saman

Hi there!

Is it possible to add an script in the generated artifact that it’s executed when the deployment has finished?

Thanks in advance!
Mikel

Yes, we have the state script framework for that. Depending on your specific needs, you can install an ArtifactCommit_Enter script that will be called just before the final commit of the update. This allows you to return an error code to rollback. If you have no need to ever force a rollback from your script, you can install an ArtifactCommit_Leave script instead.

Drew

HI, I have been using directory overlay module for some time now without any issues. Recently I updated my mender client version to 2.6.1 I keep getting this error.

. The update module is trying to update in the wrong directory. for example my install location is /home/pi/AtlasGO/src/blabla but it keeps trying to update in /home/pi/AtlasGo/ and as a result unable to find the file. I am not sure whats wrong here. Please provide any insight if possible. I am using hosted mender server.

Mender Client version 2.6.0

I am attaching the log file

2021-05-31 02:03:53 +0000 UTC info: Update module output: tar -cpf - /home/pi/AtlasGO_bin/moc_otaclient.cpp
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: /home/pi/AtlasGO_bin/moc_otaclient.cpp: Cannot stat: No such file or directory
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2021-05-31 02:03:53 +0000 UTC info: Update module output: + + cdtar -cpf - /home/pi/AtlasGO_bin/moc_logger.cpp
2021-05-31 02:03:53 +0000 UTC info: Update module output:  /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: /home/pi/AtlasGO_bin/moc_logger.cpp: Cannot stat: No such file or directory
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2021-05-31 02:03:53 +0000 UTC info: Update module output: + + cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tartar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output:  -cpf - /home/pi/AtlasGO_bin/moc_otaserver.cpp
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: /home/pi/AtlasGO_bin/moc_otaserver.cpp: Cannot stat: No such file or directory
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -cpf - /home/pi/AtlasGO_bin/.qmake.stash
2021-05-31 02:03:53 +0000 UTC info: Update module output: + cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -cpf -+  /home/pi/AtlasGO_bin/moc_otaclient.ocd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2021-05-31 02:03:53 +0000 UTC info: Update module output: 
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: /home/pi/AtlasGO_bin/moc_otaclient.o: Cannot stat: No such file or directory
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2021-05-31 02:03:53 +0000 UTC info: Update module output: + + tar -cpf - /home/pi/AtlasGO_bin/mainprocess.o
2021-05-31 02:03:53 +0000 UTC info: Update module output: cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: /home/pi/AtlasGO_bin/mainprocess.o: Cannot stat: No such file or directory
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2021-05-31 02:03:53 +0000 UTC info: Update module output: + cd+  /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar+ tar -xpf -
2021-05-31 02:03:53 +0000 UTC info: Update module output:  -cpf - /home/pi/AtlasGO_bin/otaserver.o
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Removing leading `/' from member names
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: /home/pi/AtlasGO_bin/otaserver.o: Cannot stat: No such file or directory
2021-05-31 02:03:53 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2021-05-31 02:03:53 +0000 UTC info: Update module output: + tar -cpf /var/lib/mender/dir-overlay-install/backup.tar -C /var/lib/mender/modules/v3/payloads/0000/tree/tmp .
2021-05-31 02:03:53 +0000 UTC info: Update module output: + cat /var/lib/mender/dir-overlay-install/manifest
2021-05-31 02:03:53 +0000 UTC info: Update module output: + rm /home/pi/AtlasGO_bin/moc_captureworker.o
2021-05-31 02:03:53 +0000 UTC info: Update module output: rm: cannot remove '/home/pi/AtlasGO_bin/moc_captureworker.o': No such file or directory
2021-05-31 02:03:53 +0000 UTC error: Update module terminated abnormally: exit status 1
2021-05-31 02:03:53 +0000 UTC error: transient error: Update module terminated abnormally: exit status 1
2021-05-31 02:03:53 +0000 UTC info: State transition: update-install [ArtifactInstall] -> rollback [ArtifactRollback]
2021-05-31 02:03:53 +0000 UTC info: Performing rollback
2021-05-31 02:03:53 +0000 UTC info: Update module output: + 
2021-05-31 02:03:53 +0000 UTC info: Update module output: STATE=ArtifactRollback
2021-05-31 02:03:53 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2021-05-31 02:03:53 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2021-05-31 02:03:53 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2021-05-31 02:03:53 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2021-05-31 02:03:53 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2021-05-31 02:03:53 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2021-05-31 02:03:53 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2021-05-31 02:03:53 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2021-05-31 02:03:54 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2021-05-31 02:03:54 +0000 UTC info: Update module output: + [ -ne 0 ]
2021-05-31 02:03:54 +0000 UTC info: Update module output: /usr/share/mender/modules/v3/dir-overlay: 106: [: -ne: unexpected operator
2021-05-31 02:03:54 +0000 UTC info: Update module output: + cat /var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2021-05-31 02:03:54 +0000 UTC info: Update module output: + dest_dir=/home/pi/AtlasGO_bin
2021-05-31 02:03:54 +0000 UTC info: Update module output: + test -f /var/lib/mender/dir-overlay-install/backup.tar
2021-05-31 02:03:54 +0000 UTC info: Update module output: + cp /var/lib/mender/dir-overlay-install/manifest.prev /var/lib/mender/dir-overlay-install/manifest
2021-05-31 02:03:54 +0000 UTC info: Update module output: + tar -xpf /var/lib/mender/dir-overlay-install/backup.tar -C /home/pi/AtlasGO_bin
2021-05-31 02:03:54 +0000 UTC info: Update module output: + exit 0
2021-05-31 02:03:54 +0000 UTC info: State transition: rollback [ArtifactRollback] -> update-error [ArtifactFailure]
2021-05-31 02:03:54 +0000 UTC info: Update module output: + STATE=ArtifactFailure
2021-05-31 02:03:54 +0000 UTC info: Update module output: + 
2021-05-31 02:03:54 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2021-05-31 02:03:54 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2021-05-31 02:03:54 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2021-05-31 02:03:54 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2021-05-31 02:03:54 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2021-05-31 02:03:54 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2021-05-31 02:03:54 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2021-05-31 02:03:54 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2021-05-31 02:03:54 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2021-05-31 02:03:54 +0000 UTC info: Update module output: + exit 0
2021-05-31 02:03:54 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2021-05-31 02:03:54 +0000 UTC info: Update module output: + STATE=Cleanup
2021-05-31 02:03:54 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2021-05-31 02:03:54 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2021-05-31 02:03:54 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2021-05-31 02:03:54 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2021-05-31 02:03:54 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2021-05-31 02:03:54 +0000 UTC info: Update module output: + 
2021-05-31 02:03:54 +0000 UTC info: Update module output: state_dir=/var/lib/mender/dir-overlay-install
2021-05-31 02:03:54 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2021-05-31 02:03:54 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2021-05-31 02:03:54 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2021-05-31 02:03:54 +0000 UTC info: Update module output: + exit 0
2021-05-31 02:03:54 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]

Hello @zainmehdi ,

There is no mender client version 2.6.1, so I am unsure about what have you updated.

In any case, in recent updates there have not been changes in this area that might affect the Update Modules. But maybe we have introduced a regression? Can you share the command you are using to generate the Mender Artifact?

Hi @lluiscampos Correction. The client version is 2.6.0. I am using following script to generate the artifact

#!/bin/sh
ARTIFACT_NAME="PathF-May-27-N"
DEVICE_TYPE="NanoPi-NEO4"
OUTPUT_PATH="PathF-May-27-N.mender"
DEST_DIR="/home/pi/AtlasGO_bin"
OVERLAY_TREE="AtlasGO_bin"
SCRIPT_PATH="./ArtifactCommit_Leave_01_User.sh"

./dir-overlay-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${OVERLAY_TREE} -s ${SCRIPT_PATH}

Here is the the main script

#!/bin/bash
set -e

show_help() {
  cat << EOF

Simple tool to generate Mender Artifact suitable for dir-overlay Update Module

Usage: $0 [options] file-tree [-- [options-for-mender-artifact] ]

    Options: [ -n|artifact-name -t|--device-type -d|--dest-dir -o|--output_path -h|--help ]

        --artifact-name     - Artifact name
        --device-type       - Target device type identification
        --dest-dir          - Target destination directory where to deploy the update (Defaults to "/")
        --output-path       - Path to output file. Default: file-install-artifact.mender
        --help              - Show help and exit
        file-tree           - Directory tree to bundle in the update

Anything after a '--' gets passed directly to the mender-artifact tool.

Common use-case for the dir-overlay is to install files overlayed
on the root file-system.

Example tree to deploy:

    $ tree
    .
    ├── etc
    │   └── app.conf
    └── usr
        ├── bin
        │   └── app
        └── share
            └── app
                └── data.txt

EOF
}

show_help_and_exit_error() {
  show_help
  exit 1
}

check_dependency() {
  hash "$1" || \
        (echo "Fatal error: $1 not found in PATH." && exit 1)
}

check_dependency mender-artifact

device_type=""
artifact_name=""
dest_dir="/"
dest_dir_file="dest_dir"
output_path="file-install-artifact.mender"
update_files_tar="update.tar"
directories_file="directories"
manifest_file="manifest"
file_tree=""
passthrough=0
passthrough_args=""
script_path1=""
script_path2=""

while (( "$#" )); do
  if test $passthrough -eq 1
  then
    passthrough_args="$passthrough_args $1"
    shift
    continue
  fi
  case "$1" in
    --device-type | -t)
      if [ -z "$2" ]; then
        show_help_and_exit_error
      fi
      device_type=$2
      shift 2
      ;;
    --artifact-name | -n)
      if [ -z "$2" ]; then
        show_help_and_exit_error
      fi
      artifact_name=$2
      shift 2
      ;;
    --dest-dir | -d)
      if [ -z "$2" ]; then
        show_help_and_exit_error
      fi
      dest_dir=$2
      shift 2
      ;;
    --output-path | -o)
      if [ -z "$2" ]; then
        show_help_and_exit_error
      fi
      output_path=$2
      shift 2
      ;;
    --script_path | -s)
      if [ -z "$2" ]; then
        show_help_and_exit_error
      fi
      script_path=$2
      shift 2
      ;;
    --script_path2 | -s)
      if [ -z "$2" ]; then
        show_help_and_exit_error
      fi
      script_path2=$2
      shift 2
      ;;
    -h | --help)
      show_help
      exit 0
      ;;
    --)
      passthrough=1
      shift
      ;;
    -*)
      echo "Error: unsupported option $1"
      show_help_and_exit_error
      ;;
    *)
      if [ -n "$file_tree" ]; then
        echo "File tree already specified. Unrecognized argument \"$1\""
        show_help_and_exit_error
      fi
      file_tree="$1"
      shift
      ;;
  esac
done

if [ -z "${artifact_name}" ]; then
  echo "Artifact name not specified. Aborting."
  show_help_and_exit_error
fi

if [ -z "${device_type}" ]; then
  echo "Device type not specified. Aborting."
  show_help_and_exit_error
fi

if [ -z "${file_tree}" ]; then
  echo "File tree not specified. Aborting."
  show_help_and_exit_error
fi

# Check dest-dir is an absolute path
case $dest_dir in
  /*)
    ;;
  *)
    echo "Destination dir must be an absolute path. Aborting"
    exit 1
  ;;
esac

if [ -d "${file_tree}" ]; then
  tar --same-owner -cpf ${update_files_tar} -C "${file_tree}" .
else
  echo "Error: \"${file_tree}\" is not a directory or does not exit. Aborting."
  exit 1
fi

find $file_tree -type f | sed "s|^$file_tree/||" > $manifest_file

# Create dest_dir file in plain text
echo "$dest_dir" > $dest_dir_file

mender-artifact write module-image \
  -T dir-overlay \
  -t $device_type \
  -o $output_path \
  -n $artifact_name \
  -f $update_files_tar \
  -f $dest_dir_file \
  -f $manifest_file \
  -s $script_path \
#  -s $script_path2 \
  $passthrough_args

rm $update_files_tar
rm $dest_dir_file
rm $manifest_file

echo "Artifact $output_path generated successfully:"
mender-artifact read $output_path

exit 0

Just to clarify the same artifact works with older client version and gives error with the latest version.

Thanks alot

Hi @zainmehdi

I am sorry I cannot help. I tried with my device but cannot reproduce your issue.

It seems related to having a manifest not really matching the actual files in the directory. This can happen if you have modified the files manually without using the Update Module. But this cannot be related to the Mender version, of course, so it might not be the problem.

I also had a look at your modified generator script. While not recommended (best always to keep it updated with fixes from upstream) and not necessary (you could have achieved the same using upstream script and adding -- -s ${SCRIPT_PATH}) I don’t see anything wrong with your code…

I noticed is that your generator script is based on an old upstream version, which made me think that maybe your Update Module code is not up to date neither? Again, this is not related to the Mender client version…

Hi @lluiscampos Thanks for your reply. You mentioned the scripts are not in sync with upstream. I found two links for the artifiact generation

I am using the second link. You meant that its not the latest one and I should use the first one ? I am not sure if artifact script is causing any issues but I will try that.

Hi @zainmehdi,

I am sorry I have confused you.

You downloaded at some point the generator from that “second link”, yes, but then made some modifications (adding support for state scripts). I realized this when comparing the code you posted and the upstream one (meaning: the current version in GitHub) because these differ a bit in other parts (not only the state scripts code). Hence, I thought that maybe you also had an outdated module (not only the generator) which could potentially have some bug.

I also tried to point out that the modifications you made (the code to add state scripts) is not really necessary as you can just pass these parameters with -- -s ... with the original generator. This way you don’t need to maintain your own copy that might diverge from the original (in other words, potentially missing some other fixes).

Now, note that this does not seem related to your issue. It was just a side comment.

1 Like