Directory Overlay

Description

The Directory Overlay Update Module installs a user defined file tree structure into a given destination directory in the target.

Before the deploy into the destination folder on the device, the Update Module will take a backup copy of the current contents, allowing restore of it using the rollback mechanism of the Mender client if something goes wrong. The Update Module will also delete the current installed content that was previously installed using the same module, this means that each deployment is self contained and there is no residues left on the system from the previous deployment.

Example use-cases:

  • Deploy root filesystem overlays

Specification

Module name dir-overlay
Supports rollback yes
Requires restart no
Artifact generation script yes
Full system updater no
Source code Update Module, Artifact Generator
Maintainer Community

Please be aware that Mender Update modules are meant to update parts of the system and if not configured properly they could potentially delete parts or the complete system. Always inspect the code carefully and only test modules on a systems that you can recover easily.

It is not recommended to install a Mender Artifact on your workstation

Prepare the device

This section describes how to setup your target device, i.e. the device to be updated. This will also be referred to as the device environment.

All commands outlined in this section should be run in the device environment.

Prerequisites

This update module has the following prerequisites for the device environment:

The dir-overlay module will store state information in /var/lib/mender, and that is:

  • Manifest files of current and previous installation
    • list of files
  • Backup of current installed files based on previous manifest, which is used in case of roll-back

Sample directory layout:

$ ls /var/lib/mender/dir-overlay/
backup.tar     manifest       manifest.prev

The dir-overlay directory is created by the module.

Install the Update Module

Download the latest version of this Update Module by running:

mkdir -p /usr/share/mender/modules/v3 && wget -P /usr/share/mender/modules/v3 https://raw.githubusercontent.com/mendersoftware/mender-update-modules/master/dir-overlay/module/dir-overlay && chmod +x /usr/share/mender/modules/v3/dir-overlay

Prepare the development environment on your workstation

This section describes how to set up your development environment on your workstation.

All commands outlined in this section should be run in the development environment.

Prerequisites

This Update Modules has the following prerequisites for the development environment:

Create Mender Artifacts

For convenience, an Artifact generator tool dir-overlay-artifact-gen is provided along the module. This tool will generate Mender Artifacts in the same format that the Update Module expects them.

To download dir-overlay-artifact-gen, run the following:

wget https://raw.githubusercontent.com/mendersoftware/mender-update-modules/master/dir-overlay/module-artifact-gen/dir-overlay-artifact-gen

Make it executable by running,

chmod +x dir-overlay-artifact-gen

Create example root filesystem content to deploy,

# The permissions must match DEST_DIR on the device, otherwise this module will
# overwrite the set permissions.
#
# You can check this by running the following command on the device (if DEST_DIR="/"):
#
#    stat -c %a /
#
sudo install -d -m 755 -g root -o root rootfs_overlay

# Create directory structure
sudo install -d -m 755 -g root -o root rootfs_overlay/etc
sudo install -d -m 755 -g root -o root rootfs_overlay/usr/bin
sudo install -d -m 755 -g root -o root rootfs_overlay/usr/share/app

# Create sample files
sudo touch rootfs_overlay/etc/app.conf
sudo touch rootfs_overlay/usr/bin/app
sudo touch rootfs_overlay/usr/share/app/data.txt

# You would typically chmod above files to have the appropriate mode in a production environment

The result should look like this:

$ tree rootfs_overlay/
rootfs_overlay/
β”œβ”€β”€ etc
β”‚   └── app.conf
└── usr
    β”œβ”€β”€ bin
    β”‚   └── app
    └── share
        └── app
            └── data.txt

5 directories, 3 files

Now generate your Artifact using the following command:

ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="my-device-type"
OUTPUT_PATH="my-update-1.0.mender"
DEST_DIR="/"
OVERLAY_TREE="rootfs_overlay"
./dir-overlay-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${OVERLAY_TREE}
  • ARTIFACT_NAME - The name of the Mender Artifact
  • DEVICE_TYPE - The compatible device type of this Mender Artifact
  • OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
  • DEST_DIR - The path on target device where content of OVERLAY_TREE will be installed.
  • OVERLAY_TREE - The path a folder containing the contents to be installed on top of the DEST_DIR on the device in the update.

You can either deploy this Artifact in managed mode with the Mender server (upload it under Releases in the server UI) or by using the Mender client only in Standalone deployments.

Artifact technical details

The Mender Artifact used by this Update Module has three payload files:

  • update.tar - a tarball containing all the files to be deployed
  • dest_dir - a regular file with the DEST_DIR in plain text
  • manifest - a regular file with a list of files installed in plain text

The Mender Artifact contents will look like:

Artifact my-update-1.0.mender generated successfully:
Mender artifact:
  Name: my-update-1.0
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[my-device-type]'
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   dir-overlay
    Provides: Nothing
    Depends: Nothing
    Metadata: Nothing
    Files:
      name:     update.tar
      size:     10240
      modified: 2019-04-01 12:01:41 +0000 UTC
      checksum: fabfd7f6739d35130dd05dbab42ff7fe8baa76fea22f134b254671bd52604807
    Files:
      name:     dest_dir
      size:     2
      modified: 2019-04-01 12:01:41 +0000 UTC
      checksum: f465c3739385890c221dff1a05e578c6cae0d0430e46996d319db7439f884336
    Files:
      name:     manifest
      size:     48
      modified: 2019-04-01 12:01:41 +0000 UTC
      checksum: fc582174946ec48e2782ed9357e9d4389cc0e2a3bfbc261b418d4fd5a326d472
1 Like

hmm, what is the difference between Directory and Directory Overlay modules?

Directory - Here you can choose one path and all the files inside the selected path will be updated.
Ex: /etc/conf where all files inside the conf directory will update.
Please note that, this is selecting only a single path.

Directory Overlay - Here you can select multiple directories and all files from multiple directories can be updated.
Ex: /etc/conf and /etc/bin where all files of these 2 directories can be updated.
Please note that, here the user can select and update multiple locations at once.

The advantage of each method depends on the usage of the update :slight_smile:

2 Likes

4 posts were split to a new topic: State-script support in dir-overlay Update Module

10 posts were split to a new topic: Issues with dir-overlay Update Module rollback

Hi guys,

I am getting this error on trying out this very sample directory overlay update. Can anyone help me? I am testing it on Raspberry Pi 3 module. Please do let me know if more information is required to debug the error.

2020-11-11 16:05:29 +0000 UTC info: Running Mender client version: 2.4.0
2020-11-11 16:05:36 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2020-11-11 16:05:36 +0000 UTC info: No public key was provided for authenticating the artifact
2020-11-11 16:05:36 +0000 UTC info: Update module output: + STATE=Download
2020-11-11 16:05:36 +0000 UTC info: Update module output: +
2020-11-11 16:05:36 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:36 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:36 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:36 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:36 +0000 UTC info: Update module output: + STATE=SupportsRollback
2020-11-11 16:05:36 +0000 UTC info: Update module output: +
2020-11-11 16:05:36 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:36 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:36 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:36 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:36 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:36 +0000 UTC info: Update module output: + echo Yes
2020-11-11 16:05:36 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:37 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2020-11-11 16:05:37 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2020-11-11 16:05:37 +0000 UTC info: Update module output: + STATE=ArtifactInstall
2020-11-11 16:05:37 +0000 UTC info: Update module output: +
2020-11-11 16:05:37 +0000 UTC info: Update module output: FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:37 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:37 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + [ -ne 0 ]
2020-11-11 16:05:37 +0000 UTC info: Update module output: /usr/share/mender/modules/v3/dir-overlay: 65: [: -ne: unexpected operator
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir=/home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + mkdir -p /home/pi/dir_overlay /var/lib/mender/dir-overlay-install
2020-11-11 16:05:37 +0000 UTC info: Update module output: + [ -e /var/lib/mender/dir-overlay-install/manifest ]
2020-11-11 16:05:37 +0000 UTC info: Update module output: + mkdir -p /var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + stat -c %a /home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + chmod 755 /var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + stat -c %U:%G /home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + chown root:root /var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -cpf - /home/pi/dir_overlay/etc/app.conf
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -xpf -
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Removing leading /' from member names 2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/etc/app.conf: Cannot stat: No such file or directory 2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors 2020-11-11 16:05:37 +0000 UTC info: Update module output: + + cdtar -cpf /var/lib/mender/modules/v3/payloads/0000/tree/tmp/ - 2020-11-11 16:05:37 +0000 UTC info: Update module output: /home/pi/dir_overlay/usr/bin/app 2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -xpf - 2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Removing leading /’ from member names
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/usr/bin/app: Cannot stat: No such file or directory
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -cpf - /home/pi/dir_overlay/usr/share/app/data.txt
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cd /var/lib/mender/modules/v3/payloads/0000/tree/tmp/
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -xpf -
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Removing leading `/’ from member names
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/usr/share/app/data.txt: Cannot stat: No such file or directory
2020-11-11 16:05:37 +0000 UTC info: Update module output: tar: Exiting with failure status due to previous errors
2020-11-11 16:05:37 +0000 UTC info: Update module output: + tar -cpf /var/lib/mender/dir-overlay-install/backup.tar -C /var/lib/mender/modules/v3/payloads/0000/tree/tmp .
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + rm /home/pi/dir_overlay/etc/app.conf
2020-11-11 16:05:37 +0000 UTC info: Update module output: rm: cannot remove β€˜/home/pi/dir_overlay/etc/app.conf’: No such file or directory
2020-11-11 16:05:37 +0000 UTC error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:37 +0000 UTC error: transient error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:37 +0000 UTC info: State transition: update-install [ArtifactInstall] -> rollback [ArtifactRollback]
2020-11-11 16:05:37 +0000 UTC info: Performing rollback
2020-11-11 16:05:37 +0000 UTC info: Update module output: + STATE=ArtifactRollback
2020-11-11 16:05:37 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:37 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:37 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + [ -ne 0 ]
2020-11-11 16:05:37 +0000 UTC info: Update module output: /usr/share/mender/modules/v3/dir-overlay: 106: [: -ne: unexpected operator
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cat /var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:37 +0000 UTC info: Update module output: + dest_dir=/home/pi/dir_overlay
2020-11-11 16:05:37 +0000 UTC info: Update module output: + test -f /var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:37 +0000 UTC info: Update module output: + cp /var/lib/mender/dir-overlay-install/manifest.prev /var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:37 +0000 UTC info: Update module output: cp: cannot stat β€˜/var/lib/mender/dir-overlay-install/manifest.prev’: No such file or directory
2020-11-11 16:05:37 +0000 UTC error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:37 +0000 UTC error: Rollback failed: Update module terminated abnormally: exit status 1
2020-11-11 16:05:38 +0000 UTC error: fatal error: Update module terminated abnormally: exit status 1
2020-11-11 16:05:38 +0000 UTC info: State transition: rollback [ArtifactRollback] -> update-error [ArtifactFailure]
2020-11-11 16:05:38 +0000 UTC info: Update module output: + STATE=ArtifactFailure
2020-11-11 16:05:38 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:38 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:38 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:38 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:38 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2020-11-11 16:05:38 +0000 UTC info: Update module output: + STATE=Cleanup
2020-11-11 16:05:38 +0000 UTC info: Update module output: + FILES=/var/lib/mender/modules/v3/payloads/0000/tree
2020-11-11 16:05:38 +0000 UTC info: Update module output: + backup_dir=/var/lib/mender/modules/v3/payloads/0000/tree/tmp
2020-11-11 16:05:38 +0000 UTC info: Update module output: + dest_dir_file=/var/lib/mender/modules/v3/payloads/0000/tree/files/dest_dir
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_from_deployment=/var/lib/mender/modules/v3/payloads/0000/tree/files/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + update_files_tar=/var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + state_dir=/var/lib/mender/dir-overlay-install
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file=/var/lib/mender/dir-overlay-install/manifest
2020-11-11 16:05:38 +0000 UTC info: Update module output: + manifest_file_prev=/var/lib/mender/dir-overlay-install/manifest.prev
2020-11-11 16:05:38 +0000 UTC info: Update module output: + prev_files_tar=/var/lib/mender/dir-overlay-install/backup.tar
2020-11-11 16:05:38 +0000 UTC info: Update module output: + exit 0
2020-11-11 16:05:38 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]

Thanks in advance.

Saman

Hi @SamanRatna

It is unclear to me, but the script is failing to create the backup of the previous deploy. Might it be that some files have been removed manually?

The first error is:

16:05:37 +0000 UTC info: Update module output: tar: /home/pi/dir_overlay/etc/app.conf: Cannot stat: No such file or directory

Which comes from the source code line:

            for file in $(cat $manifest_file); do
                tar -cpf - $dest_dir/$file | (cd $backup_dir/ && tar -xpf -)
            done

So, as I said, the module knows that there was a previous deployment with this module (fiel list at $manifest_file) and tries do to a backup of the files. However file /home/pi/dir_overlay/etc/app.conf does not exist any more.

Hi @lluiscampos,

Are you sure that you have set appropriate permissions? And also, can you share all the commands you had used to create the artifact?

Regards,
Saman

@SamanRatna I am not running any commands on my devices, sorry for the miss understanding. I only analyzed the output that you shared with us here.

Hi @lluiscampos,

I’m so sorry. I had actually solved the issue with file permissions and I thought you were trying to ask for solution. I didn’t realize you were trying to help me. :sweat_smile:

Thanks @lluiscampos. Good day.

Saman

Hi there!

Is it possible to add an script in the generated artifact that it’s executed when the deployment has finished?

Thanks in advance!
Mikel

Yes, we have the state script framework for that. Depending on your specific needs, you can install an ArtifactCommit_Enter script that will be called just before the final commit of the update. This allows you to return an error code to rollback. If you have no need to ever force a rollback from your script, you can install an ArtifactCommit_Leave script instead.

Drew