States script on standalone mode

Dewey · April 28, 2020, 6:50pm

I’m working on states scripts that will perform sanity checks on the updated partitions such as this one:

ArtifactCommit_Enter_00.sh

#!/bin/bash
log() {
    echo "mender:$*" >&2
}

log "$(cat /etc/mender/artifact_info): $(basename "$0") was called!"
CONTAINER_NAME=$(echo "ws")
commit=true
SERVER_VERSION=$(docker version -f "{{.Server.Version}}")
SERVER_VERSION_MAJOR=$(echo "$SERVER_VERSION"| cut -d'.' -f 1)
SERVER_VERSION_MINOR=$(echo "$SERVER_VERSION"| cut -d'.' -f 2)
SERVER_VERSION_BUILD=$(echo "$SERVER_VERSION"| cut -d'.' -f 3)
if [ "${SERVER_VERSION_MAJOR}" -ge 19 ] && \
   [ "${SERVER_VERSION_MINOR}" -ge 3 ]  && \
   [ "${SERVER_VERSION_BUILD}" -ge 8 ]; then
    echo "Docker version >= 19.03.8, update ok"
else
    echo "Docker version less than 19.03.8, update failed"
    commit=false
fi

if [ $(docker inspect -f '{{.State.Running}}' ${CONTAINER_NAME}) = "true" ]; then 
    echo "Web server is running" 
else 
    echo "Warning web server is not running!"
    commit=false 
fi

if [ "$commit" = true ] ; then
    echo "Update is successful - commit"
    exit 0
else
    echo "Update has failed - rollback"
    exit 1
fi

I added this state script to an artifact using mender-artifact and the image in ext4 type.
But I have a few questions:

When is the script runned exactly ? ArtifactCommit should be before commiting the artifact
How can I ensure that the script is ran?
Since in managed mode systemd in not used, how will the client know that it has to run this script?
With what rights will the script be run ?

kacf · April 29, 2020, 6:32am

After the reboot, while the new system is running, but before committing the artifact. You can see the state diagram here.

If it is included in the Artifact, then it is guaranteed to run (assuming there are no other errors causing a rollback before that). You can check this with mender-artifact read artifact-file.mender. Of course if you want to test it, you can write something to the data partition and check afterwards that it was done.

This is part of how Artifacts work, Mender has its own storage area for scripts inside the data partition, and will run them at the right moment during the install. Systemd is not involved at all, although Mender of course has to be running, and this usually happens with Systemd. But launching Mender using SYSV would not make a difference with regards to State Script execution.

Same rights as Mender: root account.

Dewey · April 29, 2020, 8:24am

Thank you.

Since I’m running in standalone and systemd in not involved, how will the script ArtifactCommit_Enter_00, that should be launched on the reboot, be launched ?

kacf · April 29, 2020, 9:09am

In that case it will be executed when you run mender -commit. If the script fails it will roll back instead.

Dewey · April 29, 2020, 5:55pm

It doesn’t seem to work, here is the content of the artifact:

 % mender-artifact read rpi3-release-1.1.0.mender -k ../keys/public.pem                                        
Mender artifact:
  Name: rpi3-release-1.1.0.mender
  Format: mender
  Version: 3
  Signature: signed and verified correctly
  Compatible devices: '[raspberrypi3]'
  Provides group: 
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:
    ArtifactCommit_Enter_00.sh

Updates:
    0:
    Type:   rootfs-image
    Provides:
	rootfs_image_checksum: be66eef18b9dfd8de44eddd97d6f886d0f3516f5034fe5808f01d98e431d7d22
    Depends: Nothing
    Metadata: Nothing
    Files:
      name:     latest-raspberrypi3-mender.ext4
      size:     13996392448
      modified: 2020-04-29 17:02:56 +0200 CEST
      checksum: be66eef18b9dfd8de44eddd97d6f886d0f3516f5034fe5808f01d98e431d7d22

And here is the content of the script:

#!/bin/bash

log() {
    echo "mender:$*" >&2
}

log "$(cat /etc/mender/artifact_info): $(basename "$0") was called!"

touch /data/update-1.1.0.log
echo "Sanity script started" > /data/update-1.1.0.log

CONTAINER_NAME=$(echo "ws")
commit=true
SERVER_VERSION=$(docker version -f "{{.Server.Version}}")
SERVER_VERSION_MAJOR=$(echo "$SERVER_VERSION"| cut -d'.' -f 1)
SERVER_VERSION_MINOR=$(echo "$SERVER_VERSION"| cut -d'.' -f 2)
SERVER_VERSION_BUILD=$(echo "$SERVER_VERSION"| cut -d'.' -f 3)

if [ "${SERVER_VERSION_MAJOR}" -ge 19 ] && \
   [ "${SERVER_VERSION_MINOR}" -ge 3 ]  && \
   [ "${SERVER_VERSION_BUILD}" -ge 8 ]; then
    echo "Docker version >= 19.03.8, update ok"
else
    echo "Docker version less than 19.03.8, update failed"
    commit=false
fi

if [ $(docker inspect -f '{{.State.Running}}' ${CONTAINER_NAME}) = "true" ]; then 
	echo "Web server is running" 
else 
	echo "Warning web server is not running!"
	commit=false 
fi
if [ "$commit" = true ] ; then
    echo "Update is successful - commit" > /data/update-1.1.0.log
    exit 0
else
    echo "Update has failed - rollback" > /data/update-1.1.0.log
    exit 1
fi

However, when checking on the system after an update, the file is not created on the data partition.
Any idea ?

mirzak · April 29, 2020, 6:33pm

Please share the log from when you run mender -commit

drewmoseley · April 29, 2020, 6:35pm

The artifact state scripts are not stored persistently in the filesystem. They are part of the artifact, processed during the OTA flow and then discarded.

Drew

Dewey · April 30, 2020, 7:27am

Here are the logs @mirzak:

DEBU[0000] Reading Mender configuration from file /var/lib/mender/mender.conf  module=config
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf  module=config
DEBU[0000] Reading Mender configuration from file /etc/mender/mender.conf  module=config
INFO[0000] Loaded configuration file: /etc/mender/mender.conf  module=config
WARN[0000] No server URL(s) specified in mender configuration.  module=config
WARN[0000] Server entry 1 has no associated server URL.  module=config
DEBU[0000] Merged configuration = &conf.MenderConfig{MenderConfigFromFile:conf.MenderConfigFromFile{ClientProtocol:"", ArtifactVerifyKey:"/var/lib/mender/public.pem", HttpsClient:struct { Certificate string; Key string; SkipVerify bool }{Certificate:"", Key:"", SkipVerify:false}, RootfsPartA:"/dev/mmcblk0p2", RootfsPartB:"/dev/mmcblk0p3", DeviceTypeFile:"/var/lib/mender/device_type", UpdatePollIntervalSeconds:0, InventoryPollIntervalSeconds:0, RetryPollIntervalSeconds:0, StateScriptTimeoutSeconds:0, StateScriptRetryTimeoutSeconds:0, StateScriptRetryIntervalSeconds:0, ModuleTimeoutSeconds:0, ServerCertificate:"", ServerURL:"", UpdateLogPath:"", TenantToken:"", Servers:[]client.MenderServer{client.MenderServer{ServerURL:""}}}, ModulesPath:"/usr/share/mender/modules/v3", ModulesWorkPath:"/var/lib/mender/modules/v3", ArtifactInfoFile:"/etc/mender/artifact_info", ArtifactScriptsPath:"/var/lib/mender/scripts", RootfsScriptsPath:"/etc/mender/scripts"}  module=config
DEBU[0000] Have U-Boot variable: mender_check_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_check_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: mender_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: mender_boot_part=3      module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_boot_part:3]  module=bootenv
DEBU[0000] Setting active partition from mount candidate: /dev/mmcblk0p3  module=partitions
INFO[0000] Mender running on partition: /dev/mmcblk0p3   module=cli
DEBU[0000] I/O read error for entry mender-agent.pem: open /var/lib/mender/mender-agent.pem: no such file or directory  module=dirstore
DEBU[0000] private key does not exist                    module=keystore
DEBU[0000] ModuleTimeoutSeconds not set. Defaulting to 14400 seconds  module=modules
DEBU[0000] Have U-Boot variable: mender_check_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_check_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: mender_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: upgrade_available=1     module=bootenv
DEBU[0000] List of U-Boot variables:map[upgrade_available:1]  module=bootenv
DEBU[0000] Read data from device manifest file: artifact_name=release-1.1.0  module=device
DEBU[0000] Current manifest data: release-1.1.0          module=device
Committing Artifact...
DEBU[0000] statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds  module=executor
DEBU[0000] Have U-Boot variable: mender_check_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_check_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: mender_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: upgrade_available=1     module=bootenv
DEBU[0000] List of U-Boot variables:map[upgrade_available:1]  module=bootenv
INFO[0000] Committing update                             module=dual_rootfs_device
DEBU[0000] Have U-Boot variable: mender_check_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_check_saveenv_canary:1]  module=bootenv
DEBU[0000] Have U-Boot variable: mender_saveenv_canary=1  module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_saveenv_canary:1]  module=bootenv
DEBU[0000] statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds  module=executor

The script was found during installation:

INFO[0000] installer: authenticated digital signature of artifact  module=installer
DEBU[0000] checking if device [raspberrypi3] is on compatible device list: [raspberrypi3]
  module=installer
DEBU[0000] installer: processing script: ArtifactCommit_Enter_00.sh  module=installer
DEBU[0000] installer: successfully read artifact [name: rpi3-release-1.1.0.mender; version: 3; compatible devices: [raspberrypi3]]  module=installer

mirzak · May 4, 2020, 8:22am

Hm, do not see it trying to execute the script in mender -commit.

Is /var/lib/mender a persistent location in you case? Because the script should have been unpacked there (I think under `/var/lib/mende/scripts) during the install state, and later executed during commit. So please verify that it is there and that this area is a persistent area.

Dewey · May 7, 2020, 7:58am

In fact the file is unpacked to /var/lib/mender/scripts. However, due to an unknown reason, this directory is not linked anymore with /data/mender. Could it be because I use the rootfs_overlay of mender_convert to add the public key, device_type and client.conf files on /var/lib/mender ?

Here is my rootfs_overlay:

.
└── var
└── lib
└── mender
├── mender.conf
└── public.pem
└── device_type

mirzak · May 7, 2020, 8:49am

Yes, This would remove the link.

You can add your files to /data directory in the overlay and they will put on the partition that is linked by /var/lib/mender

Example of overlay:

overlay/
└── data
    └── mender
        ├── device_type
        ├── mender.conf
        └── public.pem

Dewey · May 14, 2020, 3:38pm

Changing the root overlay did work. However, I have a new error when running the mender commit command:

time="2020-05-14T16:36:19+01:00" level=warning msg="script format mismatch: 'ArtifactCommit_Enter_00.sh' will not be run " module=executor

Script format mismatch, what is it ?

mirzak · May 15, 2020, 8:14am

Script format mismatch, what is it ?

Could be related to this change,

What do you have in /etc/mender/scripts/version ?

Dewey · May 26, 2020, 2:35pm

Sorry for the delay. The version is 3.

Topic		Replies	Views
ArtifactCommit_Leave state script not running when using standalone operation only General Discussions standalone , state-scripts	3	580	March 23, 2021
Standalone State Scripts not executed General Discussions state-scripts	3	430	August 5, 2021
Documentation of Standalone Mode & State Scripts is unclear General Discussions standalone , state-scripts , documentation	2	575	July 10, 2019
Are state scripts available for mender standalone artifact updates? General Discussions	1	726	February 19, 2020
Test and debug state scripts General Discussions state-scripts	1	360	February 22, 2022

States script on standalone mode

Related topics