SUGGESTION: Add a skip-images option to the app-update-module scripts
Unfortunately for now, if I want to use the app-update-module to update a docker-compose application, the images are by default included in the artifact.
My problem is that the images I am using are fairly big (in size), and it would be very ineffective to include them in the artifact (planing to build artifacts in CI/CD). Instead, I would like docker compose to handle the pulling of images directly from the target device.
Create Mender Artifacts
It would be nice to have something like this as a generation command:
That makes sense to have, especially for testing and in environments with very stable network.
Just note that I have seen many reports that the registry fetching tools (docker pull, etc.) work very poorly when network is unreliable. E.g. they can simply hang, there is no timeout, no resume, etc. There can also be issues with firewalls.
So the robust way to do this is to package images into the Artifact and then use delta updates to reduce the size.
In the documentation it is stated that the ‘–image’ flag is optional.
Including the Docker images is optional, excluding the images will make the devices try to pull the images from the Docker registry.
I interpret that as the images will not be bundled in the artifact, and the client will instead handling the pulling of the stated images, but I guess that is not the case @eystein?
thank you for your interest in Mender. At the moment the images are obligatory, however we have the “skip images” feature on our road map and it will be introduced soon.
thank you @DamKast for the nice idea.
@peter, when you implement this “skip images” feature, please continue to allow us to specify where the images are located so we can point the target devices to them.
do I understand correctly, that you would like to be able to set the urls for the images to pull from the device during the artifact creation? or perhaps even a registry url and only the image name:tag of particular images?
Yes - the registry url and image name:tag, just as it works currently… the only difference being that the target device pulls the images from the registry rather than the mender artifact containing the images.
I created a small script that takes an artifact generated by app_gen, unpacks it and replaces the images with a very small dummy image (around 400 bytes).
You could also just use the script as example how to use mender-artifact write together with this dummy image to create new artifacts without the need of running app_gen first.
#!/bin/bash
# Create temporary working directory
TMP=$(mktemp -d)
trap "rm -rf ${TMP}" EXIT
# Check arguments
if [ "${#}" != "2" ]; then
echo "Usage: ${0} <input_artifact> <output_artifact>"
echo "Example: ${0} artifact.mender artifact_stripped.mender"
exit 1
fi
# unpack input artifact
tar -xf ${1} -C ${TMP}
# unpack data
mkdir -p ${TMP}/data
tar -xf ${TMP}/data/0000.tar.gz -C ${TMP}/data
# unpack header
mkdir -p ${TMP}/header
tar -xf ${TMP}/header.tar.gz -C ${TMP}/header
# remove images from meta-data
cat ${TMP}/header/headers/0000/meta-data | jq 'del(.images)' > ${TMP}/metadata.json
# create the images.tar.gz with expected file structure but without actual Docker images
mkdir -p ${TMP}/empty_images/images
DUMMY_HASH="0000000000000000000000000000000000000000000000000000000000000000"
mkdir -p "${TMP}/empty_images/images/$DUMMY_HASH"
echo "# No images - will be pulled at runtime" > "${TMP}/empty_images/images/$DUMMY_HASH/url-new.txt"
echo "# No images - will be pulled at runtime" > "${TMP}/empty_images/images/$DUMMY_HASH/url-current.txt"
echo "$DUMMY_HASH" > "${TMP}/empty_images/images/$DUMMY_HASH/sums-new.txt"
echo "$DUMMY_HASH" > "${TMP}/empty_images/images/$DUMMY_HASH/sums-current.txt"
# Create a minimal valid Docker image format to satisfy docker import
mkdir -p ${TMP}/docker_image_temp
echo '{"dummy":{"latest":"'$DUMMY_HASH'"}}' > ${TMP}/docker_image_temp/repositories
echo '[]' > ${TMP}/docker_image_temp/manifest.json
tar -cf "${TMP}/empty_images/images/$DUMMY_HASH/image.img" -C ${TMP}/docker_image_temp .
rm -rf ${TMP}/docker_image_temp
tar -czf "${TMP}/data/images.tar.gz" -C ${TMP}/empty_images images
rm -rf ${TMP}/empty_images
# get artefact information
ARTIFACT_NAME=$(cat ${TMP}/header/header-info | jq -r '.artifact_provides.artifact_name')
ARTIFACT_TYPE=$(cat ${TMP}/header/header-info | jq -r '.payloads[0].type')
COMPATIBLE_DEVICES=$(cat ${TMP}/header/header-info | jq '.artifact_depends.device_type' | sed -e 's/[][ \t"\,]//g' | tr -d '\n' | sed 's/ / /g')
SOFTWARE_NAME=$(cat ${TMP}/header/headers/0000/meta-data | jq -r '.application_name')
SOFTWARE_VERSION=$(cat ${TMP}/header/headers/0000/meta-data | jq -r '.version')
# collect state scripts
STATE_SCRIPTS=""
if [ -d ${TMP}/header/scripts ]; then
for i in $(ls ${TMP}/header/scripts); do
STATE_SCRIPTS+="-s ${TMP}/header/scripts/${i} "
done
fi
# Finally create the stripped artifact
mender-artifact write module-image \
-t "${COMPATIBLE_DEVICES}" \
-o "${2}" \
-T "${ARTIFACT_TYPE}" \
-n "${ARTIFACT_NAME}" \
-f "${TMP}/data/images.tar.gz" \
-f "${TMP}/data/manifests.tar.gz" \
--software-name ${SOFTWARE_NAME} \
--software-version ${SOFTWARE_VERSION} \
--meta-data ${TMP}/metadata.json ${STATE_SCRIPTS}