Delta update rootfs error

mEK · December 11, 2019, 6:51am

Hi community:
I have a problem with delta update. I followed the Robust delta update rootfs : Robust delta update rootfs .

And first I deployed full image update that added mender-binary-delta tool. It didn’t occur any problems. Then I created a new .mender extension artifact and added some applications to it. I deployed it to our device without any problems. Then I created delta files using these two artifacts. The update failed when I tried to distribute them. I left the log file down:

2019-12-11 09:28:34 +0000 UTC info: Running Mender version 2.0.0
2019-12-11 09:28:34 +0000 UTC debug: handle update fetch state
2019-12-11 09:28:34 +0000 UTC debug: status reported, response 204 No Content
2019-12-11 09:28:34 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/vnd.mender-artifact] Content-Length:[5761536] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload] X-Frame-Options:[DENY] Server:[openresty/1.13.6.2] Accept-Ranges:[bytes] Content-Security-Policy:[block-all-mixed-content] Etag:[“b91f399b30c8e925d04b852de4ebe5fd”] Vary:[Origin] X-Amz-Request-Id:[15DF3D3A47D681FF] Connection:[keep-alive] X-Content-Type-Options:[nosniff] Last-Modified:[Sun, 08 Dec 2019 13:38:00 GMT] Date:[Wed, 11 Dec 2019 06:28:35 GMT]] 0x11f52410 5761536 [] false false map[] 0x11e7c800 0x11e6f2c0}+
2019-12-11 09:28:34 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-12-11 09:28:34 +0000 UTC debug: handle update install state
2019-12-11 09:28:34 +0000 UTC debug: status reported, response 204 No Content
2019-12-11 09:28:34 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC info: installer: authenticated digital signature of artifact
2019-12-11 09:28:34 +0000 UTC debug: checking if device [imx6ul-var-dart] is on compatibile device list: [imx6ul-var-dart]
2019-12-11 09:28:34 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-12-11 09:28:34 +0000 UTC debug: Returning artifact name release-v.2.0 from database.
2019-12-11 09:28:34 +0000 UTC debug: Read data from device manifest file: artifact_name=release-v.2.0
2019-12-11 09:28:34 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC debug: installer: successfully read artifact [name: v1.0-deltfrom-v2.0; version: 3; compatible devices: [imx6ul-var-dart]]
2019-12-11 09:28:34 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-12-11 09:28:34 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-11 09:28:34 +0000 UTC info: Update module output: field artifact_provides not found in the type-info file.
2019-12-11 09:28:34 +0000 UTC info: Update module output: Trying to parse transitional meta-data instead…
2019-12-11 09:28:34 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-12-11 09:28:39 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-12-11 09:33:33 +0000 UTC info: Update module output: Fatal failure: The expected and the actual checksums do not match!
2019-12-11 09:33:33 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1
2019-12-11 09:33:33 +0000 UTC info: State transition: update-store [Download_Enter] -> cleanup [Error]
2019-12-11 09:33:33 +0000 UTC debug: transitioning to error state
2019-12-11 09:33:33 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-11 09:33:33 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-11 09:33:33 +0000 UTC debug: Handling Cleanup state
2019-12-11 09:33:34 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-12-11 09:33:34 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-11 09:33:34 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-12-11 09:33:34 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-11 09:33:34 +0000 UTC debug: handle update status report state
2019-12-11 09:33:34 +0000 UTC debug: status reported, response 204 No Content
2019-12-11 09:33:34 +0000 UTC debug: attempting to upload deployment logs for failed update

How can I fix this problem ?

Thanks.
Best regards.

kacf · December 11, 2019, 7:56am

Hey @mEK, can you give the the output of these two commands:

mender-artifact read NEW-ARTIFACT
mender-artifact read DELTA-ARTIFACT

Replace NEW-ARTIFACT with the artifact where you added applications, and DELTA-ARTIFACT with the artifact that you created from the two previous artifacts.

mEK · December 11, 2019, 8:11am

@ubuntu:~/mender-binary-delta-1.0.0/x86_64$ mender-artifact read release-v.2.0.mender
Mender artifact:
Name: release-v.2.0
Format: mender
Version: 3
Signature: signed; verification using provided key failed
Compatible devices: ‘[imx6ul-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: rootfs-image
Provides: Nothing
Depends: Nothing
Metadata: Nothing
Files:
name: core-image-minimal-imx6ul-var-dart.ext4
size: 3829399552
modified: 2019-12-08 15:32:20 +0300 +03
checksum: 74426674aabe2d61eae2c583ef7da58b03b013abd226e04b2839bf5e822951ef

hie@ubuntu:~/mender-binary-delta-1.0.0/x86_64$ mender-artifact read v1.0-deltafrom-v2.0.mender
Mender artifact:
Name: v1.0-deltfrom-v2.0
Format: mender
Version: 3
Signature: signed; verification using provided key failed
Compatible devices: ‘[imx6ul-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: mender-binary-delta
Provides: Nothing
Depends: Nothing
Metadata:
{
“delta_algorithm”: “xdelta3”,
“rootfs_file_size”: 3829399552,
“rootfs_image_checksum”: “a089f4c0bad103e8ab445f7c545a2bbd31afa070514ee798da5ebba0459dd1ab”
}
Files:
name: core-image-minimal-imx6ul-var-dart.ext4.delta
size: 5762390
modified: 2019-12-08 16:12:56 +0300 +03
checksum: 14ac422bac36a5953cdd8773055eb14fbfcb97f71916e7265fe890fba0292b77

mEK · December 11, 2019, 10:48am

Additionally, I want to show content of mender-binary-delta file. And I realized checksum is mismatch.

What’s my problem ?

Thanks.

kacf · December 11, 2019, 11:52am

Are you sure you have not swapped the v1 and v2 artifacts? I noticed that the name of your delta artifact says “v1.0-deltfrom-v2.0”, but you’re actually going to v2 from v1, are you not?

Also, I can see from your screenshots that the other delta artifact “v2-deltafrom-v1” does have the correct checksum. The rootfs_image_checksum field from the delta artifact should match the checksum field from the new artifact.

mEK · December 12, 2019, 10:18am

I tried again with new images. If I summarize my process:

First I created an empty artifact named version-01. I placed this artifact in the target device (.sdimg extension) as specified in the document and copied the artifact with the .mender extension into the / mender-binary-delta file on the development machine. Then I created another update called version-02 on the development device, added some applications (such as telegraf). I copied this update to the same location in the / mender-binary-delta file. I then created a binary delta using these two update files:

./mender-binary-delta-generator -n 02-deltafrom-01 -o 02-deltafrom-01.mender version-01.mender version-02.mender – -k private.key
./mender-binary-delta-generator -n 01-deltafrom-02 -o 01-deltafrom-02.mender version-02.mender version-01.mender – -k private.key

In this way, after producing deltas I added to the server.

Version-01 release currently running on the board. I added the version-02, 02-deltafrom-01.mender and 01-deltafrom-02.mender update files to the server. They all remain signed. Then, as you said in the document, I deployed version-02 as full system upgrades and there was no problem. But when I deployed these delta updates (I tried 01-deltafrom-02.mender over version-02), the update could not be completed. And my failure log as follows :

2019-12-12 13:10:05 +0000 UTC info: Running Mender version 2.0.0
2019-12-12 13:10:05 +0000 UTC debug: handle update fetch state
2019-12-12 13:10:05 +0000 UTC debug: status reported, response 204 No Content
2019-12-12 13:10:05 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[X-Frame-Options:[DENY] X-Content-Type-Options:[nosniff] Content-Type:[application/vnd.mender-artifact] Accept-Ranges:[bytes] Etag:[“11b608c699fceab96cdd38d3124c6861”] X-Amz-Request-Id:[15DF97E5560A0218] Content-Security-Policy:[block-all-mixed-content] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload] Server:[openresty/1.13.6.2] Date:[Thu, 12 Dec 2019 10:10:05 GMT] Last-Modified:[Thu, 12 Dec 2019 07:42:02 GMT] Vary:[Origin] Content-Length:[5761024] Connection:[keep-alive]] 0x10c9c870 5761024 false false map 0x10b7a980 0x10cced80}+
2019-12-12 13:10:05 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2019-12-12 13:10:05 +0000 UTC debug: handle update install state
2019-12-12 13:10:05 +0000 UTC debug: status reported, response 204 No Content
2019-12-12 13:10:05 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC info: installer: authenticated digital signature of artifact
2019-12-12 13:10:05 +0000 UTC debug: checking if device [imx6ul-var-dart] is on compatibile device list: [imx6ul-var-dart]
2019-12-12 13:10:05 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-12-12 13:10:05 +0000 UTC debug: Returning artifact name version-02 from database.
2019-12-12 13:10:05 +0000 UTC debug: Read data from device manifest file: artifact_name=version-02
2019-12-12 13:10:05 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC debug: installer: successfully read artifact [name: 01-deltafrom-02; version: 3; compatible devices: [imx6ul-var-dart]]
2019-12-12 13:10:05 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-12-12 13:10:05 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-12 13:10:06 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-12-12 13:10:06 +0000 UTC info: Update module output: field artifact_provides not found in the type-info file.
2019-12-12 13:10:06 +0000 UTC info: Update module output: Trying to parse transitional meta-data instead…
2019-12-12 13:10:11 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-12-12 13:15:05 +0000 UTC info: Update module output: Fatal failure: The expected and the actual checksums do not match!
2019-12-12 13:15:05 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1
2019-12-12 13:15:05 +0000 UTC info: State transition: update-store [Download_Enter] → cleanup [Error]
2019-12-12 13:15:05 +0000 UTC debug: transitioning to error state
2019-12-12 13:15:05 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-12 13:15:05 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-12 13:15:05 +0000 UTC debug: Handling Cleanup state
2019-12-12 13:15:05 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-12-12 13:15:06 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-12 13:15:06 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]
2019-12-12 13:15:06 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-12 13:15:06 +0000 UTC debug: handle update status report state
2019-12-12 13:15:06 +0000 UTC debug: status reported, response 204 No Content
2019-12-12 13:15:06 +0000 UTC debug: attempting to upload deployment logs for failed update

When I check to artifact’s checksum, I observed like following :

mEK · December 12, 2019, 11:17am

And /etc/mender/mender.conf file included :

"ArtifactVerifyKey": "/etc/mender/artifact-verify-key.pem",
"InventoryPollIntervalSeconds": 1800,
"RetryPollIntervalSeconds": 300,
"RootfsPartA": "/dev/mmcblk1p1",
"RootfsPartB": "/dev/mmcblk1p2",
"ServerCertificate": "/etc/mender/server.crt",
"ServerURL": "https://mender.robust.com",
"TenantToken": "dummy",
"UpdatePollIntervalSeconds": 600

kacf · December 12, 2019, 7:31pm

Just now I noticed that your version of mender-binary-delta is 1.0.0. Can you try to upgrade to 1.0.1 using a rootfs-image update, and then retry? There is a fix in 1.0.1 specifically for dealing with images larger than 2GiB.

mEK · December 13, 2019, 6:22am

Okey but I couldn’t download mender-binary-delta 1.0.1 version. And @drewmoseley sent me 1.0.0 release as email. Would you please send me version 1.0.1 of mender-binary-delta? Otherwise, the download fails.

kacf · December 13, 2019, 7:36am

Take a look at the download instructions for mender-binary-delta (search for “Download mender-binary-delta”). You will need credentials either for a Hosted Mender account or an Enterprise account.

Topic		Replies	Views
Robust delta update rootfs Update Modules	12	8887	November 19, 2021
Error during delta update General Discussions	24	1509	October 12, 2021
No Artifacts found error when trying delta-updates General Discussions	31	1767	October 16, 2020
How to force rootfs-image instead of delta update General Discussions	2	309	February 11, 2021
Delta Update rootfs-image Provides field General Discussions deltaupdate	2	354	March 2, 2023

Delta update rootfs error

Related topics