Delta update rootfs error

Hi community:
I have a problem with delta update. I followed the Robust delta update rootfs : Robust delta update rootfs .

And first I deployed full image update that added mender-binary-delta tool. It didn’t occur any problems. Then I created a new .mender extension artifact and added some applications to it. I deployed it to our device without any problems. Then I created delta files using these two artifacts. The update failed when I tried to distribute them. I left the log file down:

2019-12-11 09:28:34 +0000 UTC info: Running Mender version 2.0.0
2019-12-11 09:28:34 +0000 UTC debug: handle update fetch state
2019-12-11 09:28:34 +0000 UTC debug: status reported, response 204 No Content
2019-12-11 09:28:34 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/vnd.mender-artifact] Content-Length:[5761536] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload] X-Frame-Options:[DENY] Server:[openresty/1.13.6.2] Accept-Ranges:[bytes] Content-Security-Policy:[block-all-mixed-content] Etag:[“b91f399b30c8e925d04b852de4ebe5fd”] Vary:[Origin] X-Amz-Request-Id:[15DF3D3A47D681FF] Connection:[keep-alive] X-Content-Type-Options:[nosniff] Last-Modified:[Sun, 08 Dec 2019 13:38:00 GMT] Date:[Wed, 11 Dec 2019 06:28:35 GMT]] 0x11f52410 5761536 [] false false map[] 0x11e7c800 0x11e6f2c0}+
2019-12-11 09:28:34 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-12-11 09:28:34 +0000 UTC debug: handle update install state
2019-12-11 09:28:34 +0000 UTC debug: status reported, response 204 No Content
2019-12-11 09:28:34 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC info: installer: authenticated digital signature of artifact
2019-12-11 09:28:34 +0000 UTC debug: checking if device [imx6ul-var-dart] is on compatibile device list: [imx6ul-var-dart]
2019-12-11 09:28:34 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-12-11 09:28:34 +0000 UTC debug: Returning artifact name release-v.2.0 from database.
2019-12-11 09:28:34 +0000 UTC debug: Read data from device manifest file: artifact_name=release-v.2.0
2019-12-11 09:28:34 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-11 09:28:34 +0000 UTC debug: installer: successfully read artifact [name: v1.0-deltfrom-v2.0; version: 3; compatible devices: [imx6ul-var-dart]]
2019-12-11 09:28:34 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-12-11 09:28:34 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-11 09:28:34 +0000 UTC info: Update module output: field artifact_provides not found in the type-info file.
2019-12-11 09:28:34 +0000 UTC info: Update module output: Trying to parse transitional meta-data instead…
2019-12-11 09:28:34 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-12-11 09:28:39 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-12-11 09:33:33 +0000 UTC info: Update module output: Fatal failure: The expected and the actual checksums do not match!
2019-12-11 09:33:33 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1
2019-12-11 09:33:33 +0000 UTC info: State transition: update-store [Download_Enter] -> cleanup [Error]
2019-12-11 09:33:33 +0000 UTC debug: transitioning to error state
2019-12-11 09:33:33 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-11 09:33:33 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-11 09:33:33 +0000 UTC debug: Handling Cleanup state
2019-12-11 09:33:34 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-12-11 09:33:34 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-11 09:33:34 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-12-11 09:33:34 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-11 09:33:34 +0000 UTC debug: handle update status report state
2019-12-11 09:33:34 +0000 UTC debug: status reported, response 204 No Content
2019-12-11 09:33:34 +0000 UTC debug: attempting to upload deployment logs for failed update

How can I fix this problem ?

Thanks.
Best regards.

Hey @mEK, can you give the the output of these two commands:

mender-artifact read NEW-ARTIFACT
mender-artifact read DELTA-ARTIFACT

Replace NEW-ARTIFACT with the artifact where you added applications, and DELTA-ARTIFACT with the artifact that you created from the two previous artifacts.

@ubuntu:~/mender-binary-delta-1.0.0/x86_64$ mender-artifact read release-v.2.0.mender
Mender artifact:
Name: release-v.2.0
Format: mender
Version: 3
Signature: signed; verification using provided key failed
Compatible devices: ‘[imx6ul-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: rootfs-image
Provides: Nothing
Depends: Nothing
Metadata: Nothing
Files:
name: core-image-minimal-imx6ul-var-dart.ext4
size: 3829399552
modified: 2019-12-08 15:32:20 +0300 +03
checksum: 74426674aabe2d61eae2c583ef7da58b03b013abd226e04b2839bf5e822951ef

hie@ubuntu:~/mender-binary-delta-1.0.0/x86_64$ mender-artifact read v1.0-deltafrom-v2.0.mender
Mender artifact:
Name: v1.0-deltfrom-v2.0
Format: mender
Version: 3
Signature: signed; verification using provided key failed
Compatible devices: ‘[imx6ul-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: mender-binary-delta
Provides: Nothing
Depends: Nothing
Metadata:
{
“delta_algorithm”: “xdelta3”,
“rootfs_file_size”: 3829399552,
“rootfs_image_checksum”: “a089f4c0bad103e8ab445f7c545a2bbd31afa070514ee798da5ebba0459dd1ab”
}
Files:
name: core-image-minimal-imx6ul-var-dart.ext4.delta
size: 5762390
modified: 2019-12-08 16:12:56 +0300 +03
checksum: 14ac422bac36a5953cdd8773055eb14fbfcb97f71916e7265fe890fba0292b77

Additionally, I want to show content of mender-binary-delta file. And I realized checksum is mismatch.

What’s my problem ?

Thanks.

Are you sure you have not swapped the v1 and v2 artifacts? I noticed that the name of your delta artifact says “v1.0-deltfrom-v2.0”, but you’re actually going to v2 from v1, are you not?

Also, I can see from your screenshots that the other delta artifact “v2-deltafrom-v1” does have the correct checksum. The rootfs_image_checksum field from the delta artifact should match the checksum field from the new artifact.

I tried again with new images. If I summarize my process:

First I created an empty artifact named version-01. I placed this artifact in the target device (.sdimg extension) as specified in the document and copied the artifact with the .mender extension into the / mender-binary-delta file on the development machine. Then I created another update called version-02 on the development device, added some applications (such as telegraf). I copied this update to the same location in the / mender-binary-delta file. I then created a binary delta using these two update files:

./mender-binary-delta-generator -n 02-deltafrom-01 -o 02-deltafrom-01.mender version-01.mender version-02.mender – -k private.key
./mender-binary-delta-generator -n 01-deltafrom-02 -o 01-deltafrom-02.mender version-02.mender version-01.mender – -k private.key

In this way, after producing deltas I added to the server.

Version-01 release currently running on the board. I added the version-02, 02-deltafrom-01.mender and 01-deltafrom-02.mender update files to the server. They all remain signed. Then, as you said in the document, I deployed version-02 as full system upgrades and there was no problem. But when I deployed these delta updates (I tried 01-deltafrom-02.mender over version-02), the update could not be completed. And my failure log as follows :

2019-12-12 13:10:05 +0000 UTC info: Running Mender version 2.0.0
2019-12-12 13:10:05 +0000 UTC debug: handle update fetch state
2019-12-12 13:10:05 +0000 UTC debug: status reported, response 204 No Content
2019-12-12 13:10:05 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[X-Frame-Options:[DENY] X-Content-Type-Options:[nosniff] Content-Type:[application/vnd.mender-artifact] Accept-Ranges:[bytes] Etag:[“11b608c699fceab96cdd38d3124c6861”] X-Amz-Request-Id:[15DF97E5560A0218] Content-Security-Policy:[block-all-mixed-content] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload] Server:[openresty/1.13.6.2] Date:[Thu, 12 Dec 2019 10:10:05 GMT] Last-Modified:[Thu, 12 Dec 2019 07:42:02 GMT] Vary:[Origin] Content-Length:[5761024] Connection:[keep-alive]] 0x10c9c870 5761024 [] false false map[] 0x10b7a980 0x10cced80}+
2019-12-12 13:10:05 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-12-12 13:10:05 +0000 UTC debug: handle update install state
2019-12-12 13:10:05 +0000 UTC debug: status reported, response 204 No Content
2019-12-12 13:10:05 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC info: installer: authenticated digital signature of artifact
2019-12-12 13:10:05 +0000 UTC debug: checking if device [imx6ul-var-dart] is on compatibile device list: [imx6ul-var-dart]
2019-12-12 13:10:05 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-12-12 13:10:05 +0000 UTC debug: Returning artifact name version-02 from database.
2019-12-12 13:10:05 +0000 UTC debug: Read data from device manifest file: artifact_name=version-02
2019-12-12 13:10:05 +0000 UTC debug: Read data from device manifest file: device_type=imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC debug: Current manifest data: imx6ul-var-dart
2019-12-12 13:10:05 +0000 UTC debug: installer: successfully read artifact [name: 01-deltafrom-02; version: 3; compatible devices: [imx6ul-var-dart]]
2019-12-12 13:10:05 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-12-12 13:10:05 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-12 13:10:06 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-12-12 13:10:06 +0000 UTC info: Update module output: field artifact_provides not found in the type-info file.
2019-12-12 13:10:06 +0000 UTC info: Update module output: Trying to parse transitional meta-data instead…
2019-12-12 13:10:11 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-12-12 13:15:05 +0000 UTC info: Update module output: Fatal failure: The expected and the actual checksums do not match!
2019-12-12 13:15:05 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1
2019-12-12 13:15:05 +0000 UTC info: State transition: update-store [Download_Enter] -> cleanup [Error]
2019-12-12 13:15:05 +0000 UTC debug: transitioning to error state
2019-12-12 13:15:05 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-12 13:15:05 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-12 13:15:05 +0000 UTC debug: Handling Cleanup state
2019-12-12 13:15:05 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-12-12 13:15:06 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/mender-binary-delta Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-12-12 13:15:06 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-12-12 13:15:06 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-12-12 13:15:06 +0000 UTC debug: handle update status report state
2019-12-12 13:15:06 +0000 UTC debug: status reported, response 204 No Content
2019-12-12 13:15:06 +0000 UTC debug: attempting to upload deployment logs for failed update

When I check to artifact’s checksum, I observed like following :

ss2

And /etc/mender/mender.conf file included :

"ArtifactVerifyKey": "/etc/mender/artifact-verify-key.pem",
"InventoryPollIntervalSeconds": 1800,
"RetryPollIntervalSeconds": 300,
"RootfsPartA": "/dev/mmcblk1p1",
"RootfsPartB": "/dev/mmcblk1p2",
"ServerCertificate": "/etc/mender/server.crt",
"ServerURL": "https://mender.robust.com",
"TenantToken": "dummy",
"UpdatePollIntervalSeconds": 600

Just now I noticed that your version of mender-binary-delta is 1.0.0. Can you try to upgrade to 1.0.1 using a rootfs-image update, and then retry? There is a fix in 1.0.1 specifically for dealing with images larger than 2GiB.

Okey but I couldn’t download mender-binary-delta 1.0.1 version. And @drewmoseley sent me 1.0.0 release as email. Would you please send me version 1.0.1 of mender-binary-delta? Otherwise, the download fails.

Take a look at the download instructions for mender-binary-delta (search for “Download mender-binary-delta”). You will need credentials either for a Hosted Mender account or an Enterprise account.