Mender-Artifact deployment error

mEK · September 9, 2019, 6:15pm

Hello everyone.
I am a new meander user and I am very happy to be here.
I followed the https://docs.mender.io/2.0/artifacts/signing-and-verification document and signed mender-artifact and tried to distribute it to my device with QEMU running on my virtual machine with demo server. However, the distribution cannot be completed and the log file reports:


2019-09-09 13:33:52 +0000 UTC debug: handle update fetch state
2019-09-09 13:33:52 +0000 UTC debug: status reported, response 204 No Content
2019-09-09 13:33:52 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Server:[openresty/1.13.6.2] Last-Modified:[Mon, 09 Sep 2019 12:45:54 GMT] X-Amz-Request-Id:[15C2C876AD0BD197] Content-Type:[application/vnd.mender-artifact] Content-Length:[27237376] Accept-Ranges:[bytes] Vary:[Origin] X-Frame-Options:[DENY] X-Content-Type-Options:[nosniff] Date:[Mon, 09 Sep 2019 13:33:52 GMT] Connection:[keep-alive] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=0; includeSubdomains; preload] Content-Security-Policy:[block-all-mixed-content] Etag:["1619073d9620da16b76e599f7b0a7a79"]] 0xc420199460 27237376 [] false false map[] 0xc420268e00 0xc42008e790}+
2019-09-09 13:33:52 +0000 UTC info: State transition: update-fetch [Download] -> update-store [Download]
2019-09-09 13:33:52 +0000 UTC debug: handle update install state
2019-09-09 13:33:52 +0000 UTC debug: status reported, response 204 No Content
2019-09-09 13:33:52 +0000 UTC debug: Read data from device manifest file: device_type=qemux86-64
2019-09-09 13:33:52 +0000 UTC debug: Found needed line: device_type=qemux86-64
2019-09-09 13:33:52 +0000 UTC debug: Current manifest data: qemux86-64
2019-09-09 13:33:52 +0000 UTC info: no public key was provided for authenticating the artifact
2019-09-09 13:33:52 +0000 UTC warning: installer: installing signed artifact without verification as verification key is missing
2019-09-09 13:33:52 +0000 UTC debug: checking if device [qemux86-64] is on compatibile device list: [qemux86-64]
2019-09-09 13:33:52 +0000 UTC debug: installing update core-image-minimal-qemux86-64.ext4 of size 226492416
2019-09-09 13:33:52 +0000 UTC debug: Trying to install update of size: 226492416
2019-09-09 13:33:52 +0000 UTC debug: Active partition: /dev/hda2
2019-09-09 13:33:52 +0000 UTC debug: Detected inactive partition /dev/hda3, based on active partition /dev/hda2
2019-09-09 13:33:52 +0000 UTC info: opening device /dev/hda3 for writing
2019-09-09 13:33:52 +0000 UTC info: partition /dev/hda3 size: 226492416
2019-09-09 13:35:44 +0000 UTC info: wrote 226492416/226492416 bytes of update to device /dev/hda3
2019-09-09 13:35:44 +0000 UTC debug: installer: successfully read artifact [name: release-1; version: 2; compatible devices: [qemux86-64]]
2019-09-09 13:35:44 +0000 UTC debug: status reported, response 204 No Content
2019-09-09 13:35:44 +0000 UTC info: State transition: update-store [Download] -> update-install [ArtifactInstall]
2019-09-09 13:35:44 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:35:44 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:35:44 +0000 UTC debug: status reported, response 204 No Content
2019-09-09 13:35:44 +0000 UTC debug: Inactive partition: /dev/hda3
2019-09-09 13:35:44 +0000 UTC debug: Marking inactive partition (/dev/hda3) as the new boot candidate.
2019-09-09 13:35:44 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2019-09-09 13:35:45 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=
2019-09-09 13:35:45 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:]
2019-09-09 13:35:45 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-09-09 13:35:45 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2019-09-09 13:35:45 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:35:45 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:35:45 +0000 UTC debug: handling reboot state
2019-09-09 13:35:45 +0000 UTC debug: status reported, response 204 No Content
2019-09-09 13:35:45 +0000 UTC info: rebooting device
2019-09-09 13:35:45 +0000 UTC info: Mender rebooting from active partition: /dev/hda2
2019-09-09 13:36:37 +0000 UTC debug: handling state after reboot
2019-09-09 13:36:37 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> update-verify [ArtifactReboot_Leave]
2019-09-09 13:36:37 +0000 UTC debug: handle update verify state
2019-09-09 13:36:38 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=
2019-09-09 13:36:38 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:]
2019-09-09 13:36:38 +0000 UTC debug: Have U-Boot variable: upgrade_available=1
2019-09-09 13:36:38 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:1]
2019-09-09 13:36:38 +0000 UTC info: State transition: update-verify [ArtifactReboot_Leave] -> update-commit [ArtifactCommit]
2019-09-09 13:36:38 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:36:38 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:36:38 +0000 UTC debug: handle update commit state
2019-09-09 13:36:38 +0000 UTC debug: Read data from device manifest file: artifact_name=release-1
2019-09-09 13:36:38 +0000 UTC debug: Found needed line: artifact_name=release-1
2019-09-09 13:36:38 +0000 UTC debug: Current manifest data: release-1
2019-09-09 13:36:38 +0000 UTC info: successfully running with new image release-1
2019-09-09 13:36:38 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=
2019-09-09 13:36:38 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:]
2019-09-09 13:36:38 +0000 UTC debug: Have U-Boot variable: upgrade_available=1
2019-09-09 13:36:38 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:1]
2019-09-09 13:36:38 +0000 UTC info: Commiting update
2019-09-09 13:36:38 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=
2019-09-09 13:36:38 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:]
2019-09-09 13:36:39 +0000 UTC info: Storing commit state data
2019-09-09 13:36:39 +0000 UTC info: State transition: update-commit [ArtifactCommit] -> update-status-report [none]
2019-09-09 13:36:39 +0000 UTC debug: handle update status report state
2019-09-09 13:36:56 +0000 UTC error: failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:36:56 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:36:56 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:36:56 +0000 UTC info: State transition: update-status-report [ArtifactCommit] -> update-retry-report [none]
2019-09-09 13:37:26 +0000 UTC debug: wait complete
2019-09-09 13:37:26 +0000 UTC info: State transition: update-retry-report [ArtifactCommit] -> update-status-report [ArtifactCommit]
2019-09-09 13:37:26 +0000 UTC debug: handle update status report state
2019-09-09 13:37:46 +0000 UTC error: failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:37:46 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:37:46 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:37:46 +0000 UTC info: State transition: update-status-report [ArtifactCommit] -> update-retry-report [none]
2019-09-09 13:38:16 +0000 UTC debug: wait complete
2019-09-09 13:38:16 +0000 UTC info: State transition: update-retry-report [ArtifactCommit] -> update-status-report [ArtifactCommit]
2019-09-09 13:38:16 +0000 UTC debug: handle update status report state
2019-09-09 13:38:36 +0000 UTC error: failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:38:36 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:38:36 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:38:36 +0000 UTC info: State transition: update-status-report [ArtifactCommit] -> update-retry-report [none]
2019-09-09 13:39:06 +0000 UTC debug: wait complete
2019-09-09 13:39:06 +0000 UTC info: State transition: update-retry-report [ArtifactCommit] -> update-status-report [ArtifactCommit]
2019-09-09 13:39:06 +0000 UTC debug: handle update status report state
2019-09-09 13:39:26 +0000 UTC error: failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:39:26 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:39:26 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF
2019-09-09 13:39:26 +0000 UTC info: State transition: update-status-report [ArtifactCommit] -> update-retry-report [none]
2019-09-09 13:39:26 +0000 UTC info: State transition: update-retry-report [ArtifactCommit] -> status-report-error [ArtifactFailure]
2019-09-09 13:39:26 +0000 UTC debug: transitioning to error state
2019-09-09 13:39:26 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:39:26 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:39:26 +0000 UTC error: handling report error state with status: success
2019-09-09 13:39:26 +0000 UTC info: State transition: status-report-error [ArtifactFailure] -> rollback [ArtifactRollback]
2019-09-09 13:39:26 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:39:26 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:39:26 +0000 UTC info: performing rollback
2019-09-09 13:39:26 +0000 UTC debug: Active partition: /dev/hda3
2019-09-09 13:39:26 +0000 UTC debug: Detected inactive partition /dev/hda2, based on active partition /dev/hda3
2019-09-09 13:39:26 +0000 UTC debug: Marking inactive partition (/dev/hda2) as the new boot candidate.
2019-09-09 13:39:26 +0000 UTC info: setting partition for rollback: 2
2019-09-09 13:39:26 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=
2019-09-09 13:39:26 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:]
2019-09-09 13:39:27 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-09-09 13:39:27 +0000 UTC debug: will try to rollback reboot the device
2019-09-09 13:39:27 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2019-09-09 13:39:27 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:39:27 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:39:27 +0000 UTC info: rebooting device after rollback
2019-09-09 13:39:27 +0000 UTC info: Mender rebooting from active partition: /dev/hda3
2019-09-09 13:40:17 +0000 UTC debug: handling state after rollback reboot
2019-09-09 13:40:17 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2019-09-09 13:40:17 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:40:17 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 60 seconds
2019-09-09 13:40:17 +0000 UTC debug: handle update error state
2019-09-09 13:40:17 +0000 UTC info: State transition: update-error [ArtifactFailure] -> update-status-report [none]
2019-09-09 13:40:17 +0000 UTC debug: handle update status report state
2019-09-09 13:40:18 +0000 UTC debug: status reported, response 204 No Content
2019-09-09 13:40:18 +0000 UTC debug: attempting to upload deployment logs for failed update

What is the cause of the error? How do I resolve this error?
Thanks for help

Edit: @mirzak: formatting

mirzak · September 9, 2019, 8:13pm

Welcome to the Mender community, I hope you will have a pleasent experience.

It seems that there was problems reporting status to the server after the update, hence it performed a roll-back.

Indicated by the following errors,

2019-09-09 13:39:26 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/dfd396a6-257d-4a80-ac59-f829d23a8188/status: EOF

The generated Mender Artifact that you signed, does it contain the necessary configurations to connect to docker.mender.io?

mEK · September 10, 2019, 10:14am

Thank you for reply mirzak. I downloaded the artifact from the document ready. So I didn’t make any configuration. I just signed it and uploaded it to docker.mender.io. Where do you recommend me to follow for configuration settings?

The contents of the file are as follows (/etc/mender/mender.conf) :
{
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 10,
“RootfsPartA”: “/dev/hda2”,
“RootfsPartB”: “/dev/hda3”,
“ServerCertificate”: “/etc/mender/server.crt”,
“ServerURL”: “https://docker.mender.io”,
“TenantToken”: “dummy”,
“UpdatePollIntervalSeconds”: 10
}

mirzak · September 10, 2019, 10:29am

Can you please share the exact location from which you downloaded the artifacts and also, which version of the server are you running?

The QEMU artifacts should work without any need of modifications. Did they work for you without you signing them?

Please also note that verification of signatures is not enabled on the QEMU images, so the signature will not be verified if you sign the artifacts.

mEK · September 10, 2019, 10:49am

I guess I misrepresented myself. My problem is not with validating the signature. Because when we look at the artifact on the demo server, the signature appears to be verified. My real problem is I can’t deploy this artifact. I create a deployment and select the artifact and it starts to distribute. When it reaches 80%, it reboots and my QEMU machine reboots and updates. However, when the demo reaches 90% on the server, it reboots again and rollback old artifact. I don’t understand why. I don’t think there’s a problem with signing. When I look at the log file, I think there hasn’t been an error before. What is “EOF” and how do I fix this problem.

mirzak · September 10, 2019, 11:00am

I guess I misrepresented myself. My problem is not with validating the signature

I was aware of this, it just an FYI.

I am not sure what it is, but looks like a connectivity issue.

Please share the requested information:

Where did you get the demo artifacts from?
Which version of the server are you running?

mEK · September 10, 2019, 11:21am

I downloaded this document https://docs.mender.io/2.0/getting-started/deploy-to-physical-devices Mender-Artifact tools and I am running integration-2.0.1 demo server.

mEK · September 10, 2019, 1:12pm

I am waiting for your feedback

drewmoseley · September 10, 2019, 4:29pm

The correct link for the QEMU based deployments should be https://docs.mender.io/2.0/getting-started/deploy-to-virtual-devices. Are you deploying to a QEMU device or some other? What device are you using?

It definitely seems that the new rootfs is having issues connecting to the Mender server after booting so it reboots. Can you share the details of where/how you have deployed the server?

mEK · September 11, 2019, 6:02am

I am using QEMU device and i try to deploy this device.

I downloaded the Mender-Artifact tool from the link I shared above and signed an artifact named mender-artifact. Then I uploaded it to the demo server. The installation was successful and I observed that the artifact was signed. Then I tried to distribute it to my QEMU device and failed as I explained.

I follow the deploy phsical device document. Is this wrong? Should I follow the virtual device deployment?

mEK · September 11, 2019, 6:49am

In addition, following this document QEMU, the FAST! processor emulator , I installed the QEMU environment. And I’m trying to deploy the resulting core-image-base-qemux86-64.mender artifact. I get the error when distributing this artifact file.

drewmoseley · September 11, 2019, 10:07pm

I took a crack at this setup today. I have the Mender demo server v2.0.1 running and the Mender QEMU setup from the Mender hub link. I set the MENDER_HOST_IP_ADDRESS in the local.conf file and the OTA update was successful.

One possible thing to trip over is that when you run the Mender demo server, that also sets up a QEMU device which does not work with the Mender artifact generated by the build. You may want to invoke the Mender demo environment with “./demo --no-client up -d” to avoid this.

Topic		Replies	Views
Bug report: deploying unsigned / not properly signed artifacts General Discussions	1	378	July 12, 2019
Failed to install a system-update - Artifact headers failed General Discussions yocto , tinkerboard , dunfell	8	1143	April 26, 2021
Mender rootfs artifact deploy not starting General Discussions	11	1148	June 15, 2021
Firmware deployment failed due to failed signature verification General Discussions yocto , morty , deployment	5	965	October 19, 2021
Failure updating with Mender General Discussions yocto	5	861	January 22, 2022

Mender-Artifact deployment error

Related topics