Update is succesfull but mender announces to the server that it failed

I have thud version of mender and server side is version 2.1. When updating I can read from server following error. The same is present at device. Mender does not make roll-back and the artifact_info is the new updated artifact. Where can I start to debug the issue. The error message tells me nothing.

2019-10-18 11:48:19 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2019-10-18 11:50:43 +0000 UTC error: got invalid entrypoint into the state machine: state: reboot
2019-10-18 11:50:43 +0000 UTC info: State transition: init [none] -> update-error [ArtifactFailure]
2019-10-18 11:50:43 +0000 UTC debug: transitioning to error state
2019-10-18 11:50:43 +0000 UTC debug: start executing script: ArtifactFailure_Enter_10
2019-10-18 11:50:43 +0000 UTC error: got unexpected HTTP status when reporting status: 400
2019-10-18 11:50:43 +0000 UTC error: statescript: can not send start status to server: (request_id: ): reporting status failed, bad status 400 server error message: unknown status value
2019-10-18 11:50:46 +0000 UTC error: got unexpected HTTP status when reporting status: 400
2019-10-18 11:50:46 +0000 UTC error: statescript: can not send finished status to server: (request_id: ): reporting status failed, bad status 400 server error message: unknown status value
2019-10-18 11:50:46 +0000 UTC debug: handle update error state
2019-10-18 11:50:46 +0000 UTC info: State transition: update-error [ArtifactFailure] -> update-status-report [none]
2019-10-18 11:50:46 +0000 UTC debug: handle update status report state
2019-10-18 11:50:46 +0000 UTC debug: status reported, response 204 No Content
2019-10-18 11:50:46 +0000 UTC debug: attempting to upload deployment logs for failed update

br.

AP Sihvonen

Typically has one of the following reasons:

  1. There is something wrong with your board integration, e.g the U-Boot environment configuration is not correct. Would recommend you go trough the integration checklist if you have not done that. This usually exposes these type of errors.

  2. Something went wrong during Linux kernel load/boot. These type of errors can typically only be debugged using a serial console attached to the device. But I do not think it is this one, as this would have triggered a proper rollback.

1 Like

My first post. And therefore I’m new to mender of course.

I’ve scanned through the forum for hints on what to do with my problem at hand and this thread is close enough to match my problem.

So I have a RPI3, for which I’ve built an image using Yocto warrior. Everything builds fine. I’m using the demo server to which I upload a new deployment. The update starts just fine, and looking at my RPI3 - I see that i reboots. Unfortunately I don’t have a console at hand so I can’t see what happens until he PI comes back up again.

My problem is that it always comes up with the previous version. I also have good reasons to believe it has indeed booted into the inactive partition and rejected it out of some reason. (My active partition has a static IP-address and my upgrade has an DHCP allocated one. I can see that the PI allocates a DHCP address for a while until it disappears)

The log looks as:

2019-11-08 17:30:40 +0000 UTC debug: Inactive partition: /dev/mmcblk0p3
2019-11-08 17:30:40 +0000 UTC debug: Marking inactive partition (/dev/mmcblk0p3) as the new boot candidate.
2019-11-08 17:30:40 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2019-11-08 17:30:40 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-11-08 17:30:40 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-11-08 17:30:40 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-11-08 17:30:40 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-11-08 17:30:40 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-11-08 17:30:40 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2019-11-08 17:30:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-11-08 17:30:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-11-08 17:30:40 +0000 UTC debug: handling reboot state
2019-11-08 17:30:40 +0000 UTC debug: status reported, response 204 No Content
2019-11-08 17:30:40 +0000 UTC info: rebooting device(s)
2019-11-08 17:30:40 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2019-11-08 17:31:37 +0000 UTC info: Running Mender version 2.1.1
2019-11-08 17:31:37 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]

This would’ve been really simple if there were a reason for the update to not function, but the only difference between the partition that works is the choice of IP addressing.

My questions are:

  • (obviously) Is this is questioned before?
  • Is the only possibility for this to happen that my upgrade image is faulty somehow?
  • Do I have a u-boot problem after all?
  • Is there a better way to debug this?

Welcome to the Mender community @edzeppelin.

  • Is the only possibility for this to happen that my upgrade image is faulty somehow?

The Mender client must be able to connect to the Mender server for the update to succeed, booting the image is not enough to mark it as “successful”. Would be helpful if you posted the full log of a failed deployment and then we can see if it fails on connectivity or something else.

  • Do I have a u-boot problem after all?

Do not have enough information to state this. I suspect that this is not U-Boot related. The RPi3 integration in Yocto is fairly well tested.

  • Is there a better way to debug this?

Serial console will always provide better insights on what is going on.

I decided to buy a serial dongle and my problem turned out to be completely self-inflicted. I’ve forgotten that one of my start-scripts actually restarts the machine at first boot.

The restart is considered as a failed installation and my RPI reverts back to the previous image. (Just as it’s supposed to)

So Mea Culpa. Thanks a lot for your quick reply.

Great, thanks for reporting back