Bad status 500 server error after failed update

General Discussions
1

Hi,

I somehow got myself into a state where I’m seeing message from mender client after a failed update attempt:

reporting status failed, bad status 500 server error message: internal error
  • Running mender version 2.1.0 on an old release of yocto (sumo).
  • Using hosted mender
  • Reason for failed update is unknown.
  • Have completed mender -install and mender -commit successfully on the same device

Here are the full journalctl mender logs after a mender -check-update attempt

Sep 22 15:35:50 baidnn mender[501]: time="2020-09-22T15:35:50Z" level=info msg="forced wake-up from sleep" module=state
Sep 22 15:35:50 baidnn mender[501]: time="2020-09-22T15:35:50Z" level=info msg="State transition: update-retry-report [none] -> update-status-report [none]" module=mender
Sep 22 15:35:50 baidnn mender[501]: time="2020-09-22T15:35:50Z" level=error msg="got unexpected HTTP status when reporting status: 500" module=client_status
Sep 22 15:35:50 baidnn mender[501]: time="2020-09-22T15:35:50Z" level=error msg="error reporting update status: (request_id: ): reporting status failed, bad status 500 server error message: internal error" module=mender
Sep 22 15:35:50 baidnn mender[501]: time="2020-09-22T15:35:50Z" level=error msg="failed to send status to server: transient error: (request_id: ): reporting status failed, bad status 500 server error message: internal error" module=state
Sep 22 15:35:50 baidnn mender[501]: time="2020-09-22T15:35:50Z" level=info msg="State transition: update-status-report [none] -> update-retry-report [none]" module=mender
Sep 22 15:36:20 baidnn mender[501]: time="2020-09-22T15:36:20Z" level=info msg="forced wake-up from sleep" module=state
Sep 22 15:36:20 baidnn mender[501]: time="2020-09-22T15:36:20Z" level=info msg="State transition: update-retry-report [none] -> update-status-report [none]" module=mender
Sep 22 15:36:20 baidnn mender[501]: time="2020-09-22T15:36:20Z" level=error msg="got unexpected HTTP status when reporting status: 500" module=client_status
Sep 22 15:36:20 baidnn mender[501]: time="2020-09-22T15:36:20Z" level=error msg="error reporting update status: (request_id: ): reporting status failed, bad status 500 server error message: internal error" module=mender
Sep 22 15:36:20 baidnn mender[501]: time="2020-09-22T15:36:20Z" level=error msg="failed to send status to server: transient error: (request_id: ): reporting status failed, bad status 500 server error message: internal error" module=state
Sep 22 15:36:20 baidnn mender[501]: time="2020-09-22T15:36:20Z" level=info msg="State transition: update-status-report [none] -> update-retry-report [none]" module=mender

Here’s my mender.conf minus tenant token:

{
    "InventoryPollIntervalSeconds": 1800,
    "RetryPollIntervalSeconds": 300,
    "RootfsPartA": "/dev/mmcblk0p1",
    "RootfsPartB": "/dev/mmcblk0p29",
    "ServerURL": "https://hosted.mender.io",
    "TenantToken": "--removed--",
    "UpdatePollIntervalSeconds": 1800
}

The only similar conversations I could find were at [Self-hosted Mender 1.7] Deployment Stalls at 99% and SSL CA certificate setup on mender production, both of which don’t appear to relate to my setup.

I already tried removing the device from the mender server and then re-adding, same result.

Any suggestions about what might be happening and how I could recover from this?

2

@merlin something for you, mby?

3

@dwalkes We had a short outage in the storage backend on Hosted Mender (around the time this was posted) which may have caused this issue.
Could you please try again and see if it persists?