Failed to report status: EOF

Hello!

A collegue of mine is suffering from a problem. His device reproducably rolls back deployments when it receives them in his home. Deployment seems to work fine first, but when booting into the new artifact, it cannot report status to the server.

Here’s the error message:

Failed to report status: Put "https://hosted.mender.io/api/devices/v1/deployments/device/deployments/0bf971b7-58fc-4457-b413-45342e1617dd/status": EOF

I would like to verify what this error message means exactly.

I read it as:

  • Network connection is up
  • Device could set up a TLS connection to hosted.mender.io
  • Device sends an HTTP PUT, but instead of a response it receives only EOF.

But I don’t thank that to be true. It would mean something specifically eats the http data inside a TLS connection. That’s hard to believe, given that nothing should be able to peek inside the TLS stream.

Given that error message, can I even trust that network was operational when that https PUT was sent?

Thanks,
Manuel

After the rollback, these two errors are printed into the log:

error: Got unexpected HTTP status when reporting status: 500
error: statescript: Can not send start status to server: (request_id: ): reporting status failed, bad status 500 server error message: failed to parse server response: json: cannot unmarshal object into Go struct field .error of type string

The problem just occured again, and this time I observed something rather interesting:

The deployment already showed up as “successful” at the mender web gui. So I thought that we couldn’t reproduce the problem anymore. My collegue insisted to reboot his device, and indeed, it did a rollback then. When switching to the mender web gui browser tab again, the deployment jumped from “successful” to “failed”.

Sounds like the server received the status update, but the device is unable to understand that. Which version of Mender client is it?

And is this Hosted Mender, or self-hosted? It could be useful to check the server logs of the api-gateway and deployments services.

Hi kacf! We are using hosted Mender with mender client 2.5.0.

I wonder if this could be related to this fix. It was fixed quite recently, and has not been included in any release yet. Mender client 3.1.0 is right around the corner though, so you could try that, or if you have the chance to patch the client by yourself, you could also try the patch I linked to above.

Oh wait, my mistake, it is included in Mender client 2.5.2! So I encourage you to try that!

We upgraded the mender client to 2.5.2 (which reports as 2.6.0) and still observe these EOF errors, but in contrast to 2.5.0 they don’t seem to cause a rollback anymore. Still, I’d like to find out what causes them, as this might cause other problems in the long run.

Can you provide some background info on the EOF error? Does it mean the http request received an empty reply? Or was the reply just not parsable by mender?

The collegue suffering from that problem is living at some rentable fully-furnished temporary home with a nonstandard internet connection. Don’t yet have complete information, but it seems to be some hotel-style internet connection. I can’t help myself but think there’s causality rather than correlation at play here.