UTC error: error forwarding from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF

Heyo, i’m trying to upload a rootfs image to a raspberry pi, but i encountered an issue which i can’t seem to solve. I tested this image with hosted Mender, and it worked fine. I switched to using my own Mender server with Docker, but when it tries to upload the artifact, it fails with the follwing logs:

2023-08-08 08:59:15 +0000 UTC info: Running Mender client version: 3.5.1
2023-08-08 08:59:15 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2023-08-08 08:59:15 +0000 UTC info: No public key was provided for authenticating the artifact
2023-08-08 08:59:15 +0000 UTC info: Opening device "/dev/mmcblk0p3" for writing
2023-08-08 08:59:15 +0000 UTC info: Native sector size of block device /dev/mmcblk0p3 is 512 bytes. Mender will write in chunks of 1048576 bytes
2023-08-08 09:04:47 +0000 UTC info: All bytes were successfully written to the new partition
2023-08-08 09:04:47 +0000 UTC info: The optimized block-device writer wrote a total of 3565 frames, where 3485 frames did need to be rewritten (i.e., skipped)
2023-08-08 09:04:47 +0000 UTC info: Wrote 3737124864/3737124864 bytes to the inactive partition
2023-08-08 09:04:47 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2023-08-08 09:04:47 +0000 UTC info: State transition: update-after-store [Download_Leave] -> mender-update-control-refresh-maps [none]
2023-08-08 09:04:47 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-08-08 09:04:47 +0000 UTC info: State transition: mender-update-control [none] -> update-install [ArtifactInstall]
2023-08-08 09:04:47 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2023-08-08 09:04:48 +0000 UTC info: State transition: update-install [ArtifactInstall] -> mender-update-control-refresh-maps [none]
2023-08-08 09:04:48 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-08-08 09:04:48 +0000 UTC info: State transition: mender-update-control [none] -> reboot [ArtifactReboot_Enter]
2023-08-08 09:04:48 +0000 UTC info: Rebooting device(s)
2023-08-08 09:04:48 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2023-08-08 09:04:48 +0000 UTC error: error forwarding from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF
2023-08-08 09:04:48 +0000 UTC info: Daemon terminated with SIGTERM
2023-08-08 09:04:48 +0000 UTC warning: error while sending close message: write tcp 127.0.0.1:40309->127.0.0.1:44270: use of closed network connection
2023-07-22 08:17:19 +0000 UTC info: Running Mender client version: 3.5.0
2023-07-22 08:17:19 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2023-07-22 08:17:19 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2023-07-22 08:17:19 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> mender-update-control-refresh-maps [none]
2023-07-22 08:17:19 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-07-22 08:17:19 +0000 UTC info: State transition: mender-update-control [none] -> update-commit [ArtifactCommit_Enter]
2023-07-22 08:17:19 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-07-22 08:17:19 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-07-22 08:17:20 +0000 UTC error: Failed to authorize with "https://eu.hosted.mender.io": authentication request rejected server error message: dev auth: unauthorized
2023-07-22 08:17:20 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-07-22 08:17:20 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-07-22 08:17:20 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-07-22 08:17:20 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-07-22 08:17:20 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-08-08 09:05:58 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2023-08-08 09:05:58 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-08-08 09:05:58 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-08-08 09:05:58 +0000 UTC error: Failed to authorize with "https://eu.hosted.mender.io": authentication request rejected server error message: dev auth: unauthorized
2023-08-08 09:05:58 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-08-08 09:05:58 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-08-08 09:05:58 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-08-08 09:05:58 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-08-08 09:05:58 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-08-08 09:06:28 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2023-08-08 09:06:28 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-08-08 09:06:28 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-08-08 09:06:28 +0000 UTC error: Failed to authorize with "https://eu.hosted.mender.io": authentication request rejected server error message: dev auth: unauthorized
2023-08-08 09:06:28 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-08-08 09:06:28 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-08-08 09:06:28 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-08-08 09:06:28 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-08-08 09:06:28 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-08-08 09:06:58 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2023-08-08 09:06:58 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-08-08 09:06:58 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-08-08 09:06:58 +0000 UTC error: Failed to authorize with "https://eu.hosted.mender.io": authentication request rejected server error message: dev auth: unauthorized
2023-08-08 09:06:58 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-08-08 09:06:58 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-08-08 09:06:58 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-08-08 09:06:58 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-08-08 09:06:58 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-08-08 09:06:58 +0000 UTC error: transient error: Tried sending status report maximum number of times.
2023-08-08 09:06:58 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> rollback [ArtifactRollback]
2023-08-08 09:06:58 +0000 UTC info: Performing rollback
2023-08-08 09:06:59 +0000 UTC info: Rolling back to the inactive partition (2).
2023-08-08 09:06:59 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2023-08-08 09:06:59 +0000 UTC info: Rebooting device(s) after rollback
2023-08-08 09:06:59 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p3
2023-08-08 09:07:00 +0000 UTC info: Daemon terminated with SIGTERM
2023-08-08 09:07:41 +0000 UTC info: Running Mender client version: 3.5.1
2023-08-08 09:07:41 +0000 UTC info: Mender shut down in state: rollback-reboot
2023-08-08 09:07:41 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-08-08 09:07:41 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-08-08 09:07:41 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2023-08-08 09:07:41 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2023-08-08 09:07:41 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2023-08-08 09:07:41 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-08-08 09:07:41 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-08-08 09:07:41 +0000 UTC info: successfully received new authorization data from server https://menderio.duckdns.org
2023-08-08 09:07:41 +0000 UTC info: Local proxy started
2023-08-08 09:07:41 +0000 UTC info: Reauthorization successful

Thanks to anyone who might be able to help

Hi @FearlessMailbox,

Did you try to migrate the device from Hosted to your own server through an update? The websocket related error that you spotted is not the root cause in my opinion, it is this:

2023-08-08 09:06:58 +0000 UTC error: Failed to authorize with "https://eu.hosted.mender.io": authentication request rejected server error message: dev auth: unauthorized

So your device is trying to connect to the EU Hosted Mender instance.

Greetz,
Josef

That might very well be the problem. I’ll report back here in an hour to let you know. Thanks!