Connection issue with mender server

General Discussions
1

I am using raspberry pi 4 and trying to implement OTA with mender. I have some problem on reporting status to mender. Here are some error log.

– Logs begin at Fri 2020-10-23 10:22:45 HKT, end at Fri 2020-10-23 10:27:15 HKT. –
Oct 23 10:22:48 raspberrypi systemd[1]: Started Mender OTA update service.
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“Loaded configuration file: /var/lib/mender/mender.conf”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“Loaded configuration file: /etc/mender/mender.conf”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“Mender running on partition: /dev/mmcblk0p2”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“State transition: init [none] -> init [none]”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“Handling loaded state: rollback-reboot”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“Running Mender client version: 2.4.0”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“Mender shut down in state: rollback-reboot”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg="State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“State transition: update-error [ArtifactFailure] -> cleanup [Error]”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“State transition: cleanup [Error] -> update-status-report [none]”
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=error msg="Failed to report status: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/cea14727-f9a9-49d1-a
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=error msg="Failed to report status: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/cea14727-f9a9-49d1-a
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=error msg="error reporting update status: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/d
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=error msg="error reporting update status: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/d
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=error msg="Failed to send status to server: transient error: reporting status failed: Put https://hosted.mender.io/api/devices/v1/d
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=error msg="Failed to send status to server: transient error: reporting status failed: Put https://hosted.mender.io/api/devices/v1/d
Oct 23 10:22:54 raspberrypi mender[431]: time=“2020-10-23T10:22:54+08:00” level=info msg=“State transition: update-status-report [none] -> update-retry-report [none]”

2

That looks like a network connectivity issue. Are you sure your Pi has proper networking when in this situation?
Drew

3

I think already have connected to internet. I pretty sure for that

4

Well, when you do an OTA deployment of a full system update, the network settings may be overwritten if you have not managed to move them to the data partition. The entire root filesystem is replaced in this case so you need to arrange to move bits into the data partition that you need to persist across updates. Exactly how you do that will depend on what OS you are running and what network manager you are using.
Drew

5

Thanks for the answer, I am using raspbian on pi4 is there any methods to move bits into the data partition. Also, there is another question when I using mender-convert to convert the latest raspbian and cannot boot the converted image.

6

2020-10-28 06:35:39 +0000 UTC info: Running Mender client version: 2.4.0
2020-10-28 06:35:40 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2020-10-28 06:35:41 +0000 UTC info: No public key was provided for authenticating the artifact
2020-10-28 06:35:41 +0000 UTC info: Opening device “/dev/mmcblk0p3” for writing
2020-10-28 06:35:41 +0000 UTC info: Native sector size of block device /dev/mmcblk0p3 is 512 bytes. Mender will write in chunks of 1048576 bytes
2020-10-28 06:54:55 +0000 UTC info: All bytes were successfully written to the new partition
2020-10-28 06:54:55 +0000 UTC info: The optimized block-device writer wrote a total of 4234 frames, where 42 frames did need to be rewritten (i.e., skipped)
2020-10-28 06:54:56 +0000 UTC info: Wrote 4439666688/4439666688 bytes to the inactive partition
2020-10-28 06:54:56 +0000 UTC error: Failed to report status: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: read tcp 192.165.1.143:33496->34.196.112.130:443: read: connection reset by peer
2020-10-28 06:54:56 +0000 UTC error: error reporting update status: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: read tcp 192.165.1.143:33496->34.196.112.130:443: read: connection reset by peer
2020-10-28 06:54:56 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2020-10-28 06:54:56 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2020-10-28 06:54:57 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2020-10-28 06:54:57 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2020-10-28 06:54:57 +0000 UTC info: Rebooting device(s)
2020-10-28 06:54:57 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2020-02-13 16:19:41 +0000 UTC info: Running Mender client version: 2.4.0
2020-02-13 16:19:41 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2020-02-13 16:19:41 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2020-02-13 16:19:41 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]
2020-02-13 16:19:41 +0000 UTC error: Failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:19:41 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:19:41 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:19:41 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2020-02-13 16:20:11 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2020-02-13 16:20:11 +0000 UTC error: Failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:20:11 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:20:11 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:20:11 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2020-02-13 16:20:41 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2020-02-13 16:20:41 +0000 UTC error: Failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:20:41 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:20:41 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:20:41 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2020-02-13 16:21:11 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2020-02-13 16:21:11 +0000 UTC error: Failed to report status: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:21:11 +0000 UTC error: error reporting update status: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:21:11 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put https://docker.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp 192.168.1.1:443: connect: network is unreachable
2020-02-13 16:21:11 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2020-02-13 16:21:11 +0000 UTC error: transient error: Tried sending status report maximum number of times.
2020-02-13 16:21:11 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> rollback [ArtifactRollback]
2020-02-13 16:21:11 +0000 UTC info: Performing rollback
2020-02-13 16:21:11 +0000 UTC info: Setting partition for rollback: 2
2020-02-13 16:21:11 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2020-02-13 16:21:11 +0000 UTC info: Rebooting device(s) after rollback
2020-02-13 16:21:11 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p3
2020-10-28 06:55:09 +0000 UTC info: Running Mender client version: 2.4.0
2020-10-28 06:55:09 +0000 UTC info: Mender shut down in state: rollback-reboot
2020-10-28 06:55:09 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2020-10-28 06:55:09 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2020-10-28 06:55:09 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2020-10-28 06:55:09 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2020-10-28 06:55:09 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2020-10-28 06:55:10 +0000 UTC error: Failed to report status: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp: lookup hosted.mender.io on 192.165.1.1:53: dial udp 192.165.1.1:53: connect: network is unreachable
2020-10-28 06:55:10 +0000 UTC error: error reporting update status: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp: lookup hosted.mender.io on 192.165.1.1:53: dial udp 192.165.1.1:53: connect: network is unreachable
2020-10-28 06:55:10 +0000 UTC error: Failed to send status to server: transient error: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/83c86c68-3713-49f9-b16c-61a8a7663a76/status: dial tcp: lookup hosted.mender.io on 192.165.1.1:53: dial udp 192.165.1.1:53: connect: network is unreachable
2020-10-28 06:55:10 +0000 UTC info: State transition: update-status-report [none] -> update-retry-report [none]
2020-10-28 06:58:04 +0000 UTC info: State transition: update-retry-report [none] -> update-status-report [none]

The latest error log

7

As @drewmoseley mentioned, check your networking setup and configuration.

the private ipv4 address 192.168.1.1:443 cannot be connected prior to rollback for docker.mender.io
the public ip4 address 192.165.1.1 cannot be connected to and is not serving DNS on port 53 even if you could connect after rollback

8

We don’t have anything as a stock feature because it will be so dependent on your setup. I think raspbian uses Network Manager and if you move the /etc/NetworkManager/system-connections directory to somewhere under /data and then make a symlink from the original location it will work.