Mender client not rolling back when there is not internet connection

aro86 · July 25, 2019, 2:26pm

I have been using Mender since version 1.5.
When I started the integration of Mender in my product I tested that after an update, if there is not Internet connection Mender Client performs a rollback. This was for me a killing feature, because a device will never install an update that will set it out of connection and so, new updates can been send to it in the future.

Now I am using Mender 2.0. And I see that this behabiur is not working any more, log can be found at the end. I don0’t know if it is a bug, a feature that has been removed or perhaps I am not undestanding something.

Thanks for the help in advance.

mender[203]: level=info msg="Loaded configuration file: /etc/mender/mender.conf" module=config
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="Loaded configuration file: /etc/mender/mender.conf" module=config
mender[203]: level=info msg="Mender running on partition: /dev/mmcblk0p3" module=main
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="Mender running on partition: /dev/mmcblk0p3" module=main
mender[203]: level=info msg="State transition: init [none] -> init [none]" module=mender
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="State transition: init [none] -> init [none]" module=mender
mender[203]: level=info msg="handling loaded state: reboot" module=state
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="handling loaded state: reboot" module=state
mender[203]: level=info msg="Running Mender version 2.0.0" module="deployment_logger"
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="Running Mender version 2.0.0" module="deployment_logger"
mender[203]: level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available" module=modules
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available" module=modules
mender[203]: level=info msg="State transition: init [none] -> after-reboot [ArtifactReboot_Leave]" module=mender
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="State transition: init [none] -> after-reboot [ArtifactReboot_Leave]" module=mender
mender[203]: level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]" module=mender
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]" module=mender
mender[203]: level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58863->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T15:58:25Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58863->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58863->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T15:58:25Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58863->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58863->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T15:58:25Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58863->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T15:58:25Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T15:59:25Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47993->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T15:59:25Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47993->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47993->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T15:59:25Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47993->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47993->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T15:59:25Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47993->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T15:59:25Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:00:26Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:32958->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T16:00:26Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:32958->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:32958->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T16:00:26Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:32958->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:32958->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T16:00:26Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:32958->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:00:26Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:01:26Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57696->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T16:01:26Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57696->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57696->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T16:01:26Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57696->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57696->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T16:01:26Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57696->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:01:26Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:02:26Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40466->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T16:02:26Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40466->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40466->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T16:02:26Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40466->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40466->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T16:02:26Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40466->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:02:26Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:03:26Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58513->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T16:03:26Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58513->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58513->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T16:03:26Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58513->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58513->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T16:03:26Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:58513->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:03:26Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:04:26Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47213->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T16:04:26Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47213->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47213->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T16:04:26Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47213->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47213->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T16:04:26Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:47213->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:04:26Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:05:26Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
mender[203]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57028->[::1]:53: read: connection refused" module="client_status"
mender[203]: time="2018-01-28T16:05:26Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57028->[::1]:53: read: connection refused" module="client_status"
mender[203]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57028->[::1]:53: read: connection refused" module=mender
mender[203]: time="2018-01-28T16:05:26Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57028->[::1]:53: read: connection refused" module=mender
mender[203]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57028->[::1]:53: read: connection refused" module=state
mender[203]: time="2018-01-28T16:05:26Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/c3c5a2a1-68ff-4102-add8-0df83ef33fc5/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:57028->[::1]:53: read: connection refused" module=state
mender[203]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
mender[203]: time="2018-01-28T16:05:26Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender

Edit: @mirzak: formating

kacf · July 26, 2019, 6:42am

What are your RetryPollIntervalSeconds and UpdatePollIntervalSeconds settings in the client configuration? Low values of the former, combined with high values of the latter, can cause the client to retry a lot of times before it gives up. It’s essentially a “UpdatePollIntervalSeconds / RetryPollIntervalSeconds” based formula.

aro86 · July 26, 2019, 7:39am

UpdatePollIntervalSeconds =“18000”
RetryPollIntervalSeconds = "300

So it is like check for OTA every 5h and the retry is like 5m.
I think those times are reasonable.

5h is good for not overloading the server but good for noticing an OTA.
5m to to retry communication with the server when it can be overloaded seems good to me.

In the other hand, if I understand correctly, what you are telling me with the formula is that we are going to perform as much retries as the division of “UpdatePollIntervalSeconds / RetryPollIntervalSeconds” pausing RetryPollIntervalSeconds between retries. So at the end, we are not going to perform a rollback before the time defined in UpdatePollIntervalSeconds, this means that I have to choose a time that either will overload the server but will perform a quick rollback or a long time that will not overload the server but will perform a quickly rollback.

aro86 · July 26, 2019, 8:48am

Here there is a test where the times have been changed and we can see that the rollback is not performed when the update commit could no be done and the UpdatePollIntervalSeconds time has been overpassed (and I have waited several minutes more and the log loop continues like shown).

$ cat /etc/mender/mender.conf
{
  "ClientProtocol": "https",
  "RootfsPartA": "/dev/mmcblk0p2",
  "RootfsPartB": "/dev/mmcblk0p3",
  "UpdatePollIntervalSeconds": 300,
  "InventoryPollIntervalSeconds": 86400,
  "RetryPollIntervalSeconds": 10,
  "ServerURL": "https://mender-development.XXXXXXXXX.com:7443",
  "ArtifactVerifyKey": "XXXX"
}

$ journalctl | grep mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="Loaded configuration file: /etc/mender/mender.conf" module=config
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="Loaded configuration file: /etc/mender/mender.conf" module=config
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="Mender running on partition: /dev/mmcblk0p2" module=main
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="Mender running on partition: /dev/mmcblk0p2" module=main
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="State transition: init [none] -> init [none]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="State transition: init [none] -> init [none]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="handling loaded state: reboot" module=state
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="handling loaded state: reboot" module=state
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="Running Mender version 2.0.0" module="deployment_logger"
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="Running Mender version 2.0.0" module="deployment_logger"
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available" module=modules
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available" module=modules
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="State transition: init [none] -> after-reboot [ArtifactReboot_Leave]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="State transition: init [none] -> after-reboot [ArtifactReboot_Leave]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:54285->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:54285->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:54285->[::1]:53: read: connection refused" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:54285->[::1]:53: read: connection refused" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:54285->[::1]:53: read: connection refused" module=state
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:54285->[::1]:53: read: connection refused" module=state
Jan 28 16:00:23 XXXXXXXXX mender[453]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:23 XXXXXXXXX mender[453]: time="2018-01-28T16:00:23Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:33 XXXXXXXXX mender[453]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:33 XXXXXXXXX mender[453]: time="2018-01-28T16:00:33Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:33 XXXXXXXXX mender[453]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39067->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:00:33 XXXXXXXXX mender[453]: time="2018-01-28T16:00:33Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39067->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:00:33 XXXXXXXXX mender[453]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39067->[::1]:53: read: connection refused" module=mender
Jan 28 16:00:33 XXXXXXXXX mender[453]: time="2018-01-28T16:00:33Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39067->[::1]:53: read: connection refused" module=mender
Jan 28 16:00:33 XXXXXXXXX mender[453]: level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39067->[::1]:53: read: connection refused" module=state
Jan 28 16:00:33 XXXXXXXXX mender[453]: time="2018-01-28T16:00:33Z" level=error msg="Failed to send status report to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39067->[::1]:53: read: connection refused" module=state
Jan 28 16:00:33 XXXXXXXXX mender[453]: level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:33 XXXXXXXXX mender[453]: time="2018-01-28T16:00:33Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:43 XXXXXXXXX mender[453]: level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:43 XXXXXXXXX mender[453]: time="2018-01-28T16:00:43Z" level=info msg="State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]" module=mender
Jan 28 16:00:43 XXXXXXXXX mender[453]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:00:43 XXXXXXXXX mender[453]: time="2018-01-28T16:00:43Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:00:43 XXXXXXXXX mender[453]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused" module=mender


....

Jan 28 16:06:57 XXXXXXXXX mender[453]: level=info msg="State transition: update-retry-report [none] -> update-status-report [none]" module=mender
Jan 28 16:06:57 XXXXXXXXX mender[453]: time="2018-01-28T16:06:57Z" level=info msg="State transition: update-retry-report [none] -> update-status-report [none]" module=mender
Jan 28 16:06:57 XXXXXXXXX mender[453]: level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39718->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:06:57 XXXXXXXXX mender[453]: time="2018-01-28T16:06:57Z" level=error msg="failed to report status: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39718->[::1]:53: read: connection refused" module="client_status"
Jan 28 16:06:57 XXXXXXXXX mender[453]: level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39718->[::1]:53: read: connection refused" module=mender
Jan 28 16:06:57 XXXXXXXXX mender[453]: time="2018-01-28T16:06:57Z" level=error msg="error reporting update status: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39718->[::1]:53: read: connection refused" module=mender
Jan 28 16:06:57 XXXXXXXXX mender[453]: level=error msg="failed to send status to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39718->[::1]:53: read: connection refused" module=state
Jan 28 16:06:57 XXXXXXXXX mender[453]: time="2018-01-28T16:06:57Z" level=error msg="failed to send status to server: transient error: reporting status failed: Put https://mender-development.XXXXXXXXX.com:7443/api/devices/v1/deployments/device/deployments/d244a63e-5620-428c-befa-faaa0dad147c/status: dial tcp: lookup mender-development.XXXXXXXXX.com on [::1]:53: read udp [::1]:39718->[::1]:53: read: connection refused" module=state
Jan 28 16:06:57 XXXXXXXXX mender[453]: level=info msg="State transition: update-status-report [none] -> update-retry-report [none]" module=mender
Jan 28 16:06:57 XXXXXXXXX mender[453]: time="2018-01-28T16:06:57Z" level=info msg="State transition: update-status-report [none] -> update-retry-report [none]" module=mender

Edit: @mirzak: formatting

…

kacf · July 29, 2019, 8:15am

Correct, with a minimum of three retries, but it looks like you’ve got way more than that. I will try to test this later today.

kacf · August 5, 2019, 9:16am

I cannot reproduce this, it rolled back just fine for me. What exactly have you set the intervals to in your latest test?

aro86 · August 5, 2019, 10:17am

$ cat /etc/mender/mender.conf
{
“ClientProtocol”: “https”,
“RootfsPartA”: “/dev/mmcblk0p2”,
“RootfsPartB”: “/dev/mmcblk0p3”,
“UpdatePollIntervalSeconds”: 300,
“InventoryPollIntervalSeconds”: 86400,
“RetryPollIntervalSeconds”: 10,
“ServerURL”: “https://mender-development.XXXXXXXXX.com:7443”,
“ArtifactVerifyKey”: “XXXX”
}

kacf · August 5, 2019, 1:35pm

Thanks, now I reproduced it, I see you added https://tracker.mender.io/browse/MEN-2676, so I added more details there.

aro86 · August 5, 2019, 1:46pm

Great! Thanks

ster · August 5, 2019, 4:17pm

On the tracker, you mentioned that

For existing users, a workaround for this problem is to ensure that the UpdatePollIntervalSeconds / RetryPollIntervalSeconds formula doesn’t produce a high number

Can you please provide the exact number which would allow to stay safe, if possible?

kacf · August 6, 2019, 6:30am

Staying below 5 should definitely be safe.

Our tests indicate that more cycles than this are needed in order to trigger the state loop detection.

mirzak · October 7, 2019, 10:58am

9 posts were split to a new topic: Issues with upgrading from 1.7.0 to 2.1.0

Topic		Replies	Views
Using mender with unreliable internet connection General Discussions	1	263	October 3, 2022
Mender 2.0 update failure because wifi network not immediately available General Discussions	9	1514	June 19, 2019
Unexpected rollback updating device General Discussions rocko	17	1473	April 21, 2020
Wait for network connection before uploading status report General Discussions	12	748	October 1, 2020
Issues with rollback using Directory Update Module General Discussions	15	1404	September 26, 2019

Mender client not rolling back when there is not internet connection

Related Topics