Environment is:
Hosted Mender
Mender Client 2.1.0 running on Thud
We are trying to update to Dunfell using the default Mender Dunfell layers
When we try and update the device, we get this error over and over again:
Mar 04 10:19:08 ifu-25 mender[14270]: time="2021-03-04T10:19:08Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]" module=mender
Mar 04 10:21:08 ifu-25 mender[14270]: time="2021-03-04T10:21:08Z" level=info msg="State transition: fetch-install-retry-wait [Download_Enter] -> update-fetch [Download_Enter]" module=mender
Mar 04 10:21:09 ifu-25 mender[14270]: time="2021-03-04T10:21:09Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]" module=mender
Mar 04 10:21:09 ifu-25 mender[14270]: time="2021-03-04T10:21:09Z" level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available" module=modules
Mar 04 10:21:09 ifu-25 mender[14270]: time="2021-03-04T10:21:09Z" level=info msg="installer: authenticated digital signature of artifact" module=installer
Mar 04 10:21:09 ifu-25 mender[14270]: time="2021-03-04T10:21:09Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: type_info provides values not yet supported" module=state
Mar 04 10:21:09 ifu-25 mender[14270]: time="2021-03-04T10:21:09Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]" module=mender
Mar 04 10:25:09 ifu-25 mender[14270]: time="2021-03-04T10:25:09Z" level=info msg="State transition: fetch-install-retry-wait [Download_Enter] -> update-fetch [Download_Enter]" module=mender
Mar 04 10:25:11 ifu-25 mender[14270]: time="2021-03-04T10:25:11Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]" module=mender
Mar 04 10:25:11 ifu-25 mender[14270]: time="2021-03-04T10:25:11Z" level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available" module=modules
Mar 04 10:25:11 ifu-25 mender[14270]: time="2021-03-04T10:25:11Z" level=info msg="installer: authenticated digital signature of artifact" module=installer
Mar 04 10:25:11 ifu-25 mender[14270]: time="2021-03-04T10:25:11Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: type_info provides values not yet supported" module=state
Mar 04 10:25:11 ifu-25 mender[14270]: time="2021-03-04T10:25:11Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]" module=mender
We have tried adding MENDER_ARTIFACT_EXTRA_ARGS = "--no-checksum-provide" to build, and we think that is taking affect.
Original artifact (without --no-checksum-provide):
Artifact after adding --no-checksum-provide
The root-image.checksum value disappears.
Our original images (from Thud) look like this in Mender UI:
I notice they don’t have a rootfs-image.version.
Help? Where do we go from here 
had similar issue recently, try adding the following to MENDER_ARTIFACT_EXTRA_ARGS
--no-default-clears-provides --no-default-software-version --no-checksum-provide
as your using mender-client 2.1.0
2 Likes
Thanks @dellgreen, that works!
I also realised that passing in -v 2 to force a v2 artifact also allow it to update.
But now a different problem… the update is still failing with this:
2021-03-04 15:43:53 +0000 UTC info: Running Mender version 2.1.0
2021-03-04 15:43:53 +0000 UTC debug: handle update fetch state
2021-03-04 15:43:54 +0000 UTC debug: status reported, response 204 No Content
2021-03-04 15:43:54 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Date:[Thu, 04 Mar 2021 15:43:55 GMT] Last-Modified:[Thu, 04 Mar 2021 15:09:30 GMT] Etag:["9df43abfd9798c516ec7a288a146d28d-17"] X-Amz-Request-Id:[GM6HSBDGRRJ52ZQC] Expires:[Thu, 04 Mar 2021 15:19:29 GMT] Accept-Ranges:[bytes] Content-Type:[application/vnd.mender-artifact] Content-Length:[171537920] Server:[AmazonS3] X-Amz-Id-2:[flxVwwV13bSd19wqvYJ2go4wNwcqfGiHBsCYl2SyV6B3iX2ZpYOR4bqvu1WjBCbNoUA9GCfnDA8=]] 0xf2c870 171537920 [] false false map[] 0xf9c180 0xfb8120}+
2021-03-04 15:43:54 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2021-03-04 15:43:54 +0000 UTC debug: handle update install state
2021-03-04 15:43:54 +0000 UTC debug: status reported, response 204 No Content
2021-03-04 15:43:54 +0000 UTC debug: Read data from device manifest file: device_type=orange-pi-pc-plus
2021-03-04 15:43:54 +0000 UTC debug: Current manifest data: orange-pi-pc-plus
2021-03-04 15:43:54 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2021-03-04 15:43:54 +0000 UTC info: installer: authenticated digital signature of artifact
2021-03-04 15:43:54 +0000 UTC debug: checking if device [orange-pi-pc-plus] is on compatible device list: [orange-pi-pc-plus]
2021-03-04 15:43:54 +0000 UTC debug: installer: successfully read artifact [name: ifu-orangepi-265459003-c09d1d71-mbe-test-dunfell; version: 3; compatible devices: [orange-pi-pc-plus]]
2021-03-04 15:43:54 +0000 UTC debug: Trying to install update of size: 687865856
2021-03-04 15:43:54 +0000 UTC debug: Active partition: /dev/mmcblk2p2
2021-03-04 15:43:54 +0000 UTC debug: Detected inactive partition /dev/mmcblk2p3, based on active partition /dev/mmcblk2p2
2021-03-04 15:43:54 +0000 UTC info: native sector size of block device /dev/mmcblk2p3 is 512, we will write in chunks of 1048576
2021-03-04 15:43:54 +0000 UTC info: opening device /dev/mmcblk2p3 for writing
2021-03-04 15:43:54 +0000 UTC info: partition /dev/mmcblk2p3 size: 687865856
2021-03-04 15:46:50 +0000 UTC info: wrote 687865856/687865856 bytes of update to device /dev/mmcblk2p3
2021-03-04 15:46:51 +0000 UTC debug: status reported, response 204 No Content
2021-03-04 15:46:51 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2021-03-04 15:46:51 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2021-03-04 15:46:51 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:46:51 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:46:51 +0000 UTC debug: status reported, response 204 No Content
2021-03-04 15:46:51 +0000 UTC debug: Inactive partition: /dev/mmcblk2p3
2021-03-04 15:46:51 +0000 UTC debug: Marking inactive partition (/dev/mmcblk2p3) as the new boot candidate.
2021-03-04 15:46:51 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2021-03-04 15:46:51 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2021-03-04 15:46:51 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2021-03-04 15:46:51 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2021-03-04 15:46:51 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2021-03-04 15:46:51 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2021-03-04 15:46:51 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2021-03-04 15:46:51 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:46:51 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:46:51 +0000 UTC debug: handling reboot state
2021-03-04 15:46:51 +0000 UTC debug: status reported, response 204 No Content
2021-03-04 15:46:51 +0000 UTC info: rebooting device(s)
2021-03-04 15:46:51 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk2p2
2021-03-04 15:47:15 +0000 UTC info: Running Mender client version: 2.5.0
2021-03-04 15:47:15 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2021-03-04 15:47:15 +0000 UTC error: Dual rootfs configuration not found when resuming update. Recovery may fail.
2021-03-04 15:47:15 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2021-03-04 15:47:15 +0000 UTC error: transient error: Stub module: Cannot execute ArtifactVerifyReboot
2021-03-04 15:47:15 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2021-03-04 15:47:15 +0000 UTC info: Performing rollback
2021-03-04 15:47:15 +0000 UTC error: Unable to roll back with a stub module, but will try to reboot to restore state
2021-03-04 15:47:15 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2021-03-04 15:47:15 +0000 UTC info: Rebooting device(s) after rollback
2021-03-04 15:47:15 +0000 UTC info: Daemon terminated with SIGTERM
2021-03-04 15:47:40 +0000 UTC info: Running Mender version 2.1.0
2021-03-04 15:47:40 +0000 UTC info: Update was interrupted in state: rollback-reboot
2021-03-04 15:47:40 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2021-03-04 15:47:40 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2021-03-04 15:47:40 +0000 UTC debug: transitioning to error state
2021-03-04 15:47:40 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2021-03-04 15:47:40 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2021-03-04 15:47:40 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2021-03-04 15:47:40 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2021-03-04 15:47:40 +0000 UTC debug: Have U-Boot variable: upgrade_available=0
2021-03-04 15:47:40 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:0]
2021-03-04 15:47:40 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2021-03-04 15:47:40 +0000 UTC debug: handling state after rollback reboot
2021-03-04 15:47:40 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2021-03-04 15:47:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:47:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:47:40 +0000 UTC debug: handle update error state
2021-03-04 15:47:40 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2021-03-04 15:47:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:47:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:47:40 +0000 UTC debug: Handling Cleanup state
2021-03-04 15:47:40 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2021-03-04 15:47:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2021-03-04 15:47:40 +0000 UTC debug: handle update status report state
2021-03-04 15:47:40 +0000 UTC error: failed to report status: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/8a24bd91-386a-4173-9f9a-c8d55b4170aa/status: dial tcp: lookup hosted.mender.io on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: socket: address family not supported by protocol
2021-03-04 15:47:40 +0000 UTC error: error reporting update status: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/8a24bd91-386a-4173-9f9a-c8d55b4170aa/status: dial tcp: lookup hosted.mender.io on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: socket: address family not supported by protocol
2021-03-04 15:47:40 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/8a24bd91-386a-4173-9f9a-c8d55b4170aa/status: dial tcp: lookup hosted.mender.io on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: socket: address family not supported by protocol
2021-03-04 15:47:40 +0000 UTC info: State transition: update-status-report [none] -> update-retry-report [none]
2021-03-04 15:52:38 +0000 UTC debug: wait complete
2021-03-04 15:52:38 +0000 UTC info: State transition: update-retry-report [none] -> update-status-report [none]
2021-03-04 15:52:38 +0000 UTC debug: handle update status report state
2021-03-04 15:52:39 +0000 UTC debug: status reported, response 204 No Content
2021-03-04 15:52:39 +0000 UTC debug: attempting to upload deployment logs for failed update
The important bit seems to be:
2021-03-04 15:47:15 +0000 UTC error: Dual rootfs configuration not found when resuming update. Recovery may fail.
2021-03-04 15:47:15 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2021-03-04 15:47:15 +0000 UTC error: transient error: Stub module: Cannot execute ArtifactVerifyReboot
2021-03-04 15:47:15 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2021-03-04 15:47:15 +0000 UTC info: Performing rollback
2021-03-04 15:47:15 +0000 UTC error: Unable to roll back with a stub module, but will try to reboot to restore state
2021-03-04 15:47:15 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2021-03-04 15:47:15 +0000 UTC info: Rebooting device(s) after rollback
This looks like the split-config change.
https://docs.mender.io/troubleshoot/yocto-project-runtime
as I hadn’t deployed my fleet of devices yet, i just factory flashed the devices with the newly updated factory image to proceed, and bypassed the migration steps.
If you updating existing device in the field then you will need to follow the instructions in the docs
Thanks, yes, I found that after Googling
But the steps in the documentation don’t actually work
ERROR: Nothing RPROVIDES 'mender-migrate-configuration' (but /builds/rugged-networks/onvp/ifu/yocto/ifu-orangepi/.build-opi/../sources/meta-rnl/meta-rnl/recipes-core/images/manufacture-ifu-image.bb, /builds/rugged-networks/onvp/ifu/yocto/ifu-orangepi/.build-opi/../sources/meta-rnl/meta-rnl/recipes-core/images/ifu-image.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'mender-migrate-configuration' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['mender-migrate-configuration']
ERROR: Required build target 'manufacture-ifu-image' has no buildable providers.
Missing or unbuildable dependency chain was: ['manufacture-ifu-image', 'mender-migrate-configuration']
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
ERROR: Job failed: exit code 1
Looks like it has been renamed from mender-migrate-configuration to mender-client-migrate-configuration
How do I go about getting the docs updated? Wasted a crazy amount of time on this stuff today and I can’t be the first person.
Hi @jakew009 our docs are in github. We happily take pull requests.
Thanks,
Drew
Thanks @drewmoseley
I’m still not 100% sure what the correct procedure should be, but hopefully if I open a PR, someone can guide me 
I submitted a PR for a few updates to the docs here but I haven’t got time to look into why the tests have failed for it atm
The update has now gone through, but I did notice this line:
Mar 04 17:31:07 ifu-25 mender[333]: time="2021-03-04T17:31:07Z" level=info msg="Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactCommit_Enter_10_migrate-configuration\n/var/lib/mender/scripts/ArtifactCommit_Enter_10_migrate-configuration: line 32: mender-transient.conf: Read-only file system\nmv: can't rename 'mender-transient.conf': Read-only file system\n\n---------- end of script output"
Is that anything to worry about?
The full log was:
Mar 04 17:31:04 ifu-25 systemd[1]: Started Mender OTA update service.
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="Loaded configuration file: /etc/mender/mender.conf"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="Mender running on partition: /dev/mmcblk2p3"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="State transition: init [none] -> init [none]"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="Handling loaded state: reboot"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="Running Mender client version: 2.5.0"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="Update Module path \"/usr/share/mender/modules/v3\" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="State transition: init [none] -> after-reboot [ArtifactReboot_Leave]"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]"
Mar 04 17:31:04 ifu-25 mender[333]: time="2021-03-04T17:31:04Z" level=info msg="State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]"
Mar 04 17:31:07 ifu-25 mender[333]: time="2021-03-04T17:31:07Z" level=info msg="Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactCommit_Enter_10_migrate-configuration\n/var/lib/mender/scripts/ArtifactCommit_Enter_10_migrate-configuration: line 32: mender-transient.conf: Read-only file system\nmv: can't rename 'mender-transient.conf': Read-only file system\n\n---------- end of script output"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="Committing update"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: update-commit [ArtifactCommit_Enter] -> update-after-first-commit [none]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: update-after-first-commit [none] -> update-after-commit [ArtifactCommit_Leave]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: update-after-commit [ArtifactCommit_Leave] -> cleanup [none]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: cleanup [none] -> update-status-report [none]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: update-status-report [none] -> idle [Idle]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: idle [Idle] -> check-wait [Idle]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
Mar 04 17:31:08 ifu-25 mender[333]: time="2021-03-04T17:31:08Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 3"
Mar 04 17:31:09 ifu-25 mender[333]: time="2021-03-04T17:31:09Z" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
Mar 04 17:31:09 ifu-25 mender[333]: time="2021-03-04T17:31:09Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Mar 04 17:31:09 ifu-25 mender[333]: time="2021-03-04T17:31:09Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
Mar 04 18:01:05 ifu-25 mender[333]: time="2021-03-04T18:01:05Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Mar 04 18:01:06 ifu-25 mender[333]: time="2021-03-04T18:01:06Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
