Issues with upgrading from 1.7.0 to 2.1.0

General Discussions
1

I’m having the same issue after the update from mender 1.7.0 to 2.1.0.
I’m using the latest release of mender server and mender-client 2.1.0 for yocto warrior.

2019-10-03 08:32:32 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-10-03 08:32:32 +0000 UTC debug: handle update install state
2019-10-03 08:32:32 +0000 UTC debug: status reported, response 204 No Content
2019-10-03 08:32:32 +0000 UTC debug: Read data from device manifest file: device_type=MYDEVICENAME
2019-10-03 08:32:32 +0000 UTC debug: Current manifest data: MYDEVICENAME
2019-10-03 08:32:32 +0000 UTC info: Update Module path “/usr/share/mender/modules/v3” could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-03 08:32:33 +0000 UTC info: installer: authenticated digital signature of artifact
2019-10-03 08:32:33 +0000 UTC debug: checking if device [MYDEVICENAME] is on compatible device list: [MYDEVICENAME]
2019-10-03 08:32:33 +0000 UTC error: Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: handleHeaderReads: Cannot load handler for unknown Payload type ‘rootfs-image’
2019-10-03 08:32:33 +0000 UTC info: State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]
2019-10-03 08:32:33 +0000 UTC debug: handle fetch install retry state
2019-10-03 08:32:33 +0000 UTC debug: wait 1m0s before next fetch/install attempt
2019-10-03 08:33:33 +0000 UTC debug: wait complete
2019-10-03 08:33:33 +0000 UTC info: State transition: fetch-install-retry-wait [Download_Enter] -> update-fetch [Download_Enter]
2019-10-03 08:33:33 +0000 UTC debug: handle update fetch state
2019-10-03 08:33:33 +0000 UTC debug: status reported, response 204 No Content

I’m using this values, so probably it’s causing issues:

{
“ArtifactVerifyKey”: “/etc/mender/artifact-verify-key.pem”,
“InventoryPollIntervalSeconds”: 28800,
“RetryPollIntervalSeconds”: 300,
“ServerURL”: “https://MY_SERVER_URL”,
“TenantToken”: “dummy”,
“UpdatePollIntervalSeconds”: 1800
}

Do you have suggestions?

2

Hi,

I think you are being affected by a feature in warrior to split mender.conf into persistent and transient parts. See this commit.

The easiest, is just to disable the feature as the first Changelog indicates.

Please let me know if this fixes your issue!

We will soon release a warrior tag, and for it we will document better this change that affects upgrading from yocto to warrior.

3

See also the related ticket MEN-2757.

4

I had the same issue also from 2.1.0 to 2.1.0.

There are some possible scenarios:

  • update from an existing 1.7.0 to a 2.1.0
  • create a new device with 2.1.0 and update to another firmware always with 2.1.0
  • update a device with 2.1.0 to a firmware with a future release of mender (for instance 3.0.0)
  • update an older device with 1.7.0 to a firmware with a future release of mender (for instance 3.0.0)

What should I do in these cases.
If I disable the split I’ll be ok forever? Or should I be prepared to handle different updates if the starting version is different? Because If I have an old device with 1.7.0 it’s common to update directly to the latest version (with mender 3.0.0) instead of passing to 2.1.0 and then to 3.0.0.

5

I updated my device to firmware 2.1.0-alpha2 from 2.1.0-alpha1. Both warrior firmwares have mender client 2.1.0 inside.
After the error above, now the current version on mender server contains “_INCONSISTENT”.

6

If I disable the split I’ll be ok forever? Or should I be prepared to handle different updates if the starting version is different? Because If I have an old device with 1.7.0 it’s common to update directly to the latest version (with mender 3.0.0) instead of passing to 2.1.0 and then to 3.0.0.

You’ll be ok in warrior, but we don’t guarantee that it will be supported in later branches, so it’s advised to perform the migration. In general we don’t support skipping minor versions; this applies to both the Mender software and the Yocto branches.

7

I’m trying to understand which is the best way to manage some possible situation, so I have another question.

If I have this situation:

  • device with my firmware 3.0.0 (with client 1.7.0
  • device with my firmware 4.0.0 (with client 2.1.0)

Let say that I want to release 4.1.0 (still with client 2.1.0), I should build 2 firmwares: 4.1.0 to upgrade devices already at 4.0.0 and 4.1.0-with-split to upgrade older devices still at 3.0.0.

If yes, mender server covers this scenario showing warnings or other stuff? Because If I release an update to devices without migration I can have issues.

PS: what do you mean with “we don’t support skipping minor versions”?

Thanks

8

That’s what I would recommend, yes.

PS: what do you mean with “we don’t support skipping minor versions”?

It means we don’t test going from 2.1.0 to 2.3.0, but we do test going from 2.1.0 to 2.2.0. In the vast majority of cases, both upgrades scenarios will work correctly though, it’s just that we don’t guarantee that it does. The best policy is just to always stay up to date!

But this particular case is not so much about the Mender client version, but about the Yocto branch. Going from 1.7.0 til 2.1.0 using only thud would work just fine, AFAIK. It is going from thud to warrior that introduces the new mender-split-config default. Yocto branches are not guaranteed to remain compatible with each other (the Yocto Project doesn’t guarantee this either), but we try to provide workarounds for such cases, and the migration is one of them.

9

Yes, I’m agree with you. I always want to stay up to date in particular for security reasons. However, for embedded devices that require software certifications it’s extremely difficult to release a firmware update every mender release. So It’s really common to skip some versions. I’m saying this only to give a feedback and to explain my scenario where updates are checked and approved by other companies very slowly.

Let’s recap a little bit:

  • Now I have many devices with mender 1.7.0, the first recommended thing that I should do to update to warrior with mender 2.1.0 is to create a “.mender” update able to split configuration to do the migration. To do that I have to build a “.mender” file with all default values because ‘split-mender-config’ is enabled by default.
  • Next, all future updates of warrior firmwares must disable the split building a new “.mender” file with this line in local.conf: "PACKAGECONFIG_remove = “split-mender-config”.

PS: I suggest to write a very detailed post about this topic when you decide to release the final version, because It’s not so easy to understand all cases and implication for companies and some examples could be really useful :+1: .

10

To continue the discussion, I’m doing some random experiments based on this commit text Fix and strict check for mender-migrate-configuration MEN-2757 · mendersoftware/meta-mender@1da44a8 · GitHub but I’m still confused.

My info:
mender server: 2.1.0
current client: mender 1.7.0 thud branch
firmware that I’m trying to release: mender 2.1.0 warrior branch

In every combination using mender server 2.1.0 and with or without package-split and/or adding IMAGE_INSTALL_append = " mender-migrate-configuration" and MENDER_PERSISTENT_CONFIGURATION_VARS = "RootfsPartA RootfsPartB to the mender file on warrior, I’m receiving this error message in “journalctl -u mender” on my CURRENT firmware with mender 1.7.0:

level=error msg=“update install failed: installer: failed to read and install update: read install failed: installer: failed to read and install update: reader: unsupported version: 3” module=state

Next I see “level=info msg=“State transition: update-store [Download] → fetch-install-retry-wait [Download] → fetch-install-retry-wait [Download]” module=mender” and again it restarts with the previous message. It seems blocked in a loop.

What does it mean?

11

Mender 1.7 requires mender-artifact version 2. It does not understand version 3 artifacts. You can force the older artifact version for the intermediate install by setting:

PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"

in your local.conf file. Once you have migrated then you can remove that and start to use the new features of mender-artifact v3.

12

I added those lines in local.conf as you suggested:

PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"

But it appears this while building:

NOTE: Resolving any missing task queue dependencies
NOTE: preferred version 2.0.% of mender-artifact-native not available (for item mender-artifact-native)
NOTE: versions of mender-artifact-native available: 2.4.0 2.4.1 3.0.0 3.0.1 3.1.0 master-gitAUTOINC+77326b288c

I also tried to add and/or remove this in my local.conf “PACKAGECONFIG_remove = " split-mender-config”" but the problem about artifact version persists. In my rootfs /etc/mender/script/version there is always “3”, also after a “bitbake -c clean mender && bitbake -c cleansstate mender && bitbake -c my-image && bitbake -c cleansstate my-image”.

The only way that I found to build a version 2 artifact with mender 2.1.0 warrior is modifying .mender file with mender-artifact with “-v 2” option.
Is it a bug or I’m missing something else?

13

I found an issue about caching. In particular about .conf files. If I add “PACKAGECONFIG_remove = " split-mender-config”" to my local.conf file after other previous builds, I expect that /data/mender/mender.conf doesn’t exist and /etc/mender/mender.conf should exists.
However, /data/mender/mender.conf always exists and it contains this:
{
“RootfsPartA”: “/dev/mmcblk1p2”,
“RootfsPartB”: “/dev/mmcblk1p3”
}
And my /data/mender/mender.conf contains:
{
“ArtifactVerifyKey”: “/etc/mender/artifact-verify-key.pem”,
“InventoryPollIntervalSeconds”: 28800,
“RetryPollIntervalSeconds”: 300,
“ServerURL”: “MY_SERVER_URL”,
“TenantToken”: “dummy”,
“UpdatePollIntervalSeconds”: 1800
}
To only way to restore the right behaviour is to run “bitbake -c clean mender && bitbake -c cleansstate mender”.

14

Also I want to recap a little bit because I’m going crazy. I’m still getting failed updates.

To create a firmware to upgrade an existing device with mender thud 1.7.0 to mender warrior 2.1.0 I have to add these lines in local.conf:

PACKAGECONFIG_remove = " split-mender-config"
MENDER_PERSISTENT_CONFIGURATION_VARS = "RootfsPartA RootfsPartB"
IMAGE_INSTALL_append = " mender-migrate-configuration"
PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"

This will generate a firmware without split (so no /data/mender/mender.conf in /tmp/work/…/rootfs) but able to do the split on the device while installing (??? or split won’t be executed???). On my device I cannot find any mender.conf in /data. I’m not sure if this is the right behaviour or not.
To be able to create a .mender with artifact version 2, I used mender-artifact to modify it with -v 2.
In /etc/mender/scripts/version the values changes from 2 to 3. (is it correct???).

Now I can create updates for the device with split enabled, so to create other updates on warrior branch, my local.conf should be with all these disabled (commented):

#PACKAGECONFIG_remove = " split-mender-config"
#MENDER_PERSISTENT_CONFIGURATION_VARS = "RootfsPartA RootfsPartB"
#IMAGE_INSTALL_append = " mender-migrate-configuration"
#PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
#PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"

And the final .mender must have artifact version = 3 (is it right?)
The important thing is that a device with mender 1.7.0 thud MUST install an update with the migration enabled before to install other firmwares on warrior with 2.1.0.
Is it correct? Is this the expected configuration and behaviour?

The first update completed successfully, so I migrated to mender 2.1.0 warrior.
However the next update on mender 2.1.0 warrior to again mender 2.1.0 warrior without adding “PACKAGECONFIG_remove = " split-mender-config”" failed with this report:

2019-10-09 10:39:04 +0000 UTC info: Running Mender version 2.1.0
2019-10-09 10:39:04 +0000 UTC debug: handle update fetch state
2019-10-09 10:39:04 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 10:39:05 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Accept-Ranges:[bytes] Content-Length:[90674688] Content-Security-Policy:[block-all-mixed-content] Content-Type:[application/vnd.mender-artifact] Date:[Wed, 09 Oct 2019 10:39:05 GMT] Etag:["6a7183d8dab3938d68c6199c3e6580ef"] Last-Modified:[Wed, 09 Oct 2019 10:08:29 GMT] Server:[Minio/RELEASE.2018-11-22T02-51-56Z (linux; amd64)] Vary:[Origin] X-Amz-Request-Id:[15CBF456078A5847] X-Minio-Deployment-Id:[7440c984-0cb7-41ab-9e65-c77bf1a96b2a] X-Xss-Protection:[1; mode=block]] 0x240e990 90674688 [] false false map[] 0x2476880 0x2467560}+
2019-10-09 10:39:05 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-10-09 10:39:05 +0000 UTC debug: handle update install state
2019-10-09 10:39:05 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 10:39:05 +0000 UTC debug: Read data from device manifest file: device_type=mydevice-staging
2019-10-09 10:39:05 +0000 UTC debug: Current manifest data: mydevice-staging
2019-10-09 10:39:05 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-09 10:39:05 +0000 UTC info: installer: authenticated digital signature of artifact
2019-10-09 10:39:05 +0000 UTC debug: checking if device [mydevice-staging] is on compatible device list: [mydevice-staging]
2019-10-09 10:39:05 +0000 UTC debug: installer: successfully read artifact [name: 2.1.0-alpha.8; version: 3; compatible devices: [mydevice-staging]]
2019-10-09 10:39:05 +0000 UTC debug: Trying to install update of size: 813694976
2019-10-09 10:39:05 +0000 UTC debug: Active partition: /dev/mmcblk1p3
2019-10-09 10:39:05 +0000 UTC debug: Detected inactive partition /dev/mmcblk1p2, based on active partition /dev/mmcblk1p3
2019-10-09 10:39:05 +0000 UTC info: native sector size of block device /dev/mmcblk1p2 is 512, we will write in chunks of 1048576
2019-10-09 10:39:06 +0000 UTC info: opening device /dev/mmcblk1p2 for writing
2019-10-09 10:39:06 +0000 UTC info: partition /dev/mmcblk1p2 size: 813694976
2019-10-09 10:43:10 +0000 UTC info: wrote 813694976/813694976 bytes of update to device /dev/mmcblk1p2
2019-10-09 10:43:12 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 10:43:12 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2019-10-09 10:43:12 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2019-10-09 10:43:12 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:12 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:12 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 10:43:12 +0000 UTC debug: Inactive partition: /dev/mmcblk1p2
2019-10-09 10:43:12 +0000 UTC debug: Marking inactive partition (/dev/mmcblk1p2) as the new boot candidate.
2019-10-09 10:43:12 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2019-10-09 10:43:12 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 10:43:12 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 10:43:12 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 10:43:12 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 10:43:12 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-10-09 10:43:12 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2019-10-09 10:43:12 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:12 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:12 +0000 UTC debug: handling reboot state
2019-10-09 10:43:13 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 10:43:13 +0000 UTC info: rebooting device(s)
2019-10-09 10:43:13 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk1p3
2019-10-09 10:43:52 +0000 UTC info: Running Mender version 2.1.0
2019-10-09 10:43:52 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-09 10:43:52 +0000 UTC error: Dual rootfs configuration not found when resuming update. Recovery may fail.
2019-10-09 10:43:52 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2019-10-09 10:43:52 +0000 UTC error: transient error: Stub module: Cannot execute ArtifactVerifyReboot
2019-10-09 10:43:53 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2019-10-09 10:43:53 +0000 UTC debug: transitioning to error state
2019-10-09 10:43:53 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:53 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:53 +0000 UTC info: performing rollback
2019-10-09 10:43:53 +0000 UTC error: Unable to roll back with a stub module, but will try to reboot to restore state
2019-10-09 10:43:53 +0000 UTC debug: will try to rollback reboot the device
2019-10-09 10:43:53 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2019-10-09 10:43:53 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:53 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:43:53 +0000 UTC info: rebooting device(s) after rollback
2019-10-09 10:44:18 +0000 UTC info: Running Mender version 2.1.0
2019-10-09 10:44:18 +0000 UTC info: Update was interrupted in state: rollback-reboot
2019-10-09 10:44:18 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-09 10:44:18 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2019-10-09 10:44:18 +0000 UTC debug: transitioning to error state
2019-10-09 10:44:18 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 10:44:18 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 10:44:19 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 10:44:19 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 10:44:19 +0000 UTC debug: Have U-Boot variable: upgrade_available=0
2019-10-09 10:44:19 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:0]
2019-10-09 10:44:19 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2019-10-09 10:44:19 +0000 UTC debug: handling state after rollback reboot
2019-10-09 10:44:19 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2019-10-09 10:44:19 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:44:19 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:44:19 +0000 UTC debug: handle update error state
2019-10-09 10:44:19 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2019-10-09 10:44:19 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:44:19 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:44:19 +0000 UTC debug: Handling Cleanup state
2019-10-09 10:44:19 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-10-09 10:44:19 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 10:44:19 +0000 UTC debug: handle update status report state
2019-10-09 10:44:19 +0000 UTC error: failed to report status: Put https://MY_SERVER_URL/api/devices/v1/deployments/device/deployments/6b932a5d-c1c4-4229-bd82-dc7df4d509a3/status: dial tcp: lookup MY_SERVER_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 10:44:19 +0000 UTC error: error reporting update status: reporting status failed: Put https://MY_SERVER_URL/api/devices/v1/deployments/device/deployments/6b932a5d-c1c4-4229-bd82-dc7df4d509a3/status: dial tcp: lookup MY_SERVER_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 10:44:19 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://MY_SERVER_URL/api/devices/v1/deployments/device/deployments/6b932a5d-c1c4-4229-bd82-dc7df4d509a3/status: dial tcp: lookup MY_SERVER_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 10:44:19 +0000 UTC info: State transition: update-status-report [none] -> update-retry-report [none]
2019-10-09 10:48:18 +0000 UTC debug: SIGUSR1 signal received.
2019-10-09 10:48:18 +0000 UTC debug: Sent wake up!
2019-10-09 10:48:18 +0000 UTC info: forced wake-up from sleep
2019-10-09 10:48:18 +0000 UTC info: State transition: update-retry-report [none] -> update-status-report [none]
2019-10-09 10:48:18 +0000 UTC debug: handle update status report state
2019-10-09 10:48:20 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 10:48:20 +0000 UTC debug: attempting to upload deployment logs for failed update
15

And these changes where applied when you are on the warrior branch right?

able to do the split on the device while installing (??? or split won’t be executed???). On my device I cannot find any mender.conf in /data

The split should occure during the updates utilizing a state-script.

To be able to create a .mender with artifact version 2, I used mender-artifact to modify it with -v 2.

You can add the following to local.conf to achieve the same result:

MENDER_ARTIFACT_EXTRA_ARGS_append = " -v 2"

Now I can create updates for the device with split enabled, so to create other updates on warrior branch, my local.conf should be with all these disabled (commented):

That is correct.

2019-10-09 10:43:52 +0000 UTC error: Dual rootfs configuration not found when resuming update. Recovery may fail.

It seems that the migration did not succeed. Do you have a /data/mender/mender.conf file at this point?

16

One way to further debug this would be to try to extract the ArtifactCommit_Enter_10_migrate-configuration script from the .mender when you have the following enabled:

PACKAGECONFIG_remove = " split-mender-config"
MENDER_PERSISTENT_CONFIGURATION_VARS = "RootfsPartA RootfsPartB"
IMAGE_INSTALL_append = " mender-migrate-configuration"
PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"

You can just run tar xvf <path to .mender> to be able to access the state-script.

You can try running this script “manually” to see if there are errors in it or if it successfully is able to migrate the data.

17

This is correct behavior based on looking at this part of the code

18

I now suspect that this is where it went wrong for you, and you might have not re-added the migration state-script when you re-created the Mender Artifact with -v 2

19

yes

Ok, so It didn’t work.

Thank you for the trick, it’s very useful.

Not right now, but I will try again the procedure from scratch also using the MENDER_ARTIFACT_EXTRA_ARGS_append = " -v 2" and I’ll post the result.

Very useful.

Good point, Ok, so I retry to follow your suggestion and I’ll post the results again in some hours.

20

I tried to add this in my local conf

PACKAGECONFIG_remove = " split-mender-config"
MENDER_PERSISTENT_CONFIGURATION_VARS = "RootfsPartA RootfsPartB"
IMAGE_INSTALL_append = " mender-migrate-configuration"
PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"
MENDER_ARTIFACT_EXTRA_ARGS_append = " -v 2"

with also the suggested MENDER_ARTIFACT_EXTRA_ARGS_append = " -v 2"
I signed the .mender file using the sign command, instead of write with -v 2.
It worked! I migrated from 1.7.0 to 2.1.0 warrior with success. The split worked as expected.
These are my config files after the migration from 1.7.0 to 2.1.0:
/etc/mender/mender.conf:

{
  "ArtifactVerifyKey": "/etc/mender/artifact-verify-key.pem",
  "InventoryPollIntervalSeconds": 28800,
  "RetryPollIntervalSeconds": 300,
  "ServerURL": "MY_SERVER",
  "TenantToken": "dummy",
  "UpdatePollIntervalSeconds": 1800
}

and /data/mender/mender.conf is:

{
  "RootfsPartA": "/dev/mmcblk1p2",
  "RootfsPartB": "/dev/mmcblk1p3"
}

If I run mount at this point, I get that the current partition is /dev/mmcblk1p3. It’s ok.

Then I tried to upgrade again to a 2.1.0 warrior firmware with artifact version 3 with all this stuff commented:

#PACKAGECONFIG_remove = " split-mender-config"
#MENDER_PERSISTENT_CONFIGURATION_VARS = "RootfsPartA RootfsPartB"
#IMAGE_INSTALL_append = " mender-migrate-configuration"
#PREFERRED_VERSION_pn-mender-artifact = "2.0.%"
#PREFERRED_VERSION_pn-mender-artifact-native = "2.0.%"
#MENDER_ARTIFACT_EXTRA_ARGS_append = " -v 2"

But the result is a failure with rollback. This is the log:

2019-10-09 14:11:32 +0000 UTC info: Running Mender version 2.1.0
2019-10-09 14:11:32 +0000 UTC debug: handle update fetch state
2019-10-09 14:11:32 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 14:11:32 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Accept-Ranges:[bytes] Content-Length:[90675712] Content-Security-Policy:[block-all-mixed-content] Content-Type:[application/vnd.mender-artifact] Date:[Wed, 09 Oct 2019 14:11:32 GMT] Etag:["cae6a22392cad63831f072a4ab3d0d97"] Last-Modified:[Wed, 09 Oct 2019 14:11:05 GMT] Server:[Minio/RELEASE.2018-11-22T02-51-56Z (linux; amd64)] Vary:[Origin] X-Amz-Request-Id:[15CBFFEE0A849925] X-Minio-Deployment-Id:[7440c984-0cb7-41ab-9e65-c77bf1a96b2a] X-Xss-Protection:[1; mode=block]] 0x200e820 90675712 [] false false map[] 0x2076880 0x2067560}+
2019-10-09 14:11:32 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-10-09 14:11:33 +0000 UTC debug: handle update install state
2019-10-09 14:11:33 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 14:11:33 +0000 UTC debug: Read data from device manifest file: device_type=mydevice-staging
2019-10-09 14:11:33 +0000 UTC debug: Current manifest data: mydevice-staging
2019-10-09 14:11:33 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-09 14:11:33 +0000 UTC info: installer: authenticated digital signature of artifact
2019-10-09 14:11:33 +0000 UTC debug: checking if device [mydevice-staging] is on compatible device list: [mydevice-staging]
2019-10-09 14:11:33 +0000 UTC debug: installer: processing script: ArtifactCommit_Enter_10_migrate-configuration
2019-10-09 14:11:33 +0000 UTC debug: installer: successfully read artifact [name: 2.1.0-alpha.6; version: 3; compatible devices: [mydevice-staging]]
2019-10-09 14:11:33 +0000 UTC debug: Trying to install update of size: 813694976
2019-10-09 14:11:33 +0000 UTC debug: Active partition: /dev/mmcblk1p3
2019-10-09 14:11:33 +0000 UTC debug: Detected inactive partition /dev/mmcblk1p2, based on active partition /dev/mmcblk1p3
2019-10-09 14:11:33 +0000 UTC info: native sector size of block device /dev/mmcblk1p2 is 512, we will write in chunks of 1048576
2019-10-09 14:11:33 +0000 UTC info: opening device /dev/mmcblk1p2 for writing
2019-10-09 14:11:33 +0000 UTC info: partition /dev/mmcblk1p2 size: 813694976
2019-10-09 14:15:38 +0000 UTC info: wrote 813694976/813694976 bytes of update to device /dev/mmcblk1p2
2019-10-09 14:15:39 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 14:15:39 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2019-10-09 14:15:39 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2019-10-09 14:15:39 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:15:39 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:15:40 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 14:15:40 +0000 UTC debug: Inactive partition: /dev/mmcblk1p2
2019-10-09 14:15:40 +0000 UTC debug: Marking inactive partition (/dev/mmcblk1p2) as the new boot candidate.
2019-10-09 14:15:40 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2019-10-09 14:15:40 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 14:15:40 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 14:15:40 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 14:15:40 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 14:15:40 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-10-09 14:15:40 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2019-10-09 14:15:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:15:40 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:15:40 +0000 UTC debug: handling reboot state
2019-10-09 14:15:40 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 14:15:40 +0000 UTC info: rebooting device(s)
2019-10-09 14:15:40 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk1p3
2019-10-09 14:16:20 +0000 UTC info: Running Mender version 2.1.0
2019-10-09 14:16:20 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-09 14:16:20 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: upgrade_available=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:1]
2019-10-09 14:16:20 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2019-10-09 14:16:20 +0000 UTC debug: handling state after reboot
2019-10-09 14:16:20 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]
2019-10-09 14:16:20 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:20 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:20 +0000 UTC debug: start executing script: ArtifactCommit_Enter_10_migrate-configuration
2019-10-09 14:16:20 +0000 UTC error: failed to report status: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:20 +0000 UTC error: statescript: can not send start status to server: reporting status failed: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:20 +0000 UTC error: failed to report status: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:20 +0000 UTC error: statescript: can not send finished status to server: reporting status failed: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:20 +0000 UTC error: transient error: error calling enter script for (error) update-commit state: error running enter state script(s) for ArtifactCommit_Enter state: statescript: error executing 'ArtifactCommit_Enter_10_migrate-configuration': 1 : exit status 1
2019-10-09 14:16:20 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> rollback [ArtifactRollback]
2019-10-09 14:16:20 +0000 UTC debug: transitioning to error state
2019-10-09 14:16:20 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:20 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:20 +0000 UTC info: performing rollback
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: upgrade_available=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:1]
2019-10-09 14:16:20 +0000 UTC debug: Active partition: /dev/mmcblk1p2
2019-10-09 14:16:20 +0000 UTC debug: Detected inactive partition /dev/mmcblk1p3, based on active partition /dev/mmcblk1p2
2019-10-09 14:16:20 +0000 UTC debug: Marking inactive partition (/dev/mmcblk1p3) as the new boot candidate.
2019-10-09 14:16:20 +0000 UTC info: setting partition for rollback: 3
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 14:16:20 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 14:16:20 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 14:16:20 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-10-09 14:16:20 +0000 UTC debug: will try to rollback reboot the device
2019-10-09 14:16:20 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2019-10-09 14:16:20 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:20 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:20 +0000 UTC info: rebooting device(s) after rollback
2019-10-09 14:16:20 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk1p2
2019-10-09 14:16:47 +0000 UTC info: Running Mender version 2.1.0
2019-10-09 14:16:47 +0000 UTC info: Update was interrupted in state: rollback-reboot
2019-10-09 14:16:47 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2019-10-09 14:16:47 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2019-10-09 14:16:47 +0000 UTC debug: transitioning to error state
2019-10-09 14:16:47 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-10-09 14:16:47 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-10-09 14:16:47 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-10-09 14:16:47 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-10-09 14:16:47 +0000 UTC debug: Have U-Boot variable: upgrade_available=0
2019-10-09 14:16:47 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:0]
2019-10-09 14:16:47 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2019-10-09 14:16:47 +0000 UTC debug: handling state after rollback reboot
2019-10-09 14:16:47 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2019-10-09 14:16:47 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:47 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:47 +0000 UTC debug: handle update error state
2019-10-09 14:16:47 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2019-10-09 14:16:47 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:47 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:47 +0000 UTC debug: Handling Cleanup state
2019-10-09 14:16:47 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-10-09 14:16:48 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-10-09 14:16:48 +0000 UTC debug: handle update status report state
2019-10-09 14:16:48 +0000 UTC error: failed to report status: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:48 +0000 UTC error: error reporting update status: reporting status failed: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:48 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://MY_WEBSITE_URL/api/devices/v1/deployments/device/deployments/343c0962-6dbe-4f55-b364-0286fe5a2dbe/status: dial tcp: lookup MY_WEBSITE_URL on [2001:4860:4860::8888]:53: dial udp [2001:4860:4860::8888]:53: connect: network is unreachable
2019-10-09 14:16:48 +0000 UTC info: State transition: update-status-report [none] -> update-retry-report [none]
2019-10-09 14:17:44 +0000 UTC debug: SIGUSR1 signal received.
2019-10-09 14:17:44 +0000 UTC debug: Sent wake up!
2019-10-09 14:17:44 +0000 UTC info: forced wake-up from sleep
2019-10-09 14:17:44 +0000 UTC info: State transition: update-retry-report [none] -> update-status-report [none]
2019-10-09 14:17:44 +0000 UTC debug: handle update status report state
2019-10-09 14:17:46 +0000 UTC debug: status reported, response 204 No Content
2019-10-09 14:17:46 +0000 UTC debug: attempting to upload deployment logs for failed update