Already installed, device failed to update

Hi,
All my updates fail with the “already installed” message.

I use mender convert 2.6.0 and I have image artifact_info different in /etc/mender/artifact_info between the new image and the image installed on the PLC.
Any clues on what I might be doing wrong ?

Is it potentially related to this warning ?
root@unipi-s103-2:~# mender show-artifact
WARN[0000] Could not resolve path link: root.overlay Attempting to continue
gruener-afce431-mender

Hello @gabrielSoudry ,

Your screenshot does not show already installed (they would be skipped). This looks like a failure and you should check the View Log button (post here if you need help).

Skipped / already installed is when Artifact name is the same on your device and the Artifact you are trying to deploy. /etc/mender/artifact_info is deprecated, this is now stored in a binary database on the device (/data/mender/mender_state IIRC). To resolve that the simplest is to deploy some noop dummy update (e.g. a script Update Module that runs echo or something to that effect). Then deploy your real update and it should get installed.

Hope this helps.

Hello @eystein
Thank you very much for your reply, it is indeed an “already installed” it’s just when I click on it in the history it just says failed :

Logs: 
2022-02-15 15:13:12 +0000 UTC info: Running Mender client version: 3.2.0
2022-02-15 15:13:13 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2022-02-15 15:13:13 +0000 UTC info: No public key was provided for authenticating the artifact
2022-02-15 15:13:13 +0000 UTC info: Opening device "/dev/mmcblk0p3" for writing
2022-02-15 15:13:13 +0000 UTC info: Native sector size of block device /dev/mmcblk0p3 is 512 bytes. Mender will write in chunks of 1048576 bytes
2022-02-15 15:18:32 +0000 UTC info: All bytes were successfully written to the new partition
2022-02-15 15:18:32 +0000 UTC info: The optimized block-device writer wrote a total of 860 frames, where 676 frames did need to be rewritten (i.e., skipped)
2022-02-15 15:18:32 +0000 UTC info: Wrote 900759552/900759552 bytes to the inactive partition
2022-02-15 15:18:33 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2022-02-15 15:18:33 +0000 UTC info: State transition: update-after-store [Download_Leave] -> mender-update-control-refresh-maps [none]
2022-02-15 15:18:33 +0000 UTC info: Validating the Update Info: https://s3.amazonaws.com/hosted-mender-artifacts/614b5452e57ff9c652547348/7804ea84-f2dc-4bea-a93d-64e30130bbef?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXXXXXXXX_request&X-Amz-Date=20220215T151833Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=4613ba0fa553e52d300ab12c13e81aea77f67c4c736c89ed20cab469e098a7ab [name: gruener-gabriel_test2-mender; devices: [gruener]]
2022-02-15 15:18:33 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2022-02-15 15:18:33 +0000 UTC info: State transition: mender-update-control [none] -> update-install [ArtifactInstall]
2022-02-15 15:18:33 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2022-02-15 15:18:33 +0000 UTC info: State transition: update-install [ArtifactInstall] -> mender-update-control-refresh-maps [none]
2022-02-15 15:18:33 +0000 UTC info: Validating the Update Info: https://s3.amazonaws.com/hosted-mender-artifacts/614b5452e57ff9c652547348/7804ea84-f2dc-4bea-a93d-64e30130bbef?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXXXXXXXXXXX2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220215T151833Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=4613ba0fa553e52d300ab12c13e81aea77f67c4c736c89ed20cab469e098a7ab [name: gruener-gabriel_test2-mender; devices: [gruener]]
2022-02-15 15:18:33 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2022-02-15 15:18:33 +0000 UTC info: State transition: mender-update-control [none] -> reboot [ArtifactReboot_Enter]
2022-02-15 15:18:34 +0000 UTC info: Rebooting device(s)
2022-02-15 15:18:34 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2022-02-15 15:18:34 +0000 UTC error: error forwarding from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF
2022-02-15 15:18:34 +0000 UTC warning: error while sending close message: write tcp [::1]:39869->[::1]:54382: use of closed network connection
2022-02-15 15:18:34 +0000 UTC info: Daemon terminated with SIGTERM
2022-02-15 15:19:05 +0000 UTC info: Running Mender client version: 3.2.0
2022-02-15 15:19:05 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2022-02-15 15:19:05 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2022-02-15 15:19:05 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> mender-update-control-refresh-maps [none]
2022-02-15 15:19:05 +0000 UTC info: Device unauthorized; attempting reauthorization
2022-02-15 15:19:05 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2022-02-15 15:19:07 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2022-02-15 15:19:07 +0000 UTC info: Local proxy started
2022-02-15 15:19:07 +0000 UTC info: Reauthorization successful
2022-02-15 15:19:07 +0000 UTC error: transient error: The deployment was aborted from the server
2022-02-15 15:19:07 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> rollback [ArtifactRollback]
2022-02-15 15:19:07 +0000 UTC info: Performing rollback
2022-02-15 15:19:07 +0000 UTC info: Rolling back to the inactive partition (2).
2022-02-15 15:19:07 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2022-02-15 15:19:07 +0000 UTC info: Rebooting device(s) after rollback
2022-02-15 15:19:07 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p3
2022-02-15 15:19:08 +0000 UTC info: Daemon terminated with SIGTERM
2022-02-15 15:19:35 +0000 UTC info: Running Mender client version: 3.2.0
2022-02-15 15:19:35 +0000 UTC info: Mender shut down in state: rollback-reboot
2022-02-15 15:19:35 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2022-02-15 15:19:35 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2022-02-15 15:19:35 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2022-02-15 15:19:35 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2022-02-15 15:19:35 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2022-02-15 15:19:35 +0000 UTC info: Device unauthorized; attempting reauthorization
2022-02-15 15:19:35 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2022-02-15 15:19:36 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2022-02-15 15:19:36 +0000 UTC info: Local proxy started
2022-02-15 15:19:36 +0000 UTC info: Reauthorization successful

For the binary :
/data/mender/mender_state IIRC
I don’t have this file on my PLC:

root@unipi-s103-2:/data/mender# ls       
deployments.0001.1ad00d82-c929-4780-bf8a-5894ac1f7516.log  mender-agent.pem
deployments.0002.d79ed734-2fdd-4b6b-93d1-9f833e113493.log  mender-store
deployments.0003.98a61060-177f-41ba-8443-b4b1f9ce071e.log  mender-store-lock
deployments.0004.e88054a3-1c80-44f1-b8ac-ea30c0e8d033.log  mender.conf
deployments.0005.0d57d5df-2377-4ff6-8ffb-96efcdf18cef.log  scripts
device_type

PS : Thanks for the tips with the Update Module script :slight_smile:

Hi @gabrielSoudry ,

I don’t quite understand what is happening here: the screenshot says Already installed but in the logs the Mender client is processing the update. If it’s “Already installed” then it should not process the update. Are these two from the same deployment?

Sorry, the correct name for the client internal database is “mender-store”. You should never need to do this in a real environment, but for testing / developing you can remove mender-store and mender-store-lock then restart the client. This will “reset” the client (make it forget all the state, including installed software).

What I think may be the solution to your issue is to upgrade to Mender client 3.2.1. I see you are running Mender 3.2.0 and you may be hitting this bug: [MEN-5421] [mender-client] gets rate-limited by the mender-server - Northern.tech AS Even though I do not see any 429 errors in your log, this is what I would try first.

1 Like

Hi,
Yes it’s the same update, what is a little strange is that it completes the download of the update before saying that it is the same … I will try to update my mender client thank you. Maybe it’s related to the fact that my raspberry pi is in read-only mode ? (only the data partition is persisted on my raspberry)

Hi,
Thank you very much, roll back the client to 3.1.0 fixes the problem. I am not able to install the mender client in 3.2.1 using mender_convert, it installs mender 3.2.0 even with “MENDER_CLIENT_VERSION=latest” any idea?

Same problem : Update fails with "The deployment was aborted from the server" - #3 by bao-eng

Hello

Yes,

Upgrade to mender-convert 2.6.2. See release notes here. This version includes a fix for mender-convert to look for the client packages in the new repositories, so then it will correctly resolve “latest” to 3.2.1

1 Like

Great, thank you very much @lluiscampos and @eystein :tada:

Update : it does not work also on mender 3.2.1 but only in MENDER_CLIENT_VERSION=“3.1.0”, I will use this version until Mender keeps rolloing back after update is fix

1 Like