How can i roll back the update using this module?
Hi Lucas,
So if you have installed it in standalone mode, and assuming you have not committed it yet, you can rollback the artifact by running mender -rollback.
But if you were using managed mode, the the install/commit was done automatically and the rollback is only performed if there are errors during the process.
I think you will understand this better if you have a look at the Architecture part of our docs, ref: https://docs.mender.io/2.1/architecture/overview
I performed an update for testing and used the command with sudo permissions. I got this:
So, regardless of anything happening, I tried to upload a new artifact again and now I have the following error:
I’m running the pre-build client.
Can you please copy the error log in text? I cannot read the full log from an image.
I performed the authentication procedure again and went back to working normally.
I sent a new artifact, the folder appeared in my home directory but when I run the -rollback command I get this:
sudo mender -rollback
INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config
INFO[0000] No dual rootfs configuration present module=main
ERRO[0000] There is nothing to commit module=standalone
WARN[0000] There is nothing to commit module=main
Still not clear to me what are you trying to achieve.
Are you running mender in managed mode or in standalone mode? If you are running in managed mode you cannot manually perform a rollback. The rollback mechanism will be handled automatically by mender if the installation fails.
Thanks for the reply,
I am running from a locally hosted server. I believe it is managed mode. I now understand that mender automatically handles rollback with this directory module. But what I mean is if there is an error in an application because of the update and not in the installation of the artifact itself, how will Mender handle it? Can Mender somehow at least send me a debug info?
I recommend you have a look at this tutorial on Update Modules, to better understand the rollback mechanism: https://docs.mender.io/2.1/devices/update-modules
The logs are sent to the server, you should be able to find them in the finished deployment.
@bonini_lucas I would also recommend looking into a pre-commit state script that will allow you to plugin your own post-install checks. It’s basically a shell script that you provide that Mender will call and if you detect an error condition, you return the value 1 and the managed mode update will be rolled back. Full details are here: https://docs.mender.io/2.1/artifacts/state-scripts
@lluiscampos @drewmoseley Thanks so much for the support.
I will read these documents. But, is there somewhere where I can find script examples of these cases?
We have some example state scripts that simply return “OK” to show how they can be added to a Yocto build here: https://github.com/mendersoftware/meta-mender/tree/master/meta-mender-demo/recipes-mender/example-state-scripts/files
You can do something similar only having appropriate checks and returning “1” in case they fail.
Hello,
I was performing rollback tests and tried to reproduce this part of the documentation:
You can experiment with the rollback mechanism by forcing the update to fail. For example, creating a directory under
/var/wwwwith the same name as one of the files contained in the incoming Artifact will trigger an error in thecpcommand. Once this happens, the Update Module should restore the previous files.
But it did not work. I got the folder I was ordered to create and on the mender server I got this:
inside the client:
sudo mender -show-artifact
web-file-4.0_INCONSISTENT
Logs:
2019-09-24 18:28:02 +0000 UTC info: Running Mender version 2.1.0
2019-09-24 18:28:02 +0000 UTC debug: handle update fetch state
2019-09-24 18:28:02 +0000 UTC debug: status reported, response 204 No Content
2019-09-24 18:28:03 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Date:[Tue, 24 Sep 2019 18:28:03 GMT] Etag:[“6864c457d91d952e6aeb756d249c2b59”] X-Amz-Request-Id:[15C77338D0B8F668] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload] Vary:[Origin] Connection:[keep-alive] Content-Security-Policy:[block-all-mixed-content] Last-Modified:[Tue, 24 Sep 2019 17:35:59 GMT] X-Content-Type-Options:[nosniff] Server:[openresty/1.13.6.2] Content-Type:[application/vnd.mender-artifact] Content-Length:[5120] Accept-Ranges:[bytes] X-Frame-Options:[DENY]] 0xc0c8e0 5120 false false map 0xe3e280 0xc7ca20}+
2019-09-24 18:28:03 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2019-09-24 18:28:03 +0000 UTC debug: handle update install state
2019-09-24 18:28:03 +0000 UTC debug: status reported, response 204 No Content
2019-09-24 18:28:03 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-09-24 18:28:03 +0000 UTC debug: Current manifest data: raspberrypi3
2019-09-24 18:28:03 +0000 UTC info: no public key was provided for authenticating the artifact
2019-09-24 18:28:03 +0000 UTC debug: checking if device [raspberrypi3] is on compatible device list: [raspberrypi3]
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-09-24 18:28:03 +0000 UTC debug: Returning artifact name web-file-4.0_INCONSISTENT from database.
2019-09-24 18:28:03 +0000 UTC debug: Read data from device manifest file: artifact_name=unknown
2019-09-24 18:28:03 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-09-24 18:28:03 +0000 UTC debug: Current manifest data: raspberrypi3
2019-09-24 18:28:03 +0000 UTC debug: installer: successfully read artifact [name: web-file-4.0; version: 3; compatible devices: [raspberrypi3]]
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.SupportsRollback
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file SupportsRollback /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC debug: Update module output: Yes
2019-09-24 18:28:03 +0000 UTC debug: Module supports rollback
2019-09-24 18:28:03 +0000 UTC debug: status reported, response 204 No Content
2019-09-24 18:28:03 +0000 UTC info: State transition: update-store [Download_Enter] → update-after-store [Download_Leave]
2019-09-24 18:28:03 +0000 UTC info: State transition: update-after-store [Download_Leave] → update-install [ArtifactInstall]
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: status reported, response 204 No Content
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.InstallUpdate
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file ArtifactInstall /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC info: Update module output: rm:
2019-09-24 18:28:03 +0000 UTC info: Update module output: missing operand
2019-09-24 18:28:03 +0000 UTC info: Update module output: Try ‘rm --help’ for more information.
2019-09-24 18:28:03 +0000 UTC error: Update module terminated abnormally: exit status 123
2019-09-24 18:28:03 +0000 UTC error: transient error: Update module terminated abnormally: exit status 123
2019-09-24 18:28:03 +0000 UTC info: State transition: update-install [ArtifactInstall] → rollback [ArtifactRollback]
2019-09-24 18:28:03 +0000 UTC debug: transitioning to error state
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC info: performing rollback
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.Rollback
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file ArtifactRollback /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC error: Update module terminated abnormally: exit status 1
2019-09-24 18:28:03 +0000 UTC error: rollback failed: Update module terminated abnormally: exit status 1
2019-09-24 18:28:03 +0000 UTC error: fatal error: Update module terminated abnormally: exit status 1
2019-09-24 18:28:03 +0000 UTC debug: Setting artifact name to web-file-4.0_INCONSISTENT
2019-09-24 18:28:03 +0000 UTC info: State transition: rollback [ArtifactRollback] → update-error [ArtifactFailure]
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: handle update error state
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.Failure
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file ArtifactFailure /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC info: State transition: update-error [ArtifactFailure] → cleanup [Error]
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: Handling Cleanup state
2019-09-24 18:28:03 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]
2019-09-24 18:28:03 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-24 18:28:03 +0000 UTC debug: handle update status report state
2019-09-24 18:28:03 +0000 UTC debug: status reported, response 204 No Content
2019-09-24 18:28:03 +0000 UTC debug: attempting to upload deployment logs for failed update
What I think is that the rollback seems to be looking for content in an empty folder. And then it shows the artifact as inconsistent, regardless of whether I make new deploys
Hmm. The relevant part of the log appears to be
2019-09-24 18:28:03 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/web-file ArtifactRollback /var/lib/mender/modules/v3/payloads/0000/tree
2019-09-24 18:28:03 +0000 UTC error: Update module terminated abnormally: exit status 1
2019-09-24 18:28:03 +0000 UTC error: rollback failed: Update module terminated abnormally: exit status 1
@lluiscampos do you know why the update module when trying to rollback could exit abnormally? The only thing I see is if the prev_files.tar file is missing it will error out.
Earlier there are these errors also, during ArtifactInstall. Maybe the backup is never created?
@kacf @drewmoseley @bonini_lucas yes, that is what is puzzling me as well (I had some PMs with @bonini_lucas on this), that the actual failure is in the rm command in ArtifactInstall stage, not in the rollback.
It must be a bug in this web-file Update Module. The code is here: https://docs.mender.io/2.1/devices/update-modules#advanced-example-update-module-with-rollback
One obvious error in the code is that the first install always assume there are files in /var/www, which might not be the case. So instead of find /var/www -maxdepth 1 -type f | xargs rm we should first check if there are any files there.
@bonini_lucas, please try replacing that line with something like:
if $(ls /var/www >/dev/null 2>&1); then
find /var/www -maxdepth 1 -type f | xargs rm
if
Better yet, just:
find /var/www -maxdepth 1 -type f | xargs -r rm
Notice the -r argument. Then you don’t need the if.
1 Like