2019-04-29 13:25:26 +0000 UTC info: Running Mender version 2.0.0b1
2019-04-29 13:25:26 +0000 UTC debug: handle update fetch state
2019-04-29 13:25:26 +0000 UTC debug: status reported, response 204 No Content
2019-04-29 13:25:27 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Date:[Tue, 30 Apr 2019 13:25:54 GMT] Content-Type:[application/vnd.mender-artifact] Accept-Ranges:[bytes] Etag:["775fd42bf8c16b5d63b4ce75f026bb19"] X-Xss-Protection:[1; mode=block] Server:[openresty/1.13.6.2] Last-Modified:[Tue, 30 Apr 2019 13:19:39 GMT] X-Amz-Request-Id:[159A436C82E1FD16] X-Frame-Options:[DENY] X-Content-Type-Options:[nosniff] Content-Length:[5120] Connection:[keep-alive] Content-Security-Policy:[block-all-mixed-content] Vary:[Origin] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload]] 0x15843a0 5120 [] false false map[] 0x14a0580 0x177e120}+
2019-04-29 13:25:27 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-04-29 13:25:27 +0000 UTC debug: handle update install state
2019-04-29 13:25:27 +0000 UTC debug: status reported, response 204 No Content
2019-04-29 13:25:27 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-04-29 13:25:27 +0000 UTC debug: Found needed line: device_type=raspberrypi3
2019-04-29 13:25:27 +0000 UTC debug: Current manifest data: raspberrypi3
2019-04-29 13:25:27 +0000 UTC info: no public key was provided for authenticating the artifact
2019-04-29 13:25:27 +0000 UTC debug: checking if device [raspberrypi3] is on compatibile device list: [raspberrypi3]
2019-04-29 13:25:27 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-04-29 13:25:27 +0000 UTC debug: Returning artifact name 20190430-install-ntp_INCONSISTENT from database.
2019-04-29 13:25:27 +0000 UTC debug: Read data from device manifest file: artifact_name=unknown
2019-04-29 13:25:27 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-04-29 13:25:27 +0000 UTC debug: Found needed line: device_type=raspberrypi3
2019-04-29 13:25:27 +0000 UTC debug: Current manifest data: raspberrypi3
2019-04-29 13:25:30 +0000 UTC debug: installer: successfully read artifact [name: 20190430-install-ntp; version: 3; compatible devices: [raspberrypi3]]
2019-04-29 13:25:30 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-04-29 13:25:30 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-04-29 13:25:30 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-04-29 13:25:30 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-04-29 13:25:30 +0000 UTC debug: Executing ModuleInstaller.SupportsRollback
2019-04-29 13:25:30 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command SupportsRollback /var/lib/mender/modules/v3/payloads/0000/tree
2019-04-29 13:25:30 +0000 UTC debug: Module does not support rollback
2019-04-29 13:25:31 +0000 UTC debug: status reported, response 204 No Content
2019-04-29 13:25:31 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2019-04-29 13:25:31 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2019-04-29 13:25:31 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-04-29 13:25:31 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-04-29 13:25:33 +0000 UTC debug: status reported, response 204 No Content
2019-04-29 13:25:33 +0000 UTC debug: Executing ModuleInstaller.InstallUpdate
2019-04-29 13:25:33 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command ArtifactInstall /var/lib/mender/modules/v3/payloads/0000/tree
2019-04-29 13:25:33 +0000 UTC info: Update module output: Reading package lists...
2019-04-29 13:25:33 +0000 UTC info: Update module output:
2019-04-29 13:25:33 +0000 UTC info: Update module output: Building dependency tree...
2019-04-29 13:25:33 +0000 UTC info: Update module output:
2019-04-29 13:25:33 +0000 UTC info: Update module output: Reading state information...
2019-04-29 13:25:33 +0000 UTC info: Update module output:
2019-04-29 13:25:34 +0000 UTC info: Update module output: The following additional packages will be installed:
2019-04-29 13:25:34 +0000 UTC info: Update module output: libopts25
2019-04-29 13:25:34 +0000 UTC info: Update module output: Suggested packages:
2019-04-29 13:25:34 +0000 UTC info: Update module output: ntp-doc
2019-04-29 13:25:35 +0000 UTC info: Update module output: The following NEW packages will be installed:
2019-04-29 13:25:35 +0000 UTC info: Update module output: libopts25 ntp
2019-04-29 13:29:42 +0000 UTC info: Running Mender version 2.0.0b1
2019-04-29 13:29:42 +0000 UTC error: Update was interrupted in state: update-install
2019-04-29 13:29:42 +0000 UTC debug: Setting artifact name to 20190430-install-ntp_INCONSISTENT
2019-04-29 13:29:42 +0000 UTC info: State transition: init [none] -> update-error [ArtifactFailure]
2019-04-29 13:29:42 +0000 UTC debug: transitioning to error state
2019-04-29 13:29:42 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-04-29 13:29:42 +0000 UTC debug: handle update error state
2019-04-29 13:29:42 +0000 UTC debug: Executing ModuleInstaller.Failure
2019-04-29 13:29:42 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command ArtifactFailure /var/lib/mender/modules/v3/payloads/0000/tree
2019-04-29 13:29:42 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2019-04-29 13:29:42 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-04-29 13:29:42 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-04-29 13:29:43 +0000 UTC debug: Handling Cleanup state
2019-04-29 13:29:43 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-04-29 13:29:43 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-04-29 13:29:43 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-04-29 13:29:43 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-04-29 13:29:43 +0000 UTC debug: handle update status report state
2019-04-29 13:29:46 +0000 UTC debug: status reported, response 204 No Content
2019-04-29 13:29:46 +0000 UTC debug: attempting to upload deployment logs for failed update
Raspberry stops working and it is not reachable anymore.
Any suggestion?
This message is output when the client starts, and is an indication that Mender was restarted as part of what happened in the update module. This is treated by the client as an error and results in a failed update. Can you think of anything in your script that would kill the mender process?
I’m going to continue this discussion because of another issue I’ve found
I develop a “shell-command” script to update content of a file using following script
#!/bin/bash
touch /usr/share/mender/modules/v3/install-update
cat > /usr/share/mender/modules/v3/install-update << 'ENDOFFILE'
#!/bin/bash
set -e
STATE="$1"
FILES="$2"
# working directory for update
update_dir="/tmp/update"
# files for backup
prev_files=/tmp/prev.tar
# file containing new version
update_file="$FILES"/files/update.tar
case "$STATE" in
NeedsArtifactReboot)
echo "No"
;;
SupportsRollback)
echo "Yes"
;;
ArtifactInstall)
# update file doesn't exist in package
echo "Checking if update package exists..."
if [ ! -f "$update_file" ]; then
echo "Update package does not exist! Exiting..."
exit 1
fi
echo "Update package found!"
## stop services
# Java APP
echo "Stopping EASServer..."
sudo -u pi pm2 stop EASServer
sudo service EASServer stop
echo "done!"
# unpack update file
echo "Extracting Update files..."
sudo rm -rf "$update_dir"
mkdir -p "$update_dir"
sudo tar -xf "$update_file" -C "$update_dir"
echo "done!"
# save previous files
echo "Saving backup for web package..."
sudo tar -cf "$prev_files_web" /home/pi/EASServer/
echo "done!"
# copy new files
sudo cp -r "$update_dir"/* /home/pi/EASServer/
# assign directory to user
echo "Applying permission..."
sudo chown -R pi:pi /home/pi/EASServer/
echo "done!"
# save reference for this upgrade
touch "$prev_files.lock"
;;
ArtifactRollback)
## stop services
# Java APP
echo "Stopping EASServer..."
sudo -u pi pm2 stop EASServer
sudo service EASServer stop
echo "done!"
if [ -f "$prev_files.lock" ]; then
# remove Nodejs app
sudo rm -rf /home/pi/EASServer/
# create directory
sudo mkdir -p /home/pi/EASServer/
# copy new files
sudo tar -xf "$prev_files" -C /
# assign directory to user
sudo chown -R pi:pi /home/pi/EASServer/
fi
## restart services
# Nodejs APP
echo "Starting EASServer..."
sudo -u pi pm2 start EASServer
# Java APP
sudo service EASServer start
echo "done!"
# delete all temporary files
echo "Deleting temporary files..."
sudo rm -f "$prev_files"
sudo rm -f "$prev_files.lock"
echo "done!"
sudo rm -rf "$update_dir"
;;
ArtifactCommit)
echo "Deleting temporary files..."
# delete all temporary files
sudo rm -f "$prev_files"
sudo rm -f "$prev_files.lock"
sudo rm -rf "$update_dir"
echo "done!"
## restart services
# Nodejs APP
echo "Starting EASServer..."
sudo -u pi pm2 start EASServer
# Java APP
sudo service EASServer start
echo "done!"
;;
esac
exit 0
ENDOFFILE
This is error returned by mender client
info: Running Mender version 2.0.0b1
2019-05-08 13:45:35 +0000 UTC debug: handle update fetch state
2019-05-08 13:45:35 +0000 UTC debug: status reported, response 204 No Content
2019-05-08 13:45:35 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Date:[Wed, 08 May 2019 13:45:34 GMT] Content-Security-Policy:[block-all-mixed-content] X-Amz-Request-Id:[159CB923D693EEF7] X-Frame-Options:[DENY] Connection:[keep-alive] Last-Modified:[Wed, 08 May 2019 13:43:12 GMT] Vary:[Origin] X-Content-Type-Options:[nosniff] Server:[openresty/1.13.6.2] Content-Length:[5632] Strict-Transport-Security:[max-age=63072000; includeSubdomains; preload] X-Xss-Protection:[1; mode=block] Content-Type:[application/vnd.mender-artifact] Accept-Ranges:[bytes] Etag:["8a4a574f5f19cefb8bd73262a9ad933c"]] 0x140cb40 5632 [] false false map[] 0x14d4180 0x185c9c0}+
2019-05-08 13:45:35 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-05-08 13:45:35 +0000 UTC debug: handle update install state
2019-05-08 13:45:35 +0000 UTC debug: status reported, response 204 No Content
2019-05-08 13:45:35 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-05-08 13:45:35 +0000 UTC debug: Found needed line: device_type=raspberrypi3
2019-05-08 13:45:35 +0000 UTC debug: Current manifest data: raspberrypi3
2019-05-08 13:45:35 +0000 UTC info: no public key was provided for authenticating the artifact
2019-05-08 13:45:35 +0000 UTC debug: checking if device [raspberrypi3] is on compatibile device list: [raspberrypi3]
2019-05-08 13:45:35 +0000 UTC debug: Executing ModuleInstaller.Initialize
2019-05-08 13:45:35 +0000 UTC debug: Returning artifact name 20190508-update-script_INCONSISTENT from database.
2019-05-08 13:45:35 +0000 UTC debug: Read data from device manifest file: artifact_name=rev-20190502
2019-05-08 13:45:35 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-05-08 13:45:35 +0000 UTC debug: Found needed line: device_type=raspberrypi3
2019-05-08 13:45:35 +0000 UTC debug: Current manifest data: raspberrypi3
2019-05-08 13:45:35 +0000 UTC debug: installer: successfully read artifact [name: 20190508-update-script; version: 3; compatible devices: [raspberrypi3]]
2019-05-08 13:45:35 +0000 UTC debug: Executing ModuleInstaller.PrepareStoreUpdate
2019-05-08 13:45:35 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command Download /var/lib/mender/modules/v3/payloads/0000/tree
2019-05-08 13:45:35 +0000 UTC debug: Executing ModuleInstaller.StoreUpdate
2019-05-08 13:45:35 +0000 UTC debug: Executing ModuleInstaller.FinishStoreUpdate
2019-05-08 13:45:35 +0000 UTC debug: Executing ModuleInstaller.SupportsRollback
2019-05-08 13:45:35 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command SupportsRollback /var/lib/mender/modules/v3/payloads/0000/tree
2019-05-08 13:45:35 +0000 UTC debug: Module does not support rollback
2019-05-08 13:45:36 +0000 UTC debug: status reported, response 204 No Content
2019-05-08 13:45:36 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2019-05-08 13:45:36 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: status reported, response 204 No Content
2019-05-08 13:45:36 +0000 UTC debug: Executing ModuleInstaller.InstallUpdate
2019-05-08 13:45:36 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command ArtifactInstall /var/lib/mender/modules/v3/payloads/0000/tree
2019-05-08 13:45:36 +0000 UTC info: Update module output: /usr/share/mender/modules/v3/shell-command: 12: /usr/share/mender/modules/v3/shell-command: /var/lib/mender/modules/v3/payloads/0000/tree/files/update-script.sh: not found
2019-05-08 13:45:36 +0000 UTC error: Update module terminated abnormally: exit status 127
2019-05-08 13:45:36 +0000 UTC error: transient error: Update module terminated abnormally: exit status 127
2019-05-08 13:45:36 +0000 UTC debug: Setting artifact name to 20190508-update-script_INCONSISTENT
2019-05-08 13:45:36 +0000 UTC info: State transition: update-install [ArtifactInstall] -> update-error [ArtifactFailure]
2019-05-08 13:45:36 +0000 UTC debug: transitioning to error state
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: handle update error state
2019-05-08 13:45:36 +0000 UTC debug: Executing ModuleInstaller.Failure
2019-05-08 13:45:36 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command ArtifactFailure /var/lib/mender/modules/v3/payloads/0000/tree
2019-05-08 13:45:36 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: Handling Cleanup state
2019-05-08 13:45:36 +0000 UTC debug: Executing ModuleInstaller.Cleanup
2019-05-08 13:45:36 +0000 UTC debug: Calling module: /usr/share/mender/modules/v3/shell-command Cleanup /var/lib/mender/modules/v3/payloads/0000/tree
2019-05-08 13:45:36 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-05-08 13:45:36 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-05-08 13:45:36 +0000 UTC debug: handle update status report state
2019-05-08 13:45:36 +0000 UTC debug: status reported, response 204 No Content
2019-05-08 13:45:36 +0000 UTC debug: attempting to upload deployment logs for failed update
Update module output: /usr/share/mender/modules/v3/shell-command: 12: /usr/share/mender/modules/v3/shell-command: /var/lib/mender/modules/v3/payloads/0000/tree/files/update-script.sh: not found
However, I don’t see in your Update Module where do you call update-script.sh. Clearly not in line 12.
Are you sure that you are posting the right code for the module?
There is a misunderstanding here. You are writing your own Update Module, named install-update, but then your Artifact is for another module, shell-command(*). This is wrong, the Artifact shall have the type of the Update Module that needs to handle it.
You need to change your mender-artifact options to use -T install-update.
(*) Side note, shell-command module has been renamed to script from the beta to the final 2.0 release
Dear @lluiscampos,
I’m calling shell-command topic. This script should run as each file included in mender artifact. File in mender artifact is called update-script.sh
Okey, I think I finally understood what are you doing. Your script is trying to create other Update Module called install-update, and there is where I got confused before. Sorry.
I cannot see any obvious error. The update-script.sh file should be there…
If you have remote access to the device, I would try to install this Artifact in standalone mode, so that you can debug a bit. In standalone mode, the file tree will be present in /var/lib/mender/... between install and commit/rollback, so then you have the chance to inspect and manually run the script.
I tried to manually copy update-script.sh to /var/lib/mender/modules/v3/payloads/0000/tree/files/update-script.sh and run it from there.
It works like a charm.
Good. So there must be something related to the Artifact or to the shell-command Update Module (that, as I said, is deprecated and you should use the script one, but that cannot be the issue).
What I suggested is:
First eliminate that script that you have manually created at /var/lib/mender/.../update-script.sh
Eliminate also the Update Module that you have probably created when running the previous script (the file at /usr/share/mender/modules/v3/install-update)
Copy the Artifact to the device with scp
Run mender -install filename, where filename is the name of the Artifact
Now, that should fail at this point. Please copy us here the output.
And depending on the error situation, the payload tree at /var/lib/mender/... might be there, so run a find command on there and copy us here the output
Here is command output.
I have changed mender-client to production version 2.0.0 (no more beta)
I have created a symlink from “script” to “shell-command”
root@XXXX-YYYY-ZZZZ:/home/pi# ls -l /usr/share/mender/modules/v3/
total 24
-rwxr-xr-x 1 root root 140 May 7 13:51 deb
-rwxr-xr-x 1 root root 1378 May 7 13:51 directory
-rwxr-xr-x 1 root root 805 May 7 13:51 docker
-rwxr-xr-x 1 root root 150 May 7 13:51 rpm
-rwxr-xr-x 1 root root 207 May 7 13:51 script
lrwxrwxrwx 1 root root 35 May 16 13:14 shell-command -> /usr/share/mender/modules/v3/script
-rwxr-xr-x 1 root root 1343 May 7 13:51 single-file
Command output
root@XXXX-YYYY-ZZZZ:/home/pi# mender -install update-v2_r0.mender
INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config
INFO[0000] No dual rootfs configuration present module=main
INFO[0000] Start updating from local image file: [update-v2_r0.mender] module=standalone
Installing Artifact of size 5632...
INFO[0000] no public key was provided for authenticating the artifact module=installer
. 100% 5 KiB
INFO[0000] Update module output: /usr/share/mender/modules/v3/shell-command: 12: /usr/share/mender/modules/v3/shell-command: /var/lib/mender/modules/v3/payloads/0000/tree/files/update-v2.sh: not found module=modules
ERRO[0000] Update module terminated abnormally: exit status 127 module=modules
ERRO[0000] Installation failed: Update module terminated abnormally: exit status 127 module=standalone
ERRO[0000] Update module terminated abnormally: exit status 127 module=main
After the error there are no files and also there isn’t path /var/lib/mender/modules/v3/payloads/0000/tree/files/
Path ends at /var/lib/mender/modules/v3/payloads/0000
I think I found the problem. The updatev2 script has Windows-like line endings CRLF, so when executing on the Raspberry Pi it will actually error with:
./updatev2: /bin/bash^M: bad interpreter: No such file or directory
I don’t know, however, how you managed to execute it locally, maybe you were using bash updatev2 instead of just ./updatev2 ?
For a Linux script, you must use Unix endings. Please modify your script, repack the Artifact, and retry. Tip: you can use dos2unix tool for this.