Hello there, I have been updating my beaglebone via debian update on Mender Hosted Demo, and I’m having a really high rate of failure for debian updates. Tagging @tranchitella because this is a repost from a support thread.
An example log of the common failure(s) can be found in the attached text file (renamed to yaml so I could upload it, sorry!).
deb_failure_bbb.yaml (6.3 KB)
The highlights are:
1. It seems the device rebooted in ArtifactCommit, but this is unrelated to your modified Update Module, since it doesn’t contain that state
a. I do have the NeedsArtifactReboot state/option set to Automatic, and it should reboot once the Debian is installed. Looking at the timestamps, and knowing that our gateway takes about thirty seconds to a minute to reboot, I’m thinking it’s rebooting as I would have expected here – is that an incorrect assumption on when the reboot should occur? Looking at the logs, is it possible that Mender is just handling the expected reboot poorly on startup? I’ve provided the logs I think are interesting below. At 14:55:04 it definitely begins the reboot process, and then a little less than thirty seconds later at 14:55:32, the client starts back up again and everything after the startup message is within the same second. Is there something we can add to the ArtifactCommit state like a sync or sleep that might help?
2021-05-20 14:55:04 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2021-05-20 14:55:04 +0000 UTC info: Rebooting device(s)
2021-05-20 14:55:32 +0000 UTC info: Running Mender client version: 2.3.1
2021-05-20 14:55:32 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2021-05-20 14:55:32 +0000 UTC info: Running Mender client version: 2.3.1
2021-05-20 14:55:32 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2021-05-20 14:55:32 +0000 UTC error: Mender shut down in state: after-reboot
2021-05-20 14:55:32 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> update-commit [ArtifactCommit_Enter]
2. According to the logs, the deb update module is not available anymore, are you shipping it as part of your package?
a. No, the only Mender file we edit on the gateway with our Module Updates is a Mender Inventory file – more details on what we do touch are in item 4. Is it possible that Mender is misconfigured and the deb failure is a waterfall effect of a different missing dependency?
3. We can also see networking issues in contacting the server, and this makes it hard to understand what triggered what.
a. Are these networking issues server side? I noticed that the errors mention ‘server misbehaving’. This gateway is connected directly to my router and hasn’t had other networking issues that I’m aware of, however if there is a connectivity check you’d like me to run to verify, I have no problem doing that.
4. Can you please provide us more information about the content of the package you are installing?
a. Sure thing; we are installing our own binary that runs as a service on the gateway. I’ll add the list of files/directories we touch for your reference:
i. /usr/local/bin – this is where we house the binaries called by our service
ii. /usr/local/env – we hold some files here for configuration use
iii. /usr/lib/systemd/system – we modify our service file here
iv. /etc/artis – our own directory for data, this is symlinked to the persistent /data partition
v. /usr/local/man/man1 – we add our documentation here
vi. /etc/dhcp/dhclient-enter-hooks.d – for ethernet configurations
vii. /etc/init.d – custom initialization script lives here
viii. /usr/share/mender/inventory – add a custom inventory file that holds data we send to Mender
After pulling together the above responses, I also monitored an update via serial terminal – as soon as I was able to log in, I checked if the file /usr/share/mender/modules/v3/deb was there, and it was – and shortly after the update failed saying it doesn’t exist. I did notice that the error doesn’t include a state to check when calling the file, is it possible there is a file of states getting checked during the process that is missing instead?
Edit: forgot to post the current deb file contents:
ubuntu@artis-mender:~$ cat /usr/share/mender/modules/v3/deb
#!/bin/sh
set -e
STATE="$1"
FILES="$2"
case "$STATE" in
ArtifactInstall)
yes | dpkg -i "$FILES"/files/*.deb
;;
NeedsArtifactReboot)
echo "Automatic"
;;
esac
exit 0