Mender OTA update fails with demo artifact on ubuntu 18.04 x86

Rohita83 · October 29, 2020, 1:49pm

Hi

I am using mender demo server version integration-2.1.0. I have hosted it on AWS EC2 instance.
Mender demo artifact deployment was working up to last week. But it has stopped working suddenly.

I am getting below error for OTA update:

2020-10-29 10:31:32 +0000 UTC info: Running Mender client version: b6dfc02
2020-10-29 10:31:33 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2020-10-29 10:31:34 +0000 UTC info: No public key was provided for authenticating the artifact
2020-10-29 10:31:41 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2020-10-29 10:31:41 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2020-10-29 10:31:41 +0000 UTC info: State transition: update-install [ArtifactInstall] -> update-commit [ArtifactCommit_Enter]
2020-10-29 10:31:45 +0000 UTC info: Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_90_install_systemd_unit
Job for mender-demo-artifact.service failed because the control process exited with error code.
See “systemctl status mender-demo-artifact.service” and “journalctl -xe” for details.

---------- end of script output
2020-10-29 10:31:46 +0000 UTC error: transient error: error executing leave script for update-install state: error running leave state script(s) for ArtifactInstall state: statescript: error executing ‘ArtifactInstall_Leave_90_install_systemd_unit’: 1 : exit status 1
2020-10-29 10:31:46 +0000 UTC info: State transition: update-install [ArtifactInstall] -> rollback [ArtifactRollback]
2020-10-29 10:31:47 +0000 UTC info: Performing rollback
2020-10-29 10:31:47 +0000 UTC info: State transition: rollback [ArtifactRollback] -> update-error [ArtifactFailure]
2020-10-29 10:31:47 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2020-10-29 10:31:47 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]

is there something changed in mender server or what is causing it to fail ?

@mirzak Any idea why am i facing this issue ?

drewmoseley · October 29, 2020, 2:09pm

What does your state script var/lib/mender/scripts/ArtifactInstall_Leave_90_install_systemd_unit do? It seems to be causing an error which will instruct Mender to rollback.

Rohita83 · October 30, 2020, 5:23am

Script ArtifactInstall_Leave_90_install_systemd_unit

#!/bin/sh
set -e
cat > /lib/systemd/system/mender-demo-artifact.service <<EOF
[Install]
WantedBy=multi-user.target

[Service]
Type=notify
ExecStart=/var/www/localhost/entrypoint.sh
EOF

systemctl daemon-reload
systemctl restart mender-demo-artifact
systemctl enable mender-demo-artifact

Mender demo-service error

Oct 30 10:12:00 alto-VirtualBox systemd[1]: Starting mender-demo-artifact.service…
Oct 30 10:12:03 alto-VirtualBox entrypoint.sh[15596]: /usr/share/mender/inventory/mender-inventory-geo: Unable to get the geolocalization data from ipvigilante.com
Oct 30 10:12:03 alto-VirtualBox systemd[1]: mender-demo-artifact.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 30 10:12:03 alto-VirtualBox systemd[1]: mender-demo-artifact.service: Failed with result ‘exit-code’.
Oct 30 10:12:03 alto-VirtualBox systemd[1]: Failed to start mender-demo-artifact.service.

This was working until few days back but suddenly stopped working

Rohita83 · October 30, 2020, 6:08am

@drewmoseley

We are also getting the error to fetch image in OTA update from URL’s https://s3.docker.mender.io:9000
docker.mender.io.

=================================================
2020-10-29 14:41:49 +0000 UTC info: Running Mender client version: b6dfc02
2020-10-29 14:43:22 +0000 UTC info: Running Mender client version: b6dfc02
2020-10-29 14:43:22 +0000 UTC error: Mender shut down in state: update-fetch
2020-10-29 14:43:22 +0000 UTC info: State transition: init [none] -> idle [Idle]
2020-10-29 14:43:24 +0000 UTC info: Running Mender client version: b6dfc02
2020-10-29 14:45:36 +0000 UTC error: Can not fetch update image: Get https://s3.docker.mender.io:9000/mender-artifact-storage/e5f6524c-2125-47d4-859a-b887ed18cca3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20201029%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201029T144324Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=7893eb09429ee8ee9b1463a52ece672800130340d3d280f151a00a82189d1a4b: dial tcp 34.207.126.151:9000: connect: connection timed out

I tried to ping urls but it not reachable.
ping s3.docker.mender.io

O/P: ping: s3.docker.mender.io: Name or service not known

Could you plz assist me for the same?

drewmoseley · October 30, 2020, 4:49pm

It seems that for some reason this systemctl restart is not working. I don’t know why but since you have “set -e” in your script, that will force the script to return an error code which is the signal to Mender to roll back. You need to either ignore the errors and explicitly return 0 or troubleshoot the failure to start the mender-demo-artifact script.

Using the Mender demo server requires these names but it also requires them to exist in /etc/hosts since you have to explicitly assign an IP address. For some reason your name resolution is not working. What are the contents of /etc/hosts and /etc/resolv.conf? What happens if you run ping 8.8.8.8?

Rohita83 · November 3, 2020, 1:09pm

@drewmoseley

$ cat hosts
127.0.0.1 localhost

The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.88.250.172 docker.mender.io s3.docker.mender.io

Question: where (s3 bucket) OTA uploaded mender artifacts (xyz**.menter files**) stores in mender demo server integration-2.0 hosted on my EC2 install??
Actually, I want to stores .mender artifacts on my own aws account S3 bucket that was uploaded on the mender demo server. is it possible it using mender demo server?? if yes what changes suppose to make?

drewmoseley · November 3, 2020, 3:05pm

I’d suggest you move to a production server before setting up S3. That will ensure you are not using a shared certificate and the like that are part of the demo setup. It will also allow you to use your own domain name for the server rather than using the demo names and hacking it with the /etc/hosts file modifications.

The docs for configuring the storage proxy are here.

Drew

Rohita83 · November 4, 2020, 11:39am

@drewmoseley
As per your suggestion, I have setup the production mender server by using url Production installation | Mender documentation and also i am able to upload the default mender artifact successfully. current prod.yml file having data as below screenshot

I had tried to replace DEPLOYMENTS_AWS_AUTH_KEY as my AWS & DEPLOYMENTS_AWS_AUTH_SECRET with my aws access key & secret key. and also DEPLOYMENTS_AWS_URI as https://s3-us-west-1.amazonaws.com. Now after that, i am unable to upload mender artifacts. can you plz assist me that what wrong i make ??

I had also tried to update /home/ubuntu/mender-server/docker-compose.storage.s3.yml parameters below

        DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
        DEPLOYMENTS_AWS_AUTH_KEY: ${AWS_ACCESS_KEY_ID}
        DEPLOYMENTS_AWS_AUTH_SECRET: ${AWS_SECRET_ACCESS_KEY}
        DEPLOYMENTS_AWS_REGION: us-west-1
        DEPLOYMENTS_AWS_URI: https://s3-us-west-1.amazonaws.com
        DEPLOYMENTS_AWS_BUCKET: mybucket

But still getting issue to upload artifacts with error msg on console as “Artifacts couldn’t be uploaded” & “Check the connection”.

where can I see the mender server (ec2 instance hosted) the logs? Could you please assist me the steps need an update in .yml so that uploaded artifacts should store on my AWS account S3 bucket?

drewmoseley · November 4, 2020, 2:49pm

Did you include docker-compose.storage.s3.yml in the run script? By default it calls the Minio version.

Rohita83 · November 4, 2020, 5:30pm

@drewmoseley
Thanks for quick response.
yes, I have added docker-compose.storage.s3.yml in the run script as below.

exec docker-compose
-p menderproduction
-f …/docker-compose.yml
-f …/docker-compose.storage.minio.yml
-f …/docker-compose.storage.s3.yml \

docker-compose.storage.s3.yml file data is as below

   mender-deployments:
       # S3 access configuration - override with your own values
       # Keys have to grant access to default bucket: mender-artifact-storage
       environment:
           DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
           DEPLOYMENTS_AWS_AUTH_KEY: XXXXXXXXXXXXXXX
           DEPLOYMENTS_AWS_AUTH_SECRET: XXXXXXXXXXXXXXXX
           #DEPLOYMENTS_AWS_URI: https://s3.amazonaws.com
           DEPLOYMENTS_AWS_BUCKET: mender-public-bucket
           DEPLOYMENTS_AWS_REGION: us-west-1
           DEPLOYMENTS_AWS_URI: https://s3-us-west-1.amazonaws.com

I am able to upload artifacts on console but I didn’t get it in s3 bucket “mender-public-bucket”. currenlly i have made this s3 bucket “mender-public-bucket” as public. should bucket public?
what should be DEPLOYMENTS_AWS_URI value? is it right in my above .yml file?

drewmoseley · November 9, 2020, 6:18pm

I’m not sure about the permissions but I suspect public is not correct. The URI that is generated for deployments will allow the devices to find it properly. @merlin can you comment?

I think you need to remove -f …/docker-compose.storage.minio.yml from your run script.

peter · December 3, 2020, 4:33pm

hello @Rohita83

thanks for using Mender.
https://s3.amazonaws.com should work, and the bucket most certainly does not/should not be public, thats why you need AUTH_KEY and AUTH_SECRET settings.
do you have some logs from the deployments service and from the mender-client, so we can see what is the problem?

peter

Topic		Replies	Views
Unable to rollout the OS OTA update Update Modules os-update	5	289	September 27, 2023
Getting "error: fw_setenv returned failure: exit status 2" after "Committing update" OTA update General Discussions debian , x86 , mender-client , ota	47	2221	June 22, 2021
Failure updating with Mender General Discussions yocto	5	862	January 22, 2022
Mender OTA Update successful but reported by Enterprise server as a failure General Discussions	5	582	February 7, 2022
No public key was provided for authenticating the artifact General Discussions	3	1178	September 28, 2020

Mender OTA update fails with demo artifact on ubuntu 18.04 x86

Mender demo-service error

The following lines are desirable for IPv6 capable hosts

Related topics