Stop mender update after header mismatch

General Discussions
, ,
1

Hi,

We’re trying out with hosted.mender.io and if the header mismatch occurs in mender artifact it’s waiting 1 minute and trying again (this occurs 3 times). I want to change this behaviour as failure in dashboard after 1 failure.

Can this be done with hosted mender?
Can we change this behaviour in mender client?

Thanks in advance.

2

Hi @danie I’m guessing this occurs because you have specified retries when you created the deployment. Those retries will always happen when a deployment fails. You should be able to just eliminate the retries when you create the deployment and this issue will go away.

Drew

3

Here while creating the deployment the retries section specifies “Don’t retry” yet, it’s trying 3 times

Screenshot from 2020-12-01 11-36-45
Screenshot from 2020-12-01 11-36-45608×591 22.7 KB

Do I need to change it somewhere else?

4

I do not think that this is related to the server side settings.

The Mender client does not distinguish between errors and it will use RetryPollIntervalSeconds for any error it gets during the update cycle. I thin this is what you are seeing.

If you can share the Mender client log, we can make a final confirmation that this is the case.

1 Like
5

I’m appending the log for your reference. NOTE: It is big and the whole log took 3 and half minutes.

-- Logs begin at Thu 1970-01-01 00:00:06 UTC. --
Dec 01 09:57:35 imx6ulevk mender[293]: time="2020-12-01T09:57:35Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Dec 01 09:57:35 imx6ulevk mender[293]: time="2020-12-01T09:57:35Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Dec 01 09:57:35 imx6ulevk mender[293]: time="2020-12-01T09:57:35Z" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
Dec 01 09:57:36 imx6ulevk mender[293]: time="2020-12-01T09:57:36Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Dec 01 09:57:37 imx6ulevk mender[293]: time="2020-12-01T09:57:37Z" level=info msg="Correct request for getting image from: https://s3.amazonaws.com/hosted-mender-artifacts/5f02d290608e2bc4c8a6b7d0/2e089875-5b43-4259-8992-898b70480461?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAQWI25QR6MOCESDWO%2F20201201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201201T095736Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=648dfb1359a7e3362a42ad3b95ac879fd9cf2c841291309b248b7b14e3df114d [name: demo-false-signed; devices: [imx6ulevk]]"
Dec 01 09:57:37 imx6ulevk mender[293]: time="2020-12-01T09:57:37Z" level=info msg="State transition: update-check [Sync] -> update-fetch [Download_Enter]"
Dec 01 09:57:37 imx6ulevk mender[293]: time="2020-12-01T09:57:37Z" level=info msg="Running Mender client version: 2.4.1-build2-dirty"
Dec 01 09:57:43 imx6ulevk mender[293]: time="2020-12-01T09:57:43Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]"
Dec 01 09:57:44 imx6ulevk mender[293]: time="2020-12-01T09:57:44Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 09:57:44 imx6ulevk mender[293]: time="2020-12-01T09:57:44Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 09:57:44 imx6ulevk mender[293]: time="2020-12-01T09:57:44Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]"
Dec 01 09:58:44 imx6ulevk mender[293]: time="2020-12-01T09:58:44Z" level=info msg="State transition: fetch-install-retry-wait [Download_Enter] -> update-fetch [Download_Enter]"
Dec 01 09:58:57 imx6ulevk mender[293]: time="2020-12-01T09:58:57Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]"
Dec 01 09:58:58 imx6ulevk mender[293]: time="2020-12-01T09:58:58Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 09:58:58 imx6ulevk mender[293]: time="2020-12-01T09:58:58Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 09:58:58 imx6ulevk mender[293]: time="2020-12-01T09:58:58Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]"
Dec 01 09:59:58 imx6ulevk mender[293]: time="2020-12-01T09:59:58Z" level=info msg="State transition: fetch-install-retry-wait [Download_Enter] -> update-fetch [Download_Enter]"
Dec 01 09:59:59 imx6ulevk mender[293]: time="2020-12-01T09:59:59Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]"
Dec 01 10:00:00 imx6ulevk mender[293]: time="2020-12-01T10:00:00Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 10:00:00 imx6ulevk mender[293]: time="2020-12-01T10:00:00Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 10:00:00 imx6ulevk mender[293]: time="2020-12-01T10:00:00Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]"
Dec 01 10:01:00 imx6ulevk mender[293]: time="2020-12-01T10:01:00Z" level=info msg="State transition: fetch-install-retry-wait [Download_Enter] -> update-fetch [Download_Enter]"
Dec 01 10:01:18 imx6ulevk mender[293]: time="2020-12-01T10:01:18Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]"
Dec 01 10:01:18 imx6ulevk mender[293]: time="2020-12-01T10:01:18Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 10:01:18 imx6ulevk mender[293]: time="2020-12-01T10:01:18Z" level=error msg="Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: reader: invalid signature: crypto/rsa: verification error"
Dec 01 10:01:18 imx6ulevk mender[293]: time="2020-12-01T10:01:18Z" level=info msg="State transition: update-store [Download_Enter] -> fetch-install-retry-wait [Download_Enter]"
Dec 01 10:01:18 imx6ulevk mender[293]: time="2020-12-01T10:01:18Z" level=info msg="State transition: fetch-install-retry-wait [Download_Enter] -> update-error [ArtifactFailure]"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=info msg="State transition: update-error [ArtifactFailure] -> cleanup [Error]"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=info msg="State transition: cleanup [Error] -> update-status-report [none]"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=info msg="State transition: update-status-report [none] -> idle [Idle]"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=info msg="authorization data present and valid"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=info msg="State transition: idle [Idle] -> check-wait [Idle]"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Dec 01 10:01:19 imx6ulevk mender[293]: time="2020-12-01T10:01:19Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Dec 01 10:01:20 imx6ulevk mender[293]: time="2020-12-01T10:01:20Z" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
Dec 01 10:01:20 imx6ulevk mender[293]: time="2020-12-01T10:01:20Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Dec 01 10:01:20 imx6ulevk mender[293]: time="2020-12-01T10:01:20Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
Dec 01 10:01:21 imx6ulevk mender[293]: time="2020-12-01T10:01:21Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
Dec 01 10:01:21 imx6ulevk mender[293]: time="2020-12-01T10:01:21Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Dec 01 10:01:21 imx6ulevk mender[293]: time="2020-12-01T10:01:21Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Dec 01 10:01:22 imx6ulevk mender[293]: time="2020-12-01T10:01:22Z" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
Dec 01 10:01:22 imx6ulevk mender[293]: time="2020-12-01T10:01:22Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Dec 01 10:01:22 imx6ulevk mender[293]: time="2020-12-01T10:01:22Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
Dec 01 10:01:23 imx6ulevk mender[293]: time="2020-12-01T10:01:23Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
6

Indeed, this is the Mender client not distinguishing between error types, and it is working trough the “normal” retry loop.

I seem to recall that we have a ticket for this. @kacf do you know?

7

I remember we talked about it, but I’m not sure if a ticket was filed. At least I cannot find one, so feel free to report it.