Mender-Client failed to update deployment status

General Discussions
22

after getting storage_proxy to work i do now get another behaviour from the client:

Jul 15 13:55:23 qemux86-64 mender-update[334]: record_id=1045 severity=info time="2025-Jul-15 13:55:23.080686" name="Global" msg="Deployment with ID 46e2d6fe-049b-4a88-96c2-ac220014375e started."
Jul 15 13:55:23 qemux86-64 mender-update[334]: record_id=1046 severity=info time="2025-Jul-15 13:55:23.096298" name="Global" msg="Sending status update to server"
Jul 15 13:55:23 qemux86-64 mender-update[334]: record_id=1047 severity=warning time="2025-Jul-15 13:55:23.357163" name="http_resumer:client" msg="bad version: GET https://x.x.x.x.sslip.io/mender-server-459007-mender-storage-dev/7e888ca6-957e-4b65-9a8e-cac181fa222a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOGLXXXXXXXX%2F20250715%2Feurope-west3%2Fs3%2Faws4_request&X-Amz-Date=20250715T135523Z&X-Amz-Expires=86400&X-Amz-Signature=d9a314bc9b788e7a0b2cac1e8821ca06e04243a242c69cfdebf02e413834f847&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B+filename%3D%22pdncrash.log.mender%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject: "
Jul 15 13:55:23 qemux86-64 mender-update[334]: record_id=1048 severity=info time="2025-Jul-15 13:55:23.359578" name="http_resumer:client" msg="Resuming download after 60 seconds"
Jul 15 13:56:27 qemux86-64 mender-update[334]: record_id=1049 severity=warning time="2025-Jul-15 13:56:27.959440" name="http_resumer:client" msg="bad version: GET https://x.x.x.x.sslip.io/mender-server-459007-mender-storage-dev/7e888ca6-957e-4b65-9a8e-cac181fa222a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOGLXXXXXXXX%2F20250715%2Feurope-west3%2Fs3%2Faws4_request&X-Amz-Date=20250715T135523Z&X-Amz-Expires=86400&X-Amz-Signature=d9a314bc9b788e7a0b2cac1e8821ca06e04243a242c69cfdebf02e413834f847&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B+filename%3D%22pdncrash.log.mender%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject: "
Jul 15 13:56:27 qemux86-64 mender-update[334]: record_id=1050 severity=info time="2025-Jul-15 13:56:27.963505" name="http_resumer:client" msg="Resuming download after 60 seconds"

When pasting the url into my browser i get this (from storage.googleapis.com ):

SignatureDoesNotMatchAccess denied.The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.AWS4-HMAC-SHA256 20250715T135523Z 20250715/europe-west3/s3/aws4_request 9003d9e150d66fd6b37facb3796a1030b5c35b26da71b16b879d15f40b2c064aGET /mender-server-459007-mender-storage-dev/7e888ca6-957e-4b65-9a8e-cac181fa222a X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOGLXXXXXXXXX%2F20250715%2Feurope-west3%2Fs3%2Faws4_request&X-Amz-Date=20250715T135523Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3D%22pdncrash.log.mender%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject%3A host:storage.googleapis.com host UNSIGNED-PAYLOAD

would love to get this fixed!

23

Hi all,

we have identified the bug. The main patch can be found at fix: Clear response buffer before every HTTP request by vpodzime · Pull Request #1803 · mendersoftware/mender · GitHub, so for those of you building from master or willing to manually patch, please give it a try.

I will follow up once the Client releases are available.

Greetz,
Josef

1 Like
24

Howdy everybody, especially @nicktech @simonbuehler @aashish,

The fixed release is now available as Mender Client | Mender documentation,

and specifically for Yocto as Release scarthgap-v2025.07 · mendersoftware/meta-mender · GitHub and Release kirkstone-v2025.07 · mendersoftware/meta-mender · GitHub.

Greetz,
Josef

25

Great, one question about best practice::

Using kas-container in dev and CI, so I included the meta-mender-community kas includes for scarthgap , but they are fixed to a stable commit.
Now I would need a newer revision of the client which is not available in this revision, so what’s the best way to use the kas includes and bump some newer package updates?

26

Hi @simonbuehler,

Literally the easiest way would be to send a commit hash bump PR to meta-mender-community. I update those whenever I find the time and/or need, but there’s no fixed cadence. Maybe I can get later to it today or Monday, but I just can’t tell yet.

Greetz,
Josef

27

i see but further questions have popped up, using kas build, scarthgap as target on cm4.

my main yml:

header:
  version: 16
  includes:
  - repo: meta-mender-community
    file: kas/include/mender-full.yml
  - repo: meta-mender-community
    file: kas/include/raspberrypi.yml

which includes from mender-base.yml :

[...]

repos:
[...]
  meta-mender:
    url: https://github.com/mendersoftware/meta-mender.git
    commit: 52b332b61d26c97b8e6cc1badb80db0bb6a1666b

    layers:
      meta-mender-core:

and my yaml continues with:

repos:

  meta-mender-community:
    url: https://github.com/mendersoftware/meta-mender-community.git
    branch: scarthgap # is this needed ? i guess - to not switch branches in future without noticing

  meta-mender: # overrides / extends definition in mender-base.yml from includes
    url: https://github.com/mendersoftware/meta-mender.git # unneeded ?
    tag: scarthgap-v2025.09 # Core of question - use tags or commits? conflicts with fixed commit from include? use branch also?

    commit: 52b332b61d26c97b8e6cc1badb80db0bb6a1666b # overwrites included making the whole include (regarding version) useless?
    layers:
      meta-mender-core: # duplicate - not needed?

So it boils down to - whats best practice to follow a LTS release with important cherrypicks?

28

Hi @simonbuehler,

In a nutshell - it depends.

The meta-mender-community kas files are not meant to serve as a continually updated distribution. There are two main starting points:

  • the scarthgap branch which will get version bumps whenever I or somebody else gets around to do them, and will use fixed SHA tags to provide a stable, known good & building starting point
  • the scarthgap-next branch which will track the upstream branches, but on the other hand not be up to date with the scarthgap branch and occasionally rebase.

kas can be run with the --update flag, so you can use that in conjunction with the scarthgap branch to stay on top. This is probably the most straightforward.

The alternative is to create a build configuration for yourself in your pipeline/repository which tracks upstream branches.

In general, I would suggest to have a two branch strategy.

  • one with fixed commit hashes, and that one goes to prod
  • one which tracks the upstream branch tips, and after successful validation of a build, you bump prod to it.

Greetz,
Josef

1 Like
29

Thanks Josef for your reply, this makes it clear!

this was my lazy assumption, thanks for pointing this out!