Failed to verify the manifest signature: The predicted (387) and the actual (384) length differ

General Discussions
,
1

Hi,
I’m trying to use AWS KMS key to sign the Mender artifact and it seems to work just fine on my build machine but it fails to verify it on the install target:

After succesfully running mender-convert
Extract the tar file:

$ mkdir temp
$ cd temp/
$ tar -xf ../mender-artifact-x86_64.mender

$ ls -l
drwxrwxr-x 1 username username  24 Feb  5 11:52 data
-rw------- 1 username username 379 Feb  5 10:14 header.tar.zst
-rw------- 1 username username 242 Jan  1  1970 manifest
-rw------- 1 username username  31 Jan  1  1970 version

Sign manifest with kms sign:



$ aws kms sign --key-id alias/bogus-key-alias --message fileb://manifest --message-type RAW --signing-algorithm RSASSA_PKCS1_V1_5_SHA_256 --query Signature --output text > manifest.sig

$ ls -l
drwxrwxr-x 1 username username  24 Feb  5 11:52 data
-rw------- 1 username username 379 Feb  5 10:14 header.tar.zst
-rw------- 1 username username 242 Jan  1  1970 manifest
-rw-rw-r-- 1 username username 513 Feb  5 11:53 manifest.sig
-rw------- 1 username username  31 Jan  1  1970 version

base64 decode the manifest to we can verify it:
$ base64 -d manifest.sig > manifest.sig.raw

Verify (the public key saved previously):
$ openssl dgst -sha256 -verify /tmp/bogus-public-key.pem -signature manifest.sig.raw manifest
Verified OK

Create a new tar archive that contains mainfest.sig and validate:

$ tar cf ../mender-artifact-x86_64-x86_64-signed.mender version manifest manifest.sig header.tar.zst data/0000.tar.zst

$ mender-artifact validate ../mender-artifact-x86_64-x86_64-signed.mender -k /tmp/bogus-public-key.pem
Artifact file ‘../mender-artifact-x86_64-x86_64-signed.mender’ validated successfully

So far so good, when I try to install it on my target device it fails:



$ sudo mender-update install ./mender-artifact-x86_64-x86_64-signed.mender
record_id=1 severity=error time=“2026-Feb-06 10:19:27.996135” name=“Global” msg=“Signature verification Error: Failed to verify the manifest signature: The predicted (387) and the actual (384) length differ”
Streaming failed.
System not modified.
Could not fulfill request: Signature verification Error: Failed to verify the manifest signature: The predicted (387) and the actual (384) length differ

I have the same public key used with “mender-artifact verify” saved in /etc/mender/artifact-verify-key.pem and have it configured in my config file:
$ grep ArtifactVerifyKey /etc/mender/mender.conf
“ArtifactVerifyKey”: “/etc/mender/artifact-verify-key.pem”,

I think the error message is a red herring, I have verified the key length a thousand times and using “strace” I can see that mender-update does not even try to open my public key so we fail before that step:

$ sudo strace -e trace=openat -f mender-update install ./mender-aftifact-x86_64-signed.mender 2>&1 | grep -E ‘mender.conf|artifact-verify-key.pem’
openat(AT_FDCWD, “/var/lib/mender/mender.conf”, O_RDONLY) = 3
openat(AT_FDCWD, “/etc/mender/mender.conf”, O_RDONLY) = 3

I have tried installing using hosted Mender and locally as described above and both fail with the same error message. Worth noting is that hosted Mender accept the artifact I upload (I messed it up earlier and then it complained and refused the uploaded artifact).

Versions used:

  • On the build machine:
    $ mender-artifact --version
    mender-artifact version 4.2.0

  • On the install target:
    $ mender-update --version
    5.0.3

Edit: Added printed manifest from the signed mender file (replaced my actual name and versions with “mender-artifact-x86_64”). What I wanted to show was that it says “Signature: signed and verified correctly”

$ mender-artifact read mender-artifact-x86_64-signed.mender --print-manifest -k /tmp/bogus-public-key.pem
Reading Artifact.......................... - 100 %
Mender Artifact:
  Name: mender-artifact-x86_64
  Format: mender
  Version: 3
  Signature: signed and verified correctly
  Compatible devices: [mender-artifact-x86_64]
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts: []

Updates:
  - Type: rootfs-image
    Provides:
      rootfs-image.checksum: f4c3b6029124a385b4aff53b9b6b910187662b4ecd8e8b24464ab6ac46fbc013
      rootfs-image.version: mender-artifact-x86_64
    Depends: {}
    Clears Provides: [artifact_group, rootfs_image_checksum, rootfs-image.*]
    Metadata: {}
    Files:
      - checksum: f4c3b6029124a385b4aff53b9b6b910187662b4ecd8e8b24464ab6ac46fbc013
        modified: 2026-02-05 10:13:43 +0000 UTC
        name: rootfs.img
        size: 16106127360

If someone have any clue on what is causing this error it would make me very happy, thank you in advance.

Best regards,
Thomas

2

Hi @tmartensson,

Thanks for reaching out! It definitely sounds strange and I can’t directly put my finger on the reason. A couple of ideas to try:

  1. instead of manually signing, can you try the mechanism integrated in mender-artifact? It should be able to interact with AWS KMS directly.
  2. can you try, just for understanding, a build with a Mender Client V3.x? Since V4.x the client is implemented in C++, and does not share the Go implementation of V3.x and mender-artifact anymore.

Greetz,
Josef

3

Thanks for the reply @TheYoctoJester, I have a hard time trying to figure out the first point to use my aws kms key directly with mender-artifact. I see flags for Google and Azure KMS, but I do not see any flags for AWS KMS. If you can point me in the right direction that would be much appreciated.

So I went for the second option and uninstalled version 5.0.3 of the client and:

$ wget https://downloads.mender.io/repos/debian/pool/main/m/mender-client/mender-client_3.5.2-1+ubuntu+jammy_amd64.deb

$ apt install ./mender-client_3.5.2-1+ubuntu+jammy_amd64.deb

$ mender --log-level info install mender-artifact-x86_64.mender
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/vda2
INFO[0000] Mender running on partition: /dev/vda2
INFO[0000] Start updating from local image file: [mender-artifact-x86_64.mender]
Installing Artifact of size 1582581760...
INFO[0000] Installer: authenticated digital signature of artifact
ERRO[0000] Unknown device_type. Continuing with update
INFO[0000] Opening device "/dev/vda3" for writing
INFO[0000] Native sector size of block device /dev/vda3 is 512 bytes. Mender will write in chunks of 1048576 bytes
.............................................................  -  99 %INFO[0055] All bytes were successfully written to the new partition
INFO[0055] The optimized block-device writer wrote a total of 15361 frames, where 4893 frames did need to be rewritten (i.e., skipped)
INFO[0055] Wrote 16106127360/16106127360 bytes to the inactive partition
INFO[0055] Enabling partition with new image installed to be a boot candidate: 3
Use 'commit' to update, or 'rollback' to roll back the update.
At least one payload requested a reboot of the device it updated.

Since it printed this line:
INFO[0000] Installer: authenticated digital signature of artifact

it seems that version 3 of the client works fine. Also something I forgot to mention in my first post, but as you probably found out in this reply, we use Ubuntu 22.04 as our base image.

Best regards
Thomas

4

Hi @tmartensson,

Thanks for the report! That seems indeed to be a kind-of corner case / semi-regression. I’ve also checked the mender-artifact documentation, there is no direct AWS integration for that, sorry. Was confusing it with Azure :frowning:

So a guess I have right now is that at some point there are newlines inserted into the signature. Can you open manifest.sig, inspect it accordingly and eventually remove them? In a pipeline this can obviously be done with sed.

In the meantime, I’ll check with the devs if this is intended behaviour or something they want to address.

Greetz,
Josef

5

Thank you @TheYoctoJester! :clap:
that was indeed the issue, adding | tr -d '\n' > manifest.sig in my aws kms sign command solved the issue and the signed artifact now installs without any issue.

Thank you again, this one would have taken me ages to figure out :laughing:

Best regards,
Thomas

1 Like