The checksum of the read byte-stream does not match the expected checksum

I am trying to do an OS update (not delta) and for some reason can not get it to work. It complains about the checksum mismatch, but I can not figure out where it is getting the ‘calculated’ output from.

# mender-update install /mnt/revel-disk-v0.0.8-20260108.1146-x86_64-mender.mender
Installing artifact...
record_id=1 severity=info time="2026-Jan-08 22:10:42.331851" name="Global" msg="Sending SIGTERM to PID 7791"
record_id=2 severity=info time="2026-Jan-08 22:10:42.332037" name="Global" msg="PID 7791 exited with status 15"
record_id=3 severity=error time="2026-Jan-08 22:10:42.332066" name="Global" msg="Streaming failed: Shasum mismatch error: The checksum of the read byte-stream does not match the expected checksum, (expected): fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6 (calculated): e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
record_id=4 severity=error time="2026-Jan-08 22:10:42.339857" name="Global" msg="Shasum mismatch error: The checksum of the read byte-stream does not match the expected checksum, (expected): fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6 (calculated): e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Streaming failed.
System not modified.
Could not fulfill request: Shasum mismatch error: The checksum of the read byte-stream does not match the expected checksum, (expected): fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6 (calculated): e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

I took the artifact and dumped the contents to check everything:

fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6  data/0000/rootfs.img
509fb823a0cfb31f3b0bec6669abdaa78dc7788dd156f672d6502ad5e743f386  header.tar.gz
96bcd965947569404798bcbdb614f103db5a004eb6e364cfc162c146890ea35b  version

# sha256sum data/rootfs.img
fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6  data/rootfs.img

# sha256sum header.tar.gz
509fb823a0cfb31f3b0bec6669abdaa78dc7788dd156f672d6502ad5e743f386  header.tar.gz

# sha256sum version
96bcd965947569404798bcbdb614f103db5a004eb6e364cfc162c146890ea35b  version

This all shows the rootfs and everything else with the correct sha.

I just have no idea how it is calculating the incorrect e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

just a follow up. we figured out that the calculated sha is of an empty string…

# echo -n "" | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  - 

so no idea why mender is taking my artifact and sha nothing.

as shown above when i expand the artifact it has files and the files all look good.

Hi @phynias,

Thanks for reaching out! Yes the payload obviously should not be empty, or considered to be empty. The first obvious questions would be:

  • which versions of the Mender Client and mender-artifact tools are you using?
  • how did the artifact get created? Through a Yocto or mender-convert process, or manually? If so, how?
  • mender-artifact read should be able to give details about the artifact, can you post its output?

Greetz,
Josef

the version of mender-client that is installed by mender-convert looks like package version 5.1.0 of mender-client4, but the –version of mender-auth and mender-update are 2cae2f4b

artifact is created by installing ubuntu onto a proxmox vm, running ansible against it and then taking the proxmox img and converting it to raw then running mender-convert against it. this provides me with a img.gz that i throw into pxe and can install and boot fine.

mender-artifact version tried. 3.9 and 4.2

mender-artifact read:

Reading Artifact...
.............................................................. - 100 %
Mender artifact:
  Name: v0.0.13
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[x86_64]'
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   rootfs-image
    Provides:
	rootfs-image.checksum: 6fd522a985bcc17542a270cd26217e7bff78f50fdd8ccd135194cb6a1019bfb3
	rootfs-image.version: v0.0.13
    Depends: Nothing
    Clears Provides: ["artifact_group", "rootfs_image_checksum", "rootfs-image.*"]
    Metadata: Nothing
    Files:
      name:     rootfs.img
      size:     12951595008
      modified: 2026-01-12 19:00:24 +0000 UTC
      checksum: 6fd522a985bcc17542a270cd26217e7bff78f50fdd8ccd135194cb6a1019bfb3

I also decided to create an artifact with MENDER_ARTIFACT_COMPRESSION=none and the error changed to this:

Installing artifact…
record_id=1 severity=error time=“2026-Jan-12 20:08:37.766869” name=“Global” msg=“Failed to initialize the Archive handle: Failed to read from the archive stream: ‘Unrecognized archive format’ error code: 84”
record_id=2 severity=info time=“2026-Jan-12 20:08:37.801552” name=“Global” msg=“Sending SIGTERM to PID 341661”
record_id=3 severity=info time=“2026-Jan-12 20:08:37.801695” name=“Global” msg=“PID 341661 exited with status 15”
record_id=4 severity=error time=“2026-Jan-12 20:08:37.801774” name=“Global” msg=“Streaming failed: Parse error: archive_read_next failed in LibArchive. Error code: -30 Error message: Unrecognized archive format”
record_id=5 severity=error time=“2026-Jan-12 20:08:37.810241” name=“Global” msg=“Parse error: archive_read_next failed in LibArchive. Error code: -30 Error message: Unrecognized archive format”
Streaming failed.
System not modified.
Could not fulfill request: Parse error: archive_read_next failed in LibArchive. Error code: -30 Error message: Unrecognized archive format
Reading Artifact...
.............................................................. - 100 %
Mender Artifact:
  Name: v0.0.13
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: [x86_64]
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts: []

Updates:
  - Type: rootfs-image
    Provides:
      rootfs-image.checksum: 4dccd622fef069435729dfe9341b6fdb1f68350d7e5c2fcae222f1e9cb456a66
      rootfs-image.version: v0.0.13
    Depends: {}
    Clears Provides: [artifact_group, rootfs_image_checksum, rootfs-image.*]
    Metadata: {}
    Files:
      - checksum: 4dccd622fef069435729dfe9341b6fdb1f68350d7e5c2fcae222f1e9cb456a66
        modified: 2026-01-12 20:00:56 +0000 UTC
        name: rootfs.img
        size: 12951595008

another small update. i took the rootfs from the .mender and i dd it to the rootfsb on the machine and then set

sudo grub-editenv /boot/grub/grubenv set mender_boot_part=3
sudo grub-editenv /boot/grub/grubenv set mender_boot_part_hex=3

and rebooted and it booted fine into the new part.

some more info.

currently mender-convert install mender-client4 v5.0.3. if i install my image onto a machine, then rremove the mender-client4 and install the older mender-client v3 version the mender install works with my artifact.

Hi @phynias,

Thanks a lot for the details! It seems there is possibly a couple of things going on.

  • the mender install working with Client v3. So this works with the very same artifact which fails with Client v4.x/v5.x? In that case it’s a regression that we definitely look into.
  • the Proxmox route, so you’re explicitly using a VM, not a CT? If so, would you mind sharing a bit more details on the machine configuration and setup process? I do have Proxmox infrastructure here too, and I’d like to try and reproduce this as a possible use case which might be helpful to others too.

Greetz,
Josef

Hi @phynias,

The pointer from the developers was circling back to the initial post:

Please note the missing “0000” in your checksum verification. So the current guess is that there is something off with the artifact. Was there any post processing on it after creation with mender-artifact?

For better understanding, can you drop a tar -tf of it?

Greetz,
Josef

artifact isn’t created with mender-artifact. i use mender-convert on the raw proxmox disk.

and no i do not modify the .mender package in anyway once created.
i’ve also untarred it and that is in the initial post i made above.

I took the artifact and dumped the contents to check everything:


fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6  data/0000/rootfs.img
509fb823a0cfb31f3b0bec6669abdaa78dc7788dd156f672d6502ad5e743f386  header.tar.gz
96bcd965947569404798bcbdb614f103db5a004eb6e364cfc162c146890ea35b  version

# sha256sum data/rootfs.img
fcf616db0573d3540b7bb73bf1f39ed32004c1bceb64bd61b53fcec9b35ea6b6  data/rootfs.img

# sha256sum header.tar.gz
509fb823a0cfb31f3b0bec6669abdaa78dc7788dd156f672d6502ad5e743f386  header.tar.gz

# sha256sum version
96bcd965947569404798bcbdb614f103db5a004eb6e364cfc162c146890ea35b  version

also not sure what you mean by “Please note the missing “0000” in your checksum verification.” missing from where exactly?

Hi @phynias,

see below, I have marked the 0000 in bold. If mender-convert is used and no manual artifact creation then this is really something we’d like to understand. Can you drop full tar -tf output on it?

Additionally, I’d like to reproduce the case. Installing Ubuntu 24.04 on Proxmox, then (theoretically) some application installation and configuration, then mender-convert on the image. Which configuration of mender-convert did you use? Or if it is a fully custom one, can you maybe provide a sanitized, minimal version which I can use to try it out here?

Greetz,
Josef

I did some more testing today and I am almost positive that this is a mender-artifact thing. mender-convert calls mender-artifact so if i take the rootfs.img from mender-convert and try to run mender-artifact myself i still see the above errors. on top of that if i do compression none and try to install that i get:

Installing artifact...
record_id=1 severity=error time="2026-Jan-19 23:19:53.835339" name="Global" msg="Failed to initialize the Archive handle: Failed to initalize the 'libarchive' C bindings. LibArchive error message: 'Unrecognized archive format' error code: 84"
record_id=2 severity=info time="2026-Jan-19 23:19:53.865742" name="Global" msg="Sending SIGTERM to PID 1011000"
record_id=3 severity=info time="2026-Jan-19 23:19:53.865879" name="Global" msg="PID 1011000 exited with status 15"
record_id=4 severity=error time="2026-Jan-19 23:19:53.869552" name="Global" msg="Parse error: archive_read_next failed in LibArchive. Error code: -30 Error message: Unrecognized archive format"
Installation failed. System not modified.
Could not fulfill request: Parse error: archive_read_next failed in LibArchive. Error code: -30 Error message: Unrecognized archive format

i’ve tried mender-artifact 4.2.0, 4.1.0, 4.1.1

i know the rootfs.img is good because i can take that and dd it to the other rootfs and boot it and it works fine. so mender-artifact is doing things i just dont understand. it takes a perfectly fine rootfs.img and then packages in a way that seems utterly broken. paths dont make sense, like you said above, the 0000 directory is weird to me.

in the manifest it says
8ae4b98c0dfa4c762681799da27cacbaf4c61fdcf2cd9064b0c7019920afd98b data/0000/rootfs.img

but then you look in the data dir and there’s just a file in there called 0000.tar or 0000.tar.gz if compression is gzip. but if you untar that file you would assume it would untar into 0000/rootfs.img but it just expands into rootfs.img. you would think it would expand much like header.tar{.gz} which expands to

headers/0000/type-info
headers/0000/meta-data

not to mention even in the manifest it has the header.tar{.gz} checksum on the .tar{.gz} but the checksum for the rootfs.img is on the .img file living in data/0000 instead of the compressed file???

which leads me to believe that mender-update is looking for a file that doesn’t exist inside the package

as for how to replicate this.

here is my user-data storage settings for a ubuntu 25.04 install:

  storage:
    swap:
      size: 0
    config:
      - type: disk
        match:      # select largest ssd...
          size: largest
        id: disk0    # ...and call it disk0
        ptable: gpt # use gpt partitions on disk0
        wipe: superblock
      - type: partition # create partitions on disk0
        number: 1
        id: efi-partition
        device: disk0
        size: 256M
        flag: boot        # uefi partition needs boot flag
        grub_device: true # and must be the grub device?
      - type: partition
        number: 2
        id: roota-partition
        device: disk0
        size: 10G
      - type: partition
        number: 3
        id: rootb-partition
        device: disk0
        size: 10G
      - type: partition
        number: 4
        id: data-partition
        device: disk0
        size: -1
      - type: format # format partitions on disk0
        id: efi-format
        volume: efi-partition
        fstype: fat32 # ESP gets FAT32
        label: ESP
      - type: format
        id: roota-format
        volume: roota-partition
        fstype: ext4 # / (root) gets ext4, xfs, btrfs
        label: ROOTA
      - type: format
        id: rootb-format
        volume: rootb-partition
        fstype: ext4 # / (root) gets ext4, xfs, btrfs
        label: ROOTB
      - type: format
        id: data-format
        volume: data-partition
        fstype: ext4
        label: DATA
      - type: mount # mount formatted partitions on disk0
        id: root-mount # / (root) gets mounted first
        device: roota-format
        path: /
      - type: mount
        id: efi-mount # /boot/efi gets mounted next
        device: efi-format
        path: /boot/efi
      - type: mount
        id: data-mount
        device: data-format
        path: /data

the rest is pretty basic.

then qemu-img convert -O raw /dev/pve/vm-9001-disk-1 ./disk-v0.0.19-$(date +“%Y%m%d.%H%M”).img

then my mender-convert config is

# Compression
MENDER_COMPRESS_DISK_IMAGE="gzip"
MENDER_ARTIFACT_COMPRESSION="gzip"

# EFI / GRUB
MENDER_GRUB_EFI_INTEGRATION="y"
MENDER_BOOT_PART_MOUNT_LOCATION="/boot/efi"
MENDER_PARTITION_SCHEME="gpt"

# Storage device
MENDER_ENABLE_PARTUUID="y"
MENDER_BOOT_PART="/dev/disk/by-partuuid/66aa2bd1-1329-4b40-94c4-1d1cebd51acc"
MENDER_ROOTFS_PART_A="/dev/disk/by-partuuid/98309743-8135-4000-85fd-00108ae8d601"
MENDER_ROOTFS_PART_B="/dev/disk/by-partuuid/872f9f2e-7f80-4cbf-a694-fd7f98aa114e"
MENDER_DATA_PART="/dev/disk/by-partuuid/8f3f7347-a2df-4674-b642-ef3e16a47798"

# Partition numbers
MENDER_BOOT_PART_NUMBER="1"
MENDER_ROOTFS_PART_A_NUMBER="2"
MENDER_ROOTFS_PART_B_NUMBER="3"
MENDER_DATA_PART_NUMBER="4"

# Partition sizes
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="512"
MENDER_STORAGE_TOTAL_SIZE_MB="40960" # end up with roughly 20G drives

# Rootfs sizing
IMAGE_ROOTFS_SIZE="20070400"   # 19600 MiB
IMAGE_ROOTFS_EXTRA_SPACE="0"
IMAGE_OVERHEAD_FACTOR="1.0"

# Data partition
MENDER_DATA_PART_GROWFS="y"

# Image behavior
MENDER_SPARSE_IMAGE="y"
MENDER_COPY_BOOT_GAP="n"

# Client
MENDER_CLIENT_INSTALL="y"
MENDER_CLIENT_VERSION="4.0.7"

# Device type
MENDER_DEVICE_TYPE="x86_64"

…and here is the wilder part. if i take the rootfs.img and then tar gzip to get rootfs.tar.gz and then i take that and run

mender-artifact write rootfs-image       
--file rootfs.tar.gz       
--output-path v0.0.19-25.mender       
--artifact-name v0.0.19-25       
--device-type x86_64       
--compression none

i can take that that .mender file and mender-update install it just fine. no errors from mender-update. of course even though the install works the new rootf wont work because all it did was dd the tar.gz to the disk making the disk not boot. but the bigger thing is mender-update allows me to install it.

@phynias
Perfect, thanks a lot for the details. I should be able to reproduce it with those, hopefully will get around to it in the next couple of days. I was knocked out with the season’s “fun” a few days and now am running behind, sorry. Will let you know how it goes!

Greetz,
Josef