record_id=29 severity=trace time="2025-Mar-20 15:52:49.055568" name="Global" msg="Entry name: headers/0000/meta-data"
record_id=30 severity=trace time="2025-Mar-20 15:52:49.055782" name="Global" msg="StringToType: headers/0000/meta-data"
record_id=31 severity=trace time="2025-Mar-20 15:52:49.056995" name="Global" msg="sub-header: looking for meta-data"
record_id=32 severity=trace time="2025-Mar-20 15:52:49.057215" name="Global" msg="Parsing the header meta-data"
record_id=33 severity=trace time="2025-Mar-20 15:52:49.057851" name="Global" msg="Received json load error: Failed to parse JSON from stream: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal"
record_id=34 severity=trace time="2025-Mar-20 15:52:49.058091" name="Global" msg="Received an empty Json body. Not treating this as an error"
record_id=35 severity=trace time="2025-Mar-20 15:52:49.058385" name="Global" msg="sub-header: parsed the meta-data"
Installing artifact...
record_id=36 severity=error time="2025-Mar-20 15:52:49.059771" name="Global" msg="Type error: Invalid JSON type to get children from"
It says here mender-artifact/Documentation/artifact-format-v3.md at 4.0.0 · mendersoftware/mender-artifact · GitHub that:
meta-data
Format: JSON
Meta data about the image. This depends on the type in header-info. For rootfs-image there are no additional information needed and the file might be empty.
Hi @ernstp-se,
Thanks for reaching out. Reading through the documentation and code, it sounds like this might be a regression from Client 3.x to 4.x/5.x, but I have to verify it. I take it that you’re using a custom-crafted mechanism to create the artifact? Did this work up to now, or is it a recent addition?
Judging from the code additionally, a very quick workaround might be to not include the meta-data key at all, as it is considered fully optional.
Greetz,
Josef
mender-artifact 3.10.0 + mender-client 3.10.0: ok
mender-artifact 3.10.0 + mender-client 4.0.6: fail
mender-artifact 4.0.0 + mender-client 4.0.6: fail
Yeah right, this is a regression, mender-client 3.10.0 accepts these files.
Not using any custom mechanism to create the artifact, meta-mender with the image type mender for both kirkstone and scarthgap seems to create files like this using the mender-artifact tool.
@ernstp-se strange. Ok, I’ll look into it. Can you drop the commit revision of meta-mender that you’re using?
Greetz,
Josef
The .mender artifact seems to be created with empty “header-info” by both
commit df6b158fb00fb5c9d524bff277122adfc1a5e6ff (tag: kirkstone-v2023.02)
and
commit 694ac37314167a2b5445f503c1baaeb3b1b15a4e (scarthgap)
Hi @ernstp-se,
This indeed seems like a bad regression. I cannot understand (yet) how it went unnoticed.
Could you share the headers of the exact artifact that is triggering this issue? I crafted this command that should do the job, but check the output before pasting it here in case it has some sensitive information from your end:
tar -xf YOUR-ARTIFACT.mender header.tar.gz
tar -xf header.tar.gz
echo header-info; cat header-info; echo;
for file in $(find headers/ -type f); do echo $file; cat $file; echo; done
Something like that. Thank you!
@ernstp-se Can you also share with us the output of mender-update show-provides from the device?
# tar -xf tmp/deploy/images/esmi-fdp/esmi-image-master-esmi-fdp.mender
# tar -xf header.tar.gz
# echo header-info; cat header-info; echo;
header-info
{"payloads":[{"type":"rootfs-image"}],"artifact_provides":{"artifact_name":"localbuild-20250320114522"},"artifact_depends":{"device_type":["esmi-fdp"]}}
# for file in $(find headers/ -type f); do echo $file; cat $file; echo; done
headers/0000/meta-data
headers/0000/type-info
{"type":"","artifact_provides":{"rootfs-image.update-module.deb.mender_update_module":"deb","rootfs-image.update-module.deb.version":"localbuild-20250320114522","rootfs-image.update-module.directory.mender_update_module":"directory","rootfs-image.update-module.directory.version":"localbuild-20250320114522","rootfs-image.update-module.docker.mender_update_module":"docker","rootfs-image.update-module.docker.version":"localbuild-20250320114522","rootfs-image.update-module.rpm.mender_update_module":"rpm","rootfs-image.update-module.rpm.version":"localbuild-20250320114522","rootfs-image.update-module.script.mender_update_module":"script","rootfs-image.update-module.script.version":"localbuild-20250320114522","rootfs-image.update-module.single-file.mender_update_module":"single-file","rootfs-image.update-module.single-file.version":"localbuild-20250320114522","rootfs-image.version":"24.12.9999"},"clears_artifact_provides":["artifact_group","rootfs_image_checksum","rootfs-image.*"]}
Thank you @ernstp-se
The header is just fine. You can see that the meta-data is a purely empty file, which is allowed, and also from your logs you can see a trace one saying: “Received an empty Json body. Not treating this as an error”
So the parsing of the Artifact header succeeds, and indeed the Client starts the installation.
Then, the error triggers.
It is a big mystery for me right now. The only relevant place in the code that GetChildren() is used is here, but that should be called before we do “Installing Artifact…” (and it is also called even before when performing the API call to the server).
I need to experiment a bit more to find out the root cause.
Can you provide the output of mender-update show-provides ?
Oh I see, I was focused on that error but it does actually continue a little bit.
mender-update -l trace show-provides
record_id=1 severity=debug time="2025-Mar-21 14:09:03.382745" name="Global" msg="Failed to load config from '/var/lib/mender/mender.conf': Failed to open '/var/lib/mender/mender.conf': No such file or directory"
Could not fulfill request: Type error: Invalid JSON type to get children from
We have been using /etc/mender/mender.conf but that is no longer used… ?
mender-update -c /etc/mender/mender.conf -l trace show-provides
record_id=1 severity=debug time="2025-Mar-21 14:14:27.017670" name="Global" msg="Failed to load config from '/var/lib/mender/mender.conf': Failed to open '/var/lib/mender/mender.conf': No such file or directory"
Could not fulfill request: Type error: Invalid JSON type to get children from
env | grep MENDER is empty.
@ernstp-se Bingo! That must be the real problem.
First, please note that the error message there is just a debug entry - it warns that it did correctly parsed /etc/mender/mender.conf but failed on the extra one in the persistent partition /var/lib/.... It might be that you are just not using it on your deployment. But this is not an issue.
The issues is in parsing the provides from the database, as I suspected. Although it is still a mystery why the client can do the API call for asking for deployments if it cannot even read the stored provides (unless it is defaulting to the v1 deployments/next 
)
I unfortunately don’t have more time today to help out but the next step is to dump the database and see what are the contents of the artifact_provides key. I can help on that next week, of mabye @TheYoctoJester can give you the recipe.
Also please let’s look at the logs of the API calls when the client checks for deployments, I bet there is an error there and it is just silently doing a fallback to the v1 deployments/next API call.
Have a good weekend,
Lluis
1 Like
What is the best way to dump the database?
After running rm /var/lib/mender/mender-store* I could run mender-update install. Just trying to understand the problem, not suggesting this is a solution…
@ernstp-se I’ve been offline few days, just today catching up and wanted to provide you the recipe to do the backup + dump the database… I see it is too late now 
What is the best way to dump the database?
With lmdb_dump.
After running rm /var/lib/mender/mender-store* I could run mender-update install
There was something corrupted in the database, hence the error from show-provides but I wanted to find out how you ended up in that situation. Did you take a backup of the database? 
Otherwise we cannot investigate further.
Please report back if you get back to this state.
I did make a copy of the database. And now I actually reproduce the issue. Not sure how yet though!
Where can I find lmdb_dump?