Mender-update 5.0.4 — malformed timestamp on non-UTC systems causes HTTP 400 on log upload

General Discussions
1

Hi, I just started setting up mender client on our Jetson Devices running 22.04 Linux for Tegra and ran into an issue:

Running mender-update 5.0.4 on Ubuntu 22.04 (Jammy) / ARM64 with system timezone
set to Europe/Berlin. Every deployment fails to upload logs to the server with:

Could not send deployment status: Bad response error: Got unexpected response 400
from logs API: parsing time “2026-03-02T17:20:05.293319Z00Z”: extra text: “00Z”

The timestamp “Z00Z” is malformed — it looks like the client formats the UTC time
with a trailing “Z” and then appends a UTC offset string on top, producing an invalid
RFC3339 timestamp that the server rejects.

Environment:

  • mender-update 5.0.4, installed via ubuntu/jammy/stable APT repo
  • Ubuntu 22.04, ARM64 (NVIDIA Jetson Orin Nano)
  • Timezone: Europe/Berlin (CET, +0100), NTP synced, TZ env var not set
  • date -u +"%Y-%m-%dT%H:%M:%S.%6NZ" produces a clean timestamp — issue is in
    the client’s formatting, not the system

Impact: deployment logs never reach the server, status reporting broken.

Workaround: unknown — does setting the system timezone to UTC fix this?

Has anyone else seen this on non-UTC systems? Is there a known fix or preferred
channel for client bugs?

Edit: i tried switching to UTC but still no luck unfortunately :frowning:

2

I tried downgrading to 5.0.1 and that works - so maybe a regression?

3

Hello @sandesh-nomadic and thank you for report.

Indeed this is a regression introduced in 5.0.4. Versions 5.0.3 and earlier are not affected.

I was able to reproduce and we created this ticket to follow it up:

One detail is that this bug only triggers if there is a corrupted log in the system. Or at least if mender-update detects that the log is corrupted.

So either your log is corrupted for some other reason or our the mechanism from mender-update to detect corruption in the log is flawed for some other issue.

If you have access to the device, would you mind sharing the deployment logs with us?

ls /var/lib/mender/deployments.*.log
cat /var/lib/mender/deployments.*.log

Thank you!

Lluís

4

Hello @lluiscampos ,

thank you for the answer and that you are able to reproduce - that’s great!

As I was just setting up mender I ran into a number of issues which I could not validate individually. I was already wondering how this was not yet reported if it was happening for all logs. I later found out that the log was indeed corrupt, because I ran a pip install inside a script and that printed non-ASCII characters (e.g. green progress bar etc.).

I fixed this by updating my update module to strip all non-ASCII characters and set pip to –no-color but did not know that this specifically then caused the time error. So I cannot share the logs right now but you can reproduce by just running a pip install that logs color to the terminal.

This means I should be able to update to 5.0.4 now that I have fixed the corrupt log, right?

What is the behavior if I encounter a corrupt log again?

  • on 5.0.3 it is not published because it is corrupt? Or will it still be uploaded somehow?
  • on 5.0.4 it is not published because we get this server rejected error

I am just wondering whether I can safely update because I don’t lose any logs with the new version.

Thanks,

Sandesh

5

Hello,

Sorry for my last response.

Before 5.0.4 the a client with corrupted logs will just fail to report them to the server, getting 400 by the server. We had bug forever, at least since 4.0.0.

Now, 5.0.4 has for the first time an attempted fix for this issue, which as you experienced is buggy. So in this regard the attempt to rescue the valid logs fails and ends up with also 400 errors from the server (what an irony, isn’t it?)

In short it is safe for you to run 5.0.4 as long as you don’t use that pip install command. It is as safe as 5.0.3 or earlier, meaning both will fail to report logs if there is binary data there.

We are working on a better fix as we speak, you can see the PR here:

So hopefully you will get this fix in 5.0.5 and 5.1.1 soon enough :wink:

Cheers,
Lluís

6

Hey Lluis,

I have a follow-up question: When I experience this issue, next to not seeing the log, the device also does not accept any further updates. Will this also be fixed in the issue you linked? Thank you!

Cheers,

Sandesh

7

Hi @sandesh-nomadic,

I have just discussed this with the developers, and the second issue you describe does not seem to be related. Can you provide a bit more detail on what you are seeing? Obvious things to check:

  • does the device check correctly for deployments?
  • or does the installation fail?
  • can you obtain the logs manually from a local device?

On which version are you now, specifically? Did you bump the Mender Client version manually via compilation, or did you stay with 5.0.4?

Greetz,
Josef

8

Hi Josef,

thank you for checking! The device does not check for updates, so the update always stays at pending in the platform. The logs just state the error in log upload (every few seconds) and nothing else, hence I assumed that it was stuck with a malformed log. I do not see the typical logs that it received a new update and starts installing it etc.

I found out that in one of these cases the log was just consisting of zeros \x00 and claude assumed it might be from a power outage during update which led to a file write. This then causes the log upload to fail and more importantly prevents further updates.

9

Hi @sandesh-nomadic,

That really sounds weird. Is the Mender service running? Can you check systemctl status on both mender-updated and mender-authd, and eventually the ownership/permission of the files in /data/mender? I’ve never heard of, or seen a similar situation so I really wonder what’s going on.

Greetz,
Josef