Mender-update 5.0.4 — malformed timestamp on non-UTC systems causes HTTP 400 on log upload

General Discussions
1

Hi, I just started setting up mender client on our Jetson Devices running 22.04 Linux for Tegra and ran into an issue:

Running mender-update 5.0.4 on Ubuntu 22.04 (Jammy) / ARM64 with system timezone
set to Europe/Berlin. Every deployment fails to upload logs to the server with:

Could not send deployment status: Bad response error: Got unexpected response 400
from logs API: parsing time “2026-03-02T17:20:05.293319Z00Z”: extra text: “00Z”

The timestamp “Z00Z” is malformed — it looks like the client formats the UTC time
with a trailing “Z” and then appends a UTC offset string on top, producing an invalid
RFC3339 timestamp that the server rejects.

Environment:

  • mender-update 5.0.4, installed via ubuntu/jammy/stable APT repo
  • Ubuntu 22.04, ARM64 (NVIDIA Jetson Orin Nano)
  • Timezone: Europe/Berlin (CET, +0100), NTP synced, TZ env var not set
  • date -u +"%Y-%m-%dT%H:%M:%S.%6NZ" produces a clean timestamp — issue is in
    the client’s formatting, not the system

Impact: deployment logs never reach the server, status reporting broken.

Workaround: unknown — does setting the system timezone to UTC fix this?

Has anyone else seen this on non-UTC systems? Is there a known fix or preferred
channel for client bugs?

Edit: i tried switching to UTC but still no luck unfortunately :frowning:

2

I tried downgrading to 5.0.1 and that works - so maybe a regression?

3

Hello @sandesh-nomadic and thank you for report.

Indeed this is a regression introduced in 5.0.4. Versions 5.0.3 and earlier are not affected.

I was able to reproduce and we created this ticket to follow it up:

One detail is that this bug only triggers if there is a corrupted log in the system. Or at least if mender-update detects that the log is corrupted.

So either your log is corrupted for some other reason or our the mechanism from mender-update to detect corruption in the log is flawed for some other issue.

If you have access to the device, would you mind sharing the deployment logs with us?

ls /var/lib/mender/deployments.*.log
cat /var/lib/mender/deployments.*.log

Thank you!

Lluís

4

Hello @lluiscampos ,

thank you for the answer and that you are able to reproduce - that’s great!

As I was just setting up mender I ran into a number of issues which I could not validate individually. I was already wondering how this was not yet reported if it was happening for all logs. I later found out that the log was indeed corrupt, because I ran a pip install inside a script and that printed non-ASCII characters (e.g. green progress bar etc.).

I fixed this by updating my update module to strip all non-ASCII characters and set pip to –no-color but did not know that this specifically then caused the time error. So I cannot share the logs right now but you can reproduce by just running a pip install that logs color to the terminal.

This means I should be able to update to 5.0.4 now that I have fixed the corrupt log, right?

What is the behavior if I encounter a corrupt log again?

  • on 5.0.3 it is not published because it is corrupt? Or will it still be uploaded somehow?
  • on 5.0.4 it is not published because we get this server rejected error

I am just wondering whether I can safely update because I don’t lose any logs with the new version.

Thanks,

Sandesh