Mender-client startup error loading configuration file

General Discussions
1

I have switched from demo to production using mender-convert but I’m getting an error on startup of the OTA update service. I have no idea where to look for this, below is the error log.

-- Logs begin at Tue 2024-01-30 09:17:01 GMT, end at Tue 2024-01-30 09:48:38 GMT. --
Jan 30 09:32:51 xxxxxxxxxxx systemd[1]: Started Mender OTA update service.
Jan 30 09:32:52 xxxxxxxxxxx mender[422]: time="2024-01-30T09:32:52Z" level=error msg="Error loading configuration from file: /var/lib/mender/mender.conf (Error parsing mender configuration file: invalid character '}' looking for beginning of object key string)"
Jan 30 09:32:52 xxxxxxxxxxx mender[422]: time="2024-01-30T09:32:52Z" level=error msg="Error parsing mender configuration file: invalid character '}' looking for beginning of object key string"
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Failed with result 'exit-code'.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Service RestartSec=100ms expired, scheduling restart.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Scheduled restart job, restart counter is at 1.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: Stopped Mender OTA update service.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: Started Mender OTA update service.
Jan 30 09:32:52 xxxxxxxxxxx mender[543]: time="2024-01-30T09:32:52Z" level=error msg="Error loading configuration from file: /var/lib/mender/mender.conf (Error parsing mender configuration file: invalid character '}' looking for beginning of object key string)"
Jan 30 09:32:52 xxxxxxxxxxx mender[543]: time="2024-01-30T09:32:52Z" level=error msg="Error parsing mender configuration file: invalid character '}' looking for beginning of object key string"
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Failed with result 'exit-code'.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Service RestartSec=100ms expired, scheduling restart.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Scheduled restart job, restart counter is at 2.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: Stopped Mender OTA update service.
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: Started Mender OTA update service.
Jan 30 09:32:52 xxxxxxxxxxx mender[558]: time="2024-01-30T09:32:52Z" level=error msg="Error loading configuration from file: /var/lib/mender/mender.conf (Error parsing mender configuration file: invalid character '}' looking for beginning of object key string)"
Jan 30 09:32:52 xxxxxxxxxxx mender[558]: time="2024-01-30T09:32:52Z" level=error msg="Error parsing mender configuration file: invalid character '}' looking for beginning of object key string"
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 09:32:52 xxxxxxxxxxx systemd[1]: mender-client.service: Failed with result 'exit-code'.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Service RestartSec=100ms expired, scheduling restart.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Scheduled restart job, restart counter is at 3.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: Stopped Mender OTA update service.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: Started Mender OTA update service.
Jan 30 09:32:53 xxxxxxxxxxx mender[566]: time="2024-01-30T09:32:53Z" level=error msg="Error loading configuration from file: /var/lib/mender/mender.conf (Error parsing mender configuration file: invalid character '}' looking for beginning of object key string)"
Jan 30 09:32:53 xxxxxxxxxxx mender[566]: time="2024-01-30T09:32:53Z" level=error msg="Error parsing mender configuration file: invalid character '}' looking for beginning of object key string"
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Failed with result 'exit-code'.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Service RestartSec=100ms expired, scheduling restart.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Scheduled restart job, restart counter is at 4.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: Stopped Mender OTA update service.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: Started Mender OTA update service.
Jan 30 09:32:53 xxxxxxxxxxx mender[575]: time="2024-01-30T09:32:53Z" level=error msg="Error loading configuration from file: /var/lib/mender/mender.conf (Error parsing mender configuration file: invalid character '}' looking for beginning of object key string)"
Jan 30 09:32:53 xxxxxxxxxxx mender[575]: time="2024-01-30T09:32:53Z" level=error msg="Error parsing mender configuration file: invalid character '}' looking for beginning of object key string"
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Failed with result 'exit-code'.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Service RestartSec=100ms expired, scheduling restart.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Scheduled restart job, restart counter is at 5.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: Stopped Mender OTA update service.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Start request repeated too quickly.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: mender-client.service: Failed with result 'exit-code'.
Jan 30 09:32:53 xxxxxxxxxxx systemd[1]: Failed to start Mender OTA update service.
2

According to the lines above, it looks like /var/lib/mender/mender.conf is not formatted correctly. Would you mind sharing the file (ensure to redact any sensitive data)?

3

It is only

{
  "RootfsPartA": "/dev/mmcblk0p2",
  "RootfsPartB": "/dev/mmcblk0p3"
}

John
4

I tried removing /var/lib/mender/mender.conf and /etc/mender/mender.conf and I got the following

Jan 16 14:05:29 xxxxxxxxxxx systemd[1]: Started Mender OTA update service.
Jan 16 14:05:30 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:30Z" level=info msg="'UpdateControlMapExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 2*UpdatePollIntervalSeconds"
Jan 16 14:05:30 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:30Z" level=info msg="'UpdateControlMapBootExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 600 seconds"
Jan 16 14:05:30 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:30Z" level=info msg="No configuration files present. Using defaults"
Jan 16 14:05:30 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:30Z" level=warning msg="No server URL(s) specified in mender configuration."
Jan 16 14:05:30 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:30Z" level=warning msg="Server entry 1 has no associated server URL."
Jan 16 14:05:30 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:30Z" level=info msg="No dual rootfs configuration present"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="Bootstrap Artifact installed successfully"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="State transition: init [none] -> init [none]"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="State transition: init [none] -> idle [Idle]"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="State transition: idle [Idle] -> check-wait [Idle]"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="Output (stderr) from command \"/usr/share/mender/inventory/mender-inventory-geo\": /usr/share/mender/inventory/mender-inventory-geo: Unable to get IP info from ipinfo.io"
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Jan 16 14:05:31 xxxxxxxxxxx mender[572]: time="2024-01-16T14:05:31Z" level=warning msg="No server URL(s) specified in mender configuration."
Jan 16 14:05:31 xxxxxxxxxxx mender[572]: time="2024-01-16T14:05:31Z" level=warning msg="Server entry 1 has no associated server URL."
Jan 16 14:05:31 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:31Z" level=info msg="Output (stderr) from command \"/usr/share/mender/inventory/mender-inventory-provides\": time=\"2024-01-16T14:05:31Z\" level=warning msg=\"No server URL(s) specified in mender configuration.\""
Jan 16 14:05:32 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:32Z" level=info msg="Output (stderr) from command \"/usr/share/mender/inventory/mender-inventory-provides\": time=\"2024-01-16T14:05:31Z\" level=warning msg=\"Server entry 1 has no associated server URL.\""
Jan 16 14:05:32 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:32Z" level=info msg="Device unauthorized; attempting reauthorization"
Jan 16 14:05:32 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:32Z" level=info msg="Device keys not present or bootstrap forced, generating"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=info msg="Output (stderr) from command \"/usr/share/mender/identity/mender-device-identity\": using interface /sys/class/net/eth0"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=error msg="Failure occurred while executing authorization request: Method: Post, URL: https:///api/devices/v1/authentication/auth_requests"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=error msg="Failed to authorize with \"\": Unknown url.Error type: http: no Host in request URL"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=warning msg="Reauthorization failed with error: transient error: authorization request failed"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=error msg="Failed to submit inventory data: transient error: authorization request failed"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=error msg="inventory submit failed: transient error: authorization request failed"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=warning msg="Failed to refresh inventory: failed to submit inventory data: inventory submit failed: transient error: authorization request failed"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=info msg="State transition: inventory-update [Sync] -> inventory-update-retry-wait [Sync]"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=info msg="Handle update inventory retry state try: 0"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=info msg="Output (stderr) from command \"/usr/share/mender/inventory/mender-inventory-geo\": /usr/share/mender/inventory/mender-inventory-geo: Unable to get IP info from ipinfo.io"
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2"
Jan 16 14:05:39 xxxxxxxxxxx mender[683]: time="2024-01-16T14:05:39Z" level=warning msg="No server URL(s) specified in mender configuration."
Jan 16 14:05:39 xxxxxxxxxxx mender[683]: time="2024-01-16T14:05:39Z" level=warning msg="Server entry 1 has no associated server URL."
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=info msg="Output (stderr) from command \"/usr/share/mender/inventory/mender-inventory-provides\": time=\"2024-01-16T14:05:39Z\" level=warning msg=\"No server URL(s) specified in mender configuration.\""
Jan 16 14:05:39 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:39Z" level=info msg="Output (stderr) from command \"/usr/share/mender/inventory/mender-inventory-provides\": time=\"2024-01-16T14:05:39Z\" level=warning msg=\"Server entry 1 has no associated server URL.\""
Jan 16 14:05:40 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:40Z" level=info msg="Device unauthorized; attempting reauthorization"
Jan 16 14:05:40 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:40Z" level=info msg="Output (stderr) from command \"/usr/share/mender/identity/mender-device-identity\": using interface /sys/class/net/eth0"
Jan 16 14:05:40 xxxxxxxxxxx mender[424]: time="2024-01-16T14:05:40Z" level=error msg="Failure occurred while executing authorization request: Method: Post, URL: https:///api/devices/v1/authentication/auth_requests"

So it does look that the parsing of the .conf files is going wrong.

John

5

@TheYoctoJester Could this be related to problems with having to use 4.1.1 with version set to 3.5.2?

John

6

Solved:
The error message specified /var/lib/mender/mender.conf but the actual error was in /etc/mender/mender.conf! mender-convert had left a trailing comma before the terminating }.

John

1 Like