Issue connecting client to open source production server

General Discussions
1

We are hoping to get a production setup running that will allow us to ensure our raspberrypi4 devices can connect to our server and do some simple protoype testing. I was able to get the demo version working but not the production one.
We are intending to use Mender on its own (no kubernetes, no yocto) so I installed the mender server 3.7.7 on my desktop machine using the installation instructions from the v2.7 docs and was able to get it up and running at http://192.168.2.114. However, after installing the client on the raspberrypi4 from the apt repository (deb bookworm on raspian os) and configuring the setup parameters, I’m unable to get the device to connect. I’ve also tried installing the client from source and using the express-install method as well.
Did the mender-setup as follows:

Enter a name for the device type (e.g. raspberrypi3): [raspberrypi4]

Are you connecting this device to hosted.mender.io? [Y/n] n

Demo server uses a self-signed certifcate for “docker.mender.io” and modifies device’s /etc/hosts with the server’s IP address (Required if using Mender demo server.)
Do you want to configure the client for a demo server? [Y/n] n

Set the URL of the Mender Server: [https://docker.mender.io] http://192.168.2.114

Set the location of the certificate of the server; leave blank if using http (not recommended) or a certificate from a known authority (filepath, for example /etc/mender/server.crt):

Demo intervals uses short poll and retry intervals (Recommended for testing.)
Do you want to run the client in demo mode? [Y/n] y
Mender setup successfully.

I attempted to look into the logs but journalctl -u mender-client returns: “-- No entries --”.
I also tried to view the current status with journalctl -u mender-client -f but it just hangs and returns nothing.
systemctl status mender-client returns: “Unit mender-client.service could not be found.”
How can I troubleshoot this issue?

EDIT: I was able to see the output of mender-connect service:

Oct 28 17:40:31 raspberrypi systemd[1]: Started mender-connect.service - Mender Connect service.
Oct 28 17:40:31 raspberrypi mender-connect[6944]: time=“2024-10-28T17:40:31Z” level=info msg=“Loaded configuration file: /etc/mender/mender-connect.conf”
Oct 28 17:40:31 raspberrypi mender-connect[6944]: time=“2024-10-28T17:40:31Z” level=warning msg=“ShellArguments is empty, defaulting to [–login]”
Oct 28 17:40:31 raspberrypi mender-connect[6944]: time=“2024-10-28T17:40:31Z” level=warning msg="call to GetJWTToken on the Mender D-Bus API failed: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name io.mender.AuthenticationManager was not provided by any .service f

and here is the output of mender-updated:
Oct 28 19:35:46 raspberrypi mender-update[7785]: record_id=5 severity=error time=“2024-Oct-28 19:35:46.396187” name=“Global” msg=“Failed to submit inventory: Unauthorized error: Cannot submit API request”
Oct 28 19:35:46 raspberrypi mender-update[7785]: record_id=6 severity=error time=“2024-Oct-28 19:35:46.666987” name=“Global” msg=“Failed to set new authentication data on HTTP request”
Oct 28 19:35:46 raspberrypi mender-update[7785]: record_id=7 severity=error time=“2024-Oct-28 19:35:46.667108” name=“Global” msg=“Request to check new deployments failed: Cannot submit API request”

2

Hi @G462,

Can you check the logs of mender-authd? That one is the service which handles authentication with the server and provides the connection to mender-updated and mender-connect.

Greetz,
Josef

4

Tried resetting everything up and now mender-authd shows the following:

Oct 31 20:03:54 client mender-auth[66451]: record_id=52 severity=info time=“2024-Oct-31 20:03:54.550549” name=“Global” msg="Authentication error trying server ‘http://192.168.2.114’: Connection refused: POST http://192.168.2.114/api/devices/v1/authentication/auth_requests: "

Oct 31 20:03:54 client mender-auth[66451]: record_id=53 severity=error time=“2024-Oct-31 20:03:54.550750” name=“Global” msg=“Failed to fetch new token: Authentication error: No more servers to try for authentication”

Oct 31 20:03:55 client mender-auth[67207]: using interface /sys/class/net/eth0

5

Hi @G462,

some thoughts:

  • I think it should be https://192.168.2.114.
  • is there a firewall on the host, like Ubuntus ufw?
  • can you check API access manually, for example using curl, as documented here? Mender API docs
    Greetz,
    Josef
6

The logs are slightly different using https://192.168.2.114:

Nov 04 08:51:20 client mender-auth[13352]: using interface /sys/class/net/eth0

Nov 04 08:51:20 client mender-auth[2338]: record_id=1040 severity=info time=“2024-Nov-04 08:51:20.323808” name=“Global” msg=“Signing with: /var/lib/mender/mender-agent.pem”

Nov 04 08:51:20 client mender-auth[2338]: record_id=1041 severity=error time=“2024-Nov-04 08:51:20.505267” name=“http_client” url=“https://192.168.2.114/api/devices/v1/authentication/auth_requests” msg=“https: Failed to perform the SSL handshake: certificate verify failed”

Nov 04 08:51:20 client mender-auth[2338]: record_id=1042 severity=info time=“2024-Nov-04 08:51:20.505836” name=“Global” msg="Authentication error trying server ‘https://192.168.2.114’: certificate verify failed: POST https://192.168.2.114/api/devices/v1/authentication/auth_requests: "

Nov 04 08:51:20 client mender-auth[2338]: record_id=1043 severity=error time=“2024-Nov-04 08:51:20.505988” name=“Global” msg=“Failed to fetch new token: Authentication error: No more servers to try for authentication”

I tried running the curl command on the server to check API access:

curl -v -X POST https://hosted.mender.io/api/management/v1/useradm/auth/login
-H ‘Content-Type: application/json’
-H ‘Accept: application/jwt’

and received the following:

{“error”:“invalid or missing auth header”,“request_id”:“ac8cd6d0-907a-4a73-a324-6522bdbe3c32”}

Should this have generated a JWT token? How can I get a valid JWT token?

7

Hi @G462, this is your key:

The Mender Client can’t check the certificate of 192.168.2.114, and hence it rejects the server. As you won’t be able to get LetsEncrypt on a local IP, the correct approach is to add the certificate to the device and configure the Client accordingly, see Configuration options | Mender documentation.

For the curl check, it shows that the server is alive and responding, I think that’s good enough as we already know https is the area you need to look into.

Greetz,
Josef

10

Hi @TheYoctoJester ,
I’m confident I added the certificate to the device and configured the client to point to it and I’m getting the same SSL error as here: Issue connecting client to open source production server - #6 by G462. Is there anything else I can check?

11

My issue is the same as this unresolved post where its setup does not include kubernetes, only difference is I’m using the latest version (3.7.7):

12

Hi @G462,

I’m about 99% sure that there is something off with your certificate setup. Things to watch out for:

  • device RTC time vs. certificate date and expiry
  • certificate file permissions
  • hostname matches

For a very brutal test, you can add this to /etc/mender/mender.conf, adhering to JSON syntax:

    "SkipVerify": true

If the client connects successfully then, both server and client are functional, but the certificate chain is not. That this is not production ready should be very obvious.

Greetz,
Josef

13

Hey @TheYoctoJester , thanks for the suggestions. This helped me pinpoint the cause of the issue.
The SAN field in my certificate was set to DNS:192.168.2.114 and DNS:*.192.168.2.114 and this was causing issues because these entries were being interpreted as DNS names rather than IP address. My solution was to create a configuration file (san.cnf):

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = 192.168.2.114

[req_ext]
subjectAltName = @alt_names

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = 192.168.2.114

and then generate the new certificate to ensure that the IP address was correctly included in the SAN field as an IP address, not a DNS name:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout private.key -out certificate.crt -config san.cnf

Then it was just a matter of replacing the cert and key on my server and updating the cert on my client after pointing the mender.conf to it.