# Issue connecting client to open source production server

**URL:** <https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281>\
**Category:** General Discussions\
**Created:** [October 28, 2024, 4:32pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281 "2024-10-28T16:32:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![G462](https://avatars.discourse-cdn.com/v4/letter/g/b5e925/32.png) [@G462](https://hub.mender.io/u/G462)\
**Post date:** [October 28, 2024, 4:32pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/1 "2024-10-28T16:32:45Z")

</div>

We are hoping to get a production setup running that will allow us to ensure our raspberrypi4 devices can connect to our server and do some simple protoype testing. I was able to get the demo version working but not the production one.  
We are intending to use Mender on its own (no kubernetes, no yocto) so I installed the mender server 3.7.7 on my desktop machine using the installation instructions from the v2.7 docs and was able to get it up and running at [http://192.168.2.114](http://192.168.2.114). However, after installing the client on the raspberrypi4 from the apt repository (deb bookworm on raspian os) and configuring the setup parameters, I’m unable to get the device to connect. I’ve also tried installing the client from source and using the express-install method as well.  
Did the mender-setup as follows:

> Enter a name for the device type (e.g. raspberrypi3): [raspberrypi4]
> 
> Are you connecting this device to [hosted.mender.io](http://hosted.mender.io)? [Y/n] n
> 
> Demo server uses a self-signed certifcate for “[docker.mender.io](http://docker.mender.io)” and modifies device’s /etc/hosts with the server’s IP address (Required if using Mender demo server.)  
> Do you want to configure the client for a demo server? [Y/n] n
> 
> Set the URL of the Mender Server: [[https://docker.mender.io](https://docker.mender.io)] [http://192.168.2.114](http://192.168.2.114)
> 
> Set the location of the certificate of the server; leave blank if using http (not recommended) or a certificate from a known authority (filepath, for example /etc/mender/server.crt):
> 
> Demo intervals uses short poll and retry intervals (Recommended for testing.)  
> Do you want to run the client in demo mode? [Y/n] y  
> Mender setup successfully.

I attempted to look into the logs but journalctl -u mender-client returns: “-- No entries --”.  
I also tried to view the current status with journalctl -u mender-client -f but it just hangs and returns nothing.  
systemctl status mender-client returns: “Unit mender-client.service could not be found.”  
How can I troubleshoot this issue?

EDIT: I was able to see the output of mender-connect service:

Oct 28 17:40:31 raspberrypi systemd[1]: Started mender-connect.service - Mender Connect service.  
Oct 28 17:40:31 raspberrypi mender-connect[6944]: time=“2024-10-28T17:40:31Z” level=info msg=“Loaded configuration file: /etc/mender/mender-connect.conf”  
Oct 28 17:40:31 raspberrypi mender-connect[6944]: time=“2024-10-28T17:40:31Z” level=warning msg=“ShellArguments is empty, defaulting to [–login]”  
Oct 28 17:40:31 raspberrypi mender-connect[6944]: time=“2024-10-28T17:40:31Z” level=warning msg="call to GetJWTToken on the Mender D-Bus API failed: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name io.mender.AuthenticationManager was not provided by any .service f

and here is the output of mender-updated:  
Oct 28 19:35:46 raspberrypi mender-update[7785]: record\_id=5 severity=error time=“2024-Oct-28 19:35:46.396187” name=“Global” msg=“Failed to submit inventory: Unauthorized error: Cannot submit API request”  
Oct 28 19:35:46 raspberrypi mender-update[7785]: record\_id=6 severity=error time=“2024-Oct-28 19:35:46.666987” name=“Global” msg=“Failed to set new authentication data on HTTP request”  
Oct 28 19:35:46 raspberrypi mender-update[7785]: record\_id=7 severity=error time=“2024-Oct-28 19:35:46.667108” name=“Global” msg=“Request to check new deployments failed: Cannot submit API request”

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [October 29, 2024, 6:12am UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/2 "2024-10-29T06:12:05Z")

</div>

Hi @G462,

Can you check the logs of `mender-authd`? That one is the service which handles authentication with the server and provides the connection to `mender-updated` and `mender-connect`.

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![G462](https://avatars.discourse-cdn.com/v4/letter/g/b5e925/32.png) [@G462](https://hub.mender.io/u/G462)\
**Post date:** [October 31, 2024, 7:50pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/4 "2024-10-31T19:50:52Z")

</div>

Tried resetting everything up and now mender-authd shows the following:

> Oct 31 20:03:54 client mender-auth[66451]: record\_id=52 severity=info time=“2024-Oct-31 20:03:54.550549” name=“Global” msg="Authentication error trying server ‘[http://192.168.2.114](http://192.168.2.114)’: Connection refused: POST [http://192.168.2.114/api/devices/v1/authentication/auth\_requests:](http://192.168.2.114/api/devices/v1/authentication/auth_requests:) "
> 
> Oct 31 20:03:54 client mender-auth[66451]: record\_id=53 severity=error time=“2024-Oct-31 20:03:54.550750” name=“Global” msg=“Failed to fetch new token: Authentication error: No more servers to try for authentication”
> 
> Oct 31 20:03:55 client mender-auth[67207]: using interface /sys/class/net/eth0

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [November 1, 2024, 8:04am UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/5 "2024-11-01T08:04:38Z")

</div>

Hi @G462,

some thoughts:

- I think it should be `https://192.168.2.114`.
- is there a firewall on the host, like Ubuntus `ufw`?
- can you check API access manually, for example using `curl`, as documented here? [Mender API docs](https://docs.mender.io/api/?shell#management-api-user-administration-and-authentication-login)  
Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![G462](https://avatars.discourse-cdn.com/v4/letter/g/b5e925/32.png) [@G462](https://hub.mender.io/u/G462)\
**Post date:** [November 4, 2024, 4:05pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/6 "2024-11-04T16:05:41Z")

</div>

The logs are slightly different using [https://192.168.2.114](https://192.168.2.114):

> Nov 04 08:51:20 client mender-auth[13352]: using interface /sys/class/net/eth0
> 
> Nov 04 08:51:20 client mender-auth[2338]: record\_id=1040 severity=info time=“2024-Nov-04 08:51:20.323808” name=“Global” msg=“Signing with: /var/lib/mender/mender-agent.pem”
> 
> Nov 04 08:51:20 client mender-auth[2338]: record\_id=1041 severity=error time=“2024-Nov-04 08:51:20.505267” name=“http\_client” url=“[https://192.168.2.114/api/devices/v1/authentication/auth\_requests](https://192.168.2.114/api/devices/v1/authentication/auth_requests)” msg=“https: Failed to perform the SSL handshake: certificate verify failed”
> 
> Nov 04 08:51:20 client mender-auth[2338]: record\_id=1042 severity=info time=“2024-Nov-04 08:51:20.505836” name=“Global” msg="Authentication error trying server ‘[https://192.168.2.114](https://192.168.2.114)’: certificate verify failed: POST [https://192.168.2.114/api/devices/v1/authentication/auth\_requests:](https://192.168.2.114/api/devices/v1/authentication/auth_requests:) "
> 
> Nov 04 08:51:20 client mender-auth[2338]: record\_id=1043 severity=error time=“2024-Nov-04 08:51:20.505988” name=“Global” msg=“Failed to fetch new token: Authentication error: No more servers to try for authentication”

I tried running the curl command on the server to check API access:

> curl -v -X POST [https://hosted.mender.io/api/management/v1/useradm/auth/login](https://hosted.mender.io/api/management/v1/useradm/auth/login)   
> -H ‘Content-Type: application/json’   
> -H ‘Accept: application/jwt’

and received the following:

> {“error”:“invalid or missing auth header”,“request\_id”:“ac8cd6d0-907a-4a73-a324-6522bdbe3c32”}

Should this have generated a JWT token? How can I get a valid JWT token?

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [November 5, 2024, 11:32am UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/7 "2024-11-05T11:32:56Z")

</div>

Hi @G462, this is your key:

> [@G462](#):
>
> msg=“https: Failed to perform the SSL handshake: certificate verify failed”

The Mender Client can’t check the certificate of `192.168.2.114`, and hence it rejects the server. As you won’t be able to get LetsEncrypt on a local IP, the correct approach is to add the certificate to the device and configure the Client accordingly, see [Configuration options | Mender documentation](https://docs.mender.io/client-installation/configuration/configuration-options#httpsclient).

For the `curl` check, it shows that the server is alive and responding, I think that’s good enough as we already know `https` is the area you need to look into.

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![G462](https://avatars.discourse-cdn.com/v4/letter/g/b5e925/32.png) [@G462](https://hub.mender.io/u/G462)\
**Post date:** [November 13, 2024, 8:13pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/10 "2024-11-13T20:13:04Z")

</div>

Hi @TheYoctoJester ,  
I’m confident I added the certificate to the device and configured the client to point to it and I’m getting the same SSL error as here: [Issue connecting client to open source production server - #6 by G462](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/6). Is there anything else I can check?

---

<div class="post-metadata">

**Author:** ![G462](https://avatars.discourse-cdn.com/v4/letter/g/b5e925/32.png) [@G462](https://hub.mender.io/u/G462)\
**Post date:** [November 13, 2024, 9:10pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/11 "2024-11-13T21:10:19Z")

</div>

My issue is the same as this unresolved post where its setup does not include kubernetes, only difference is I’m using the latest version (3.7.7):

> [@TLS Handshake Verification Failure in Mender 3.4 Production Installation](https://hub.mender.io/t/tls-handshake-verification-failure-in-mender-3-4-production-installation/7171):
>
> I’m using Mender 3.4 production installation with Docker. I was able to set up the Mender server and log into the dashboard successfully. However, when attempting to connect a device using the Mender client, I encountered a TLS handshake verification failure. While running the client, I provided the same TLS certificate used for the server. I need help troubleshooting and resolving this issue. Here is the logs Setting up the Mender client: The client will regularly poll the server to check for…

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [November 14, 2024, 3:51pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/12 "2024-11-14T15:51:12Z")

</div>

Hi @G462,

I’m about 99% sure that there is something off with your certificate setup. Things to watch out for:

- device RTC time vs. certificate date and expiry
- certificate file permissions
- hostname matches

For a very brutal test, you can add this to `/etc/mender/mender.conf`, adhering to JSON syntax:

```auto
    "SkipVerify": true

```

If the client connects successfully then, both server and client are functional, but the certificate chain is not. That this is not production ready should be very obvious.

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![G462](https://avatars.discourse-cdn.com/v4/letter/g/b5e925/32.png) [@G462](https://hub.mender.io/u/G462)\
**Post date:** [November 14, 2024, 9:36pm UTC](https://hub.mender.io/t/issue-connecting-client-to-open-source-production-server/7281/13 "2024-11-14T21:36:50Z")

</div>

Hey @TheYoctoJester , thanks for the suggestions. This helped me pinpoint the cause of the issue.  
The SAN field in my certificate was set to DNS:192.168.2.114 and DNS:\*.192.168.2.114 and this was causing issues because these entries were being interpreted as DNS names rather than IP address. My solution was to create a configuration file (san.cnf):

> [req]  
> distinguished\_name = req\_distinguished\_name  
> req\_extensions = req\_ext  
> x509\_extensions = v3\_req  
> prompt = no
> 
> [req\_distinguished\_name]  
> CN = 192.168.2.114
> 
> [req\_ext]  
> subjectAltName = @alt\_names
> 
> [v3\_req]  
> subjectAltName = @alt\_names
> 
> [alt\_names]  
> IP.1 = 192.168.2.114

and then generate the new certificate to ensure that the IP address was correctly included in the SAN field as an IP address, not a DNS name:

> openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout private.key -out certificate.crt -config san.cnf

Then it was just a matter of replacing the cert and key on my server and updating the cert on my client after pointing the mender.conf to it.
