Hello,
I’m upgrading our self-hosted server from 2.6.x to 3.2.1. I believe we’re running into issues due to changes in how the certificates/private keys are generated.
If I use the keygen script to generate a brand new set of keys, everything works fine i.e. the website loads and I can see our existing artifacts, deployments, etc, but the server keys are now different than the ones stored on the devices. If I try to copy our old keys, mender-deployments keeps restarting and I get the following error:
RequestError: send request failed mender-deployments_1 | caused by: Put "https://my-mender-url/mender-artifact-storage": EOF
I suspect the problem is because instead of having separate api-gateway and storage-proxy certs/keys, you now just have one certificate and key. I’ve tried concatenating the existing api-gateway and storage-proxy certs and private key into a single cert.crt and private.key file, but that hasn’t worked.
How can we migrate our old style keys to be compatible with the new layout?
I think the problem is due to the fact that the single certificate/key that is currently used now has a subject alternative name that includes a wildcard (e.g. *.mycomain.com). Presumably, some of the services have been changed to point to a subdomain.
If this is true, it sounds like a pretty breaking change - up until version 3.0.2 the keygen script generated separate keys for the storage-proxy and api-gateway which pointed to a fixed domain. In 3.1.0, this changed and only one key was generated that had a wildcard domain. The problem with this is that it seems that the services are now actually using the subdomain and so existing certificates cannot be used in place of this single certificate.
Hope the Mender team can shed light on this.
Are you using self signed certificate for server in LAN production? if you are I would recommend also generating your own self signed CA cert and create your server certificates signed by the CA cert. then you only need to install the public part of your CA cert on your devices. Then you are free to create/update your server cert as needed over time as long as it’s signed by the same CA cert.
If your not using self-signed and using a 3rd party Certificate Authority, and you OS doesnt already have the CA cert and its intermediates certs (if any), then add them to your devices rather than the server cert itself for the same reasons as above.
One thing that has to be noted is that Mender needs the certificate to have filled the DNS part of Alternative Names in the certificate that is stored in the traefik container.
You can see this using the following command.
openssl x509 -in CERTIFICATE -text
Look at the DNS part in the example below.
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
16:b3:a8:9f:fd:2c:31:2f:b1:fe:41:a3:83:86:e6:a6:ff:a8:e1:b6
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = #YOUR DOMAIN NAME
Validity
Not Before: Mar 25 10:18:16 2022 GMT
Not After : Mar 22 10:18:16 2032 GMT
Subject: CN = #YOUR DOMAIN NAME
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:ad:1c:a3:23:47:0c:e7:87:5c:1a:ee:a3:fb:5e:
62:14:1b:c5:d6:24:ed:b0:bc:26:0a:01:07:9d:d9:
d0:71:ba:ed:62:19:3e:4d:1a:4e:0d:ae:0a:14:82:
50:5d:1d:7f:3d:f8:b1:24:be:f2:8c:10:3c:eb:bd:
17:7f:28:ab:c9:00:5f:b8:93:5e:f9:47:40:cc:14:
e2:2b:da:c3:ee:4f:07:97:26:94:94:a1:eb:9d:1d:
8d:dd:55:c3:80:2c:f1:51:e8:53:93:84:a2:5c:94:
6a:94:b4:ba:78:1b:e4:90:74:be:da:e4:1c:b8:94:
12:c9:d6:80:85:1b:f2:52:83:ea:e5:5c:82:62:03:
02:99:8a:04:9b:6c:46:c7:dd:51:35:8a:74:58:63:
b3:13:08:b8:2c:ce:77:04:1e:e0:4e:48:f9:54:5a:
34:b4:e7:49:a7:7b:f0:68:d7:da:cd:98:c6:92:0d:
21:d1:d6:06:ce:07:e9:dc:d2:80:63:90:0b:e0:2a:
6d:f1:16:33:59:34:1e:5c:71:33:5e:21:c2:40:56:
c0:61:fa:7f:60:b1:41:e2:14:60:80:14:a3:98:10:
71:4d:ac:01:9e:51:87:8d:62:d9:67:a2:6b:17:d5:
bb:80:44:c5:5c:8f:51:ab:a4:a8:7c:ee:15:2b:8f:
1f:4d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
38:FE:EA:25:2D:AA:2E:13:2E:A5:0C:33:40:95:DE:BD:22:4D:20:04
X509v3 Authority Key Identifier:
keyid:38:FE:EA:25:2D:AA:2E:13:2E:A5:0C:33:40:95:DE:BD:22:4D:20:04
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:#YOUR DOMAIN NAME, DNS:*.#YOUR DOMAIN NAME
The second DNS (your domain with a “*.” before is important.
1 Like
Thanks for the reply. However, the questions remains is that is there a direct upgrade path to 3.2.1? From what I can tell it looks like there isn’t and it is necessary to update the devices to change the certificates before the server is upgraded to 3.2.1, after which the server upgrade can take place.
I wanted to confirm that this was indeed the case. If so, I think it would be necessary to update the Mender documentation, because this is potentially a pretty breaking upgrade.
Are you using self-signed certificates?
I’m afraid I cannot think of a workaround, because pinning the device to a specific server certificate is normally reserved for non-production setups or ones where it’s easy to change them. For production, you would normally add the CA cert (self-signed CA cert or 3rd party if not already in ca-certificates package) to the device so that you can rotate your server certificates as they are signed by the same CA cert. You can almost certainly achieve pinning by providing your server public CA.cert to mender client instead, or the actual server certificate.
I appreciate having gone back and re-read the Mender docs, this could be clearer.
Caveat: my knowledge of mender is quite old now, and we use our own written client to interact with our mender server, so I’m recalling from memory. However, the principle of certificate management should be the same whether it’s mender software or other TLS client/server setups you may have.
Maybe others with more recent knowledge of mender than I may be able to think of a workaround.