I just did a clean install of 2.6.0 and my devices aren’t registering with the server as a new pending device.
I can see a log on the client device that reads "authorization request failed: Unknown url. Error type: certificate file signed by an unknown authority., openssl verify rc: 20 server cert file: /etc/mender/server.crt."
I can’t imagine that this error is because my server is sitting behind a reverse proxy. Could it?
Thanks!
Regards,
Joe
Hi @jmeirow did you change the server cert as part of the clean install? If you have devices that you want to migrate you should use the same cert on both the 2.5 and 2.6 installs as the public key is already stored on your devices.
Drew
Hi @drewmoseley. Yes, I copied the new server.crt from the 2.6 install to the location on my build-server where I run mender-convert. I’ve compared the files using cksum and “sum -r” and they appear to be identical.
OK. What does sudo journalctl -u mender-client show when run on the client?
Drew
@drewmoseley, in the spirit of as much information as possible, though I have to believe this is unrelated, I am routing the port 9000 storage traffic through port 80, via modifications to the prod.yml and the nginx.conf.
openssl verify error 20 errors in general usually mean you don’t have the full certificate chain available for verification on your device. So either you need to install ca-certificates package and/or concat your CA-cert/s into the same file in " /etc/mender/server.crt"
@dellgreen , you’re right: I definitely have not concatenated the certificate from our reverse proxy server into /etc/mender/server.crt. I’m going to give that a try. I also need to understand how the ca-certificates package is utilized within our organization and if there’s anything I need to do with that with respect to our Mender production server.
@dellgreen - I saved the SSL cert from our proxy server in base64 x509 format and concatenated it to the server.crt. I rebuilt the Menderized image for my Raspberry Pi, using the updated server.crt in the mender-convert process. I am encountering the same error. (openssl verify rc: 20).
I assume the order of concatenation is immaterial, true?
Also, does the Mender client software log to the device the certificate it’s encountering that it cannot find in the server.crt file?
@peter may be able to help here.
I am fairly certain there would be a log message if it could not find the file.
Drew
Just an observation… If you look closely at the photo I posted earlier, it’s not saying it can’t find the certificate, rather, the log states certificate signed by an unknown authority. Not sure why it’s saying this… Our reverse proxy cert’s CA is GoDaddy.
and do you have the GoDaddy CA certificate installed on your device? this is something i would expect to be in the ca-certificates package installed on your device
@dellgreen - yes I do. What I do is navigate to my Mender server in Chrome, click on the SSL “key” icon and then save the certificate in Base 64 X509 format. I then concatenate that file on to the end of the server.crt
on your device try the following to see if you have everything you need, it should highlight if your are missing any intermediate certs.
openssl s_client -connect <your-mender-server:port> -showcerts -CAfile <your-crt-file>
@dellgreen - Thanks! I will try this.
Here is the output (stdout) from the command. I deleted the contents of the certificate body before posting this:
Server certificate
subject=OU = Domain Control Validated, CN = *.lcecorp.com
issuer=C = US, ST = Arizona, L = Scottsdale, O = “GoDaddy.com, Inc.”, OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: D21FEBF865B718FB6741645E00170A4A172FC9F550F354CC54E2C1D02A35617B
Session-ID-ctx:
Master-Key: 2514BD6B4051B7F6727DCE6B1D6AA469BA1EE7C8F9554DB5167FE16355964455E8A03710F267B34463A042D2C69445D6
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 600 (seconds)
TLS session ticket:
0000 - 79 32 09 22 97 b7 69 35-5d 4a d5 69 f6 5a ca 94 y2."…i5]J.i.Z…
0010 - 7a 29 4c 53 7e 1a 8a 55-45 41 d1 78 08 e4 96 d2 z)LS~…UEA.x…
0020 - 9c a7 44 5b 34 0b fe a8-d2 c9 83 70 a3 09 37 f1 …D[4…p…7.
0030 - f2 4d bd ad 68 e1 4b 47-ad 98 75 4a 69 bd 2b de .M…h.KG…uJi.+.
0040 - 8a cc b1 18 36 a6 ad c8-38 c1 6a 85 f5 be d8 ae …6…8.j…
0050 - ed d9 df af 73 6a f2 54-bf 1e 37 fc 30 d3 d5 a8 …sj.T…7.0…
0060 - 26 6e 9a 72 69 b8 a9 c8-6b 3d 71 36 ee 94 15 9d &n.ri…k=q6…
0070 - e0 a7 46 3c ce 11 14 10-5a 13 4b 9f 14 b6 fb 91 …F<…Z.K…
0080 - d1 42 6d 4f 8a 35 b7 86-40 c3 69 d2 ff 7e 5f 74 .BmO.5…@.i…~t
0090 - 1a eb a1 a6 29 73 b5 80-4b bc d0 d2 b7 f2 65 08 …)s…K…e.
00a0 - 6a 80 99 e8 6b f7 7c 9d-b9 02 76 b2 a3 0e 65 06 j…k.|…v…e.
00b0 - ec 22 7b fe 32 09 5d 08-73 b8 43 ee fb 9b 49 a2 ."{.2.].s.C…I.
00c0 - bf ce 4f c8 4f ff d2 de-cc e4 5f 99 6f f9 f6 8a …O.O….o…
Start Time: 1613692793
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no
Here is some std err from that command:
depth=0 OU = Domain Control Validated, CN = *.lcecorp.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.lcecorp.com
verify error:num=21:unable to verify the first certificate
verify return:1
Taking another whack at this… What do you mean by “do you have the GoDaddy cert installed on your device?”?
It’s concatenated to the end of server.crt. That is all. Do I need to do something else with it as well?
whatever you have concatenated in your crt file on your device, doesn’t look like it contains the go-daddy G2 certificate authority certificate which your mender server certificate is signed with.
“Go Daddy Secure Certificate Authority - G2” comes with the ca-certificates package/recipe that you need to install on the device or add to your crt file.
For example, if you have a linux laptop or server take a look in /etc/ssl/certs and you should see the go-daddy-g2-certifiate authority file (“Go_Daddy_Root_Certificate_Authority_-_G2.pem”).
I appended the file Go_Daddy_Root_Certificate_Authority_-_G2.pem to the end of server.crt. BTW, that same file is in /etc/ssl/certs on my device as well.
It is still getting the same error. I did discover that when catting the SSL cert I saved, I saved it on a Windows system so I had ^M at the end of each line of the cert, but I removed them with a global replace in vi and still get the error. Just in case there’s a minor variation I’m not picking up on, I’ll paste the stdout and stderr of the openssl s_client command below:
Verify return code: 21 (unable to verify the first certificate)
Server certificate
subject=OU = Domain Control Validated, CN = *.lcecorp.com
issuer=C = US, ST = Arizona, L = Scottsdale, O = “GoDaddy.com, Inc.”, OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: E61B459E83053B93DEDEF154103637A3088CB29740B62AA6F42B64D8D70FE0AD
Session-ID-ctx:
Master-Key: 0A3892E87ED036EBAAA3879DBDE84FFD60C4E0A3296CB9C65B876A5C8CBA517D79A24FA708F5D975C64F1CD2B216B83D
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 600 (seconds)
TLS session ticket:
0000 - 79 32 09 22 97 b7 69 35-5d 4a d5 69 f6 5a ca 94 y2."…i5]J.i.Z…
0010 - fb 8f a5 96 8e 58 90 09-77 eb 32 1f 63 2e f6 72 …X…w.2.c…r
0020 - bb 1c 18 8a 97 73 63 68-5a 71 e3 7e cd 1b 36 46 …schZq.~…6F
0030 - 7e b5 e5 11 ca a1 06 4e-17 93 79 9b c5 c0 92 8b ~…N…y…
0040 - dd c2 d8 f1 3c 11 b0 f5-0d b7 c3 30 5e d5 a6 45 …<…0^…E
0050 - f6 40 6d 89 c3 a2 d1 b4-b0 e1 60 52 e2 ad aa 45 .@m…`R…E
0060 - 0a 59 8c 0e df a0 98 03-db d3 5d 78 47 c4 4e ca .Y…]xG.N.
0070 - 4f 99 94 ec 69 fc 0a 36-bd 76 13 d0 0d 37 e3 0e O…i…6.v…7…
0080 - 9a e9 3b 75 56 41 67 7f-e0 df ec be da 77 4f e0 …;uVAg…wO.
0090 - 48 8f 7e e4 6d b4 ae b0-4b 4c 35 0b 18 e0 58 22 H.~.m…KL5…X"
00a0 - 89 6b ff fe de 9a 77 46-3a 32 77 48 af e6 d7 f7 .k…wF:2wH…
00b0 - 0b cc a4 89 da 12 04 ee-ec b0 d5 7b d5 48 54 70 …{.HTp
00c0 - dc 8f 44 4f a6 16 b5 d5-f0 88 42 09 32 ee 3b 56 …DO…B.2.;V
Start Time: 1613702672
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended
and the stderr…
depth=0 OU = Domain Control Validated, CN = *.lcecorp.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.lcecorp.com
verify error:num=21:unable to verify the first certificate
verify return:1
I’m wondering if the cert on our reverse proxy server is installed incorrectly. Other than the name of the server, the last four posts of this from a different forum match my situation exactly:
https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1656054