SSL CA certificate setup on mender production

General Discussions
1

Hi, we have installed version 2.2 of Mender in production, with CA certificate.
The mender client is up to date, from mender-client_2.1.1-1_armhf.deb.
The device is a Raspberrypi cm3 Raspbian based.

The device has the certificate in server.crt of /etc/mender/mender.conf, since without it, and omitting the certificate in mender.conf as it puts in the documentation, we were not able to connect, by saying that the certificate It was not validated by a trusted authority, when it is.
Of course on the server side we have final wilcard domain certificate, along with the intermediate certificate concatenated in server.crt

I don’t know at what point we may be failing, or that we may have bad about it.

I would like to know why you can give that error, and if it is correct that you have to give the certificate to the client, it is a certificate authority that we have on the server with our domain.

Thanks in advance, and congratulations on the project :slight_smile:

(Mender accept the device, but show this error, when click on accept)

When I accept a pending device, the error goes out but accepts it equally, however when decomissing the error comes out and does not execute it.
If I can make dissmis to the device, and there is no error.

There was a problem updating the device authorization status: cannot PUT / api / management / v2 / devauth / devices / 5ddbafb4c71c970001c2dfae / auth / 5ddbb046c71c970001c2dfb3 / status (500) ... [Request ID: 4356254d]
There was a problem decommissioning the device: cannot DELETE /api/management/v2/devauth/devices/5ddbafb4c71c970001c2dfae (500)... [Request ID: 5cc10952]

in the client side :

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: **level=error msg="Error receiving scheduled update data: (request_id: ): client not authorized server error message: failed to parse server response: invalid character '<' looking for beginning of value" module=mender**

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=info msg="State transition: update-check [Sync] -> error [Error]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=info msg="handling error state, current error: transient error: (request_id: ): client not authorized server error message: failed to parse server response: invalid character '<' looking f

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=info msg="State transition: error [Error] -> idle [Idle]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=info msg="State transition: idle [Idle] -> authorize-wait [Idle]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=info msg="State transition: authorize-wait [Idle] -> authorize [Sync]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: **level=error msg="update check failed: transient error: (request_id: ): client not authorized server error message: failed to parse server response: invalid character '<' looking for beginning of value" module=state**

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: level=info msg="State transition: update-check [Sync] -> error [Error]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: level=info msg="handling error state, current error: transient error: (request_id: ): client not authorized server error message: failed to parse server response: invalid character '<' looking for beginning of value" module=sta

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: level=info msg="State transition: error [Error] -> idle [Idle]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: level=info msg="State transition: idle [Idle] -> authorize-wait [Idle]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: level=info msg="State transition: authorize-wait [Idle] -> authorize [Sync]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: **level=error msg="authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state**

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=error msg="authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: time="2019-11-25T11:43:19+01:00" level=info msg="State transition: authorize [Sync] -> authorize-wait [Idle]" module=mender

nov 25 11:43:19 ftr-B827EB118CFF mender[24812]: level=info msg="State transition: authorize [Sync] -> authorize-wait [Idle]" module=mender


nov 25 15:32:01 ftr-B827EB118CFF mender[896]: time="2019-11-25T15:32:01+01:00" level=info msg="Device unauthorized; attempting reauthorization" module=client

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: level=warning msg="Reauthorization failed with error: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=client

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: time="2019-11-25T15:32:02+01:00" level=warning msg="Reauthorization failed with error: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=client

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: level=error msg="got unexpected HTTP status when submitting to inventory: 401" module=client_inventory

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: time="2019-11-25T15:32:02+01:00" level=error msg="got unexpected HTTP status when submitting to inventory: 401" module=client_inventory

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: level=warning msg="failed to refresh inventory: failed to submit inventory data: (request_id: ): inventory submit failed, bad status 401 server error message: failed to parse server response: invalid character '<' looking for beginning of value" module=state

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: time="2019-11-25T15:32:02+01:00" level=warning msg="failed to refresh inventory: failed to submit inventory data: (request_id: ): inventory submit failed, bad status 401 server error message: failed to parse server response: invalid character '<' looking for beginning of value" module=state

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]" module=mender

nov 25 15:32:02 ftr-B827EB118CFF mender[896]: time="2019-11-25T15:32:02+01:00" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]" module=mender

Edit: @mirzak: formatting

2

If your Mender server uses a CA-signed certificate then you should not have the cert in the client as it will be handled by the SSL code in Golang libraries.

I wonder if having both is causing some confusion on validation?

3

If I don’t add the server.crt in / etc / mender /, and I remove the line from the mender.conf so that it looks like this:

    {
      "InventoryPollIntervalSeconds": 1200,
      "RetryPollIntervalSeconds": 300,
      "ServerURL": "https://deploy.mydomain.com/",
      "UpdatePollIntervalSeconds": 1800
    }

The client returns this:

Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: time = "2019-11-25T16: 01: 02 + 01: 00" level = error msg = "Failure occurred while executing authorization request: & url.Error {Op : \ "Post \", URL: \ "https: //deploy.blackdevice.com/api/devices/v1/authentication/auth_requests \", Err: x509.UnknownAuthorityError {Cert: (* x509.Certificate) (0x21ec000)
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: time = "2019-11-25T16: 01: 02 + 01: 00" level = error msg = "Certificate is signed by unknown authority." module = client_auth
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: time = "2019-11-25T16: 01: 02 + 01: 00" level = error msg = "If you are using a self-signed certificate, make sure it is available locally to the Mender client in /etc/mender/server.crt and is configured properly in /etc/mender/mender.conf. " module = client_auth
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: time = "2019-11-25T16: 01: 02 + 01: 00" level = error msg = "See https://docs.mender.io/troubleshooting / mender-client # certificate-signed-by-unknown-authority for more information. " module = client_auth
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: level = error msg = "If you are using a self-signed certificate, make sure it is available locally to the Mender client in /etc/mender/server.crt and is configured properly in /etc/mender/mender.conf. " module = client_auth
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: time = "2019-11-25T16: 01: 02 + 01: 00" level = error msg = "authorize failed: transient error: authorization request failed: certificate signed by unknown authority: Post https://deploy.blackdevice.com/api/devices/v1/authentication/auth_requests: x509: certificate signed by unknown authori
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: time = "2019-11-25T16: 01: 02 + 01: 00" level = info msg = "State transition: authorize [Sync] -> authorize-wait [Idle] "module = mender
Nov 25 16:01:02 ftr-B827EB118CFF mender [2065]: level = error msg = "See https://docs.mender.io/troubleshooting/mender-client#certificate-signed-by-unknown-authority for more information . " module = client_auth
Nov 25 04:01:02 ftr-B827EB118CFF mender [2065]: level = error msg = "authorize failed: transient error: authorization request failed: certificate signed by unknown authority: Post https://deploy.blackdevice.com/api/ devices / v1 / authentication / auth_requests: x509: certificate signed by unknown authority "module = state
Nov 25 04:01:02 ftr-B827EB118CFF mender [2065]: level = info msg = "State transition: authorize [Sync] -> authorize-wait [Idle]" module = mender
4

Thinking about this issue, I wonder, if the .key of / production / keys-generated / keys for deviceauth and useradm should I also modify them with the .key of my CA certificate, or should I use those generated by keygen, right now I am using those generated by keygen for that part.

The process I used was:
generate with keygen all the certificates of both / certs and / keys, and then modify the .key and .cert certificates of / production / keys-generated / certs / for api-gateway and storage proxy, together with the server.crt.

But those of / keys-generated / keys I left the self-generated, can the problem be there?

Update :
I just tried this idea, and it seems that if I use our CA key in / production / keys-generated / keys / for deviceauth and useradm, but storage and api don’t communicate well and it doesn’t work …

I don’t know what other combination to try to get the client to accept ok the certificates signed by CA entity.

5

Have you tried just running the keygen script and using those unmodified? I suspect that will work as we have tested that. If you clean out keys-generated and start fresh, it will tell us if the issue is in the post-processing you are doing or not.

6

Hi, we have resolved this issue.

Finally, Mender needs the intermediate certificates of the CA authority within the individual certificate file in each service :
/production/keys-generated/certs/api-gateway/cert.crt
/production/keys-generated/certs/storage-proxy/cert.crt
so these files would look exactly like
/production/keys-generated/certs/server.crt

It is also advisable to download the root certificate files of your CA and include them in the server path in /etc/ssl/certs/ if they are not already there.

With this, clients can now connect perfectly to the mender server, and everything works fine.

Thanks for your help.

1 Like