A general question which many will probably ask: what would be options if one day we want to migrate to Hosted Mender while having a number of devices deployed and connected to self-hosted instance?
There’s basically 2 parts in this question:
switching devices: can it be done with a single mender update? At first glance, it should be possible, yet not sure how artifactCommit stage will behave when the client will start trying commit installation to a clean new server which isn’t aware about this deployment. Which brings to
backend database migration. Probably will be required as well? Absolutely necessary or not really?
Would be nice to receive some info from Mender guys as you probably thought about this scenario.
What is suggested procedure? Will it involve an additional cost for custom technical support (for databse migration for example)?
Hi @ster migrating the clients is supported by setting up multiple servers in your mender.conf file. See https://docs.mender.io/2.1/client-configuration/configuration-file/configuration-options#servers for specifics. Basically, you have to do a bootstrap update where you include both the old and the new server in an image. Let the deployment complete. Then the client will attempt to authenticate with the new server and finish the deployment on the old server. Once it is admitted on the new server you can then deploy another update to remove the old server from the mender.conf file.
Stuck with the problem trying to migrate devices from one Mender server to another, and seems that multiple server setup not helps here.
Could anyone point me out if any issue could be found in the setup?
server1: Mender 2.0, self-signed certificate.
server2: Mender 2.3, AWS-signed certificate.
Client device: Mender 1.7.1
Adding a new artifact, mender.conf has:
“Servers”: [
{“ServerUrl”: “https://server2”},
{“ServerUrl”: “https://server1”}
]
“ServerCertificate”: “/etc/mender/server.crt”,
When deploying, I can see this errors in logs:
Jul 07 11:45:26 host mender[172]: time=“2020-07-07T11:45:26Z” level=warning msg=“Reauthorization failed with error: transient error: authorization request failed: (request_id: ): authentic
ation request rejected server error message: dev auth: unauthorized” module=client
Jul 07 11:45:26 host mender[172]: time=“2020-07-07T11:45:26Z” level=warning msg="Server “https://server2” failed to serve request “/api/devices/v1/deployments/device/de
ployments/d75f90d1-0979-4c8f-9fa8-c8ac24010fed/status”. Attempting “https://server1"” module=client
Jul 07 11:45:26 host mender[172]: time=“2020-07-07T11:45:26Z” level=error msg=“failed to report status: Put https://server1/api/devices/v1/deployments/device/deployments/d75f90d1
-0979-4c8f-9fa8-c8ac24010fed/status: http: ContentLength=21 with Body length 0” module=“client_status”
Jul 07 11:45:26 host mender[172]: time=“2020-07-07T11:45:26Z” level=error msg=“error reporting update status: reporting status failed: Put https://server1/api/devices/v1/deployme
nts/device/deployments/d75f90d1-0979-4c8f-9fa8-c8ac24010fed/status: http: ContentLength=21 with Body length 0” module=mender
Jul 07 11:45:26 host mender[172]: time=“2020-07-07T11:45:26Z” level=error msg=“failed to send status to server: transient error: reporting status failed: Put https://server1/api/
devices/v1/deployments/device/deployments/d75f90d1-0979-4c8f-9fa8-c8ac24010fed/status: http: ContentLength=21 with Body length 0” module=state
Jul 07 11:45:27 host mender[172]: time=“2020-07-07T11:45:27Z” level=info msg=“State transition: update-status-report [ArtifactCommit] -> update-retry-report [none]” module=mender
Device is registered on server2, but never sends inventory updates. It also didn’t report back to the server2 about deployment result.
If force restart Mender service, it start sends the update to the server2 and new deployment could be started to remove old server. Deployment result was never sent to the old server still.
I’ve tried to switch servers order, deployment successful but device will not register at the new server at all.
But the idea was the automatic device migration from old server to the new server.
My guess is that this is an issue with the certificate in use which is definitely going to be different between the servers. We are working on making this smoother but in the meantime you may need to use a statescript with some extra logic on your end to update the server.crt file.
I think if you deploy an update that adds the second ServerURL but does not update /etc/mender/server.crt, then the deployment should complete from server1. You can then use a ArtifactCommit_Leave state script to update /etc/mender/server.crt (or just delete it and remove the conf file entry if your new server uses a CA-signed cert). At this point you may need to restart Mender.
Please let us know if you try this method and what success you have with it.
i successfully setup two self-hosted servers in mender.conf.
However, i have still problems migrating from a self-hosted to a mender-hosted server.
Using the following configuration
the client gets 401 on authorization requests to self-hosted-server, while the web frontend of self-hosted-server shows no authorization attempts. This failure disappears when i remove the TenantToken entry of the configuration (but then i won’t be able to use hosted.mender.io).
Moving the TenantToken into corresponding Server entry
Hello @Julien, I feel like this has worked for me in the past but it’s been some time since I tested it. I know we are looking at modifying the conf format so that you can do something similar to your second test above but I don’t know when it will be done.
I have the same reaction as you, I thought this would work. Logs from both the client and the backend would be useful here (the self hosted one will be enough for now).
As a test that the client-server communication works, i want to get the (unauthorized) device under “Pending” in the web ui of the self-hosted server.
At first i consider the configuration with only the self-hosted server.
On the client side, everything seems normal:
tenant token: " module=auth
authorization data: {{\"mac\":\"(mac address)\"} -----BEGIN PUBLIC KEY----------END PUBLIC KEY-----\n}" module=auth
making an authorization request () to server https://self-hosted" module=client_auth
got response: &{401 Unauthorized 401 HTTP/1.1 1 1 map[Date:[Mon, 14 Sep 2020 08:23:07 GMT] Vary:[Accept-Encoding] Access-Control-Allow-Origin:[*] Server:[openresty/1.13.6.2] Content-Type:[application/json; charset=utf-8] Connection:[keep-alive] X-Authentication-Version:[unknown] X-Men-Requestid:[e9c45734-de01-4b12-83f2-d088bf3b6c3a]] 0x2bcde40 -1 [] false true map[] 0x2888a80 0x2818f00}" module=client_auth
authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state
The client gets 401 because it is not authorized.
On the server side:
dev auth: unauthorized: dev auth: unauthorized" file=response_helpers.go func=rest_utils.RestErrWithWarningMsg line=55 request_id=e9c45734-de01-4b12-83f2-d088bf3b6c3a
401 15431μs POST /api/devices/v1/authentication/auth_requests HTTP/1.0 - Go-http-client/1.1" file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=60 request_id=e9c45734-de01-4b12-83f2-d088bf3b6c3a
mender-api-gateway_1 | (device-ip)- - [14/Sep/2020:08:23:07 +0000] "POST /api/devices/v1/authentication/auth_requests HTTP/1.1" 401 111 "-" "Go-http-client/1.1" "-" request ID "e9c45734-de01-4b12-83f2-d088bf3b6c3a" 0.016
And the client gets visible under Pending. All fine.
Then i consider the configuration with both the self-hosted server and the mender-hosted one.
The log file look very similar except that the token is transmitted.
tenant token: (removed)" module=auth
authorization data: {{\"mac\":\"(mac address)\"} (tenant token)-----BEGIN PUBLIC KEY----------END PUBLIC KEY-----\n}" module=auth
making an authorization request () to server https://self-hosted" module=client_auth
got response: &{401 Unauthorized 401 HTTP/1.1 1 1 map[Date:[Mon, 14 Sep 2020 08:32:33 GMT] Content-Type:[application/json; charset=utf-8] Vary:[Accept-Encoding] X-Men-Requestid:[9a4064bc-78c5-4441-821e-fca4a88c3dd6] Access-Control-Allow-Origin:[*] Server:[openresty/1.13.6.2] Connection:[keep-alive] X-Authentication-Version:[unknown]] 0x280cf90 -1 [] false true map[] 0x2888600 0x2818ea0}" module=client_auth
Failed to authorize \"https://self-hosted\"; attempting \"https://hosted.mender.io\"." module=mender
On the server side:
dev auth: unauthorized: dev auth: unauthorized" file=response_helpers.go func=rest_utils.RestErrWithWarningMsg line=55 request_id=9a4064bc-78c5-4441-821e-fca4a88c3dd6 sub=5f2cf28a80b5418fe5f3585e tenant_id=5f2cf28a80b5418fe5f3585e
401 19794μs POST /api/devices/v1/authentication/auth_requests HTTP/1.0 - Go-http-client/1.1" file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=60 request_id=9a4064bc-78c5-4441-821e-fca4a88c3dd6 sub=5f2cf28a80b5418fe5f3585e tenant_id=5f2cf28a80b5418fe5f3585e
mender-api-gateway_1 | (device-ip)- - [14/Sep/2020:08:32:33 +0000] "POST /api/devices/v1/authentication/auth_requests HTTP/1.1" 401 110 "-" "Go-http-client/1.1" "-" request ID "9a4064bc-78c5-4441-821e-fca4a88c3dd6" 0.028
dev auth: unauthorized: dev auth: unauthorized" file=response_helpers.go func=rest_utils.RestErrWithWarningMsg line=55 request_id=9d38b75b-9723-4e15-a68d-94b6a9be6b46 sub=5f2cf28a80b5418fe5f3585e tenant_id=5f2cf28a80b5418fe5f3585e
401 4731μs POST /api/devices/v1/authentication/auth_requests HTTP/1.0 - Go-http-client/1.1" file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=60 request_id=9d38b75b-9723-4e15-a68d-94b6a9be6b46 sub=5f2cf28a80b5418fe5f3585e tenant_id=5f2cf28a80b5418fe5f3585e
POST /api/devices/v1/authentication/auth_requests HTTP/1.1" 401 110 "-" "Go-http-client/1.1" "-" request ID "9d38b75b-9723-4e15-a68d-94b6a9be6b46" 0.006
This is the only difference. The self-hosted server runs the Open Source edition.
For both, the client gets a 401 authorization failure. This is correct, since the client is not authorized. But in the case of the second configuration, the device does not appear in the web ui in the Pending tab of the self-hosted server. It appears in the Pending tab of the mender-hosted server web ui, though.
Sorry for the delay. We have verified this as a bug and have reported a ticket. I hope this will be resolved relatively soon; since it breaks migration I expect it to get high priority.
