Deployment doesn't deploy

Running Mender production server 2.6.0.

Uploaded artifact and created a deployment prior to any devices being registered.

Deployment status is “queued to start”, as it is waiting for a device.

Connected a device and it registers as pending. Approved the device.
There is now a connected device.

The deployment is still queued and not downloading.

SSH’d into the mender server and tried to telnet to port 9000 and receive:

telnet: could not resolve 127.0.0.1:9000/telnet: Name or service not known

Is there a way to tell if the storage proxy is running?
Also, if there’s a log on the Mender server that I could look into that would be great.

FWIW, on the device, the log reports no problems. It’s just happily checking for updates, checking inventory, etc. No sign of an inability to download the deployment…it’s just not doing it.

How did you create a deployment in the first place without adding any devices to the deployment first? (Unless you are using some of the enterprise features for matching devices?)

Yeah, as @dellgreen said this can’t work in on-prem. It requires Enterprise.

Got it. I think that was user error.

I did create a deployment with the a device connected. It still doesn’t update but now I am getting logs on the device. From the log I am guessing that our reverse proxy is misconfigured and routing port 9000 to some other port, either 443 this machine or another, as the logs contains complaining about an invalid character “<” in the response.

Please allow me to investigate this and update before anyone spends any more time on this. Thx.

OK, pretty sure we have port 9000 open. We’re getting this in the deployment failure log:

2021-02-24 19:05:28 +0000 UTC info: Running Mender client version: 5d46f39
2021-02-24 19:06:28 +0000 UTC error: Can not fetch update image: Get “https://:9000/mender-artifact-storage/adc40e83-3ea9-48f3-8f08-6c8ec3158fa1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210224T190528Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=a203faaeef80394da788667f9ed0c2a4087ffce215c16349d4381a1b1c53b1da”: read tcp 10.0.0.36:36508->:9000: read: connection reset by peer
2021-02-24 19:06:28 +0000 UTC error: Update fetch failed: update fetch request failed: Get “https://.com:9000/mender-artifact-storage/adc40e83-3ea9-48f3-8f08-6c8ec3158fa1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210224T190528Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=a203faaeef80394da788667f9ed0c2a4087ffce215c16349d4381a1b1c53b1da”: read tcp 10.0.0.36:36508->9000: read: connection reset by peer
2021-02-24 19:06:28 +0000 UTC info: State transition: update-fetch [Download_Enter] → fetch-install-retry-wait [Download_Enter]
2021-02-24 19:07:28 +0000 UTC info: State transition: fetch-install-retry-wait [Download_Enter] → update-fetch [Download_Enter]
2021-02-24 19:08:29 +0000 UTC error: Can not fetch update image: Get “https://:9000/mender-artifact-storage/adc40e83-3ea9-48f3-8f08-6c8ec3158fa1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210224T190528Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=a203faaeef80394da788667f9ed0c2a4087ffce215c16349d4381a1b1c53b1da”: read tcp 10.0.0.36:36510->:9000: read: connection reset by peer
2021-02-24 19:08:29 +0000 UTC error: Update fetch failed: update fetch request failed: Get “https://:9000/mender-artifact-storage/adc40e83-3ea9-48f3-8f08-6c8ec3158fa1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210224T190528Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=a203faaeef80394da788667f9ed0c2a4087ffce215c16349d4381a1b1c53b1da”: read tcp 10.0.0.36:36510->174.136.145.86:9000: read: connection reset by peer
2021-02-24 19:08:29 +0000 UTC info: State transition: update-fetch [Download_Enter] → fetch-install-retry-wait [Download_Enter]
2021-02-24 19:08:29 +0000 UTC info: State transition: fetch-install-retry-wait [Download_Enter] → update-error [ArtifactFailure]
2021-02-24 19:08:29 +0000 UTC info: State transition: update-error [ArtifactFailure] → cleanup [Error]
2021-02-24 19:08:29 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]

Can you manually download that URL onto the device?

Drew,

When I run wget along with the URL above, I get a few normal, promising looking messages, then I get this:

GnuTLS: Error in the pull function
Unable to establish SSL connection

Hmm. I wonder if something is not happy re the supported crypto protocols, versions, etc between the reverse proxy and the storage proxy? I see posts across the web for similar errors that suggest adding --secure-protocol=TLSv1 to the wget options. But to be honest I’m just reading what others have written and don’t understand it.

@peter do you have any ideas here?

Drew

Waiting for a confirmation… We may not have opened port 9000 on our firewall when we opened it on the reverse proxy. I’ll confirm and close this thread if that turns out to be the case.