Upgrading from 1.7.1 to 2.0.0. (mender-deployments_1 | RequestError: send request failed)

General Discussions
,
#1

Hello MenderHub,

First off, I’m very new to Mender and whilst I have experience using git and docker I can’t seem to figure this one out.

The company I work for has a Mender setup running 1.7.1. so, an update is overdue. I learned that you can’t just update to 2.5.0 because of the DB so I set my sights on 2.0.1. and made a snapshot of the VM. Then I hit this brick wall:

mender-deployments_1 | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1 | time="2021-01-12T14:21:23Z" level=info msg="Deployments Service, version unknown starting up"
mender-deployments_1 | time="2021-01-12T14:21:23Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=migrations.Migrate line=48
mender-deployments_1 | time="2021-01-12T14:21:23Z" level=info msg="migrating deployment_service" file=migrations.go func=migrations.MigrateSingle line=70
mender-deployments_1 | time="2021-01-12T14:21:23Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
mender-deployments_1 | time="2021-01-12T14:21:23Z" level=info msg="DB migrated to version 1.2.1" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://**{our domain}**:9000/mender-artifact-storage: dial tcp [172.17.0.10:9000](http://172.17.0.10:9000): connect: connection refused

Initially I figured it was an issue with the network and looped back our domain in the hosts file (not it), then continued to check the used ports in the docker network (these are open and accessible), next up I checked and regenerated the Minio auth keys (no-go), and lastly regenerated the certificates also to no avail.

I feel as though I’m missing something obvious, so any advice or even a place to start would be greatly appreciated.

Thanks in advance!

#2

just to clarify, did you upgrade from 1.7.1 to 2.0.1, if so you could try rolling back your VM snapshot to 1.7.1 and then try 1.7.1 to 2.0.0 as an incremental step before then doing 2.0.0 to 2.0.1.

#3

@Kronos as @dellgreen suggested, we only test migrating from one version to the very next version. Otherwise it would require too many combinations. Please let us know if you have issues moving to 2.0.0.

@merlin @tranchitella FYI.

Drew

#4

Thank you for the quick suggestions,

I’ve retried updating, this time to 2.0.0. but I’m still greeted with the same error.

Any other ideas?

#5

It looks like your storage proxy is not functioning. (ie port 9000).
Can you provide docker logs?
@tranchitella can you help?

#6

@Kronos can you please check the storage-proxy container is running? You should be able to reach it on port 9000 with a simple curl from any of the containers running in the Mender docker-composition. If the storage proxy is not working, please provide us with the logs, including the ones from the minio container.

#7

Apologies for the late response,

Docker tells me the proxy is up and running on port 9000. Running:
“docker exec -ti menderproduction_mender-conductor_1 curl menderproduction_storage-proxy_1:9000” gives me:

curl: (7) Failed to connect to menderproduction_storage-proxy_1 port 9000: Connection refused

And “curl localhost:9000” or “curl http://localhost:9000” give me:

curl: (52) Empty reply from server

curl: (56) Recv failure: Connection reset by peer

Respectively. Docker logs for the storage proxy are empty, so I can’t provide those.

To give a bit of context, this is what the updated mender instance looks like in docker. If you’d like any logs let me know.

image
image1658×212 15.8 KB

#8

hello @Kronos

thank you for using Mender.
there are two things:

  • storage proxy runs in the container, so inside the container you should be able to reach its port:
    docker exec -ti menderproduction_mender-storage-proxy_1 nc -z -v 127.0.0.1 9000
  • docker as per configuration maps the 9000 port of the host to the 9000 port inside the container, so you should be able to reach the port from the host running docker:
    nc -z -v 127.0.0.1 9000

what are the above commands show you?

best regards,
peter