Issue: mender-deployments SerializationError

Hi,

If i take the docker-compose.storage.minio.yml into account, this means that we now have the storage available under either the s3.docker.mender.io, which is an internal domain, or we take the website_url and add the path /mender-artifact-storage, right? Will test that

I checked it and the issue on my side seemed to be that my bucket wasn’t named mender-artifact-storage, but something slightly different. I adjusted the path in the docker-compose.storage.minio.yml and now it looks good. So it should be $DEPLOYMENTS_AWS_BUCKET

Problem with your prod.yml is that the s3.docker.mender.io must be replaced somehow with my mender-url. The frontend runs normally, because all the containers can see s3.docker.mender.io, that’s what the alias does. But this way, the request to pull an artifact from s3.docker.mender.io is sent to the devices, and they cant resolve the url… The other way around, using my mender-url instead of s3.docker.mender.io, i can also get it to work, that the frontend forwards my requests correctly and i can download all the artifacts from the releases menu, where the artifacts are resolved via my mender-url/my-bucket. but somehow deployments cant handle that and creates this serialization error

@kjaskiewiczz @dbasner

I have used same files and minio as a storage proxy but still deployment has failed.

ErrorLog-1.txt.yml (5.5 KB)

2021-06-03 05:55:03 +0000 UTC error: Can not fetch update image: Get "https://s3.docker.mender.io/mender-artifact-storage/df2248bf-b75c-40d1-a48c-da275a803bd5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210603%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210603T055500Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=5ede7f6f9388d58138b6a6e577bdb5edcc9d3e01f7c88b9a10cbfa876d2338d5": dial tcp: lookup s3.docker.mender.io: no such host

2021-06-03 05:55:03 +0000 UTC error: Update fetch failed: update fetch request failed: Get "https://s3.docker.mender.io/mender-artifact-storage/df2248bf-b75c-40d1-a48c-da275a803bd5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210603%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210603T055500Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=5ede7f6f9388d58138b6a6e577bdb5edcc9d3e01f7c88b9a10cbfa876d2338d5": dial tcp: lookup s3.docker.mender.io: no such host

prod.yml (5.4 KB)
docker-compose.storage.minio.yaml (1.0 KB)

Any suggestion to resolve this issue. I have followed Production installation | Mender documentation mender production server v2.7

We just moved from mender server 2.4.0 to a clean installation of 2.7.0.
But we are experiencing exactly the same problem (with same error logs) as Rohita83 and some others.

Instead that the mender-client uses our public mender server URI it uses the mender-server internal https://s3.docker.mender.io/ URI. The latter is not publicly known on the internet and causes for this reason the error.

Hopefully someone can give us some tips to resolve the issue. Thanks a lot.

And to give some more context (as Dave took over from me). For the 2.4.0. installation we used two different URL’s for Mender: 1 public facing one for the the API side and another one for the port 9000 access to the deployment server. This because we have a URL provider for the public facing one that does not open port 9000.

So the problem we are now facing is that with the changes in 2.7.0 we don’t know how we are supposed to change our configuration to make it work. We got as far as making the web interface work, but the clients don’t seem able to reach the deployment side of things.

This of course makes using Mender for our purposes unsuitable, so we need some help to get this working under the new setup of Mender Server.

The information in this thread has up to now not helped us and we are in the same boat as Rohita83.

Any help to get our Mender Server working again as intended would be very much appreciated.

Why are you using s3.docker.mender.io? I think the docs may have a bug as the latest version does seem to require that but the older version defaults to $API_GATEWAY_NAME which I think is correct. Can you try to change that and see if it makes a difference?

Drew

apart from following Drew’s advice, could you please send me the /etc/hosts from the host, output of docker ps and also could you enter the deployments container and print the environment with something like that:

docker exec -it `docker ps | grep deployments | awk '{print($1);}'` /bin/bash -c "set"

peter

Thanks for the suggestion Drew. But changing it to $API_GATEWAY_NAME makes it MINIO is not accessible anymore.

Hi Peter,

Thanks for the support. Please find the requested information below:

/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6

docker ps

CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS                   PORTS                          NAMES
b62ced3bc1e9   mendersoftware/deployments:mender-2.7.0              "/entrypoint.sh --co…"   8 minutes ago   Up 8 minutes             8080/tcp                       menderproduction_mender-deployments_1
9bf38c7d14c3   traefik:v2.4                                         "/entrypoint.sh --ac…"   9 minutes ago   Up 9 minutes             80/tcp, 0.0.0.0:443->443/tcp   menderproduction_mender-api-gateway_1
c2dff0db2412   mendersoftware/deviceauth:mender-2.7.0               "/usr/bin/deviceauth…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-device-auth_1
418daac77c17   mendersoftware/inventory:mender-2.7.0                "/usr/bin/inventory …"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-inventory_1
9b234e753bf9   mendersoftware/create-artifact-worker:mender-2.7.0   "/usr/bin/workflows …"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-create-artifact-worker_1
9b151803751b   mendersoftware/workflows-worker:mender-2.7.0         "/usr/bin/workflows …"   9 minutes ago   Up 9 minutes                                            menderproduction_mender-workflows-worker_1
f08c6163aece   mendersoftware/deviceconnect:mender-2.7.0            "/usr/bin/deviceconn…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-deviceconnect_1
035a1a319d09   mendersoftware/useradm:mender-2.7.0                  "/usr/bin/useradm --…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-useradm_1
8425244bfdea   mendersoftware/deviceconfig:mender-2.7.0             "/usr/bin/deviceconf…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-deviceconfig_1
cf4c212bbea5   mendersoftware/workflows:mender-2.7.0                "/usr/bin/workflows …"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-workflows-server_1
8fc676cc5976   minio/minio:RELEASE.2019-04-23T23-50-36Z             "/usr/bin/docker-ent…"   9 minutes ago   Up 9 minutes (healthy)   9000/tcp                       menderproduction_minio_1
d1636faae9f4   nats:2.1.9-alpine3.12                                "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             4222/tcp, 6222/tcp, 8222/tcp   menderproduction_mender-nats_1
6404c9233f0b   mendersoftware/gui:mender-2.7.0                      "/entrypoint.sh nginx"   9 minutes ago   Up 9 minutes (healthy)   80/tcp, 8080/tcp               menderproduction_mender-gui_1
9b3d5cbf671b   mongo:4.4                                            "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             27017/tcp                      menderproduction_mender-mongo_1

docker exec -it menderproduction_mender-deployments_1 env

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b62ced3bc1e9
TERM=xterm
STORAGE_BACKEND_CERT=/etc/ssl/certs/s3.docker.mender.io.crt
DEPLOYMENTS_AWS_AUTH_KEY=mender-deployments
DEPLOYMENTS_AWS_AUTH_SECRET=*secret_replaced*
DEPLOYMENTS_AWS_URI=https://s3.docker.mender.io
HOME=/root

You need to ensure that all instances of s3.docker.mender.io are replaced. It looks like it is using that name somewhere else.

@drewmoseley

You mean, we should remove the lines having “s3.docker.mender.io” from “prod.yml.template” as below screenshot?

Yes. That likely also means you need to regenerate your certificates since that name is likely embedded in them.

Has anyone solved the issue?
Deployments have not worked for us either since V2.7.0. I can upload and download artifacts and trigger deployments via the web interface without any problems. But the devices can not load them, it appears the “SerializationError” already mentioned here.

Maybe an interesting discovery. It seams the api-gateway forward the request from the device to the gui container instead of the deployments container. According to the api-gateway log the reqest is sent to docker internal ip 10.21.255.130, but this is the gui container.

xxx.xxx.xxx.xxx - - [11/Jun/2021:15:35:39 +0000] "POST /api/devices/v1/deployments/device/deployments/next HTTP/1.1" 503 76 "-" "-" 64 "deployments@docker" "http://10.21.255.130:8080" 2ms
xxx.xxx.xxx.xx - - [11/Jun/2021:15:35:40 +0000] "GET /api/devices/v1/deployments/device/deployments/next?artifact_name=my_artifact&device_type=my_device HTTP/1.1" 503 76 "-" "-" 65 "deployments@docker" "http://10.21.255.130:8080" 3ms

gui container network:

"Networks": {
                "menderproduction_mender": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "mender-gui",
                        "80257243e507"
                    ],
                    "NetworkID": "7934f1c0955f6550f3f122521673638200f09e1239eb9b0103827d2c638ee95f",
                    "EndpointID": "1e5b9f576a9338b89d4281ae0210c3a87dc571d0e0186ae6ad85db8c91973508",
                    "Gateway": "10.21.255.129",
                    "IPAddress": "10.21.255.130",
                    "IPPrefixLen": 26,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:15:ff:82",
                    "DriverOpts": null
                }

deployment container network:

"Networks": {
                "menderproduction_mender": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "mender-deployments",
                        "711cf37570be"
                    ],
                    "NetworkID": "7934f1c0955f6550f3f122521673638200f09e1239eb9b0103827d2c638ee95f",
                    "EndpointID": "d8763abce27472628f08815aaf0bc0f3557dd80017663b472eea6d303ab65e6c",
                    "Gateway": "10.21.255.129",
                    "IPAddress": "10.21.255.143",
                    "IPPrefixLen": 26,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:15:ff:8f",
                    "DriverOpts": null
                }

Could this be the cause, or rather a symptom?

Same thing here. At this point I’m forced to downgrade to 2.6. It’s a shame because the new features and the Web app restyling were great in 2.7.

Widening the net in case others have ideas. cc @Alan @kacf @kjaskiewiczz @lluiscampos @oleorhagen

Drew

Guys, pulling directly from master branch and reinstalling the server worked for me.

These are the differences I have in prod.yml between 2.7 tagged version and master.

You can see changes in the certificate lines, that now involves traefik. I think this can be the trick. Someone more experienced can explain the meaning of these edited lines?

@Dave and all, could you please check this by Drew: Fix broken STORAGE_PROXY_DOMAIN_NAME by drewmoseley · Pull Request #1468 · mendersoftware/mender-docs · GitHub with the above screen by @andrea.nencini (thanks) you should be able to figure it out, before we fix it.

peter

The latest version from master branch works also for me (no new installation, just upgrade).

Thank you @andrea.nencini !

1 Like

hello,

this was fixed and should now be working everywhere.

peter