Issue: mender-deployments SerializationError

Hello everyone!

Issue

After deploying production version of mender-server following instructions in docs, I encountered problem with creation of deployments. First of all it was taking far too long (about 4 hours without even starting the deployment - it was stuck at pending status). After checking logs from the container mender-deployments I found out that it is struggling with an error:

level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response
status code: 200, request id: 
caused by: XML syntax error on line 8: element <link> closed by </head>"

I am not sure if those two are related, but what leads me to believe so is that i was perfectly able to download the artifact and install it using command below, so i suspect that the fault isn’t at the client side of things, also client checked in with no issue, and i was able to upload artifact just fine, so that should dispel doubts about other services.

mender install 'https://my.mender.server.com/artifactlongandstrangestring' && mender commit

At first i thought there might be problem with the keys (since at first i included CA signed ones) but neither keys provided by keygen utility included with mender integration, nor keys and certificates generated by CA authority seems to have influence on that behavior (not that they should have since problem isn’t with keys nor certificates). None of other containers nor client itself report any error, only mender-deployments ant it is only this one i have described.

Have anyone encountered such behavior, or have any idea how to resolve this?

I am using Mender 2.7.0

@tranchitella @peter ping

@CezaryKierzyk it seems the deployments service is pointing to a wrong URL when connecting to the storage layer. Are you using minio or AWS S3? Can you please post the logs of the minio container? Are you using a custom domain name for your storage layer? Can you double check you can reach minio with the DNS name you set for it?

I found out that while minio container is up and running it is not listening on port 9000 and i cannot reach it via browser.

my prod.yml file looks like this:

version: '2.1'
services:

    mender-workflows-server:
        command: server --automigrate

    mender-workflows-worker:
        command: worker --automigrate --excluded-workflows generate_artifact

    mender-create-artifact-worker:
        command: --automigrate

    mender-useradm:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/keys/useradm/private.key:/etc/useradm/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-device-auth:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/keys/deviceauth/private.key:/etc/deviceauth/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-inventory:
        command: server --automigrate
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-api-gateway:
        ports:
            - "443:443"
        networks:
            mender:
                aliases:
                    - https://mender.cp.test2.ipq.co:9000
        command:
            - --accesslog=true
            - --providers.file.filename=/config/tls.toml
            - --providers.docker=true
            - --providers.docker.exposedbydefault=false
            - --entrypoints.http.address=:80
            - --entrypoints.https.address=:443
            - --entryPoints.https.transport.respondingTimeouts.idleTimeout=7200
            - --entryPoints.https.transport.respondingTimeouts.readTimeout=7200
            - --entryPoints.https.transport.respondingTimeouts.writeTimeout=7200
            - --entrypoints.http.http.redirections.entryPoint.to=https
            - --entrypoints.http.http.redirections.entryPoint.scheme=https
        volumes:
            - ./tls.toml:/config/tls.toml
            - ./production/keys-generated/certs/api-gateway/cert.crt:/certs/cert.crt:ro
            - ./production/keys-generated/certs/api-gateway/private.key:/certs/private.key:ro
            - ./production/keys-generated/certs/storage-proxy/cert.crt:/certs/s3.docker.mender.io.crt
            - ./production/keys-generated/certs/storage-proxy/private.key:/certs/s3.docker.mender.io.key
        logging:
            options:
                max-file: "10"
                max-size: "50m"
        environment:
            ALLOWED_HOSTS: mender.cp.test2.ipq.co

    mender-deployments:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/s3.docker.mender.io.crt:ro
        environment:
            STORAGE_BACKEND_CERT: /etc/ssl/certs/s3.docker.mender.io.crt
            DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
            DEPLOYMENTS_AWS_AUTH_SECRET:  mysecretminiokey
            DEPLOYMENTS_AWS_URI: https://my.mender.server.com:9000
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    minio:
        environment:
            MINIO_ACCESS_KEY: mender-deployments
            MINIO_SECRET_KEY: myminiosecretkey
        volumes:
            - mender-artifacts:/export:rw

    mender-mongo:
        volumes:
            - mender-db:/data/db:rw

volumes:
    mender-artifacts:
      external:
          name: mender-artifacts
    mender-db:
      external:
          name: mender-db

Mostly i just copy-pasted into script what was in docs and just ran it. All docker containers run in the same vps under the same ip pointed by the same domain name. I do not use AWS S3.

Minio container logs consist of this part repeating itself continuously:

minio_1                          | [REQUEST LivenessCheckHandler] [161985965.083370] [2021-05-01 09:00:50 +0000]
minio_1                          | GET /minio/health/live
minio_1                          | Host: 127.0.0.1:9000
minio_1                          | User-Agent: Go-http-client/1.1
minio_1                          | Accept-Encoding: gzip
minio_1                          | 
minio_1                          | 
minio_1                          | [RESPONSE] [161985965.083370] [2021-05-01 09:00:50 +0000]
minio_1                          | 200 OK
minio_1                          | Server: MinIO/RELEASE.2019-04-23T23-50-36Z
minio_1                          | Accept-Ranges: bytes
minio_1                          | Vary: Origin
minio_1                          | X-Xss-Protection: 1; mode=block
minio_1                          | Content-Security-Policy: block-all-mixed-content
minio_1                          | X-Amz-Request-Id: 167AE5C521E8A276
minio_1                          | X-Minio-Deployment-Id: b5a99dda-6f09-4b51-b254-4c8ca16e45b6

I managed to get it to work using mender 2.5.1 , mender 2.6.1 and 2.7.0 seems to not launch storage-proxy. It seems that after 2.5.1 in prod.yml.template there is no storage-proxy configuration section in the config, and while Minio container is in fact launched and running healthy, it is not binded to hosts port 9000. I will doublecheck that and come back with proper answer.

Cheers!

@CezaryKierzyk glad you solved your issue.

I checked the integration repository for Mender 2.6.x and the storage proxy is there: integration/prod.yml.template at 2.6.x · mendersoftware/integration · GitHub

In Mender 2.7.x things work differently because both the API gateway and the storage proxy were replaced by Traefik.

Guys,
I am getting same error in mender server v2.7 to deployment an artifact Mender V2.7 deployment getting error - #3 by Rohita83

level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response
status code: 200, request id: 
caused by: XML syntax error on line 8: element <link> closed by </head>"

hi,
I’m attaching configuration which should work if you’re using minio as a storage proxy:
docker-compose.storage.minio.yml (1.0 KB)
prod.yml (5.4 KB)

I am asking myself, if the services still are connected as mentioned in the readme for tag 2.7.0, which is

       |
  port |        +-----------------------+         +------------------------+
   443 | <----> |  API Gateway          |    +--->|  Device Authentication |
       |        |  (mender-api-gateway) |<---|    |  (mender-device-auth)  | <---+
       |        +-----------------------+    |    +------------------------+     |
       |                                     +--->|  Inventory             |     |
       |                                     |    |  (mender-inventory)    | <---+
       |                                     |    +------------------------+     |     +----------------------------------+
       |                                     +--->|  User Administration   |     +---> |  Workflows Engine                |
       |                                     |    |  (mender-useradm)      | <---+     |  (mender-workflows-server)       |
       |                                     |    +------------------------+     |     |  (mender-workflows-worker)       |
       |                                     +--->|  Deployments           |     |     |  (mender-create-artifact-worker) |
       |                 +----------------------->|  (mender-deployments)  | <---+     +----------------------------------+
       |                 |                        +------------------------+
       |                 |
       |                 v
  port |        +------------------+              +----------+
  9000 | <----> |  Storage Proxy   |<------------>|  Minio   |
       |        |  (storage-proxy) |              |  (minio) |
       |        +------------------+              +----------+

If I see this correctly, and storage proxy is gone, minio should somehow point to the api-gateway, too. So should the mender-api-gateway now expose port 9000? How does this diagram look like for the mender 2.7 - mender-api-gateway case? Like This?

       |
  port |        +-----------------------+         +------------------------+
   443 | <----> |  Traefik              |    +--->|  Device Authentication |
  port | <----> |  (mender-api-gateway) |<---|    |  (mender-device-auth)  | <---+
  9000 |        +-----------------------+    |    +------------------------+     |
       |                    ^                +--->|  Inventory             |     |
       |                    |                |    |  (mender-inventory)    | <---+
       |                    |                |    +------------------------+     |     +----------------------------------+
       |                    |                +--->|  User Administration   |     +---> |  Workflows Engine                |
       |                    |                |    |  (mender-useradm)      | <---+     |  (mender-workflows-server)       |
       |                    |                |    +------------------------+     |     |  (mender-workflows-worker)       |
       |                    |                +--->|  Deployments           |     |     |  (mender-create-artifact-worker) |
       |                    |                     |  (mender-deployments)  | <---+     +----------------------------------+
       |                    |                     +------------------------+
       |                    |
       |                    |
       |               +----------+
       |               | Minio    |
       |               |  (minio) |
       |               +----------+

Or should everybody using minio still use the proxy?

the correct diagram is here - https://docs.mender.io/2.7/server-installation/overview; we’ll update the one from integration ASAP;
you no longer need to expose port 9000; minio is using network alias and traefik has a rule for this alias (look at the mender-api-gateway networks/aliases and minio/labels from prod.yml and docker-compose.storage.minio.yml provided by me)

2 Likes

Hi,

If i take the docker-compose.storage.minio.yml into account, this means that we now have the storage available under either the s3.docker.mender.io, which is an internal domain, or we take the website_url and add the path /mender-artifact-storage, right? Will test that

I checked it and the issue on my side seemed to be that my bucket wasn’t named mender-artifact-storage, but something slightly different. I adjusted the path in the docker-compose.storage.minio.yml and now it looks good. So it should be $DEPLOYMENTS_AWS_BUCKET

Problem with your prod.yml is that the s3.docker.mender.io must be replaced somehow with my mender-url. The frontend runs normally, because all the containers can see s3.docker.mender.io, that’s what the alias does. But this way, the request to pull an artifact from s3.docker.mender.io is sent to the devices, and they cant resolve the url… The other way around, using my mender-url instead of s3.docker.mender.io, i can also get it to work, that the frontend forwards my requests correctly and i can download all the artifacts from the releases menu, where the artifacts are resolved via my mender-url/my-bucket. but somehow deployments cant handle that and creates this serialization error

@kjaskiewiczz @dbasner

I have used same files and minio as a storage proxy but still deployment has failed.

ErrorLog-1.txt.yml (5.5 KB)

2021-06-03 05:55:03 +0000 UTC error: Can not fetch update image: Get "https://s3.docker.mender.io/mender-artifact-storage/df2248bf-b75c-40d1-a48c-da275a803bd5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210603%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210603T055500Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=5ede7f6f9388d58138b6a6e577bdb5edcc9d3e01f7c88b9a10cbfa876d2338d5": dial tcp: lookup s3.docker.mender.io: no such host

2021-06-03 05:55:03 +0000 UTC error: Update fetch failed: update fetch request failed: Get "https://s3.docker.mender.io/mender-artifact-storage/df2248bf-b75c-40d1-a48c-da275a803bd5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=mender-deployments%2F20210603%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210603T055500Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=5ede7f6f9388d58138b6a6e577bdb5edcc9d3e01f7c88b9a10cbfa876d2338d5": dial tcp: lookup s3.docker.mender.io: no such host

prod.yml (5.4 KB)
docker-compose.storage.minio.yaml (1.0 KB)

Any suggestion to resolve this issue. I have followed Production installation | Mender documentation mender production server v2.7

We just moved from mender server 2.4.0 to a clean installation of 2.7.0.
But we are experiencing exactly the same problem (with same error logs) as Rohita83 and some others.

Instead that the mender-client uses our public mender server URI it uses the mender-server internal https://s3.docker.mender.io/ URI. The latter is not publicly known on the internet and causes for this reason the error.

Hopefully someone can give us some tips to resolve the issue. Thanks a lot.

And to give some more context (as Dave took over from me). For the 2.4.0. installation we used two different URL’s for Mender: 1 public facing one for the the API side and another one for the port 9000 access to the deployment server. This because we have a URL provider for the public facing one that does not open port 9000.

So the problem we are now facing is that with the changes in 2.7.0 we don’t know how we are supposed to change our configuration to make it work. We got as far as making the web interface work, but the clients don’t seem able to reach the deployment side of things.

This of course makes using Mender for our purposes unsuitable, so we need some help to get this working under the new setup of Mender Server.

The information in this thread has up to now not helped us and we are in the same boat as Rohita83.

Any help to get our Mender Server working again as intended would be very much appreciated.

Why are you using s3.docker.mender.io? I think the docs may have a bug as the latest version does seem to require that but the older version defaults to $API_GATEWAY_NAME which I think is correct. Can you try to change that and see if it makes a difference?

Drew

apart from following Drew’s advice, could you please send me the /etc/hosts from the host, output of docker ps and also could you enter the deployments container and print the environment with something like that:

docker exec -it `docker ps | grep deployments | awk '{print($1);}'` /bin/bash -c "set"

peter

Thanks for the suggestion Drew. But changing it to $API_GATEWAY_NAME makes it MINIO is not accessible anymore.

Hi Peter,

Thanks for the support. Please find the requested information below:

/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6

docker ps

CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS                   PORTS                          NAMES
b62ced3bc1e9   mendersoftware/deployments:mender-2.7.0              "/entrypoint.sh --co…"   8 minutes ago   Up 8 minutes             8080/tcp                       menderproduction_mender-deployments_1
9bf38c7d14c3   traefik:v2.4                                         "/entrypoint.sh --ac…"   9 minutes ago   Up 9 minutes             80/tcp, 0.0.0.0:443->443/tcp   menderproduction_mender-api-gateway_1
c2dff0db2412   mendersoftware/deviceauth:mender-2.7.0               "/usr/bin/deviceauth…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-device-auth_1
418daac77c17   mendersoftware/inventory:mender-2.7.0                "/usr/bin/inventory …"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-inventory_1
9b234e753bf9   mendersoftware/create-artifact-worker:mender-2.7.0   "/usr/bin/workflows …"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-create-artifact-worker_1
9b151803751b   mendersoftware/workflows-worker:mender-2.7.0         "/usr/bin/workflows …"   9 minutes ago   Up 9 minutes                                            menderproduction_mender-workflows-worker_1
f08c6163aece   mendersoftware/deviceconnect:mender-2.7.0            "/usr/bin/deviceconn…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-deviceconnect_1
035a1a319d09   mendersoftware/useradm:mender-2.7.0                  "/usr/bin/useradm --…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-useradm_1
8425244bfdea   mendersoftware/deviceconfig:mender-2.7.0             "/usr/bin/deviceconf…"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-deviceconfig_1
cf4c212bbea5   mendersoftware/workflows:mender-2.7.0                "/usr/bin/workflows …"   9 minutes ago   Up 9 minutes             8080/tcp                       menderproduction_mender-workflows-server_1
8fc676cc5976   minio/minio:RELEASE.2019-04-23T23-50-36Z             "/usr/bin/docker-ent…"   9 minutes ago   Up 9 minutes (healthy)   9000/tcp                       menderproduction_minio_1
d1636faae9f4   nats:2.1.9-alpine3.12                                "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             4222/tcp, 6222/tcp, 8222/tcp   menderproduction_mender-nats_1
6404c9233f0b   mendersoftware/gui:mender-2.7.0                      "/entrypoint.sh nginx"   9 minutes ago   Up 9 minutes (healthy)   80/tcp, 8080/tcp               menderproduction_mender-gui_1
9b3d5cbf671b   mongo:4.4                                            "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             27017/tcp                      menderproduction_mender-mongo_1

docker exec -it menderproduction_mender-deployments_1 env

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b62ced3bc1e9
TERM=xterm
STORAGE_BACKEND_CERT=/etc/ssl/certs/s3.docker.mender.io.crt
DEPLOYMENTS_AWS_AUTH_KEY=mender-deployments
DEPLOYMENTS_AWS_AUTH_SECRET=*secret_replaced*
DEPLOYMENTS_AWS_URI=https://s3.docker.mender.io
HOME=/root