Issue: mender-deployments SerializationError

Hello everyone!

Issue

After deploying production version of mender-server following instructions in docs, I encountered problem with creation of deployments. First of all it was taking far too long (about 4 hours without even starting the deployment - it was stuck at pending status). After checking logs from the container mender-deployments I found out that it is struggling with an error:

level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response
status code: 200, request id: 
caused by: XML syntax error on line 8: element <link> closed by </head>"

I am not sure if those two are related, but what leads me to believe so is that i was perfectly able to download the artifact and install it using command below, so i suspect that the fault isn’t at the client side of things, also client checked in with no issue, and i was able to upload artifact just fine, so that should dispel doubts about other services.

mender install 'https://my.mender.server.com/artifactlongandstrangestring' && mender commit

At first i thought there might be problem with the keys (since at first i included CA signed ones) but neither keys provided by keygen utility included with mender integration, nor keys and certificates generated by CA authority seems to have influence on that behavior (not that they should have since problem isn’t with keys nor certificates). None of other containers nor client itself report any error, only mender-deployments ant it is only this one i have described.

Have anyone encountered such behavior, or have any idea how to resolve this?

I am using Mender 2.7.0

@tranchitella @peter ping

@CezaryKierzyk it seems the deployments service is pointing to a wrong URL when connecting to the storage layer. Are you using minio or AWS S3? Can you please post the logs of the minio container? Are you using a custom domain name for your storage layer? Can you double check you can reach minio with the DNS name you set for it?

I found out that while minio container is up and running it is not listening on port 9000 and i cannot reach it via browser.

my prod.yml file looks like this:

version: '2.1'
services:

    mender-workflows-server:
        command: server --automigrate

    mender-workflows-worker:
        command: worker --automigrate --excluded-workflows generate_artifact

    mender-create-artifact-worker:
        command: --automigrate

    mender-useradm:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/keys/useradm/private.key:/etc/useradm/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-device-auth:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/keys/deviceauth/private.key:/etc/deviceauth/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-inventory:
        command: server --automigrate
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-api-gateway:
        ports:
            - "443:443"
        networks:
            mender:
                aliases:
                    - https://mender.cp.test2.ipq.co:9000
        command:
            - --accesslog=true
            - --providers.file.filename=/config/tls.toml
            - --providers.docker=true
            - --providers.docker.exposedbydefault=false
            - --entrypoints.http.address=:80
            - --entrypoints.https.address=:443
            - --entryPoints.https.transport.respondingTimeouts.idleTimeout=7200
            - --entryPoints.https.transport.respondingTimeouts.readTimeout=7200
            - --entryPoints.https.transport.respondingTimeouts.writeTimeout=7200
            - --entrypoints.http.http.redirections.entryPoint.to=https
            - --entrypoints.http.http.redirections.entryPoint.scheme=https
        volumes:
            - ./tls.toml:/config/tls.toml
            - ./production/keys-generated/certs/api-gateway/cert.crt:/certs/cert.crt:ro
            - ./production/keys-generated/certs/api-gateway/private.key:/certs/private.key:ro
            - ./production/keys-generated/certs/storage-proxy/cert.crt:/certs/s3.docker.mender.io.crt
            - ./production/keys-generated/certs/storage-proxy/private.key:/certs/s3.docker.mender.io.key
        logging:
            options:
                max-file: "10"
                max-size: "50m"
        environment:
            ALLOWED_HOSTS: mender.cp.test2.ipq.co

    mender-deployments:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/s3.docker.mender.io.crt:ro
        environment:
            STORAGE_BACKEND_CERT: /etc/ssl/certs/s3.docker.mender.io.crt
            DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
            DEPLOYMENTS_AWS_AUTH_SECRET:  mysecretminiokey
            DEPLOYMENTS_AWS_URI: https://my.mender.server.com:9000
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    minio:
        environment:
            MINIO_ACCESS_KEY: mender-deployments
            MINIO_SECRET_KEY: myminiosecretkey
        volumes:
            - mender-artifacts:/export:rw

    mender-mongo:
        volumes:
            - mender-db:/data/db:rw

volumes:
    mender-artifacts:
      external:
          name: mender-artifacts
    mender-db:
      external:
          name: mender-db

Mostly i just copy-pasted into script what was in docs and just ran it. All docker containers run in the same vps under the same ip pointed by the same domain name. I do not use AWS S3.

Minio container logs consist of this part repeating itself continuously:

minio_1                          | [REQUEST LivenessCheckHandler] [161985965.083370] [2021-05-01 09:00:50 +0000]
minio_1                          | GET /minio/health/live
minio_1                          | Host: 127.0.0.1:9000
minio_1                          | User-Agent: Go-http-client/1.1
minio_1                          | Accept-Encoding: gzip
minio_1                          | 
minio_1                          | 
minio_1                          | [RESPONSE] [161985965.083370] [2021-05-01 09:00:50 +0000]
minio_1                          | 200 OK
minio_1                          | Server: MinIO/RELEASE.2019-04-23T23-50-36Z
minio_1                          | Accept-Ranges: bytes
minio_1                          | Vary: Origin
minio_1                          | X-Xss-Protection: 1; mode=block
minio_1                          | Content-Security-Policy: block-all-mixed-content
minio_1                          | X-Amz-Request-Id: 167AE5C521E8A276
minio_1                          | X-Minio-Deployment-Id: b5a99dda-6f09-4b51-b254-4c8ca16e45b6

I managed to get it to work using mender 2.5.1 , mender 2.6.1 and 2.7.0 seems to not launch storage-proxy. It seems that after 2.5.1 in prod.yml.template there is no storage-proxy configuration section in the config, and while Minio container is in fact launched and running healthy, it is not binded to hosts port 9000. I will doublecheck that and come back with proper answer.

Cheers!

@CezaryKierzyk glad you solved your issue.

I checked the integration repository for Mender 2.6.x and the storage proxy is there: integration/prod.yml.template at 2.6.x · mendersoftware/integration · GitHub

In Mender 2.7.x things work differently because both the API gateway and the storage proxy were replaced by Traefik.

Guys,
I am getting same error in mender server v2.7 to deployment an artifact Mender V2.7 deployment getting error - #3 by Rohita83

level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response
status code: 200, request id: 
caused by: XML syntax error on line 8: element <link> closed by </head>"

hi,
I’m attaching configuration which should work if you’re using minio as a storage proxy:
docker-compose.storage.minio.yml (1.0 KB)
prod.yml (5.4 KB)