Hi guys.
I have a 2.7 server running on a publicly hosted machine.
The server is up and running (for example, I can preauthenticate devices, upload artifacts, connect with devices through SSH, etc.). However, today I noticed that deployments never start. On the devices, I use a custom microservice that periodically calls the device APIs to check the availability of deployments, and then proceeds to download the artifact. Until today that always worked perfectly, but now I checked that /deployments/next returns a strange 503 status code, that is not listed in the possible responses to that endpoint that I found in documentation.
Can this be related to the world-wide 503 outage we had today? It’s too strange to be a coincidence, but why other endpoints work?
Other information. My client version is 2.6. I tried a few minutes ago to install 2.7 server on a local machine, and I got the same 503 status code. So I can exclude that it is an hosting issue. The last server version I’m absolutely sure that worked was 2.6. There are some compatibility differences between some APIs in 2.6 and 2.7? I read the documentation, but I didn’t found anything on this.
It seems that also mender-client cannot parse the response that get from the server.
Thanks for your help.
I’m facing the same problem. I perform an update from a working Mender server v2.5.1 to v2.6.1 and further to 2.7.0. Now its not possible to deploy new updates, I’ve got the same errors as @andrea.nencini mention. I try also the newest version from 2.7.x Branch, but the same error occurs.
I try to go a bit deeper. I check the log of the “mender-api-gateway container” and figured out the 503 error is also mentioned there.
# private information removed from log output
[08/Jun/2021:16:46:00 +0000] "GET /api/devices/v1/deployments/device/deployments/next?artifact_name=my_artifact&device_type=my_device HTTP/1.1" 503 76 "-" "-" 255 "deployments@docker" "http://x.x.x.x:8080" 2ms
Next I checked the logs of the “mender-deployments” container. Also here you can see the 503 error.
time="2021-06-08T16:46:09Z" level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response\n\tstatus code: 200, request id: \ncaused by: XML syntax error on line 8: element <link> closed by </head>" file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=86bddb2a-95d6-49ac-8af1-090f8a6193ed
time="2021-06-08T16:46:09Z" level=info msg="503 345659ÎĽs GET /api/internal/v1/deployments/health HTTP/1.1 - Go-http-client/1.1" byteswritten=286 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/internal/v1/deployments/health qs= request_id=86bddb2a-95d6-49ac-8af1-090f8a6193ed responsetime=0.345659356 status=503 ts="2021-06-08 16:46:09.503558077 +0000 UTC" type=http
Finally i checked the logs of the " minio" container and found:
PUT /mender-artifact-storage
Host: my_host
X-Forwarded-Server: xxx
X-Real-Ip: x.x.x.x
User-Agent: aws-sdk-go/1.37.30 (go1.14.15; linux; amd64)
Accept-Encoding: gzip
Authorization: AWS4-HMAC-SHA256 Credential=mender-deployments/20210608/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=xxx
X-Amz-Content-Sha256: xxx
X-Forwarded-Port: 443
Content-Length: 0
X-Amz-Date: 20210608T164348Z
X-Forwarded-For: x.x.x.x
X-Forwarded-Host: my_host
X-Forwarded-Proto: https
[RESPONSE] [162317062.834742] [2021-06-08 16:43:48 +0000]
409 Conflict
Content-Type: application/xml
Vary: Origin
X-Xss-Protection: 1; mode=block
Content-Security-Policy: block-all-mixed-content
X-Amz-Request-Id: 1686A916287B8728
X-Minio-Deployment-Id: f2fb1b85-4945-4aee-9f95-a5c49fd71b5d
Server: MinIO/RELEASE.2019-04-23T23-50-36Z
Accept-Ranges: bytes
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>mender-artifact-storage</BucketName><Resource>/mender-artifact-storage</Resource><RequestId>1686A916287B8728</RequestId><HostId>xxx</HostId></Error>
Could this cause the error?
Thank you @dYalib , I didn’t noticed that discussion at first glance. Now I’m following that.
