Unable to Download Artifacts - Self-Hosted Mender Community Server 4.0.0

antonio-mancuso-vado · March 21, 2025, 2:18pm

Hi everyone,

I’m running into an issue with my self-hosted deployment of Mender Community Server 4.0.0.

I’m using AWS S3 as storage, and I’m unable to download artifacts from either the UI or the client.

The errors I’m seeing on the client are:

- error: Unexpected status code while fetching artifact: Not Found  
- error: HTTP stream contains a body, but a reader has not been created for it GET https://<mender-server-url>/53129dae-4973-462f-8de2-0103158d4691?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<aws-access-key-id>%2F20250321%2F<aws-region>%2Fs3%2Faws4_request&X-Amz-Date=20250321T110702Z&X-Amz-Expires=86400&X-Amz-Signature=<signature>&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B+filename%3D%22<filename>%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject:

I suspect this might be related to either the storage configuration (even though upload and delete of artifacts is working correctly) or something related to the storage_proxy.

I’m using the configuration suggested in the server installation tutorial:

s3:
  AWS_URI: "${STORAGE_ENDPOINT}"
  AWS_BUCKET: "${STORAGE_BUCKET}"
  AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
  AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"

api_gateway:
  storage_proxy:
    enabled: true
    url: "${STORAGE_ENDPOINT}"
    customRule: "PathRegexp(`^/${STORAGE_BUCKET}`)"

Has anyone encountered a similar issue or have ideas on what might be going wrong?

Thanks in advance for any help!

robgio · March 21, 2025, 4:13pm

Hi @antonio-mancuso-vado ,
can you trying specifying the AWS_REGION and the AWS_FORCE_PATH_STYLE option as false?

    AWS_URI: "https://s3.<your-aws-region>.amazonaws.com"
    AWS_BUCKET: "${STORAGE_BUCKET}"
    AWS_REGION: "${AWS_BUCKET}"
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
    AWS_FORCE_PATH_STYLE: "false"

antonio-mancuso-vado · March 21, 2025, 4:23pm

Hello @robgio, thank you for the reply, I tried implementing those changes but without success. I still have the same issue

robgio · March 21, 2025, 4:27pm

Hi @antonio-mancuso-vado could you maybe share your full values file? (without any secret of course)

antonio-mancuso-vado · March 21, 2025, 4:44pm

Here:

ingress:
  enabled: true
  annotations:
    cert-manager.io/issuer: "letsencrypt"
  ingressClassName: traefik
  path: /
  hosts:
    - <server-url>
  tls:
  # this secret must exists or it can be created from a working cert-manager instance
    - secretName: mender-ingress-tls
      hosts:
        - <server-url>
global:
  s3:
    AWS_URI: "https://s3.eu-central-1.amazonaws.com"
    AWS_BUCKET: <bucket-name>
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: <aws-access-key-id>
    AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
    AWS_FORCE_PATH_STYLE: "false"
  url: <server-url>
  mongodb:
    existingSecret: "mender-mongo"
  nats:
    existingSecret: "mender-nats-url"
  redis:
    existingSecret: "mender-redis-url"

# Disable the integrated Redis subchart:
redis:
  enabled: false

# Disable the integrated NATS subchart:
nats:
  enabled: false

# Disable the integrated MongoDB subchart:
mongodb:
  enabled: false

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://s3.eu-central-1.amazonaws.com"
    customRule: "PathRegexp(`^/<bucket-name>`)"

For context, we also tried without the api_gateway directive

robgio · March 25, 2025, 9:49am

Hi @antonio-mancuso-vado ,
it seems an integration issue with S3. Sorry, could you please try without the region in the AWS_URI? Just https://s3.amazonaws.com

So:

global:
  s3:
    AWS_URI: "https://s3.amazonaws.com"
    AWS_BUCKET: <bucket-name>
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: <aws-access-key-id>
    AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
    AWS_FORCE_PATH_STYLE: "false"

Thanks

robgio · March 25, 2025, 9:50am

Additionally, any logs from the Deployments service could help

antonio-mancuso-vado · March 25, 2025, 10:50am

Hello @robgio, I tried changin the AWS_URI as suggested but without success.

This is the logs of the deployment pod, I can’t make much out of it, sorry:

time="2025-03-25T10:35:28Z" level=info byteswritten=643 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments qs="status=pending&per_page=20&page=1&sort=desc" request_id=fcd4badb-bb4a-4dd3-9aac-3b547d1dc67c responsetime="908.653µs" status=200 ts="2025-03-25T10:35:28.777Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:28Z" level=info byteswritten=2 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments qs="status=inprogress&per_page=20&page=1&sort=desc" request_id=23dc4411-2474-4d57-b4cb-a96903d80250 responsetime=2.641ms status=200 ts="2025-03-25T10:35:28.779Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:32Z" level=info byteswritten=641 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments/f40f9b95-df09-4d66-ba26-fd418394546b qs= request_id=591be280-f363-49f8-8275-7f510fca07b2 responsetime=1.288ms status=200 ts="2025-03-25T10:35:32.895Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:33Z" level=warning byteswritten=30 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=POST path=/api/devices/v2/deployments/device/deployments/next qs= request_id=f3a4aa69-ec8f-4a88-aad8-b733d979243d responsetime="26.801µs" status=404 ts="2025-03-25T10:35:33.526Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:33Z" level=info byteswritten=773 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=GET path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs="artifact_name=REDACTED&device_type=REDACTED" request_id=b1986be9-faba-4cf0-8cfa-a912ab7a1cd3 responsetime=40.333ms status=200 ts="2025-03-25T10:35:33.916Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:34Z" level=info msg="status: {Status:downloading SubState:}" caller="http.(*DeploymentsApiHandlers).PutDeploymentStatusForDevice@api_deployments.go:1471" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea
time="2025-03-25T10:35:34Z" level=info msg="New status: downloading for device 2b6d0b18-1d35-4437-94b1-87bd06b80af1 deployment: f40f9b95-df09-4d66-ba26-fd418394546b" caller="app.(*Deployments).updateDeviceDeploymentStatus@app.go:1684" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea
time="2025-03-25T10:35:34Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/status plan=enterprise qs= request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea responsetime=8.511ms status=204 ts="2025-03-25T10:35:34.286Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:34Z" level=info msg="status: {Status:failure SubState:}" caller="http.(*DeploymentsApiHandlers).PutDeploymentStatusForDevice@api_deployments.go:1471" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2
time="2025-03-25T10:35:34Z" level=info msg="New status: failure for device 2b6d0b18-1d35-4437-94b1-87bd06b80af1 deployment: f40f9b95-df09-4d66-ba26-fd418394546b" caller="app.(*Deployments).updateDeviceDeploymentStatus@app.go:1684" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2
time="2025-03-25T10:35:34Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/status plan=enterprise qs= request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2 responsetime=4.522ms status=204 ts="2025-03-25T10:35:34.945Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:35Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/log plan=enterprise qs= request_id=fd0256c0-d5fc-49c6-a07a-2ef2de620759 responsetime=64.909ms status=204 ts="2025-03-25T10:35:35.158Z" type=HTTP/1.1 useragent=

antonio-mancuso-vado · March 25, 2025, 12:17pm

Hello @robgio, We discovered that the storage proxy must be manually disabled in 4.0.0. Removing the directive is not sufficient.

We changed it:

api_gateway:
  storage_proxy:
    enabled: false

So it was a dumb and wrong configuration on our part
Now it all works correctly. Thanks for your time.

Topic		Replies	Views
Unable to download the Artifact on the Raspberry Pi 4b device General Discussions mender-server	3	14	February 4, 2025
Working with S3 instead of Minio - no artifacts found and blank releases page General Discussions	10	3201	August 19, 2019
Unable to download artifact after upload General Discussions	1	314	April 26, 2022
Mender setup at ECS General Discussions mender-server	32	2024	April 30, 2021
Unable to access artifacts in Mender 3.0 General Discussions	11	426	August 5, 2021

Unable to Download Artifacts - Self-Hosted Mender Community Server 4.0.0

Related topics