Unable to Download Artifacts - Self-Hosted Mender Community Server 4.0.0

antonio-mancuso-vado · March 21, 2025, 2:18pm

Hi everyone,

I’m running into an issue with my self-hosted deployment of Mender Community Server 4.0.0.

I’m using AWS S3 as storage, and I’m unable to download artifacts from either the UI or the client.

The errors I’m seeing on the client are:

- error: Unexpected status code while fetching artifact: Not Found  
- error: HTTP stream contains a body, but a reader has not been created for it GET https://<mender-server-url>/53129dae-4973-462f-8de2-0103158d4691?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<aws-access-key-id>%2F20250321%2F<aws-region>%2Fs3%2Faws4_request&X-Amz-Date=20250321T110702Z&X-Amz-Expires=86400&X-Amz-Signature=<signature>&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B+filename%3D%22<filename>%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject:

I suspect this might be related to either the storage configuration (even though upload and delete of artifacts is working correctly) or something related to the storage_proxy.

I’m using the configuration suggested in the server installation tutorial:

s3:
  AWS_URI: "${STORAGE_ENDPOINT}"
  AWS_BUCKET: "${STORAGE_BUCKET}"
  AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
  AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"

api_gateway:
  storage_proxy:
    enabled: true
    url: "${STORAGE_ENDPOINT}"
    customRule: "PathRegexp(`^/${STORAGE_BUCKET}`)"

Has anyone encountered a similar issue or have ideas on what might be going wrong?

Thanks in advance for any help!

robgio · March 21, 2025, 4:13pm

Hi @antonio-mancuso-vado ,
can you trying specifying the AWS_REGION and the AWS_FORCE_PATH_STYLE option as false?

    AWS_URI: "https://s3.<your-aws-region>.amazonaws.com"
    AWS_BUCKET: "${STORAGE_BUCKET}"
    AWS_REGION: "${AWS_BUCKET}"
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
    AWS_FORCE_PATH_STYLE: "false"

antonio-mancuso-vado · March 21, 2025, 4:23pm

Hello @robgio, thank you for the reply, I tried implementing those changes but without success. I still have the same issue

robgio · March 21, 2025, 4:27pm

Hi @antonio-mancuso-vado could you maybe share your full values file? (without any secret of course)

antonio-mancuso-vado · March 21, 2025, 4:44pm

Here:

ingress:
  enabled: true
  annotations:
    cert-manager.io/issuer: "letsencrypt"
  ingressClassName: traefik
  path: /
  hosts:
    - <server-url>
  tls:
  # this secret must exists or it can be created from a working cert-manager instance
    - secretName: mender-ingress-tls
      hosts:
        - <server-url>
global:
  s3:
    AWS_URI: "https://s3.eu-central-1.amazonaws.com"
    AWS_BUCKET: <bucket-name>
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: <aws-access-key-id>
    AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
    AWS_FORCE_PATH_STYLE: "false"
  url: <server-url>
  mongodb:
    existingSecret: "mender-mongo"
  nats:
    existingSecret: "mender-nats-url"
  redis:
    existingSecret: "mender-redis-url"

# Disable the integrated Redis subchart:
redis:
  enabled: false

# Disable the integrated NATS subchart:
nats:
  enabled: false

# Disable the integrated MongoDB subchart:
mongodb:
  enabled: false

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://s3.eu-central-1.amazonaws.com"
    customRule: "PathRegexp(`^/<bucket-name>`)"

For context, we also tried without the api_gateway directive

robgio · March 25, 2025, 9:49am

Hi @antonio-mancuso-vado ,
it seems an integration issue with S3. Sorry, could you please try without the region in the AWS_URI? Just https://s3.amazonaws.com

So:

global:
  s3:
    AWS_URI: "https://s3.amazonaws.com"
    AWS_BUCKET: <bucket-name>
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: <aws-access-key-id>
    AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
    AWS_FORCE_PATH_STYLE: "false"

Thanks

robgio · March 25, 2025, 9:50am

Additionally, any logs from the Deployments service could help

antonio-mancuso-vado · March 25, 2025, 10:50am

Hello @robgio, I tried changin the AWS_URI as suggested but without success.

This is the logs of the deployment pod, I can’t make much out of it, sorry:

time="2025-03-25T10:35:28Z" level=info byteswritten=643 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments qs="status=pending&per_page=20&page=1&sort=desc" request_id=fcd4badb-bb4a-4dd3-9aac-3b547d1dc67c responsetime="908.653µs" status=200 ts="2025-03-25T10:35:28.777Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:28Z" level=info byteswritten=2 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments qs="status=inprogress&per_page=20&page=1&sort=desc" request_id=23dc4411-2474-4d57-b4cb-a96903d80250 responsetime=2.641ms status=200 ts="2025-03-25T10:35:28.779Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:32Z" level=info byteswritten=641 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments/f40f9b95-df09-4d66-ba26-fd418394546b qs= request_id=591be280-f363-49f8-8275-7f510fca07b2 responsetime=1.288ms status=200 ts="2025-03-25T10:35:32.895Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:33Z" level=warning byteswritten=30 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=POST path=/api/devices/v2/deployments/device/deployments/next qs= request_id=f3a4aa69-ec8f-4a88-aad8-b733d979243d responsetime="26.801µs" status=404 ts="2025-03-25T10:35:33.526Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:33Z" level=info byteswritten=773 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=GET path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs="artifact_name=REDACTED&device_type=REDACTED" request_id=b1986be9-faba-4cf0-8cfa-a912ab7a1cd3 responsetime=40.333ms status=200 ts="2025-03-25T10:35:33.916Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:34Z" level=info msg="status: {Status:downloading SubState:}" caller="http.(*DeploymentsApiHandlers).PutDeploymentStatusForDevice@api_deployments.go:1471" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea
time="2025-03-25T10:35:34Z" level=info msg="New status: downloading for device 2b6d0b18-1d35-4437-94b1-87bd06b80af1 deployment: f40f9b95-df09-4d66-ba26-fd418394546b" caller="app.(*Deployments).updateDeviceDeploymentStatus@app.go:1684" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea
time="2025-03-25T10:35:34Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/status plan=enterprise qs= request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea responsetime=8.511ms status=204 ts="2025-03-25T10:35:34.286Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:34Z" level=info msg="status: {Status:failure SubState:}" caller="http.(*DeploymentsApiHandlers).PutDeploymentStatusForDevice@api_deployments.go:1471" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2
time="2025-03-25T10:35:34Z" level=info msg="New status: failure for device 2b6d0b18-1d35-4437-94b1-87bd06b80af1 deployment: f40f9b95-df09-4d66-ba26-fd418394546b" caller="app.(*Deployments).updateDeviceDeploymentStatus@app.go:1684" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2
time="2025-03-25T10:35:34Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/status plan=enterprise qs= request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2 responsetime=4.522ms status=204 ts="2025-03-25T10:35:34.945Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:35Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/log plan=enterprise qs= request_id=fd0256c0-d5fc-49c6-a07a-2ef2de620759 responsetime=64.909ms status=204 ts="2025-03-25T10:35:35.158Z" type=HTTP/1.1 useragent=

antonio-mancuso-vado · March 25, 2025, 12:17pm

Hello @robgio, We discovered that the storage proxy must be manually disabled in 4.0.0. Removing the directive is not sufficient.

We changed it:

api_gateway:
  storage_proxy:
    enabled: false

So it was a dumb and wrong configuration on our part
Now it all works correctly. Thanks for your time.

robgio · July 8, 2025, 8:34am

Hello @antonio-mancuso-vado ,
the storage proxy feature is enabled by default in the Helm Chart v6; this to make the product more consistent and easy to set up. Of course you can still disable it, but it should work out-of-the-box. So if you experience any issue, feel free to report it so we can check and improve the project!

Thank you

simonbuehler · July 11, 2025, 6:54am

Hi,
for me the storage_proxy also causes failures, and I’d like to help to debug, could you tell me what you would need from logs / config?

# Mender Helm Chart Values Template
# This file is used by the combined Terraform configuration

global:
  url: https://${actual_domain}
  storage: aws
  admin:
    email: ${admin_email}
    password: ${admin_password}
  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: ${storage_bucket}
    AWS_REGION: ${region}
    AWS_FORCE_PATH_STYLE: "false"
    AWS_ACCESS_KEY_ID: ${hmac_access_key}
    AWS_SECRET_ACCESS_KEY: ${hmac_secret_key}

# Enable Kubernetes TLS secrets feature
featureGates:
  k8sTlsSecrets: true

mongodb:
  auth:
    password: ${db_password}

ingress:
  enabled: true
  ingressClassName: "traefik"
  path: /
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-${ssl_environment}"
    traefik.ingress.kubernetes.io/redirect-to-https: "true"
  hosts: 
    - ${actual_domain}
  tls:
    - secretName: mender-tls
      hosts:
        - ${actual_domain}

api_gateway:
  storage_proxy:
    enabled: false 
#doesn't work otherwise

robgio · July 11, 2025, 7:40am

Hi @simonbuehler , I see you’re using GCS as a backend, which doesn’t support virtual-hosted style, so you have to set:

  s3:
    AWS_FORCE_PATH_STYLE: "true"

Your full conf should be something like

  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: ${storage_bucket}
    AWS_REGION: "us-east-1" # ignored with GCS
    AWS_FORCE_PATH_STYLE: "true"
    AWS_ACCESS_KEY_ID: ${hmac_access_key}
    AWS_SECRET_ACCESS_KEY: ${hmac_secret_key}

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://storage.googleapis.com"
    customRule: "PathRegexp(\`^/${storage_bucket}\`)"

simonbuehler · July 11, 2025, 8:51am

Thanks for the input @robgio ,
unfortunately its still not working, i tried different variants of the customRule like

  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: ${storage_bucket}
    AWS_REGION: ${region} # ignored with GCS
    AWS_FORCE_PATH_STYLE: "true"
    AWS_ACCESS_KEY_ID: ${hmac_access_key}
    AWS_SECRET_ACCESS_KEY: ${hmac_secret_key}

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://storage.googleapis.com"
    customRule: 'PathRegexp(`^/${storage_bucket}`)' # not really
    customRule: "PathRegexp(\\`^/${storage_bucket}\\`)" # nope
    customRule: "PathRegexp(\`^/${storage_bucket}\`)" # doesn't apply", Invalid escape sequence"

all lead to 404errors , so i’m a bit lost

robgio · July 11, 2025, 9:15am

Hi @simonbuehler ,
probably I’m misunderstanding the issue here, but let me ask anyway: did you replace the environment variables here?

For example, your actual values file should looks like this:

# storage_bucket=my-mender-artifacts-storage

  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: "my-mender-artifacts-storage"
    AWS_REGION: "us-east-1" # ignored with GCS
    AWS_FORCE_PATH_STYLE: "true"
    AWS_ACCESS_KEY_ID: "<redacted>"
    AWS_SECRET_ACCESS_KEY: "<redacted>"

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://storage.googleapis.com"
    customRule: "PathRegexp(`^/my-mender-artifacts-storage`)"

simonbuehler · July 11, 2025, 9:42am

yes, they are evaluated by the terraform process like

 values = [
    templatefile("${path.module}/mender-values.yaml", {
      domain          = var.domain
      ...other vars ...
    })
  ]

it’s just not working with the storage_proxy enabled

robgio · July 11, 2025, 12:05pm

Hi @simonbuehler , can you please double check if the values file has been correctly generated?

# assuming: helm chart name is mender
# assuming: helm chart namespace is mender
helm get values mender -n mender

Can you please get to Mender UI, in the Releases page, expand one of the releases, find the “DOWNLOAD ARTIFACT” button, and right-click to copy the link address?
The address generated should be something like:

https://my-mender.example.com/my-mender-artifacts-storage

If this is the case, and the storage proxy is configured correctly, the API Gateway (traefik) is relying the requests to https://my-mender.example.com/my-mender-artifacts-storage for you towards the GCS bucket.
If you experience a 404, then it’s definitely a Traefik misconfiguration. Could you please also check the traefik config?

kubectl get cm api-gateway-traefik -o yaml -n mender

You should see this route:

        #
        # storage_proxy
        #
        storage_proxy:
          entrypoints: http
          middlewares:
          - ratelimit
          - storageProxyHeaders
          rule: "PathRegexp(`^/replace-with-your-bucket-name`)"
          priority: 65535
          service: storage_proxy
          tls: false

and this service:

        storage_proxy:
          loadBalancer:
            passHostHeader: false
            servers:
            - url: https://storage.googleapis.com

simonbuehler · July 11, 2025, 2:18pm

of course:

helm get values mender -n mender

$ helm get values mender -n mender
USER-SUPPLIED VALUES:
api_gateway:
  storage_proxy:
    customRule: PathRegexp(`^/mender-server-459007-mender-storage-dev`)
    enabled: true
    url: https://storage.googleapis.com

link is
https://x.x.x.x.sslip.io/mender-server-459007-mender-storage-dev/c13c6d5c-35a4-4926-8e33-9e6f9f5cd224

output of $ kubectl get cm api-gateway-traefik -o yaml -n mender

        #
        # storage_proxy
        #
        storage_proxy:
          entrypoints: http
          middlewares:
          - ratelimit
          - storageProxyHeaders
          rule: "PathRegexp(`^/mender-server-459007-mender-storage-dev`)"
          priority: 65535
          service: storage_proxy
          tls: false


and


        storage_proxy:
          loadBalancer:
            passHostHeader: false
            servers:
            - url: https://storage.googleapis.com

which is strange as this are the expected values? thanks for looking into this btw!

robgio · July 11, 2025, 2:25pm

Hi @simonbuehler this seems all correct, as long as the GCS name is mender-server-459007-mender-storage-dev; so the error you’re experiencing is a 404 when you download the artifact?

To better dig into this, you should double check the object url, the one you got from the link: https://x.x.x.x.sslip.io/mender-server-459007-mender-storage-dev/c13c6d5c-35a4-4926-8e33-9e6f9f5cd224 → c13c6d5c-35a4-4926-8e33-9e6f9f5cd224; do you have an object (actually, a prefix) in your GCS bucket with this name?

simonbuehler · July 14, 2025, 7:43am

Yes, there does exist this object:

but no idea why i get this 404 from (which) nginx

robgio · July 14, 2025, 7:46am

Nginx? Interesting; I’d look at the ingress/ingress controller/load balancer configuration: it looks like the issue is before Mender itself.

Topic		Replies	Views
Issue: mender-deployments SerializationError General Discussions issue , mender-server , deployment , production , integration	29	2379	July 23, 2021
Cannot Access Storage Artifact on MenderServer 2.7.0 General Discussions integration	12	658	May 28, 2021
Working with S3 instead of Minio - no artifacts found and blank releases page General Discussions	10	3227	August 19, 2019
Unable to access artifacts in Mender 3.0 General Discussions	11	453	August 5, 2021
AWS S3 storage via proxy or interface endpoint General Discussions mender-server , proxy , s3	10	4232	February 18, 2021

Unable to Download Artifacts - Self-Hosted Mender Community Server 4.0.0

Related topics