Unable to Download Artifacts - Self-Hosted Mender Community Server 4.0.0

General Discussions
1

Hi everyone,

I’m running into an issue with my self-hosted deployment of Mender Community Server 4.0.0.

I’m using AWS S3 as storage, and I’m unable to download artifacts from either the UI or the client.

The errors I’m seeing on the client are:

- error: Unexpected status code while fetching artifact: Not Found  
- error: HTTP stream contains a body, but a reader has not been created for it GET https://<mender-server-url>/53129dae-4973-462f-8de2-0103158d4691?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<aws-access-key-id>%2F20250321%2F<aws-region>%2Fs3%2Faws4_request&X-Amz-Date=20250321T110702Z&X-Amz-Expires=86400&X-Amz-Signature=<signature>&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B+filename%3D%22<filename>%22&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject:

I suspect this might be related to either the storage configuration (even though upload and delete of artifacts is working correctly) or something related to the storage_proxy.

I’m using the configuration suggested in the server installation tutorial:

s3:
  AWS_URI: "${STORAGE_ENDPOINT}"
  AWS_BUCKET: "${STORAGE_BUCKET}"
  AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
  AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"

api_gateway:
  storage_proxy:
    enabled: true
    url: "${STORAGE_ENDPOINT}"
    customRule: "PathRegexp(`^/${STORAGE_BUCKET}`)"

Has anyone encountered a similar issue or have ideas on what might be going wrong?

Thanks in advance for any help!

2

Hi @antonio-mancuso-vado ,
can you trying specifying the AWS_REGION and the AWS_FORCE_PATH_STYLE option as false?

    AWS_URI: "https://s3.<your-aws-region>.amazonaws.com"
    AWS_BUCKET: "${STORAGE_BUCKET}"
    AWS_REGION: "${AWS_BUCKET}"
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
    AWS_FORCE_PATH_STYLE: "false"
3

Hello @robgio, thank you for the reply, I tried implementing those changes but without success. I still have the same issue

4

Hi @antonio-mancuso-vado could you maybe share your full values file? (without any secret of course)

5

Here:

ingress:
  enabled: true
  annotations:
    cert-manager.io/issuer: "letsencrypt"
  ingressClassName: traefik
  path: /
  hosts:
    - <server-url>
  tls:
  # this secret must exists or it can be created from a working cert-manager instance
    - secretName: mender-ingress-tls
      hosts:
        - <server-url>
global:
  s3:
    AWS_URI: "https://s3.eu-central-1.amazonaws.com"
    AWS_BUCKET: <bucket-name>
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: <aws-access-key-id>
    AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
    AWS_FORCE_PATH_STYLE: "false"
  url: <server-url>
  mongodb:
    existingSecret: "mender-mongo"
  nats:
    existingSecret: "mender-nats-url"
  redis:
    existingSecret: "mender-redis-url"

# Disable the integrated Redis subchart:
redis:
  enabled: false

# Disable the integrated NATS subchart:
nats:
  enabled: false

# Disable the integrated MongoDB subchart:
mongodb:
  enabled: false

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://s3.eu-central-1.amazonaws.com"
    customRule: "PathRegexp(`^/<bucket-name>`)"

For context, we also tried without the api_gateway directive

6

Hi @antonio-mancuso-vado ,
it seems an integration issue with S3. Sorry, could you please try without the region in the AWS_URI? Just https://s3.amazonaws.com

So:

global:
  s3:
    AWS_URI: "https://s3.amazonaws.com"
    AWS_BUCKET: <bucket-name>
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: <aws-access-key-id>
    AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>
    AWS_FORCE_PATH_STYLE: "false"

Thanks

7

Additionally, any logs from the Deployments service could help

8

Hello @robgio, I tried changin the AWS_URI as suggested but without success.

This is the logs of the deployment pod, I can’t make much out of it, sorry:

time="2025-03-25T10:35:28Z" level=info byteswritten=643 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments qs="status=pending&per_page=20&page=1&sort=desc" request_id=fcd4badb-bb4a-4dd3-9aac-3b547d1dc67c responsetime="908.653µs" status=200 ts="2025-03-25T10:35:28.777Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:28Z" level=info byteswritten=2 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments qs="status=inprogress&per_page=20&page=1&sort=desc" request_id=23dc4411-2474-4d57-b4cb-a96903d80250 responsetime=2.641ms status=200 ts="2025-03-25T10:35:28.779Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:32Z" level=info byteswritten=641 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=GET path=/api/management/v1/deployments/deployments/f40f9b95-df09-4d66-ba26-fd418394546b qs= request_id=591be280-f363-49f8-8275-7f510fca07b2 responsetime=1.288ms status=200 ts="2025-03-25T10:35:32.895Z" type=HTTP/1.1 user_id=1632bd6b-a9d2-4333-a150-7ae338b6993e useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
time="2025-03-25T10:35:33Z" level=warning byteswritten=30 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" method=POST path=/api/devices/v2/deployments/device/deployments/next qs= request_id=f3a4aa69-ec8f-4a88-aad8-b733d979243d responsetime="26.801µs" status=404 ts="2025-03-25T10:35:33.526Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:33Z" level=info byteswritten=773 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=GET path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs="artifact_name=REDACTED&device_type=REDACTED" request_id=b1986be9-faba-4cf0-8cfa-a912ab7a1cd3 responsetime=40.333ms status=200 ts="2025-03-25T10:35:33.916Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:34Z" level=info msg="status: {Status:downloading SubState:}" caller="http.(*DeploymentsApiHandlers).PutDeploymentStatusForDevice@api_deployments.go:1471" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea
time="2025-03-25T10:35:34Z" level=info msg="New status: downloading for device 2b6d0b18-1d35-4437-94b1-87bd06b80af1 deployment: f40f9b95-df09-4d66-ba26-fd418394546b" caller="app.(*Deployments).updateDeviceDeploymentStatus@app.go:1684" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea
time="2025-03-25T10:35:34Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/status plan=enterprise qs= request_id=d78d4f03-d307-4cae-bcb4-efc25efe81ea responsetime=8.511ms status=204 ts="2025-03-25T10:35:34.286Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:34Z" level=info msg="status: {Status:failure SubState:}" caller="http.(*DeploymentsApiHandlers).PutDeploymentStatusForDevice@api_deployments.go:1471" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2
time="2025-03-25T10:35:34Z" level=info msg="New status: failure for device 2b6d0b18-1d35-4437-94b1-87bd06b80af1 deployment: f40f9b95-df09-4d66-ba26-fd418394546b" caller="app.(*Deployments).updateDeviceDeploymentStatus@app.go:1684" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 plan=enterprise request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2
time="2025-03-25T10:35:34Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/status plan=enterprise qs= request_id=412acfd4-fc71-42a4-9998-e43e5f9a61d2 responsetime=4.522ms status=204 ts="2025-03-25T10:35:34.945Z" type=HTTP/1.1 useragent=
time="2025-03-25T10:35:35Z" level=info byteswritten=0 caller="accesslog.(*AccessLogMiddleware).LogFunc@middleware.go:165" device_id=2b6d0b18-1d35-4437-94b1-87bd06b80af1 method=PUT path=/api/devices/v1/deployments/device/deployments/f40f9b95-df09-4d66-ba26-fd418394546b/log plan=enterprise qs= request_id=fd0256c0-d5fc-49c6-a07a-2ef2de620759 responsetime=64.909ms status=204 ts="2025-03-25T10:35:35.158Z" type=HTTP/1.1 useragent=
9

Hello @robgio, We discovered that the storage proxy must be manually disabled in 4.0.0. Removing the directive is not sufficient.

We changed it:

api_gateway:
  storage_proxy:
    enabled: false

So it was a dumb and wrong configuration on our part :smiley:
Now it all works correctly. Thanks for your time.

1 Like
10

Hello @antonio-mancuso-vado ,
the storage proxy feature is enabled by default in the Helm Chart v6; this to make the product more consistent and easy to set up. Of course you can still disable it, but it should work out-of-the-box. So if you experience any issue, feel free to report it so we can check and improve the project!

Thank you

11

Hi,
for me the storage_proxy also causes failures, and I’d like to help to debug, could you tell me what you would need from logs / config?

# Mender Helm Chart Values Template
# This file is used by the combined Terraform configuration

global:
  url: https://${actual_domain}
  storage: aws
  admin:
    email: ${admin_email}
    password: ${admin_password}
  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: ${storage_bucket}
    AWS_REGION: ${region}
    AWS_FORCE_PATH_STYLE: "false"
    AWS_ACCESS_KEY_ID: ${hmac_access_key}
    AWS_SECRET_ACCESS_KEY: ${hmac_secret_key}

# Enable Kubernetes TLS secrets feature
featureGates:
  k8sTlsSecrets: true

mongodb:
  auth:
    password: ${db_password}

ingress:
  enabled: true
  ingressClassName: "traefik"
  path: /
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-${ssl_environment}"
    traefik.ingress.kubernetes.io/redirect-to-https: "true"
  hosts: 
    - ${actual_domain}
  tls:
    - secretName: mender-tls
      hosts:
        - ${actual_domain}

api_gateway:
  storage_proxy:
    enabled: false 
#doesn't work otherwise
12

Hi @simonbuehler , I see you’re using GCS as a backend, which doesn’t support virtual-hosted style, so you have to set:

  s3:
    AWS_FORCE_PATH_STYLE: "true"

Your full conf should be something like

  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: ${storage_bucket}
    AWS_REGION: "us-east-1" # ignored with GCS
    AWS_FORCE_PATH_STYLE: "true"
    AWS_ACCESS_KEY_ID: ${hmac_access_key}
    AWS_SECRET_ACCESS_KEY: ${hmac_secret_key}

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://storage.googleapis.com"
    customRule: "PathRegexp(\`^/${storage_bucket}\`)"
13

Thanks for the input @robgio ,
unfortunately its still not working, i tried different variants of the customRule like

  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: ${storage_bucket}
    AWS_REGION: ${region} # ignored with GCS
    AWS_FORCE_PATH_STYLE: "true"
    AWS_ACCESS_KEY_ID: ${hmac_access_key}
    AWS_SECRET_ACCESS_KEY: ${hmac_secret_key}

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://storage.googleapis.com"
    customRule: 'PathRegexp(`^/${storage_bucket}`)' # not really
    customRule: "PathRegexp(\\`^/${storage_bucket}\\`)" # nope
    customRule: "PathRegexp(\`^/${storage_bucket}\`)" # doesn't apply", Invalid escape sequence"

all lead to 404errors , so i’m a bit lost

14

Hi @simonbuehler ,
probably I’m misunderstanding the issue here, but let me ask anyway: did you replace the environment variables here?

For example, your actual values file should looks like this:

# storage_bucket=my-mender-artifacts-storage

  s3:
    AWS_URI: "https://storage.googleapis.com"
    AWS_BUCKET: "my-mender-artifacts-storage"
    AWS_REGION: "us-east-1" # ignored with GCS
    AWS_FORCE_PATH_STYLE: "true"
    AWS_ACCESS_KEY_ID: "<redacted>"
    AWS_SECRET_ACCESS_KEY: "<redacted>"

api_gateway:
  storage_proxy:
    enabled: true
    url: "https://storage.googleapis.com"
    customRule: "PathRegexp(`^/my-mender-artifacts-storage`)"
15

yes, they are evaluated by the terraform process like

 values = [
    templatefile("${path.module}/mender-values.yaml", {
      domain          = var.domain
      ...other vars ...
    })
  ]

it’s just not working with the storage_proxy enabled

16

Hi @simonbuehler , can you please double check if the values file has been correctly generated?

# assuming: helm chart name is mender
# assuming: helm chart namespace is mender
helm get values mender -n mender

Can you please get to Mender UI, in the Releases page, expand one of the releases, find the “DOWNLOAD ARTIFACT” button, and right-click to copy the link address?
The address generated should be something like:

https://my-mender.example.com/my-mender-artifacts-storage

If this is the case, and the storage proxy is configured correctly, the API Gateway (traefik) is relying the requests to https://my-mender.example.com/my-mender-artifacts-storage for you towards the GCS bucket.
If you experience a 404, then it’s definitely a Traefik misconfiguration. Could you please also check the traefik config?

kubectl get cm api-gateway-traefik -o yaml -n mender

You should see this route:

        #
        # storage_proxy
        #
        storage_proxy:
          entrypoints: http
          middlewares:
          - ratelimit
          - storageProxyHeaders
          rule: "PathRegexp(`^/replace-with-your-bucket-name`)"
          priority: 65535
          service: storage_proxy
          tls: false

and this service:

        storage_proxy:
          loadBalancer:
            passHostHeader: false
            servers:
            - url: https://storage.googleapis.com
17

of course:

helm get values mender -n mender

$ helm get values mender -n mender
USER-SUPPLIED VALUES:
api_gateway:
  storage_proxy:
    customRule: PathRegexp(`^/mender-server-459007-mender-storage-dev`)
    enabled: true
    url: https://storage.googleapis.com

link is
https://x.x.x.x.sslip.io/mender-server-459007-mender-storage-dev/c13c6d5c-35a4-4926-8e33-9e6f9f5cd224

output of $ kubectl get cm api-gateway-traefik -o yaml -n mender 

        #
        # storage_proxy
        #
        storage_proxy:
          entrypoints: http
          middlewares:
          - ratelimit
          - storageProxyHeaders
          rule: "PathRegexp(`^/mender-server-459007-mender-storage-dev`)"
          priority: 65535
          service: storage_proxy
          tls: false


and


        storage_proxy:
          loadBalancer:
            passHostHeader: false
            servers:
            - url: https://storage.googleapis.com

which is strange as this are the expected values? thanks for looking into this btw!

18

Hi @simonbuehler this seems all correct, as long as the GCS name is mender-server-459007-mender-storage-dev; so the error you’re experiencing is a 404 when you download the artifact?

To better dig into this, you should double check the object url, the one you got from the link: https://x.x.x.x.sslip.io/mender-server-459007-mender-storage-dev/c13c6d5c-35a4-4926-8e33-9e6f9f5cd224 → c13c6d5c-35a4-4926-8e33-9e6f9f5cd224; do you have an object (actually, a prefix) in your GCS bucket with this name?

19

Yes, there does exist this object:

grafik
grafik764×388 38.9 KB

but no idea why i get this 404 from (which) nginx
grafik
grafik725×192 11.6 KB

20

Nginx? Interesting; I’d look at the ingress/ingress controller/load balancer configuration: it looks like the issue is before Mender itself.