Failed Upload of Release Artifact (100MB) via UI (v4.0.0)

General Discussions
1

Hello,

Since updating Mender to version 4.0.0, uploading a release artifact of approximately 100 MB via the User Interface (UI) consistently fails.

The progress indicator in the UI remains stuck at 0%.

  • Previous Behavior (v3.7): Uploads of these same artifacts functioned correctly.

  • Current Behavior (v4.0.0):

    • Uploading various other files of different same works.

    • Only the upload of release artifacts appears to be affected.

Logs are not very informative. The only notable entry appears only when the user manually cancels the upload: a message indicating context canceled is logged.

Composant Version/Détails
Mender Backend Version 4.0.0
Helm Chart Mender Version 6.9.0
Plateform/Orchestration Cluster Kubernetes (Kube) v1.32
Storage (Artifacts) Azure Blob Storage

Our values are the following:

mongodb:
  enabled: false
nats:
  enabled: false
redis:
  enabled: false
default:
  image:
    tag: v4.0.0
global:
  enterprise: false
  mongodb:
    URL: "***"
  nats:
    URL: "***"
    image:
      registry: public.ecr.aws
  storage: "azure"
  azure:
    AUTH_CONNECTION_STRING: "***"
    CONTAINER_NAME: mender-artifact-storage
  url: "***"
featureGates:
  k8sTlsSecrets: true
api_gateway:
  storage_proxy:
    enabled: false
  resources:
    requests:
      cpu: 100m
      memory: 500M
  image:
    registry: public.ecr.aws
  env:
    SSL: true
  minio:
    enabled: false
  certs:
    existingSecret: mender-public-tls
device_auth:
  resources:
    requests:
      cpu: 10m
  certs:
    key: "***"
useradm:
  resources:
    requests:
      cpu: 10m
  certs:
    key: "***"
tenantadm:
  enabled: false
auditlogs:
  enabled: false
devicemonitor:
  enabled: false
iot_manager:
  enabled: false
device_license_count:
  enabled: false
inventory:
  resources:
    requests:
      cpu: 10m
deployments:
  customEnvs: []
  resources:
    requests:
      cpu: 100m
    limits:
      memory: 512Mi
  directUpload:
    enabled: true
gui:
  resources:
    requests:
      cpu: 10m
create_artifact_worker:
  resources:
    requests:
      cpu: 10m
deviceconnect:
  resources:
    requests:
      cpu: 10m
deviceconfig:
  resources:
    requests:
      cpu: 10m

Do you have any idea on how to fix the issue?

Thank you in advance for your assistance.

2

Hi @mmt09 ,

which kubernetes distribution are you using? And which ingress controller?

Additionally, can you also make the same test with curl and with the mender-cli tool?
For example, with curl:

curl -v -X POST "https://my-mender.example.org/api/management/v1/deployments/artifacts" -H "Authorization: Bearer ${JWT}" -H "Content-Type: multipart/form-data" -F "artifact=@file100M.mender"

Thanks

3

Hello! Thanks for the reply.

We are running Kubernetes 1.32 with the NGINX Ingress Controller. I have tried uploading the file using both curl and mender-cli.

In both cases, the upload starts correctly, but the server never responds, eventually leading to a 504 Gateway Timeout. The file does not appear in the UI.

  • With curl, I get the message: “We are completely uploaded and fine”, but it then ends with a 504 error.

  • With mender-cli, it stays stuck on “Processing uploaded file. This may take around one minute” before freezing and returning the same 504 timeout.

It seems the data is sent, but the backend fails to process it or respond in time.

4

Hello @mmt09 ,

I fear you have to tweak a bit the Ingress Controller. As an example, you can see some example annotations in the docs:

ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
1 Like
5

Indeed it works!

Thank you very much !