Upload artifact more than 1 minute will fail

alanlineys3d · September 25, 2025, 5:43am

We want to survey “HTTP mender server” without any credential and firewall setup inside our company firewall.

But we encountered uploading large artifact with server return 502 fail, the provided pod logs is uploaded by web user interface.

Smaller artifacts are good, it will normally show up on the WebUI Release page after upload.

With seaweedfs.yml , mender-values.yml. And add a custom HTTP Ingress rule.

And cannot upload through the mender-cli it will stop around “1minute”, but we could upload artifact by adding “–direct” parameter.

We want to know if there is any setting can allow us to send large artifact more than 1 minutes through HTTP?

$ time mender-cli artifacts upload ./test_rootfs_ota.mender --server http://192.168.4.62
Configuration file not found. Continuing.
608.97 MiB / 1.79 GiB [----------------------------------------------------->__________________________________________________________________________________________________________] 33.29% 9.49 MiB p/sFAILURE: artifact upload to '127.0.0.1' failed with status 504

real	1m0.150s
user	0m0.302s
sys	0m0.381s

filer:
  s3:
    enabled: true
    enableAuth: true
    createBuckets:
      - name: "mender-artifact-storage"
  storageClass: default
s3:
  enabled: true
  enableAuth: true

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mender-ingress-http
  namespace: default
  annotations:
    # Optional: Add any Traefik-specific annotations here
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  ingressClassName: traefik
  rules:
  - http:  # No host field = accepts ANY host/IP
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mender-api-gateway
            port:
              number: 80

ingress:
  enabled: false

global:
  s3:
    AWS_URI: "http://seaweedfs-s3:8333"
    AWS_BUCKET: "mender-artifact-storage"
    AWS_ACCESS_KEY_ID: "key"
    AWS_SECRET_ACCESS_KEY: "key"
    AWS_FORCE_PATH_STYLE: "true"
  url: "http://127.0.0.1"

api_gateway:
  env:
    SSL: false
  storage_proxy:
    enabled: true
    url: "http://seaweedfs-s3:8333"
    customRule: "PathRegexp(`^/mender-artifact-storage`)"

@pod_logs/api-getway

{"ClientAddr":"10.42.0.8:39064","ClientHost":"10.42.0.8","ClientPort":"39064","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":61548965927,"OriginContentSize":11,"OriginDuration":61547728557,"OriginStatus":502,"Overhead":1237370,"RequestAddr":"127.0.0.1","RequestContentSize":653677688,"RequestCount":3758,"RequestHost":"127.0.0.1","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2025-09-25T03:39:25.020343995Z","StartUTC":"2025-09-25T03:39:25.020343995Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:40:26Z"}
{"ClientAddr":"10.42.0.8:41704","ClientHost":"10.42.0.8","ClientPort":"41704","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":61139496331,"OriginContentSize":11,"OriginDuration":61138193988,"OriginStatus":502,"Overhead":1302343,"RequestAddr":"127.0.0.1","RequestContentSize":642962104,"RequestCount":3786,"RequestHost":"127.0.0.1","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2025-09-25T03:41:26.370320367Z","StartUTC":"2025-09-25T03:41:26.370320367Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:42:27Z"}
{"ClientAddr":"10.42.0.8:50334","ClientHost":"10.42.0.8","ClientPort":"50334","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":8454549,"OriginContentSize":11,"OriginDuration":7234031,"OriginStatus":502,"Overhead":1220518,"RequestAddr":"127.0.0.1","RequestContentSize":0,"RequestCount":3889,"RequestHost":"127.0.0.1","RequestMethod":"GET","RequestPath":"/api/management/v1/tenantadm/user/tenant","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"tenantadm@file","ServiceAddr":"mender-tenantadm:8080","ServiceName":"tenantadm@file","ServiceURL":"http://mender-tenantadm:8080","StartLocal":"2025-09-25T03:48:47.84831535Z","StartUTC":"2025-09-25T03:48:47.84831535Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:48:47Z"}
{"ClientAddr":"10.42.0.8:50160","ClientHost":"10.42.0.8","ClientPort":"50160","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":61687661493,"OriginContentSize":11,"OriginDuration":61686144357,"OriginStatus":502,"Overhead":1517136,"RequestAddr":"127.0.0.1","RequestContentSize":635230719,"RequestCount":3933,"RequestHost":"127.0.0.1","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2025-09-25T03:48:56.618905267Z","StartUTC":"2025-09-25T03:48:56.618905267Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:49:58Z"}
{"ClientAddr":"10.42.0.8:50186","ClientHost":"10.42.0.8","ClientPort":"50186","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":61150128349,"OriginContentSize":11,"OriginDuration":61148869763,"OriginStatus":502,"Overhead":1258586,"RequestAddr":"127.0.0.1","RequestContentSize":607484973,"RequestCount":3960,"RequestHost":"127.0.0.1","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2025-09-25T03:49:57.420592898Z","StartUTC":"2025-09-25T03:49:57.420592898Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:50:58Z"}
{"ClientAddr":"10.42.0.8:50166","ClientHost":"10.42.0.8","ClientPort":"50166","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":61432287259,"OriginContentSize":11,"OriginDuration":61430996743,"OriginStatus":502,"Overhead":1290516,"RequestAddr":"127.0.0.1","RequestContentSize":626887725,"RequestCount":3987,"RequestHost":"127.0.0.1","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2025-09-25T03:50:58.080368408Z","StartUTC":"2025-09-25T03:50:58.080368408Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:51:59Z"}
{"ClientAddr":"10.42.0.8:50172","ClientHost":"10.42.0.8","ClientPort":"50172","ClientUsername":"-","DownstreamContentSize":11,"DownstreamStatus":502,"Duration":61323835658,"OriginContentSize":11,"OriginDuration":61322508651,"OriginStatus":502,"Overhead":1327007,"RequestAddr":"127.0.0.1","RequestContentSize":601155941,"RequestCount":4012,"RequestHost":"127.0.0.1","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2025-09-25T03:51:58.74686755Z","StartUTC":"2025-09-25T03:51:58.74686755Z","entryPointName":"http","level":"info","msg":"","time":"2025-09-25T03:53:00Z"}

@pod_logs/deployment

time="2025-09-25T03:42:11Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=8c720628-5cae-4521-96d9-6a07d414d6f3 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:42:28Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=985f783c-6ee8-46a6-8b60-da02486c99b8 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:42:28Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=985f783c-6ee8-46a6-8b60-da02486c99b8 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:43:11Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=dd1b5dc4-648e-4b31-9c27-0ad8f257a244 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:44:11Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=ffe83e11-00a9-4729-8abb-8dafbc7ebe73 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:45:11Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=f2e5ee81-5d4d-4090-9523-736c096950c0 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:46:12Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=1fcb7dc7-c5be-41d1-acfa-77652793ef02 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:47:12Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=176ae813-13a1-4899-a3ab-815770ce14bc user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:48:13Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=b66bb42e-10c1-4576-ba91-776303958476 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:48:48Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=2f1055cb-309e-44d4-b65c-10aa1af16e7d user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:48:50Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=cf0411e4-ae96-44e4-8d73-5558df6a8e82 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:48:50Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=6c7a8c71-ff1e-4363-a08a-1bb264fea82e user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:48:50Z" level=warning msg="unexpected data type (<nil>) received from distinct call: ignoring result" caller="mongo.(*DataStoreMongo).ListReleaseTags@datastore_mongo_releases.go:133" request_id=746179a8-034c-4041-be92-7b34c5b434f3 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:49:13Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=15bd9b3d-dc19-4e0d-83f4-c7a7b9ba7b94 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:49:51Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=5680b658-5051-41c5-a4c2-f04ed003dde3 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:49:59Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=922fd159-c61e-4ba4-8fd7-19b87cc16bd9 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:49:59Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=922fd159-c61e-4ba4-8fd7-19b87cc16bd9 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:50:13Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=73c3af5d-bcc9-4092-a154-57d5437ec690 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:50:52Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=cded3e13-e475-4151-94a3-8d61ad55d282 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:51:00Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=53f7d5d7-f3b2-4206-9114-44c333fde826 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:51:00Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=53f7d5d7-f3b2-4206-9114-44c333fde826 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:51:13Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=22a07b2f-19d9-4502-9523-c2ce1018bb33 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:51:53Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=f76b8436-8975-43f3-b0c9-1f322ea50a52 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:52:01Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=7ae237bf-8238-48f0-97f6-9d1b740f650d user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:52:01Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=7ae237bf-8238-48f0-97f6-9d1b740f650d user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:52:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=4de7af24-1a8d-4825-94cf-ddb593de1e10 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:52:54Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=c2244b12-993d-4153-91bc-3022c1647cf8 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:53:00Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=13fdc786-fbaf-481b-a5ea-b0780a55f77e user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:53:00Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=13fdc786-fbaf-481b-a5ea-b0780a55f77e user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:53:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=a187e430-fae1-4bd9-9da7-b0dffff8ee47 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:53:55Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=4eb8d379-f080-4e95-a3f1-dab3bd499a0f user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:54:02Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=e28131e9-51fa-4c75-8447-73b9700b1852 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:54:02Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=e28131e9-51fa-4c75-8447-73b9700b1852 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:54:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=0975bf3d-e0f0-4c99-b64a-26139c866213 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:54:56Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=f88ef4f9-cbb4-43b4-8354-65e09d31c167 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:55:02Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=18649ff4-1e8c-4162-94db-f06afcfba7c1 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:55:02Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=18649ff4-1e8c-4162-94db-f06afcfba7c1 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:55:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=e03ca5b8-bc8f-43d8-9bcd-df5d85f4c261 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:55:57Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=05f70a08-30cc-4223-8f0f-f13430d62740 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:56:03Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=4ec26ffc-6983-4cc0-9fac-a803e0a0b09b user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:56:03Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=4ec26ffc-6983-4cc0-9fac-a803e0a0b09b user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:56:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=e55257ae-f513-498b-a932-17740e546a2f user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:56:58Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=c7710a51-bf24-4700-be4e-981981683b94 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:57:04Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=621ffe39-3649-4877-84d5-d8d38ca8fdfb user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:57:04Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=621ffe39-3649-4877-84d5-d8d38ca8fdfb user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:57:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=9abfed7e-1bd6-43e1-834d-02f15df02533 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:57:59Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=7dcaf562-f511-4300-8e2e-57f7671ba3cf user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:58:04Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=3c98c5fe-ffdd-4342-8b57-58fd99a6eb29 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:58:04Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=3c98c5fe-ffdd-4342-8b57-58fd99a6eb29 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:58:14Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=892f4c22-721b-4b6d-b952-d1636bf9a417 user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:59:00Z" level=info msg="get releases method version 1.2.15" caller="mongo.(*DataStoreMongo).getReleases_1_2_15@datastore_mongo.go:689" request_id=83fd2bbc-69d0-49a2-a5a2-4234cd71f66f user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:59:04Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF: Cannot parse artifact file" caller="http.(*DeploymentsApiHandlers).newImageWithContext@api_deployments.go:753" request_id=3c9429ba-0176-4e17-8252-348c126bb2cd user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b
time="2025-09-25T03:59:04Z" level=error msg="reading artifact error: Payload: can not install Payload: rootfs.img: unexpected EOF" caller="view.(*RESTView).RenderError@view.go:63" request_id=3c9429ba-0176-4e17-8252-348c126bb2cd user_id=11beab5b-630a-4f95-96c8-b2fbafb0b69b

robgio · September 25, 2025, 2:36pm

Hi @alanlineys3d ,

could you also provide the Ingress Controller logs please? Since your ingressClass is Traefik, I assume you have one or more Traefik pods in the kube-system namespace.

You should try the failing upload again and check the Traefik logs.

Additionally, could you please also provide the tenantadm logs? I see from your logs a 502 coming from that as well.

If you feel brave, you can use this tool to extract Mender logs: warning, though: it’s still experimental:
https://github.com/oldgiova/mender-server-support-tool/blob/main/mender_server_support_tool.sh

Thanks

alanlineys3d · September 30, 2025, 4:42am

I’ve reproduced again upload by “web user interface”, the UI will seems to hang about 10minutes on 0%, then will prompt us a “Network error”.

deployments_mender-deployments-756cbb97c6-xgtcx.yml (36.6 KB)

describe_deployment_traefik.yml (4.0 KB)

traefik-bd6b7ccb9-w892j_HTTP.yml (116.6 KB)

helm_values.yaml (39.5 KB)

api-gateway_mender-api-gateway-595dd94fdc-lzqlj.yml (856.4 KB)

@list_tenants.txt

tenantadm list-tenants
Namespace: default
Timestamp: 公曆 20廿五九月卅週二十二時十七分44秒

@mongo_tenant.txt

$ cat mongo_tenantadm.txt 
mongosh use tenantadm
Namespace: default
Timestamp: 公曆 20廿五年 九月 卅日 週二 十二時十八分七秒
----------------------------------------
Current Mongosh Log ID:	68db5a0397c6aff084ce5f46
Connecting to:		mongodb+srv://<credentials>@mender-mongodb-headless.default.svc.cluster.local/?tls=false&appName=mongosh+2.5.8
Using MongoDB:		8.0.11
Using Mongosh:		2.5.8

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

------
   The server generated these startup warnings when booting
   2025-09-30T01:20:51.377+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2025-09-30T01:20:52.292+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2025-09-30T01:20:52.292+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2025-09-30T01:20:52.292+00:00: We suggest setting the contents of sysfsFile to 0.
   2025-09-30T01:20:52.292+00:00: Your system has glibc support for rseq built in, which is not yet supported by tcmalloc-google and has critical performance implications. Please set the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=0
   2025-09-30T01:20:52.292+00:00: vm.max_map_count is too low
   2025-09-30T01:20:52.292+00:00: We suggest setting swappiness to 0 or 1, as swapping can cause performance problems.
------

rs0 [primary] test> use tenantadm;
switched to db tenantadm
rs0 [primary] tenantadm> var collections = db.getCollectionNames();

rs0 [primary] tenantadm> collections.forEach(function(collectionName) {
...   print('Collection: ' + collectionName);
...   db[collectionName].find().forEach(function(doc) {
...     printjson(doc);
...   });
... });

rs0 [primary] tenantadm> quit;

alanlineys3d · September 30, 2025, 9:10am

These logs were doing terminal “mender-cli artifacts upload”

$ time mender-cli artifacts upload ./raspberrypi4.mender –server http://mender.internal.eys3d.com

Configuration file not found. Continuing.
566.59 MiB / 597.04 MiB [------------------------------------------------------------------------------------------------------------------------------------------------------->________] 94.90% 5.76 MiB p/sFAILURE: artifact upload to 'mender.internal.eys3d.com' failed with status 504

real	1m1.951s
user	0m1.059s
sys	0m1.514s

Extract the full logs : )

$ mv mender_support_2025_0930_163727.tar.yaml mender_support_2025_0930_163727.tar.gz

$ tar -xvf mender_support_2025_0930_163727.tar.gz

mender_support_20250930_163727.tar.yaml (77.3 KB)

traefik-bd6b7ccb9-w892j_HTTP_with_mender_cli.yml (184.7 KB)

By the way, in faster network environment,

we could upload artifacts by adding “–direct” parameter.

But, in the slower network environment, for example 8Mbps, even we adding “–direct” parameter, it still failed easily.

robgio · September 30, 2025, 12:14pm

Hi @alanlineys3d thanks for your report. I’m able to reproduce it. I’m still not sure where the real issue is, for now we just opened an internal issue to investigate further.

alanlineys3d · October 14, 2025, 6:06am

Any update or any logs that I should provide?

I feel it’s the same problem. That user failed at 59998118us ~= 60 secs

github.com/mendersoftware/mender-server

Issues with Artifact Upload in Mender 3.7.4

opened 12:53PM - 22 Oct 24 UTC

closed 11:03AM - 15 May 25 UTC

mehta-akshay-scanomat

**Description:** I am currently using the open-source version of Mender (3.7.…4) and encountering persistent issues when attempting to upload artifacts. Specifically, I receive a 5xx error during the upload process. When using the UI to upload an artifact, I see the message: *"Artifact couldn't be generated. Request failed with status code 502."* The logs for the `mender-deployment` service show a corresponding status code of 500: ``` time="2024-10-21T16:55:48Z" level=error msg="azblob PutObject: failed to upload object to blob: context canceled" caller="view.(*RESTView).RenderInternalError@view.go:72" request_id=66772fb8-f862-451a-83b7-046743424cc2 user_id=753afdfb-ee20-4fd3-985e-85c74fe4c56e time="2024-10-21T16:55:48Z" level=info msg="500 59998118μs POST /api/management/v1/deployments/artifacts/generate HTTP/1.1 - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" byteswritten=78 caller="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1@middleware.go:82" method=POST path=/api/management/v1/deployments/artifacts/generate qs= request_id=66772fb8-f862-451a-83b7-046743424cc2 responsetime=59.998118647 status=500 ts="2024-10-21 16:54:48.518920248 +0000 UTC" type=http user_id=753afdfb-ee20-4fd3-985e-85c74fe4c56e ``` When using `curl`, I receive a 502 Bad Gateway error: ```bash curl -X POST ${URL}/api/management/v1/deployments/artifacts \ -H 'Content-Type: multipart/form-data' \ -H "Authorization: Bearer ${JWT}" \ -F "artifact=@${ARTIFACT}" ``` Using the Mender CLI for artifact uploads results in a status code 409. I have identified a pattern: files that take longer than one minute to upload consistently fail, whereas smaller files (around 200-300 MB) that upload in under a minute succeed with a status code of 201. **Questions:** 1. Is there a default size limitation or timeout configuration that could be affecting these uploads? 2. I found some relevant configurations in the [Mender Server repository](https://github.com/mendersoftware/mender-server/blob/main/backend/services/deployments/config/config.go). I attempted to modify these settings by editing the deployment in Kubernetes: ```yaml DEPLOYMENTS_STORAGE_DEFAULT: azure DEPLOYMENTS_STORAGE_UPLOAD_EXPIRE_SECONDS: 300 DEPLOYMENTS_STORAGE_MAX_GENERATE_DATA_SIZE: 1073741824 ``` However, the issue persists. Any guidance on resolving this would be greatly appreciated! Thank you!

Topic		Replies	Views
Mender upload fails for big artifacts General Discussions	10	852	April 12, 2021
Mender-proxy/deployments unable to upload files to minio General Discussions minio , proxy , deployment	17	3819	June 18, 2020
Mender Production update from v2.2.0 to v2.3.0 General Discussions	32	3035	April 27, 2020
Unable to access artifacts in Mender 3.0 General Discussions	11	453	August 5, 2021
Can't upload signed artifacts compressed with LZMA General Discussions docker	30	2194	July 26, 2019

Upload artifact more than 1 minute will fail

Related topics