Unable to access artifacts in Mender 3.0

Dave · August 2, 2021, 9:42am

Currently we are trying to move from Mender 2.4.0 to Mender 3.0.0. The mender-server 3.0 seems to work as the UI shows the old releases, deployments and devices. Unfortunately deployments fail as access to the artifacts result in errors. This behavior can easily be mimicked by downloading the artifact using the mender-server UI.

If I download an artifact created at the time it was part of mender 2.4.0 I get the following message:

An error occurred while processing your request.

Reference #172.a4643e17.1627896281.1d704d

If I download an artifact uploaded in Mender server 3.0.0 the download starts but I get only a part of the artificact though the UI says the download was completed successfully. The artifact size is 1.7 GB but the download size only 144MB. According to the UI the upload was also successful and it shows a size 1.7 GB.

FYI, I used both the latest version of Chrome and Edge browsers.

Any idea what could be causing this problem?

Thanks in advance.

tranchitella · August 2, 2021, 9:53am

@Dave updating from Mender 2.4 to 3.0 directly is not supported, we suggest upgrading versions one by one. Can you please describe how you installed Mender and the procedure you followed to upgrade it? Can you please share the logs from the deployments service?

Dave · August 2, 2021, 12:15pm

@tranchitella I performed the following steps:

Upgraded mongo DB database to the latest version as described for Mender server 2.5.0
Stopped mender server and renamed mender server 2.4.0 folder
Followed the installation procedure ( Installation with Docker Compose | Mender documentation)
Copied artifact key as used in Mender 2.4.0
Put Minio secret as used in Mender 2.4.0 in prod.yml
Start Mender server 3.0.0

Please find deployments service logs below:

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time=“2021-08-02T12:09:05Z” level=info msg=“‘presign.secret’ not configured. Generating a random secret.” file=main.go func=main.doMain.func1 line=99
time=“2021-08-02T12:09:05Z” level=info msg=“Deployments Service, version unknown starting up”
time=“2021-08-02T12:09:05Z” level=info msg=“automigrate is ON, will apply migrations” file=migrations.go func=mongo.Migrate line=48
time=“2021-08-02T12:09:05Z” level=info msg=“migrating deployment_service” file=migrations.go func=mongo.MigrateSingle line=70
time=“2021-08-02T12:09:05Z” level=info msg=“migration to version 1.2.1 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-08-02T12:09:05Z” level=info msg=“migration to version 1.2.2 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-08-02T12:09:05Z” level=info msg=“migration to version 1.2.3 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-08-02T12:09:05Z” level=info msg=“migration to version 1.2.4 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-08-02T12:09:05Z” level=info msg=“migration to version 1.2.5 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-08-02T12:09:05Z” level=info msg=“DB migrated to version 1.2.5” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
time=“2021-08-02T12:09:23Z” level=info msg=“404 237μs POST /api/devices/v2/deployments/device/deployments/next HTTP/1.1 - Go-http-client/1.1” byteswritten=30 device_id=a1c7d29c-4ac7-4454-8074-2f48e2ee3324 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/devices/v2/deployments/device/deployments/next plan=enterprise qs= request_id=6eebdc45-ec41-4371-bfb4-4e89d1cfa700 responsetime=0.000237302 status=404 ts=“2021-08-02 12:09:23.848783825 +0000 UTC” type=http
time=“2021-08-02T12:09:23Z” level=info msg=“204 6342μs POST /api/devices/v1/deployments/device/deployments/next HTTP/1.1 - Go-http-client/1.1” byteswritten=0 device_id=a1c7d29c-4ac7-4454-8074-2f48e2ee3324 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs= request_id=ee438469-e6ec-4444-bda4-611f0bb90e1a responsetime=0.006342278 status=204 ts=“2021-08-02 12:09:23.969319767 +0000 UTC” type=http
time=“2021-08-02T12:09:26Z” level=info msg=“200 7048μs GET /api/management/v1/deployments/deployments?status=finished&per_page=20&page=1&sort=desc HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=6431 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments qs=“status=finished&per_page=20&page=1&sort=desc” request_id=9bcb6e0d-7aea-4701-9f7d-5bec49f8d2f5 responsetime=0.007048001 status=200 ts=“2021-08-02 12:09:26.730627111 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 627μs GET /api/management/v1/deployments/deployments?status=inprogress&per_page=20&page=1&sort=desc HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=2 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments qs=“status=inprogress&per_page=20&page=1&sort=desc” request_id=279db586-2c97-4595-90d6-e054c50c0858 responsetime=0.000627622 status=200 ts=“2021-08-02 12:09:26.741316728 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 6507μs GET /api/management/v1/deployments/deployments/releases HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=4341 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/releases qs= request_id=eec8177c-679a-4bda-88f0-2d28cb1765a0 responsetime=0.006507193 status=200 ts=“2021-08-02 12:09:26.808356713 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 2588μs GET /api/management/v1/deployments/deployments/7561a4df-ad98-44b4-b86f-fdaad1fc1aab/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/7561a4df-ad98-44b4-b86f-fdaad1fc1aab/statistics qs= request_id=ad411209-37a6-46cf-83df-232cb07c5937 responsetime=0.00258876 status=200 ts=“2021-08-02 12:09:26.837083187 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 445μs GET /api/management/v1/deployments/deployments/a5e84a71-c87a-4c66-abdb-165bb098b1ad/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/a5e84a71-c87a-4c66-abdb-165bb098b1ad/statistics qs= request_id=e9cdb205-f15a-4fe7-8a8e-b5a3b971ce87 responsetime=0.000445947 status=200 ts=“2021-08-02 12:09:26.848317276 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 814μs GET /api/management/v1/deployments/deployments/0acd62ea-328f-497f-ab53-b9db2a6a4899/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/0acd62ea-328f-497f-ab53-b9db2a6a4899/statistics qs= request_id=1258203d-4b73-4f67-a766-ab52ebfeb54d responsetime=0.000814067 status=200 ts=“2021-08-02 12:09:26.856270449 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 678μs GET /api/management/v1/deployments/deployments/960b08af-a412-4495-be42-a1e5836e2380/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/960b08af-a412-4495-be42-a1e5836e2380/statistics qs= request_id=7a10e1d5-a6eb-4e10-9353-85c97fbd1510 responsetime=0.000678094 status=200 ts=“2021-08-02 12:09:26.865310896 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 417μs GET /api/management/v1/deployments/deployments/98cdb24d-de1b-44ba-b432-aa425575c217/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/98cdb24d-de1b-44ba-b432-aa425575c217/statistics qs= request_id=f9693afc-805b-4e79-bec0-e23ef5aac18f responsetime=0.000417198 status=200 ts=“2021-08-02 12:09:26.87448587 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 516μs GET /api/management/v1/deployments/deployments/04c50bdf-ad86-4cf1-8720-2c21cd27a46d/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/04c50bdf-ad86-4cf1-8720-2c21cd27a46d/statistics qs= request_id=c0c28f35-0294-478f-8dd1-b4856f00ce0a responsetime=0.000516595 status=200 ts=“2021-08-02 12:09:26.881607468 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 391μs GET /api/management/v1/deployments/deployments/b8d9a988-f3d4-44e4-98ae-f68dc33a5751/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/b8d9a988-f3d4-44e4-98ae-f68dc33a5751/statistics qs= request_id=7d95a00c-8583-45ed-841b-819e0fe58639 responsetime=0.000391299 status=200 ts=“2021-08-02 12:09:26.886417732 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 689μs GET /api/management/v1/deployments/deployments/84b1f35b-3e66-4a25-90b9-089607689e2a/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/84b1f35b-3e66-4a25-90b9-089607689e2a/statistics qs= request_id=3a855664-cee3-4d68-bdca-ac26bb86b8b3 responsetime=0.00068908 status=200 ts=“2021-08-02 12:09:26.895817592 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 422μs GET /api/management/v1/deployments/deployments/85e0dd9a-7314-40cb-93b3-c803a927a5ee/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/85e0dd9a-7314-40cb-93b3-c803a927a5ee/statistics qs= request_id=8d341cd1-42be-41fb-997e-b7155ea24b83 responsetime=0.000422819 status=200 ts=“2021-08-02 12:09:26.903711778 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 1002μs GET /api/management/v1/deployments/deployments/caf58611-658a-411d-9ee0-93cf7b22034a/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/caf58611-658a-411d-9ee0-93cf7b22034a/statistics qs= request_id=051d11fd-ca72-4c79-a504-bb274a1cdfdb responsetime=0.001002988 status=200 ts=“2021-08-02 12:09:26.912852086 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 813μs GET /api/management/v1/deployments/deployments/d9a1e2aa-63e9-43f7-b466-f8190643c3ba/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/d9a1e2aa-63e9-43f7-b466-f8190643c3ba/statistics qs= request_id=964543c5-ae56-4197-83f3-14cc1a7adf23 responsetime=0.000813098 status=200 ts=“2021-08-02 12:09:26.93347104 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 426μs GET /api/management/v1/deployments/deployments/917fa20a-ee93-46a7-a192-3d5e84f5d0a2/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/917fa20a-ee93-46a7-a192-3d5e84f5d0a2/statistics qs= request_id=83a8c71b-8fdf-4666-8b02-c560a4b2916f responsetime=0.000426576 status=200 ts=“2021-08-02 12:09:26.936361633 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 524μs GET /api/management/v1/deployments/deployments/7ad2e9f4-6f60-4ef1-822f-c34d6427e6dd/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/7ad2e9f4-6f60-4ef1-822f-c34d6427e6dd/statistics qs= request_id=4d1a46db-dd71-4f3f-9c68-27f0a674dca1 responsetime=0.000524682 status=200 ts=“2021-08-02 12:09:26.955064675 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 658μs GET /api/management/v1/deployments/deployments/35ba15b2-dc37-4e97-abec-a259af57c177/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/35ba15b2-dc37-4e97-abec-a259af57c177/statistics qs= request_id=183cd614-abf4-4948-b9f4-eb47e08fa075 responsetime=0.00065897 status=200 ts=“2021-08-02 12:09:26.979485764 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:26Z” level=info msg=“200 656μs GET /api/management/v1/deployments/deployments/dca4e463-7acb-4cf8-9e7c-35bd346bbde1/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/dca4e463-7acb-4cf8-9e7c-35bd346bbde1/statistics qs= request_id=564b3890-24ef-4ab4-864c-e5ea399b6fa6 responsetime=0.000656909 status=200 ts=“2021-08-02 12:09:26.991699471 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:27Z” level=info msg=“200 628μs GET /api/management/v1/deployments/deployments/53f47de2-349c-49ff-82ce-abe6425d11d2/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=233 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/53f47de2-349c-49ff-82ce-abe6425d11d2/statistics qs= request_id=5b0dc0e7-310e-48fd-95b5-e25a5969b6ec responsetime=0.000628213 status=200 ts=“2021-08-02 12:09:27.004184117 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:27Z” level=info msg=“200 466μs GET /api/management/v1/deployments/deployments/a9107ae1-64ba-423b-9d7f-40f1d3184225/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=150 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/a9107ae1-64ba-423b-9d7f-40f1d3184225/statistics qs= request_id=229a78d4-f887-4ba9-93fb-d1eed821b839 responsetime=0.00046656 status=200 ts=“2021-08-02 12:09:27.009477411 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:27Z” level=info msg=“200 476μs GET /api/management/v1/deployments/deployments/d8f0ffaf-63fe-471a-b3a1-703b702897b4/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=150 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/d8f0ffaf-63fe-471a-b3a1-703b702897b4/statistics qs= request_id=8549ffcd-112c-4819-8128-a5824786f872 responsetime=0.000476201 status=200 ts=“2021-08-02 12:09:27.019654205 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:27Z” level=info msg=“200 533μs GET /api/management/v1/deployments/deployments/ab6f6ec3-51de-43d0-b7ff-c81968105b1d/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=150 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/ab6f6ec3-51de-43d0-b7ff-c81968105b1d/statistics qs= request_id=93087e3f-5168-46ad-b5fe-d02d31dfd3a4 responsetime=0.000533334 status=200 ts=“2021-08-02 12:09:27.026187911 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:27Z” level=info msg=“200 945μs GET /api/management/v1/deployments/deployments/a308a0b2-3380-4c85-a47d-a981a7e9ae22/statistics HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=150 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/a308a0b2-3380-4c85-a47d-a981a7e9ae22/statistics qs= request_id=0badccfd-571b-4f24-95a7-408550ab95b7 responsetime=0.000945006 status=200 ts=“2021-08-02 12:09:27.038920942 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:38Z” level=info msg=“200 6959μs GET /api/management/v1/deployments/artifacts/e9944bc9-c4bb-47bf-b692-3c2e5e4a03eb/download HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=573 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/artifacts/e9944bc9-c4bb-47bf-b692-3c2e5e4a03eb/download qs= request_id=cfeb4d39-97d2-4513-8f97-e9a31ea422e4 responsetime=0.006959074 status=200 ts=“2021-08-02 12:09:38.327295945 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:46Z” level=info msg=“200 9618μs GET /api/management/v1/deployments/artifacts/666ae908-1b7f-4683-b9c9-2df51570bc93/download HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=573 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/artifacts/666ae908-1b7f-4683-b9c9-2df51570bc93/download qs= request_id=99d1acf8-9138-4433-90b7-0e70b2cb4274 responsetime=0.009618922 status=200 ts=“2021-08-02 12:09:46.184755382 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:09:56Z” level=info msg=“204 1560μs POST /api/devices/v1/deployments/device/deployments/next HTTP/1.1 - Go-http-client/2.0” byteswritten=0 device_id=6dcc49c9-64da-4cbb-a771-116e223f40f7 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs= request_id=4291a977-2aa8-42be-9504-34ecf2efb375 responsetime=0.001560441 status=204 ts=“2021-08-02 12:09:56.501289675 +0000 UTC” type=http
time=“2021-08-02T12:10:05Z” level=info msg=“200 936μs GET /api/management/v1/deployments/deployments/releases HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=4341 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/releases qs= request_id=a1f6cc33-6be0-49d4-a89f-ce2073a1c847 responsetime=0.000936705 status=200 ts=“2021-08-02 12:10:05.310328051 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:10:23Z” level=info msg=“404 104μs POST /api/devices/v2/deployments/device/deployments/next HTTP/1.1 - Go-http-client/1.1” byteswritten=30 device_id=a1c7d29c-4ac7-4454-8074-2f48e2ee3324 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/devices/v2/deployments/device/deployments/next plan=enterprise qs= request_id=9f91dfca-4389-4803-a1b1-4a3b1b1cea8c responsetime=0.000104776 status=404 ts=“2021-08-02 12:10:23.821742011 +0000 UTC” type=http
time=“2021-08-02T12:10:24Z” level=info msg=“204 1855μs POST /api/devices/v1/deployments/device/deployments/next HTTP/1.1 - Go-http-client/1.1” byteswritten=0 device_id=a1c7d29c-4ac7-4454-8074-2f48e2ee3324 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs= request_id=ec300037-ca5e-4c33-a9b9-2c40ae582c6e responsetime=0.001855052 status=204 ts=“2021-08-02 12:10:24.034963624 +0000 UTC” type=http
time=“2021-08-02T12:10:35Z” level=info msg=“200 958μs GET /api/management/v1/deployments/deployments/releases HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55” byteswritten=4341 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/management/v1/deployments/deployments/releases qs= request_id=c3daad1c-f869-4982-b036-32b58bc15a8a responsetime=0.000958982 status=200 ts=“2021-08-02 12:10:35.482586653 +0000 UTC” type=http user_id=4bf5cb4e-2f3d-4a18-a8ef-cd8a5b1b67b9
time=“2021-08-02T12:10:36Z” level=info msg=“204 1412μs POST /api/devices/v1/deployments/device/deployments/next HTTP/1.1 - Go-http-client/2.0” byteswritten=0 device_id=22559441-426f-42ec-b739-ce59c3ea9e71 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs= request_id=abb4d400-d950-498b-991e-970a8372bb6e responsetime=0.001412747 status=204 ts=“2021-08-02 12:10:36.106357788 +0000 UTC” type=http

Thanks for the support.

tranchitella · August 4, 2021, 7:19pm

@dave the logs look good. Can I kindly ask you to check the logs of the API gateway (traefik)? How long does the download take? Can it be a timeout? Are you exposing Mender over the internet? If yes, directly from docker-compose or using something in front of it (load balancer, reverse proxy)?

Dave · August 5, 2021, 9:00am

@tranchitella and others. I did a complete new virgin install of Mender 3.0.0 server (incl. new keys/certificates/databases etc) and followed the procedure as mentioned in my previous post.

I experimented with uploading artifacts with different sizes (as shown in the dashboard) and here are my results:
0.10 GB OK. Able to download file from Mender server
0.5 GB OK. Able to download the file from Mender server
1.0 GB OK. Able to download the file from Mender server
1.58 GB OK. Able to download the file from Mender server
1.70 GB Failed. Download stops between 300-500MB
1.77 GB. Failed with message. An error occurred while processing your request. Reference xxx.xxx.xxx
1.86 GB. Failed with message. An error occurred while processing your request. Reference xxx.xxx.xxx

Apparently something goes wrong with large artifacts. Is this a known issue for Mender 3.0.0? We never experienced this in Mender 2.4.0.

Btw. our system has enough free disk space and for all uploads the Mender server mentioned it was completed successfully.

Thanks for the help.

tranchitella · August 5, 2021, 9:14am

@Dave Are you using a docker-compose setup, or Kubernetes/Helm charts? Are you exposing Mender directly or you are proxying it with another webserver (e.g. nginx)? Are the tests you are performing happening over the internet? How long does it take until the download stops?

Dave · August 5, 2021, 9:39am

@tranchitella I’m using the docker-compose setup. Downloading of 400 MB takes about ~15 second at ~25 MB/s. Our Mender server is proxied with Akamai.

But now I’m really confused . The download of the 1.70 GB artifact succeeds now. I tried it about 30 times over a timespan of two days and it always failed. The only thing that changed is that I added multiple artifacts with different sizes for testing. Could it be the case that this behavior is only present when the first artifacts is of a large size (>1.70GB)?

I’m also curious why I get the error message “An error occurred while processing your request. Reference xxx.xxx.xxx” for files larger than 1.77GB? Is there a limit configured ?

Thanks.

tranchitella · August 5, 2021, 10:13am

@Dave I don’t think having multiple artifacts is making any difference, and in any case the download is happening directly from minio, going through the Traefik API gateway. The API gateway has very high timeouts:

github.com

mendersoftware/integration/blob/master/docker-compose.yml#L49

    
      
          extends:
              file: common.yml
              service: mender-base
          # Enables the web UI and tells Traefik to listen to docker
          command: 
              - --accesslog=true
              - --entrypoints.http.address=:80
              - --entrypoints.http.http.redirections.entryPoint.scheme=https
              - --entrypoints.http.http.redirections.entryPoint.to=https
              - --entrypoints.https.address=:443
              - --entryPoints.https.transport.respondingTimeouts.idleTimeout=7200
              - --entryPoints.https.transport.respondingTimeouts.readTimeout=7200
              - --entryPoints.https.transport.respondingTimeouts.writeTimeout=7200
              - --providers.file.directory=/etc/traefik/config
          volumes:
              # So that Traefik can listen to the Docker events
              - /var/run/docker.sock:/var/run/docker.sock:ro
              # Dynamic configuration files
              - ./config/traefik/traefik.yaml:/etc/traefik/config/traefik.yaml:ro
              - ./config/traefik/traefik.middlewares.yaml:/etc/traefik/config/traefik.middlewares.yaml:ro

We are not aware of issues with large artifacts. Can it be some timeout/setting in Akamai?
Have you tried doing the same operation without Akamai in the middle?

Dave · August 5, 2021, 11:08am

@tranchitella Unfortunately I don’t have control over Akamai as it’s controlled by our IT department.
I’m happy that I can download the artifact now. Hopefully it does not appear in the future anymore.

But the other issue about the error “An error occurred while processing your request. Reference #172.a4643e17.1628161367.53853c1” when I want to download the artifact is concerning me as our firmware is nearing the 2GB.

Do you think this is a Mender issue? As the upload of the artifact completed successfully as confirmed in the UI. I also see all the details of the artifact in the Mender UI when selecting it. Probably the error message is generated by minio(?).

Thanks.

tranchitella · August 5, 2021, 11:12am

@dave can you extract the logs from deployments and minio just before and after expanding the artifact in the “Releases” section of the UI, and then clicking on the download artifact button?

Dave · August 5, 2021, 12:58pm

@tranchitella I contacted our IT department. They confirmed that the error is an Akamai error and there is a limit on file size. So the problem is at our side and we need to fix it.

I was not aware of this limit and sorry for bothering you with this.

Thanks a lot for all the support and the great product

tranchitella · August 5, 2021, 1:38pm

@dave you’re welcome, and thanks for letting me know.

Topic		Replies	Views
Unable to upload artifacts to mender server 2.3 General Discussions	6	468	July 21, 2020
Upload Artifact fails because of size General Discussions	6	1287	October 6, 2021
Mender upload fails for big artifacts General Discussions	10	662	April 12, 2021
Artifact upload error both from web (Network Error) and cli (failed with status 413) General Discussions mender-cli	3	236	October 25, 2023
Can't upload signed artifacts anymore General Discussions	8	526	March 25, 2020

Unable to access artifacts in Mender 3.0

Related Topics