NoSuchKey when uploading an artifact

General Discussions
, , , ,
1

Hello,

I have a problem uploading artifacts like system images, which are ~150MB. While smaller artifacts can easily be uploaded, big artifacts get stuck.

I am using selfhosted mender on a google VM using google cloud storage and recently updated it to integration-2.6.0. I attached the error log as well as the trace of one error case. I tried to

  • switch to a new bucket - It get’s created with in GCS and small artifacts can be uploaded, but still no success with the big artifacts
  • search through the code to see if something that might help with my search, see:

time=“2021-02-22T12:47:40Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: :
https://github.com/mendersoftware/deployments/blob/7370cc3611e02b5baec817bcee6369d143e1b78b/s3/filestorage.go#L205
reading artifact error: Payload: can not install Payload:
https://github.com/mendersoftware/mender-artifact/blob/3f57f208c23e98e24e066da28f500d1a43d2e51b/areader/reader.go#L1028
cpe-testimg-common-cloudplugedge-plus.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: :
https://github.com/mendersoftware/deployments/blob/1687ca015a1f7d4f30e33f24b40d990676bd5f67/app/app.go#L644
Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=356 request_id=c71ae7d5-c292-4757-888d-394725349e54 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
https://github.com/mendersoftware/deployments/blob/1687ca015a1f7d4f30e33f24b40d990676bd5f67/app/app.go#L290

mender-deployment config:uploadError config.yaml (1.1 KB)
I have also tried to run it with excluded storage proxy and minio - same result.

error logs:

mender-create-artifact-worker_1 | time=“2021-02-24T13:42:46Z” level=info msg=“LoadWorkflows: error loading: generate_artifact v1: Workflow already exists.” file=entry.go func=“logrus.(*Entry).Infof” line=346
mender-deployments_1 | time=“2021-02-24T13:43:49Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: : Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=356 request_id=135624ee-79b3-44f5-b9fd-6b51eb7d8d67 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:43:49Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: : Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=371 request_id=135624ee-79b3-44f5-b9fd-6b51eb7d8d67 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:43:49Z” level=error msg="Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: " file=view.go func=“view.(*RESTView).RenderError” line=53 request_id=135624ee-79b3-44f5-b9fd-6b51eb7d8d67 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-workflows-worker_1 | time=“2021-02-24T13:42:45Z” level=info msg=“LoadWorkflows: error loading: decommission_device v3: Workflow already exists.” file=entry.go func=“logrus.(*Entry).Infof” line=346
mender-workflows-worker_1 | time=“2021-02-24T13:42:45Z” level=info msg=“LoadWorkflows: error loading: provision_device v4: Workflow already exists.” file=entry.go func=“logrus.(*Entry).Infof” line=346
mender-workflows-worker_1 | time=“2021-02-24T13:42:45Z” level=info msg=“LoadWorkflows: error loading: update_device_inventory v2: Workflow already exists.” file=entry.go func=“logrus.(*Entry).Infof” line=346
mender-workflows-worker_1 | time=“2021-02-24T13:42:45Z” level=info msg=“LoadWorkflows: error loading: update_device_status v2: Workflow already exists.” file=entry.go func=“logrus.(*Entry).Infof” line=346
mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: : Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=356 request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: : Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=371 request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=error msg="Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: " file=view.go func=“view.(*RESTView).RenderError” line=53 request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63

Single request 2b09f86e-cd34-4e66-88e1-2b6d18c054c6 with error traced:

mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: : Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=356 request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=error msg=“Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: : Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=371 request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=error msg="Removing file: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: , host id: : reading artifact error: Payload: can not install Payload: cpe-testimg-common-cloudplugedge.ext4: InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: " file=view.go func=“view.(*RESTView).RenderError” line=53 request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-deployments_1 | time=“2021-02-24T13:44:26Z” level=info msg=“400 7052588μs POST /api/management/v1/deployments/artifacts HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36” byteswritten=252 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/management/v1/deployments/artifacts qs= request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 responsetime=7.052588818 status=400 ts=“2021-02-24 13:44:19.123733575 +0000 UTC” type=http user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63
mender-useradm_1 | time=“2021-02-24T13:44:19Z” level=info msg=“200 1705μs POST /api/internal/v1/useradm/auth/verify HTTP/1.1 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36” byteswritten=0 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/internal/v1/useradm/auth/verify qs= request_id=2b09f86e-cd34-4e66-88e1-2b6d18c054c6 responsetime=0.001705753 status=200 ts=“2021-02-24 13:44:19.121282212 +0000 UTC” type=http user_id=2bd1312c-8726-4be2-93ca-ef17e4ff1f63

2

@peter @tranchitella any thoughts here?

3

@dbasner from the error, it seems there is an error parsing the artifact (InvalidArgument: Invalid argument). The “removing file: NoSuchKey” error happens as a consequence, I believe.

Are you sure the artifact is correct? Can you correctly dump it using mender-artifact?

4

Hi,
I am a colleague of @dbasner. The artifact is an output of our yocto build. We don’t do any special things there. I don’t think the artifact is corrupted as this issue can be reproduced with different files.
Do you have any idea what to check on yocto side besides dumping the artifact with mender-artifact (will do this tomorrow when I am back on my workstation)
Cheers
Ruben

5

@ruben have you tried uploading the same artifact to a local Mender server running on docker-compose, with minio? Just to clear out it is not a problem with the artifact itself.

6

mender-artifact dump worked without errors. I was able to dump all components (scripts, files, metadata and cmdline).

7

I used the local demo server (./demo up) and was able to upload files on my local setup
I am still not sure if it might be something regarding google cloud storage. Is there anything besides the config shown that we might need?

8

@dbasner If it works with a local demo server, I assume the issue is Google Cloud Storage-specific. Unfortunately, I don’t have direct experience in using Google Cloud Storage with Mender. There are no additional configuration settings you can tweak at the Mender side.

9

Hi there,

I think I found the issue. With our recent update, we also updated from mender-deployments 1.9 to mender-deployments 2.2. There, the multipart s3 upload replaced the old upload mechanism. Afaik, Google Cloud Storage doesnt support this in its s3 compatibility atm, at least amazon s3 - Google cloud storage compatibility with aws s3 multipart upload - Stack Overflow indicates that. It would be interesting to see if a 2.2 but with the old s3/filestorage.go would do the trick. Also, the originally mentioned stackdriver post indicates why small uploads may work, since the first upload still seems to be successful.

Thanks so far!

10

@dbasner have you considered using minio as gateway to Google Cloud Storage?
See: MinIO | Learn how to deploy MinIO with GCS

11

Will test it, thanks

12

@tranchitella Thanks for the advise, that worked out good. After a few configs that need to be updated (somehow minio wont start with an existing bucket) we were good to go. Here is the config we finally came up with: prod-gcs-new.yaml (5.3 KB)

1 Like
13

@dbasner Glad it worked! Thanks for sharing the configuration file, very much appreciated!