Unable to download artifact after upload

General Discussions
1

Hello Everyone

Current setup:
Mender version 3.1.0 (3.2.0 does not work)
Mender deployed on Kubernetes with the use of Helm.
Using AWS S3 as artifact store.

I’m able to upload the artifact and start a deployment with a Golang script, but I’m not able to generate a pre-singed download link for the artifact.

If I use the mender web GUI and open the artifact page, I see a call to ‘/api/management/v1/deployments/artifacts/ad8ce34a-3813-4f62-b104-558b669d723b/download’ but this is dropped by the server. Chrome dev tools say ‘Canceled’.

In the pod logs of the Deployment container I see the following:

time="2022-04-22T08:58:15Z" level=info msg="404 124746μs GET /api/management/v1/deployments/artifacts/ad8ce34a-3813-4f62-b104-558b669d723b/download HTTP/1.1 - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" byteswritten=82 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/management/v1/deployments/artifacts/ad8ce34a-3813-4f62-b104-558b669d723b/download qs= request_id=d33bb6af-71a3-4fd9-aad5-c416f68ba9a1 responsetime=0.124746344 status=404 ts="2022-04-22 08:58:14.953568392 +0000 UTC" type=http user_id=576c8f31-d7e5-4413-b468-ed3f2d325d3f

Additional notes:
I can see the artifact on AWS S3, so the upload is successful and the AWS credentials are working.
Normally on AWS S3 if you want to make a pre-signed URL you first have to perform an S3.GetObject API call, I don’t see it appear in our CloudTrail logs (S3 data is enabled in cloud trail).
If we use Minio instead of AWS S3 the download works as normal.
The AWS credentials have “S3:*” permission with the bucket as a scoped resource.

Can someone provide some more insight on how to resolve this issue?
Is there a possibility to enable verbose logging?

Thanks in advance
Dimitri Vervecken

2

Hello Again

Just giving some additional information:

The AWS S3 part is configured in Helm as follows:

global:
  enterprise: false
  mongodb:
    URL: "mongodb://root:{{ mender_mongodb_root_password }}@mongodb-0.mongodb-headless.mender.svc.cluster.local:27017,mongodb-1.mongodb-headless.mender.svc.cluster.local:27017"
  nats:
    URL: "nats://nats:4222"
  s3:
    AWS_URI: "https://duracloud-image-store-{{ dtc_environment }}-eu-central-1.s3.eu-central-1.amazonaws.com"
    AWS_BUCKET: "duracloud-image-store-{{ dtc_environment }}-eu-central-1"
    AWS_REGION: "eu-central-1"
    AWS_ACCESS_KEY_ID: "{{ aws_s3_accesskey }}"
    AWS_SECRET_ACCESS_KEY: "{{ aws_s3_secretkey }}"
    AWS_FORCE_PATH_STYLE: "false"

Maybe this gives some insight.

Thanks in advance
~Dimitri