Artifact uploads using AWS_WEB_IDENTITY_TOKEN_FILE creates an empty 0kb file in S3 bucket

General Discussions
1

This is a strange one.

I am testing deployment via mender helm on EKS with OIDC service account roles. This sets up an AWS_WEB_IDENTITY_TOKEN_FILE environment variable. this role has permission to write to an s3 bucket. I verified this works with AWS CLI, and it is also detected by mender deployment. There are no access denied errors when the deployments executable starts up. The deployments application does have permission to write to the bucket, but when I used mender-ui to upload an artifact, it creates an empty file in the s3 bucket.

If I use AWS access tokens, I have no problem. So to summarize, in side the deployments pod:

# This creates an empty file on upload
AWS_SDK_LOAD_CONFIG=1 DEPLOYMENTS_AWS_AUTH_SECRET= DEPLOYMENTS_AWS_AUTH_KEY= AWS_WEB_IDENTITY_TOKEN_FILE="/var/run/secrets/eks.amazonaws.com/serviceaccount/token" AWS_ROLE_ARN=arn:aws:iam::123123123123:role/MyS3WritingRole deployments --config /etc/deployments/config.yaml

# This works fine
DEPLOYMENTS_AWS_AUTH_SECRET=<MYSECRET> DEPLOYMENTS_AWS_AUTH_KEY=<MYKEY> AWS_WEB_IDENTITY_TOKEN_FILE="" deployments --config /etc/deployments/config.yaml

There are no details logged by deployments service that give me any further information.

2

This doesn’t appear to happen on 3.1.0, only on mender-latest. Hold tough on looking into this until I have more detail.

3

Can confirm that this is only a problem with deployments:mender-master. It works fine with 3.0.0. Apologies for the noise, but hopefully this will help your current development.

4

Thanks @grandfield for your report.
I will create a bug report in our Jira tracking system.

5

just a little more info. This happens on 3.2.0 also, but not on 3.1.0, so it was introduced in that time.