Mender on Azure: Can not up/download releases after upgrades

General Discussions
1

After updating a Mender 3.6 instance on Azure to 3.7 and then to 4.0, it’s not possible to upload or download releases. We noticed that some releases were missing and that we could neither upload new releases nor download any of the remaining ones.

Rolling back to version

  • 3.7 resolved the upload issue, but some releases were still missing and we could still not download any of them.
  • 3.6 restored the missing releases, and uploading and downloading became possible again.

We performed a similar upgrade on a different Mender deployment which worked fine. The difference is, that it used S3 instead of Azure. We assume these problems are Azure-related.

Has anyone had similar problems when upgrading on Azure?

2

Hello @dbasner from 3.x to 4.x there has been a helm chart major upgrade with some breaking changes; in particular for the artifacts storage, the Storage Proxy feature is enabled by default, see: mender-helm/UPGRADE_from_v5_to_v6.md at master · mendersoftware/mender-helm · GitHub

If you want to restore the 3.x configuration with the Mender Server v4.0, you should explicitly disable the Storage Proxy:

api_gateway:
  storage_proxy:
    enabled: false

deployments:
  customEnvs: []

If that’s not the case, you could share more details of your issue, to better understand what’s happening.

3

Hi @robgio, thank you for your fast response!

We are aware of the issue and disabled the storage proxy when upgrading from version 3.7 to version 4.0. On 3.7 we already had the issue, that some releases were not shown, and we could not download any of those that were still shown. Uploading did work, though. Below is the values.yaml we used.

For the upgrade to 3.7, we changed the image and removed the nats-image that was previously set. We got an error for that, but the default 2.9.25-scratch seemed to work fine.

For 4.0, we removed the image from the global section and added it under the default section with the changed naming format. We also disabled the storage_proxy, as outlined in the [upgrade guide](https://docs.mender.io/server-installation/upgrading-from-previous-versions/upgrading-from-3.7-to-4.0). These were the only changes we made, based on the initial values.yaml that we retrieved via helm get values mender from the existing 3.6 deployment.

BTW, my colleague @eebi-sotec is mainly working on this (his account got blocked somehow…). We still believe that it might have something to do with azure’s blob storage. We are currently evaluating options on how to proceed to get a working v4.0 on azure.

USER-SUPPLIED VALUES:
api_gateway:
  storage_proxy:    # <--- set for 4.0
    enabled: false  # <--- set for 4.0
  env:
    SSL: false
device_auth:
  certs:
    key: |-
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----
default:            # <--- set for 4.0 
  image:            # <--- set for 4.0  
    tag: v4.0.0     # <--- set for 4.0     
global:
  azure:
    AUTH_SHARED_KEY_ACCOUNT_KEY: ***** 
    AUTH_SHARED_KEY_ACCOUNT_NAME: *****
    CONTAINER_NAME: *****
  enterprise: false
#  image:              # <--- removed for 4.0
#    tag: mender-3.7.0 # <--- changed for 3.7, removed for 4.0
  mongodb:
    URL: ""
  nats:
    URL: ""
  smtp:
    EMAIL_SENDER: xyz@abc.com
    SMTP_AUTH_MECHANISM: STARTTLS
    SMTP_HOST: smtp-mail.xyz.com:***
    SMTP_PASSWORD: *****
    SMTP_SSL: "true"
    SMTP_USERNAME: xyz@abc.com
  storage: azure
  url: https://xyz.com
mongodb:
  auth:
    enabled: true
    replicaSetKey: *****
    rootPassword: *****
  enabled: true
nats:
  enabled: true
#  nats:                        # <--- removed for 3.7
#    image: nats:2.9.21-scratch # <--- removed for 3.7
useradm:
  certs:
    key: |-
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----
4

Hi @dbasner could you try change from:

global:
  azure:
    AUTH_SHARED_KEY_ACCOUNT_KEY: ***** 
    AUTH_SHARED_KEY_ACCOUNT_NAME: *****
    CONTAINER_NAME: *****

to:

global:
  azure:
    AUTH_CONNECTION_STRING: ***** 
    CONTAINER_NAME: *****

Thanks