Update and statement on the Mender Client 4.0 Debian package incident

Announcements
1

Hello friends, users and contributors of Mender!

Those of you who are using the Debian package feeds in general, and the express installation script in particular, suffered installation problems of the client package in the last two days.

In this post I want to apologize for the inconvenience, show how it came to be and share the way forward.

As we analyzed the issues that were raised, we found some things where our reasoning did not match the reality of your use cases, and this caused the failures. Let me take them one by one.

  • The concept of the express installation script was to facilitate a quick and simple onboarding of devices, especially during evaluation. As such it would always install the latest released version of the client package suite. We did not expect this to be used in production flows.
  • Our debian repositories were constructed to provide the latest release, which caused the 3.5.2 client to be delisted.
  • We expected just a very short delay between the repository changes and making the release notes and documentation public.
  • The 4.0 client should serve as a full function drop in replacement.

So where did we go wrong? Three main things, as far as we can tell.

  1. The new client is a functional replacement, but not a drop-in yet. Especially the mender binary invocation was not shipped with the first iteration.
  2. We expected users of the debian package repositories to stay on “latest release”, which the user expectations were “repository stays on a single major release”.
  3. The documentation was not ready.

Those together caused the failure that you witnessed.

We sincerely apologize and regret any inconvenience and trouble this has caused.

How will we proceed?

Workaround

As a first and quick fix, you can still download the packages directly from the repositories, they are just deleted from the index. The URL pattern is:

wget https://downloads.mender.io/repos/debian/pool/main/m/mender-client/mender-client_3.5.2-1+DIST+NAME_ARCH.deb

Examples:

  • Debian buster, amd64 is:
wget https://downloads.mender.io/repos/debian/pool/main/m/mender-client/mender-client_3.5.2-1+debian+buster_amd64.deb
  • Ubuntu Focal, arm64 is:
wget https://downloads.mender.io/repos/debian/pool/main/m/mender-client/mender-client_3.5.2-1+ubuntu+focal_arm64.deb

You can install the package as usual with dpkg -ithen.

mender-convert

This issue is also affecting users relying on the LTS version of mender-convert. The LTS version of mender-convert (4.0.3) which shipped with the 3.6.3 bundle, pulls the 4.0.0 client and triggers the issue. The updated mender-convert (4.1.1) version has the client set to 3.5.2 so it wont trigger the issue.

Until the fix is in place the workaround is to explicitly set the client version to 3.5.2 in your configuration:

MENDER_CLIENT_VERSION="3.5.2"

Repository

Meanwhile, we are working on fixing the Mender Client 3.5.2 packages in the repository feed. As the version number 4.0.0 has unfortunately now been rolled out to a number of clients, we cannot downgrade easily. The solution that we will employ is bumping the so-called epoch of the repository. That means the new-epoch 3.5.2 client will take precedence, and fix broken installations. If you receive the package while still running the old epoch 3.5.2 client, it will result in a null update. We think this is the most sustainable solution.

The upcoming 4.0 client will be re-packaged under a new name which aligns with the version advance strategy of the main Debian and Ubuntu repositories.

We hope this explains the events that led up to the incident, and will do our best to sort it out in a timely manner.

Thank you for being with us - feel free to drop us a line in the thread below.

Update 1 - 2024-01-19, 10:00 UTC

The repositories are now offline for preparation of the fixed index. Manual downloads are still available.

Update 2 - 2024-01-19, 19:00 UTC

  • the repositories are back online.
  • mender-convert 4.1.1 defaults to Mender Client 3.5.2
  • the express installation script is working again.

We still will find rough edges here and there probably, but the major functionality is restored.

Josef,
Head of Developer Relations, Mender

1 Like