Hello,
I am new to Mender and trying to create my first test deployment using a single-file artifact and a Ubuntu server test device.
I have taken these steps successfully:
- Deployed Mender to EKS and ensured it is accessible from my Ubuntu server
- Created and uploaded my
single-file artifact to Mender
- Registered my Ubuntu server as a device with Mender by installing
mender-client4 using apt and entering my Mender server info
I started the deployment of my single-file artifact through the UI to my Ubuntu device but it failed, even though the file was indeed deployed to my Ubuntu device.
I sshed into the Ubuntu device and ran: sudo journalctl -u mender-updated | tail
This is what I saw (I have removed IDs and timestamps for brevity):
mender-update: severity=info name="Global" msg="Running Mender client 4.0.4"
mender-update: severity=info name="Global" msg="Deployment with ID 8d3f0784-93ad-41ec-ab54-756ba0a7d69f started."
mender-update: severity=info name="Global" msg="Sending status update to server"
mender-update: severity=error name="Global" msg="Request to push status data failed: PUT http://127.0.0.1:36069/api/devices/v1/deployments/device/deployments/8d3f0784-93ad-41ec-ab54-756ba0a7d69f/status: "
mender-update: severity=error name="Global" msg="Could not send deployment status: bad version: PUT http://127.0.0.1:36069/api/devices/v1/deployments/device/deployments/8d3f0784-93ad-41ec-ab54-756ba0a7d69f/status: "
mender-update: severity=info name="Global" msg="Installing artifact..."
mender-update: severity=info name="Global" msg="Sending status update to server"
mender-update: severity=error name="Global" msg="Request to push status data failed: PUT http://127.0.0.1:36069/api/devices/v1/deployments/device/deployments/8d3f0784-93ad-41ec-ab54-756ba0a7d69f/status: "
mender-update: severity=info name="Global" msg="Retrying status update after 60 seconds"
It appears that the install succeeded, but the Mender client was unable to update the Mender server because it is trying to access it locally. This was very confusing to me. Why is the device trying to connect to localhost when it was registered with my Mender server at a different domain?
This is what my mender.conf looks like:
$ cat /etc/mender/mender.conf
{
"HttpsClient": {},
"Security": {},
"Connectivity": {},
"DeviceTypeFile": "/var/lib/mender/device_type",
"UpdateControlMapExpirationTimeSeconds": 90,
"UpdateControlMapBootExpirationTimeSeconds": 45,
"UpdatePollIntervalSeconds": 60,
"InventoryPollIntervalSeconds": 1200,
"RetryPollIntervalSeconds": 60,
"Servers": [
{
"ServerURL": "https://mender.mydomain.com"
}
]
}
Am I missing something in my Mender config? Why would the client be able to register as a device using mender.mydomain but then try to send updates to localhost? Any help would be very much appreciated!
1 Like
Hi @JamesTann,
one thing is that mender-update connects via the mender-auth process. So the connection attempts via localhost are correct. Have you checked that the mender-auth service is up and running?
Plus, the config has some unnecessary entries. The HttpsClient, Security, Connectivity, UpdateControl* properties can all be removed, I think.
Greets,
Josef
1 Like
@TheYoctoJester
Hi Josef,
Thank you for the reply! That makes sense now that I’ve seen the authorization flow diagram in the docs… not sure how I missed that originally.
I ran the following commands as mentioned in the docs and they all seemed correct:
$ systemctl is-active mender-authd
active
$ systemctl is-enabled mender-authd
enabled
$ systemctl is-active mender-updated
active
$ systemctl is-enabled mender-updated
enabled
However, when I take a look at the logs for mender-authd, I see the following two lines repeated every minute:
severity=error name="http_forwarder" request="/api/devices/v2/deployments/device/deployments/next" msg="Error while forwarding response to client: Broken pipe: 127.0.0.1: POST /api/devices/v2/deployments/device/deployments/next: "
severity=warning name="http_client" url="https://mender.mydomain.com/api/devices/v2/deployments/device/deployments/next" msg="Client destroyed while request is still active!"
Any insights on what could be wrong here? I tried restarting the mender-authd service but these errors continue. Do I need to reset something server side with the client device?
1 Like
I also thought this might be some sort of connection issue to mender.mydomain.com since it is a newly registered domain and subdomain pointing to an ALB, but I was able to curl the endpoint in the logs and get an auth error (which makes sense, I didn’t send any credentials):
curl https://mender.mydomain.com/api/devices/v2/deployments/device/deployments/next
{"error":"no authorization header","request_id":"b6592c62-9452-499d-b932-697448d37f78"}
1 Like
Hi everybody, I’m having the same problem. Has this been resolved? I could really use some help!
Hi @JamesTann and @eschneeweiss,
Just to narrow the problem down - does it work if you use a trial account on Hosted Mender? So we know if we have to hunt server- or client side.
Greets,
Josef
Hi @TheYoctoJester and @eschneeweiss,
I tried uninstalling and reinstalling mender-client4 and went through the setup again, but am still seeing the same error.
I created a Hosted Mender account and logged in with GitHub. But when I try registering a client, it says my account does not exist. I have verified my email and password are correct by using them to sign into GitHub successfully. Does the new account take some time to register or something?
I will try and create a second Hosted Mender account with a different email address and see if that works.
Hi @JamesTann,
Ah we’re getting closer. The problem which you’re describing right now is related to the SSO provider. If you’re using GitHub, Google, or whatever provider, then the Mender server never sees your password - hence you cannot use it to authenticate a device. If you are using SSO, then you need to use the organization token as shown here: Installing | Mender documentation.
If your Hosted Mender account uses email and password, then you can use those as credentials to the interactive setup flow.
Greets,
Josef
Hi @TheYoctoJester,
Gotcha that makes sense! I found my token and ran setup, however I noticed the two mender services (mender-updated and mender-authd) are no longer present on my machine…
$ sudo systemctl start mender-authd
Failed to start mender-authd.service: Unit mender-authd.service not found.
This is likely because I removed them when trying to do a fresh reinstall but my understanding from the docs is that reinstalling the mender-client4 apt package should also install these services?
@JamesTann, yes, I’d expect the packages to bring those service files. But then I’ve never tried to remove and reinstall those (yet). Have you used the express installation flow, or the standard APT way?
Greets,
Josef
@TheYoctoJester I did the apt install.
I also just booted up a fresh VM to test, and I noticed that installing with apt automatically ran mender-setup but this did not happen on my original machine when I reinstalled. So maybe something was not cleaned up properly when I removed the package and services?
This is starting to branch, but I tried installing mender-client4 on a fresh Ubuntu Server 24 VM.
I followed the apt install instructions. Then I ran mender-setup with the Hosted instructions, using my organization token.
Finally, I started both Mender services, but mender-authd throws a new error:
Failed to trigger token fetching: Process returned non-zero exit status: While getting identity data: Process exited with status 1
I am convinced I am missing something or running the setup wrong, but I have no clue what it could be…
@TheYoctoJester Ok back on my original client server and was able to get the services running by simply running mender-update daemon and mender-auth daemon in two different terminals… this will at least suffice for testing.
I am now getting the same error on my original Ubuntu Desktop 22 client that I am getting on my new Ubuntu Server 24 VM:
severity=error name="Global" msg="Failed to trigger token fetching: Process returned non-zero exit status: While getting identity data: Process exited with status 1"
This is coming from the mender-auth component and happening in both clients with Hosted Mender and the apt install. This seems indicative of a client side issue, but it’s confusing to me that the errors are different between using Hosted Mender and my own install of Mender server.
Hi @JamesTann,
We’re getting closer, I think! The identity is a crucial piece of information about the device, as it is required to uniquely identify it. Hence, without it being available, both mender-auth and mender-update are non-functional and bailing out.
The identity is provided by a specific executable as documented here: Identity | Mender documentation. Can you check if the default one is failing on the specific OS/device combo that you’re running for some reason?
Greets,
Josef
Hi @TheYoctoJester,
Aha! I found the issue: the default identity provider is a broken symlink!
$ ls -la /usr/share/mender/identity/
total 8
drwxr-xr-x 2 root root 4096 Sep 3 15:02 .
drwxr-xr-x 5 root root 4096 Sep 3 15:02 ..
lrwxrwxrwx 1 root root 43 Aug 1 14:41 mender-device-identity -> /etc/mender/identity/mender-device-identity
$ ls -la /etc/mender/identity/
ls: cannot access '/etc/mender/identity/': No such file or directory
It appears something was not properly installed in the first place (or not reinstalled?)…
Do you know how I can fix this? I am beginning to wonder if going the apt install route is recommended since it seemed that uninstalling it did not remove the services and reinstalling it clearly did not install this identity provider…
Hi @JamesTann indeed, something is off here. The APT way is definitely the way to go. I have raised the topic internally, hopefully will have news soon.
Greets,
Josef
Hey, I tested this on a ubuntu 24.04 image, and the symlink isn’t broken there:
root@9ce792ed83ee:/# ls -la /usr/share/mender/identity/
total 8
drwxr-xr-x 2 root root 4096 Sep 12 11:10 .
drwxr-xr-x 5 root root 4096 Sep 12 11:10 ..
lrwxrwxrwx 1 root root 43 Aug 1 20:41 mender-device-identity -> /etc/mender/identity/mender-device-identity
root@9ce792ed83ee:/# ls -la /etc/mender/identity/
total 12
drwxr-xr-x 2 root root 4096 Sep 12 11:10 .
drwxr-xr-x 5 root root 4096 Sep 12 11:10 ..
-rwxr-xr-x 1 root root 1515 Aug 1 20:41 mender-device-identity
root@9ce792ed83ee:/# ls -la /etc/mender/identity/mender-device-identity
-rwxr-xr-x 1 root root 1515 Aug 1 20:41 /etc/mender/identity/mender-device-identity
Please note that mender-client4 is just a meta package to install mender-update, mender-auth and others. So by removing it or re-installing it nothing really happens:
root@6364c93f7126:/# apt-cache show mender-client4
Package: mender-client4
Version: 4.0.4-1+ubuntu+noble
Architecture: amd64
Maintainer: The Mender Team <mender@northern.tech>
Installed-Size: 23
Depends: mender-auth, mender-update, mender-snapshot, mender-setup, mender-flash
Conflicts: mender, mender-client
Replaces: mender-auth, mender-client, mender-setup, mender-snapshot, mender-update
Homepage: https://mender.io
Priority: optional
Section: admin
Filename: pool/main/m/mender-client4/mender-client4_4.0.4-1+ubuntu+noble_amd64.deb
Size: 5248
SHA256: 8a4b4921ccd3f091c7d2b8107bc8ed23dc7932952de4e02ff080c31c8e8ec9a7
SHA1: 48d95fe96015d78cdba436a5d82f400bedb15ffe
MD5sum: 28876126d6cc702206e4db99134a3b73
Description: Mender client
Mender is an open source over-the-air (OTA) software updater for embedded Linux devices.
From what I can figure out from your posts, a re-install of mender-auth should fix it, although I am still note sure how you end up in this situation.
Note that mender-updatealso has the same strategy wrt symlinks to /etc for its inventory scripts. So that one probably will be broken too in your system.
Hi @danieldrabitzius,
I just retried on a new Ubuntu 24.04 VM and was also able to install it successfully! Not sure what I was doing wrong before, but thank you so much for testing it as well.
I am now back to my original problem (although a lot more informed): mender-updated cannot send status updates to my Mender server.
I tried the same deployment with the same artifact using both my Mender server that I deployed to EKS and Hosted Mender and got the same results (this was @TheYoctoJester’s earlier suggestion to eliminate the prospect of it being a server-side issue). I am able to accept the device and the single-file-artifact I used does appear on the device successfully, but it never sends a status update back to the server to report the success (and thus is stuck on installing until it times out). There are no errors in the logs for mender-authd and the same message repeated every minute in the mender-updated logs (see below screenshots)
mender-authd logs:
mender-updated logs:
This is running in a brand new Ubuntu 24.02 VM using VirtualBox. Thanks so much for your help!
Hi @lluiscampos,
Thanks for the info! I am not sure how I ended up in this state either…
I tried reinstalling the mender-auth package with APT as you suggested but was unable to get the mender-device-identity file:
$ sudo apt remove mender-auth
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
libboost-log1.74.0 mender-flash mender-setup mender-snapshot
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
mender-auth
0 upgraded, 0 newly installed, 1 to remove and 114 not upgraded.
After this operation, 2,384 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 226581 files and directories currently installed.)
Removing mender-auth (4.0.4-1+ubuntu+jammy) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
$ sudo apt install mender-auth
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
mender-flash mender-snapshot
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
mender-auth
0 upgraded, 1 newly installed, 0 to remove and 114 not upgraded.
Need to get 0 B/570 kB of archives.
After this operation, 2,384 kB of additional disk space will be used.
Selecting previously unselected package mender-auth.
(Reading database ... 226572 files and directories currently installed.)
Preparing to unpack .../mender-auth_4.0.4-1+ubuntu+jammy_amd64.deb ...
Leaving 'diversion of /usr/share/mender/identity/mender-device-identity to /usr/
share/mender/identity/mender-device-identity.distrib by mender-auth'
Leaving 'diversion of /etc/mender/identity/mender-device-identity to /etc/mender
/identity/mender-device-identity.distrib by mender-auth'
Leaving 'diversion of /usr/share/dbus-1/system.d/io.mender.AuthenticationManager
.conf to /usr/share/dbus-1/system.d/io.mender.AuthenticationManager.conf.distrib
by mender-auth'
Unpacking mender-auth (4.0.4-1+ubuntu+jammy) ...
Setting up mender-auth (4.0.4-1+ubuntu+jammy) ...
mender-authd.service is a disabled or a static unit not running, not starting it
.
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
$ ls -la /etc/mender/identity/
total 8
drwxr-xr-x 2 root root 4096 Sep 12 09:50 .
drwxr-xr-x 3 root root 4096 Sep 12 09:47 ..
Is there something I am missing here? Do I need to run a post install command?
I also noticed this in the output of the install command but not sure what to make of it:
Leaving 'diversion of /usr/share/mender/identity/mender-device-identity to /usr/
share/mender/identity/mender-device-identity.distrib by mender-auth'
Leaving 'diversion of /etc/mender/identity/mender-device-identity to /etc/mender
/identity/mender-device-identity.distrib by mender-auth'
