Why is my device polling localhost?

General Discussions
,
1

Hello,

I am new to Mender and trying to create my first test deployment using a single-file artifact and a Ubuntu server test device.

I have taken these steps successfully:

  • Deployed Mender to EKS and ensured it is accessible from my Ubuntu server
  • Created and uploaded my single-file artifact to Mender
  • Registered my Ubuntu server as a device with Mender by installing mender-client4 using apt and entering my Mender server info

I started the deployment of my single-file artifact through the UI to my Ubuntu device but it failed, even though the file was indeed deployed to my Ubuntu device.

I sshed into the Ubuntu device and ran: sudo journalctl -u mender-updated | tail

This is what I saw (I have removed IDs and timestamps for brevity):

mender-update: severity=info name="Global" msg="Running Mender client 4.0.4"
mender-update: severity=info name="Global" msg="Deployment with ID 8d3f0784-93ad-41ec-ab54-756ba0a7d69f started."
mender-update: severity=info name="Global" msg="Sending status update to server"
mender-update: severity=error name="Global" msg="Request to push status data failed: PUT http://127.0.0.1:36069/api/devices/v1/deployments/device/deployments/8d3f0784-93ad-41ec-ab54-756ba0a7d69f/status: "
mender-update: severity=error name="Global" msg="Could not send deployment status: bad version: PUT http://127.0.0.1:36069/api/devices/v1/deployments/device/deployments/8d3f0784-93ad-41ec-ab54-756ba0a7d69f/status: "
mender-update: severity=info name="Global" msg="Installing artifact..."
mender-update: severity=info name="Global" msg="Sending status update to server"
mender-update: severity=error name="Global" msg="Request to push status data failed: PUT http://127.0.0.1:36069/api/devices/v1/deployments/device/deployments/8d3f0784-93ad-41ec-ab54-756ba0a7d69f/status: "
mender-update: severity=info name="Global" msg="Retrying status update after 60 seconds"

It appears that the install succeeded, but the Mender client was unable to update the Mender server because it is trying to access it locally. This was very confusing to me. Why is the device trying to connect to localhost when it was registered with my Mender server at a different domain?

This is what my mender.conf looks like:

$ cat /etc/mender/mender.conf 
{
    "HttpsClient": {},
    "Security": {},
    "Connectivity": {},
    "DeviceTypeFile": "/var/lib/mender/device_type",
    "UpdateControlMapExpirationTimeSeconds": 90,
    "UpdateControlMapBootExpirationTimeSeconds": 45,
    "UpdatePollIntervalSeconds": 60,
    "InventoryPollIntervalSeconds": 1200,
    "RetryPollIntervalSeconds": 60,
    "Servers": [
        {
            "ServerURL": "https://mender.mydomain.com"
        }
    ]
}

Am I missing something in my Mender config? Why would the client be able to register as a device using mender.mydomain but then try to send updates to localhost? Any help would be very much appreciated!

2

Hi @JamesTann,

one thing is that mender-update connects via the mender-auth process. So the connection attempts via localhost are correct. Have you checked that the mender-auth service is up and running?

Plus, the config has some unnecessary entries. The HttpsClient, Security, Connectivity, UpdateControl* properties can all be removed, I think.

Greets,
Josef

3

@TheYoctoJester

Hi Josef,

Thank you for the reply! That makes sense now that I’ve seen the authorization flow diagram in the docs… not sure how I missed that originally.

I ran the following commands as mentioned in the docs and they all seemed correct:

$ systemctl is-active mender-authd
active
$ systemctl is-enabled mender-authd
enabled
$ systemctl is-active mender-updated
active
$ systemctl is-enabled mender-updated
enabled

However, when I take a look at the logs for mender-authd, I see the following two lines repeated every minute:

severity=error name="http_forwarder" request="/api/devices/v2/deployments/device/deployments/next" msg="Error while forwarding response to client: Broken pipe: 127.0.0.1: POST /api/devices/v2/deployments/device/deployments/next: "
severity=warning name="http_client" url="https://mender.mydomain.com/api/devices/v2/deployments/device/deployments/next" msg="Client destroyed while request is still active!"

Any insights on what could be wrong here? I tried restarting the mender-authd service but these errors continue. Do I need to reset something server side with the client device?

4

I also thought this might be some sort of connection issue to mender.mydomain.com since it is a newly registered domain and subdomain pointing to an ALB, but I was able to curl the endpoint in the logs and get an auth error (which makes sense, I didn’t send any credentials):

curl https://mender.mydomain.com/api/devices/v2/deployments/device/deployments/next
{"error":"no authorization header","request_id":"b6592c62-9452-499d-b932-697448d37f78"}