Why is my device polling localhost?

General Discussions
,
21

Hi @JamesTann

re 404 errors on the status updates,
This must be a server issue. Basically the client is getting a “API endpoint not found”. I have never seen this before. I recommend you stick to Hosted Mender until you figure out what the issues on the client side are,

re mender-auth reinstall,
Right, the issue is that the identity script (same as inventory scripts) are interpret as configuration files by dpkg. We do this so that users can adapt, modify, remove, etc script and still upgrade. To do a reinstall, including conf files, you need to do something like apt-get purge mener-auth && apt-get install mener-auth

1 Like
22

Hi @lluiscampos,

That worked! I have now restored all the files I need on my physical Ubuntu 22 Server. After that I was able to successfully register with hosted.mender.io and deploy an artifact! This is the farthest I have made it so far.

I am now back to where I started with my device, and trying to deploy the update with my own Mender cluster on EKS.

To switch between these two Mender servers I:

  1. run sudo mender-setup and go through the interactive setup and type in my domain: mender.mydomain.com
  2. run sudo systemctl restart mender-authd and sudo systemctl restart mender-updated

Please let me know if this is the right way to switch my device between using Hosted Mender and my own.

I started digging into the Kubernetes logs for both the mender-deployments pod and the mender-api-gateway pod (are there any others I should dig through?).

mender-deployments:

level=info msg="404 82μs POST /api/devices/v2/deployments/device/deployments/next HTTP/1.1 - -" byteswritten=30 caller="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1@middleware.go:82" device_id=bc7903f6-a1a5-4d0e-aee3-f7206fab4e19 method=POST path=/api/devices/v2/deployments/device/deployments/next plan=enterprise qs= request_id=580a9bcb-ca67-4ccc-bddb-6b451fc69111 responsetime=8.2819e-05 status=404 type=http
level=info msg="204 1449μs GET /api/devices/v1/deployments/device/deployments/next?artifact_name=REDACTED&device_type=REDACTED HTTP/1.1 - -" byteswritten=0 caller="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1@middleware.go:82" device_id=bc7903f6-a1a5-4d0e-aee3-f7206fab4e19 method=GET path=/api/devices/v1/deployments/device/deployments/next plan=enterprise qs="artifact_name=REDACTED&device_type=REDACTED" request_id=cdb67bca-4882-402c-8ca4-df22729ad94f responsetime=0.001449024 status=204 type=http

mender-api-gateway:

{"ClientAddr":"10.0.25.204:2430","ClientHost":"10.0.25.204","ClientPort":"2430","ClientUsername":"-","DownstreamContentSize":54,"DownstreamStatus":404,"Duration":80030997,"OriginContentSize":54,"OriginDuration":316254,"OriginStatus":404,"Overhead":79714743,"RequestAddr":"mender.mydomain.com","RequestContentSize":123,"RequestCount":142880,"RequestHost":"mender.mydomain.com","RequestMethod":"POST","RequestPath":"/api/devices/v2/deployments/device/deployments/next","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deployments@file","ServiceAddr":"mender-gui:8080","ServiceName":"error-responder@file","ServiceURL":"http://mender-gui:8080","entryPointName":"http","level":"info","msg":""}
{"ClientAddr":"10.0.10.53:47916","ClientHost":"10.0.10.53","ClientPort":"47916","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":204,"Duration":8890821,"OriginContentSize":0,"OriginDuration":1858589,"OriginStatus":204,"Overhead":7032232,"RequestAddr":"mender.mydomain.com","RequestContentSize":0,"RequestCount":142881,"RequestHost":"mender.mydomain.com","RequestMethod":"GET","RequestPath":"/api/devices/v1/deployments/device/deployments/next?artifact_name=test2-file\u0026device_type=ubuntu-server","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deployments@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","entryPointName":"http","level":"info","msg":""}

These messages appear without a deployment even running. My ingress only has one rule pointing to the Mender API gateway. How can I further debug this on the Mender server side?

I greatly appreciate all the help!

23

@TheYoctoJester
I am seeing similar issue while using Mender Client v5.0.1 with the community edition mender-server v4.0.1 installed on our EKS.

{"timestamp":"2025-06-03T18:40:20.784635Z","level":"error","message":"Request to push status data failed: PUT http://127.0.0.1:33467/api/devices/v1/deployments/device/deployments/745b58c7-fa74-4968-b30f-e89cad631a7a/status: "}
{"timestamp":"2025-06-03T18:40:20.784929Z","level":"error","message":"Could not send deployment status: bad version: PUT http://127.0.0.1:33467/api/devices/v1/deployments/device/deployments/745b58c7-fa74-4968-b30f-e89cad631a7a/status: "}

Files are getting installed correctly, but since the deployment status update to success is failing all further API calls from mender-update are failing too until the daemon is restarted.

The same client works fine when connected to hosted.mender.io.

On the server side everything appears to be fine and seems to be an issue on the client side. But couldn’t find from the mender client source code what is causing this ‘bad version’ error and which version it is referring to.

Mender client v3(go client) works fine with our server instance, it doesn’t have any problem in updating the deployment status.