405 Method Not Allowed returned by Mender server

General Discussions
1

I am using mender-convert to create the image of Ubuntu 20 for Raspberry Pi 3. The device can successfully authenticate to the server. However, after the first checkin, mender server keeps returning 405 Method Not Allowed when calling API “PUT /api/devices/v1/inventory/device/attributes” to update inventory. The followings are the error logs captured in both mender client and mender server:

Mender Client Logs:

Nov 17 02:37:37 ubuntu mender[1148]: time="2020-11-17T02:37:37Z" level=info msg="State transition:     inventory-update [Sync] -> check-wait [Idle]"
Nov 17 02:37:38 ubuntu mender[1148]: time="2020-11-17T02:37:38Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Nov 17 02:37:38 ubuntu mender[1148]: time="2020-11-17T02:37:38Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
Nov 17 02:37:40 ubuntu mender[1148]: time="2020-11-17T02:37:40Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
Nov 17 02:37:42 ubuntu mender[1148]: time="2020-11-17T02:37:42Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 3"
Nov 17 02:37:42 ubuntu mender[1148]: time="2020-11-17T02:37:42Z" level=warning msg="Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 3"
Nov 17 02:37:42 ubuntu mender[1148]: time="2020-11-17T02:37:42Z" level=error msg="Got unexpected HTTP status when submitting to inventory: 405"
Nov 17 02:37:42 ubuntu mender[1148]: time="2020-11-17T02:37:42Z" level=error msg="Got unexpected HTTP status when submitting to inventory: 405"
Nov 17 02:37:42 ubuntu mender[1148]: time="2020-11-17T02:37:42Z" level=warning msg="Failed to refresh inventory: failed to submit inventory data: (request_id: ): inventory submit failed, bad status 405 server error message: Method not allowed"
Nov 17 02:37:42 ubuntu mender[1148]: time="2020-11-17T02:37:42Z" level=warning msg="Failed to refresh inventory: failed to submit inventory data: (request_id: ): inventory submit failed, bad status 405 server error message: Method not allowed"

Mender Server - mender inventory logs:

time="2020-11-17T10:00:17Z" level=info msg="405 251μs PUT /api/0.1.0/attributes HTTP/1.1 - Go-http-client/1.1" byteswritten=59 device_id=99e6043e-d6d6-4624-9c65-9e096ccb6e61 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=PUT path=/api/0.1.0/attributes plan=enterprise qs= request_id=40c129eb-3fb2-4bf1-93f9-498ca5d373e9 responsetime=0.000251923 status=405 ts="2020-11-17 10:00:17.856635044 +0000 UTC" type=http

Version of Mender and Ubuntu:
Mender Client (Output of command mender --version):
483c605 runtime: go1.11.5

Mender Server (Clone from repo git clone -b 2.4.0 https://github.com/mendersoftware/integration.git):
2.4.0

Ubuntu:
Ubuntu 20.04.1 LTS

2

There are some known issues with this inventory script (/usr/share/mender/inventory/mender-inventory-geo) and I would recommend to remove it.

Some addition details, https://tracker.mender.io/browse/MEN-4085

3

@mirzak Thank you for your reply. Is there any workaround for this issue? Since the updating inventory fails, deployment also does not work.

P.S.: I supplemented the version of Mender and Ubuntu in Raspberry Pi above.

4

I am also facing the issue of update inventory data fails on mender server UI inventory after removing script mender-inventory-geo ( /usr/share/mender/inventory/mender-inventory-geo ) and getting error (journalctl -f -u mender-client.service) in below screenshot.

Any suggestion or workaround to resolve this issue of “waiting for inventory data from the device”.

image
image1295×654 178 KB

image
image722×445 20 KB

5

We also face exactly the same behaviour with the same journalctl log messages !

Remote updates are successful, typing “sudo mender -show-artifact” on the device’s console correctly outputs the new artifact name.

But on the Mender Demo Server Website (https://IPv4addr/ui/#/devices) in the column “Current software” there is a minus shown instead of the corresponding artifact name.
Also in the Device inventory “Waiting for inventory data” is shown …

Whenever we repeat the above procedure the behaviour is the same: update successful, correct artifact name shown on the device but it doesn’t find its way to the Mender Server Website GUI.

We need urgent advice how to get inventory updated immediately after updates/reboot so that above information is shown correctly …

6

You seem to be running the Mender client version from master branch? Would not recommend this.

Can you please test with a tagged version instead.

7

We also get thew same ouput on “mender --version”.
But: We are using mender-convert to generate full images and mender-artifacts…
“483c605 runtime: go1.11.5” is exactly the version which mender-convert seems to install …

8

@mirzak Can we really just remove this script (/usr/share/mender/inventory/mender-inventory-geo) ? What is it used for ?
Ah, sorry: Now i saw in https://tracker.mender.io/browse/MEN-4085
that this is considered bug: “I moved the bug to Mender project, as by now we have proved there is a product issue.”

9

An interesting fact may be, that everything worked fine a few days ago.
We didn’t change anything, neither in the mender-toolchain nor on the Mender Demo Server.
We only did an apt update + full-upgrade on our Golden Device.
After that, we supplied the Golden Image to mender-convert and updated the device with the full-image and/or the mender-artifact for remote updates.
Since then, the above behaviour occured. Maybe there is an issue with dependencies of the script which changed due to update/upgrade ? Especially because this thread is very new …

10

Which revision of mender-convert? Because there have been some changes there recently which versions of the Mender client is installed.

Anyway make sure that you have the following set in your mender-convert configs,

MENDER_CLIENT_VERSION = "2.4.0"

This is the change I am referring to, https://github.com/mendersoftware/mender-convert/commit/310bcfff750add2473b31b206daea65f851deb75 and there might actually be an issue here as “latest” should be the latest released package, but it seems it is pulling a deb package built from master based on what you shared.

11

@ mirzak You saved the day:

Originally we had "MENDER_CLIENT_VERSION=“master” in our configs/mender_convert_config file.
With that setting the issue occured recently.

Then I changed it to "MENDER_CLIENT_VERSION=“latest” where, as you mentioned, an error in mender-convert occures with wget.

Now I changed it to "MENDER_CLIENT_VERSION=“2.4.0” acc. your proposal and the infos are shown again on the Mender Demo Server GUI :slight_smile:

Nevertheless there is a msg: level=warning msg=“Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 3” but i can regard this as a minor issue.

12

After changing the version of mender client to 2.4.0, the error log of 405 error disappears. Although the warning about mender-inventory-geo still exists, the client can send inventory info to server now. I changed the line in the following file:

13

“latest” would also be my favorite, but when using it (instead of “2.4.0”)
I get the following error when running mender-convert:

2020-11-19 07:37:43 [INFO] [mender-convert-modify] Installing Mender client and related files
mender-convert-modify has finished. Cleaning up…
2020-11-19 07:37:44 [ERROR] [mender-convert] mender-convert failed
2020-11-19 07:37:43 [DEBUG] [mender-convert-modify] When running: (./mender-convert-modify:99): run_and_log_cmd():

wget -Nq https://d1b0l86ne08fsf.cloudfront.net/latest/dist-packages/debian/armhf/mender-client_latest-1_armhf.deb -P work/mender-deb

2020-11-19 07:37:44 [ERROR] [mender-convert] mender-convert failed
2020-11-19 07:37:44 [ERROR] [mender-convert] mender-convert exit code: 8

14

Currently, I am using the mender_convert version 2.0.x having MENDER_CLIENT_VERSION = master GitHub - mendersoftware/mender-convert at 2.0.x

Can I use the same mender_convert version i.e. 2.0.x along with change in mender-convert configs as MENDER_CLIENT_VERSION = “2.4.0”. only

OR must I use the latest mender_convert version 2.2.x… GitHub - mendersoftware/mender-convert at 2.2.x

Please suggest.

15

Can I use the same mender_convert version i.e. 2.0.x along with change in mender-convert configs as MENDER_CLIENT_VERSION = “2.4.0”. only

That should work, and note that “latest” is not supported on 2.0.x. Only on “master” and “2.2.x” branches.

16

@mirzak,

Thank mirzak. After changing the version of mender client to 2.4.0, I have abe to successful OTA deployment.

17

@mirzak @borispenguin

I have hosted a mender server (Version: integration-2.1.0) on our EC2 instance with the region “US East (N. Virginia)us-east-1”.

OS & Architecture: Ubuntu18.04 (X86)
MENDER_CLIENT_VERSION=“2.4.0”

Issue: long time mender artifacts deployment in different network

I have observed that the Mender OTA deployment (on our device) time in my office (India) is taking around ~1hrs but the image of the same mender artifact (with same mender server UI) is taking time around (~4hrs) 4 times in my other office location device (US) network. There is no error observe in journalctl logs on the device.

I have verified internet speed as well ~87 mbps on both locations/networks.

Could anyone please assist/suggestion me that why is it taking a long time (4 times) instead of ~1 hrs here.??

18

It’s hard to say. Do you have the same polling intervals in each of the devices? Can you tell from the logs where the delay is introduced? By that I mean is it a long delay before the device starts the download or is it once it starts downloading? Do you see any retries reported in the logs?

Is it possible that you have some kind of throttling implemented in the server or the local infrastructure of your US based devices?