Missing scope identity in inventory

Hi,

We are using a self-hosted mender server 3.4, and call “Set Authentication Status” API /devices/{id}/auth/{aid}/status to accept devices.
According to documentation (https://docs.mender.io/3.4/overview/inventory after accept, the device inventory should contain an identity scope attribute with the status. However we never get this attribute while retrieving the inventory.

Currently, there is one "reserved " attribute in the identity scope: status . It stores the status (accepted , rejected , etc.) of a device and changes whether the device status changes. Therefore, you cannot modify it directly.

Are we using the wrong API to accept device?
Is this added only when using the Preauthorize API?

The missing attribute cause group deployments to fail with “no devices found”.

Thanks,
Neta

Hi,

Any thought will be welcomed.
When the “reserved” identity attribute will be missing?

Thanks,
Neta

Hi Neta,

the attribute is not called reserved, but status, and I have just checked that it exists and gets returned when accessing the inventory API. In the web UI, it is used to filter the devices according to their state so I think it is definitely there, and displayed verbatim in the device identity tab. Can you check if it is correctly displayed there?

Greetz,
Josef

Hi Josef,

Thank you for the reply.
As I wrote we are using a self-hosted mender, and more specifically we do not used the Web UI but call directly the server API with a script that we implemented.
Below is an example for getting an inventory for one of the devices. You can see that the status attribute is not present, or any other identity type attribute.

root@gc-management:/var/lib/guardicore# curl -s -X GET     -H "Accept: application/json"     -H "Authorization: Bearer $JWT"     $MENDER_SERVER_URI/api/management/v1/inventory/devices/38cc5130-f97d-42a4-a3e0-87fe6b454af7 | python -m json.tool
{
    "id": "38cc5130-f97d-42a4-a3e0-87fe6b454af7",
    "attributes": [
        {
            "name": "artifact_name",
            "value": "test-full.artifact",
            "scope": "inventory"
        },
        {
            "name": "cluster_id",
            "value": "default",
            "scope": "inventory"
        },
        {
            "name": "component_mode",
            "value": "Aggregator",
            "scope": "inventory"
        },
        {
            "name": "cpu_model",
            "value": "Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz",
            "scope": "inventory"
        },
        {
            "name": "device_type",
            "value": "x86_64",
            "scope": "inventory"
        },
        {
            "name": "geo-city",
            "value": "Tel Aviv",
            "scope": "inventory"
        },
        {
            "name": "geo-country",
            "value": "IL",
            "scope": "inventory"
        },
        {
            "name": "geo-ip",
            "value": "31.168.233.124",
            "scope": "inventory"
        },
        {
            "name": "geo-timezone",
            "value": "Asia/Jerusalem",
            "scope": "inventory"
        },
        {
            "name": "hostname",
            "value": "gc-aggregator-172-16-100-50",
            "scope": "inventory"
        },
        {
            "name": "ipv4_eth0",
            "value": "172.16.100.50/16",
            "scope": "inventory"
        },
        {
            "name": "ipv4_eth1",
            "value": "100.100.100.50/16",
            "scope": "inventory"
        },
        {
            "name": "ipv4_eth2",
            "value": "200.200.200.50/16",
            "scope": "inventory"
        },
        {
            "name": "ipv6_eth0",
            "value": "fe80::250:56ff:fe9d:2476/64",
            "scope": "inventory"
        },
        {
            "name": "ipv6_eth1",
            "value": [
                "2001:1860:1860:0:250:56ff:fe9d:c4ff/64",
                "fe80::250:56ff:fe9d:c4ff/64"
            ],
            "scope": "inventory"
        },
        {
            "name": "ipv6_eth2",
            "value": [
                "2001:2860:2860:0:250:56ff:fe9d:8006/64",
                "fe80::250:56ff:fe9d:8006/64"
            ],
            "scope": "inventory"
        },
        {
            "name": "kernel",
            "value": "Linux version 4.15.0-192-generic (buildd@lcy02-amd64-029) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #203-Ubuntu SMP Wed Aug 10 17:40:03 UTC 2022",
            "scope": "inventory"
        },
        {
            "name": "mac_eth0",
            "value": "00:50:56:9d:24:76",
            "scope": "inventory"
        },
        {
            "name": "mac_eth1",
            "value": "00:50:56:9d:c4:ff",
            "scope": "inventory"
        },
        {
            "name": "mac_eth2",
            "value": "00:50:56:9d:80:06",
            "scope": "inventory"
        },
        {
            "name": "mem_total_kB",
            "value": "2028220",
            "scope": "inventory"
        },
        {
            "name": "mender_bootloader_integration",
            "value": "bios_grub",
            "scope": "inventory"
        },
        {
            "name": "mender_client_version",
            "value": "3.3.0",
            "scope": "inventory"
        },
        {
            "name": "network_interfaces",
            "value": [
                "eth0",
                "eth1",
                "eth2"
            ],
            "scope": "inventory"
        },
        {
            "name": "os",
            "value": "Ubuntu 18.04.6 LTS",
            "scope": "inventory"
        },
        {
            "name": "rootfs_type",
            "value": "ext4",
            "scope": "inventory"
        },
        {
            "name": "update_modules",
            "value": [
                "deb",
                "directory",
                "docker",
                "rpm",
                "script",
                "single-file"
            ],
            "scope": "inventory"
        },
        {
            "name": "created_ts",
            "value": "2023-03-14T12:48:46.576Z",
            "scope": "system"
        },
        {
            "name": "updated_ts",
            "value": "2023-03-21T11:17:51.709Z",
            "scope": "system"
        },
        {
            "name": "group",
            "value": "neta",
            "scope": "system"
        },
        {
            "name": "rootfs-image.checksum",
            "value": "235704d4ee82b1a91b3a20ad466e00bad8ff167c3803c018f2a7d11c046f7443",
            "scope": "inventory"
        },
        {
            "name": "rootfs-image.version",
            "value": "1.0",
            "scope": "inventory"
        }
    ],
    "updated_ts": "2023-03-21T11:17:51.709Z"
}

From docker logs:

time="2023-03-21T13:25:01Z" level=info msg="200 5694μs GET /api/0.1.0/devices/38cc5130-f97d-42a4-a3e0-87fe6b454af7 HTTP/1.1 - curl/7.58.0" byteswritten=3226 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/0.1.0/devices/38cc5130-f97d-42a4-a3e0-87fe6b454af7 qs= request_id=d5609ab5-dce3-4325-a036-0034bb5c1c50 responsetime=0.005694285 status=200 ts="2023-03-21 13:25:01.804766419 +0000 UTC" type=http user_id=f5c10cd2-df96-4bf2-9bc5-c3edac357074

There are no identity attributes:

root@gc-management:/var/lib/guardicore# curl -s -X GET     -H "Accept: application/json"     -H "Authorization: Bearer $JWT"     $MENDER_SERVER_URI/api/management/v1/inventory/devices/38cc5130-f97d-42a4-a3e0-87fe6b454af7 | python -m json.tool | grep scope | uniq
            "scope": "inventory"
            "scope": "system"
            "scope": "inventory"

Any idea why the attribute is not shown?
Can I query directly the inventory database to search this attribute?

Thanks,
Neta

Hi Josef,

I have a Mender UI on a different environment, and even there you can see (in the attached file) that the status attribute does not exist. We are using mender server v3.4.

Thanks,
Neta

More debug information:
I see for example the accept request on the deviceauth container:

time="2023-03-22T13:28:32Z" level=info msg="204 532989μs PUT /api/management/v2/devauth/devices/9c95a5d5-669d-4c0a-a1c1-cc2a4749b63d/auth/01f94a94-48f8-407e-bb7e-d5972af4ee2b/status HTTP/1.1 - python-requests/2.26.0" byteswritten=0 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=PUT path=/api/management/v2/devauth/devices/9c95a5d5-669d-4c0a-a1c1-cc2a4749b63d/auth/01f94a94-48f8-407e-bb7e-d5972af4ee2b/status qs= request_id=52485654-10d3-442c-ac5c-76a3f3120e9d responsetime=0.53298994 status=204 ts="2023-03-22 13:28:32.162015623 +0000 UTC" type=http user_id=68a54740-2f7b-4acb-8eba-0d08573d2324

But there is no logs on the inventory container from around these times:

docker logs --since "2023-03-22T13:20:00" eda55b13456c ---> The inventory container

Above command returns nothing, no logs from that time.

Is there other mender service involved in the process?

Hi @netsolom,

Two things I could find/spot:

Greetz,
Josef

Thanks @TheYoctoJester . I will check this.

I have now noticed that the worflow-worker is trying to submit the attribute to inventory but fails.
I am not sure why, but it is trying to POST the request to port 8080, while the inventory listen on port 18084

root@gc-management:~# docker logs --since "2023-03-22T13:20:00" a8995474f7d7
time="2023-03-22T13:28:32Z" level=info msg="Worker: processing job 641b0280f2dfb272e6b7e42c workflow update_device_status" file=worker.go func=worker.workerMain line=169 worker_id=25
time="2023-03-22T13:28:32Z" level=info msg="641b0280f2dfb272e6b7e42c: started, update_device_status" file=process.go func=worker.processJob line=61 worker_id=25
time="2023-03-22T13:28:32Z" level=info msg="641b0280f2dfb272e6b7e42c: started, update_device_status task :update_device_identity_inventory" file=process.go func=worker.processJob line=65 worker_id=25
time="2023-03-22T13:28:32Z" level=info msg="processHTTPTask: starting with: method=POST uri=http://mender-inventory:8080/api/internal/v1/inventory/tenants//devices/status/accepted" file=http.go func=worker.processHTTPTask line=50 worker_id=25
time="2023-03-22T13:28:32Z" level=error msg="error: Post \"http://mender-inventory:8080/api/internal/v1/inventory/tenants//devices/status/accepted\": dial tcp: lookup mender-inventory on 127.0.0.53:53: server misbehaving" file=worker.go func=worker.workerMain line=172 worker_id=25
time="2023-03-22T13:28:32Z" level=info msg="Worker: processing job 641b0280f2dfb272e6b7e42d workflow provision_device" file=worker.go func=worker.workerMain line=169 worker_id=26
time="2023-03-22T13:28:32Z" level=info msg="641b0280f2dfb272e6b7e42d: started, provision_device" file=process.go func=worker.processJob line=61 worker_id=26
time="2023-03-22T13:28:32Z" level=info msg="641b0280f2dfb272e6b7e42d: started, provision_device task :create_device_inventory" file=process.go func=worker.processJob line=65 worker_id=26
time="2023-03-22T13:28:32Z" level=info msg="processHTTPTask: starting with: method=POST uri=http://mender-inventory:8080/api/internal/v1/inventory/tenants//devices" file=http.go func=worker.processHTTPTask line=50 worker_id=26
time="2023-03-22T13:28:32Z" level=error msg="error: Post \"http://mender-inventory:8080/api/internal/v1/inventory/tenants//devices\": dial tcp: lookup mender-inventory on 127.0.0.53:53: server misbehaving" file=worker.go func=worker.workerMain line=172 worker_id=26
netstat -tupln | grep inventory
tcp6       0      0 :::18084                :::*                    LISTEN      41315/inventory

It could be that we did not configure workflow correctly.
Can you point me to where workflow is configured with the inventory listen port?

Many thanks,
Neta

I think the mystery is solved.
I found that the workflows-worker did not receive the correct address of the mender-inventory. We did not pass it an updated INVENTORY_ADDR environment variable.
Now that the container does have the correct address, I see that upon accept the status is POSTed into inventory, and we see it when we display the device inventory.

[
    {
        "id": "a7117a6e-024a-4f20-ab6f-557e23a4a589",
        "attributes": [
            {
                "name": "status",
                "value": "accepted",
                "scope": "identity" ---------> finally it is here
            },
            {
                "name": "created_ts",
                "value": "2023-03-23T16:48:53.44Z",
                "scope": "system"
            },
            {
                "name": "updated_ts",
                "value": "2023-03-23T16:48:53.559Z",
                "scope": "system"
            }
        ],
        "updated_ts": "2023-03-23T16:48:53.559Z"
    }
]

I just wonder, why the device does not update the server with its inventory attributes now?
I guess it will happen in 8 hours since the conf file has "InventoryPollIntervalSeconds": 28800, but I would expect that first update would be immediately after status has changed.

@TheYoctoJester do you happen to know why the device does not try to update inventory immediately after being accepted by the server?
It will try after an upgrade or when reaching InventoryPollIntervalSeconds.

Hi @netsolom,

As the device has no immediate way of being accepted until the next polling cycle, it is not able to update “immediately”. This behaviour is therefore intended.

See also your identical post here

Greetz,
Josef