Recovering/Reauthorizing Decommissioned Device on Hosted Mender

General Discussions
,
1

Is there a way to see a list of decommissioned devices within the hosted.mender.io interface? Or a setting that would allow a device that was accidentally decommissioned to appear again? I’ve restarted the device a few times with no luck.

ps157:~$ sudo journalctl -f -u mender
-- Logs begin at Sat 2000-01-01 00:00:03 UTC. --
Oct 20 03:40:08 ps157 mender[207]: time="2020-10-20T03:40:08Z" level=info msg="State transition: authorize [Sync] -> authorize-wait [Idle]" module=mender
Oct 20 03:45:07 ps157 mender[207]: time="2020-10-20T03:45:07Z" level=info msg="State transition: authorize-wait [Idle] -> authorize [Sync]" module=mender
Oct 20 03:45:08 ps157 mender[207]: time="2020-10-20T03:45:08Z" level=error msg="authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state
Oct 20 03:45:08 ps157 mender[207]: time="2020-10-20T03:45:08Z" level=info msg="State transition: authorize [Sync] -> authorize-wait [Idle]" module=mender
Oct 20 03:50:07 ps157 mender[207]: time="2020-10-20T03:50:07Z" level=info msg="State transition: authorize-wait [Idle] -> authorize [Sync]" module=mender
Oct 20 03:50:09 ps157 mender[207]: time="2020-10-20T03:50:09Z" level=error msg="authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state
Oct 20 03:50:09 ps157 mender[207]: time="2020-10-20T03:50:09Z" level=info msg="State transition: authorize [Sync] -> authorize-wait [Idle]" module=mender
Oct 20 03:55:07 ps157 mender[207]: time="2020-10-20T03:55:07Z" level=info msg="State transition: authorize-wait [Idle] -> authorize [Sync]" module=mender
Oct 20 03:55:08 ps157 mender[207]: time="2020-10-20T03:55:08Z" level=error msg="authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state
Oct 20 03:55:08 ps157 mender[207]: time="2020-10-20T03:55:08Z" level=info msg="State transition: authorize [Sync] -> authorize-wait [Idle]" module=mender
2

Do not think so. The idea of decommissioning is to remove any trace of that device (or actually an authorization set).

Or a setting that would allow a device that was accidentally decommissioned to appear again?

A device which has been decommissioned should always come back in “Pending” state, should it try to connect again.

So the message you are seeing is correct,

Oct 20 03:45:08 ps157 mender[207]: time="2020-10-20T03:45:08Z" level=error msg="authorize failed: transient error: authorization request failed: (request_id: ): authentication request rejected server error message: dev auth: unauthorized" module=state

This should mean that you have an “Pending” authorization set on the server. It can be in two places, either under the “Pending” tab, or it could actually also be under “Devices” tab, as a new authorization set for an existing device. This is typically indicated by a small warning icon on the device.

3

Got it. That makes sense @mirzak

Unfortunately it’s not showing up in the pending tab and when I try to filter, there’s no devices that match the BBB’s MAC address with a new auth set.

Does it take a while to re-appear in Pending? Anything else that could be happening?

4

As soon as a device checks in and is not authorized (which is indicated by the client log), it should end up in “Pending” state.

Anything else that could be happening?

Maybe check that it is not in rejected state? This could prevent it from ending up in Pending.

https://hosted.mender.io/ui/#/devices/rejected

5

I think it’s back up! But a weird thing happened where there was a device with a similar (or the same HWAddress) but a different hostname that was receiving the new auth set requests. Is it possible for two BBB’s to have the same Mac Address?

6

Is it possible for two BBB’s to have the same Mac Address?

No, this is not allowed if the MAC is used as device identity.

If two devices have the same device identifier, the Mender server would just see them as the same device but with different authorization sets (different keys). Which matches I believe what you saw.

7

We had this happen as well a few weeks ago on our self hosted Mender server. A device was decommissioned and did not show up in the Pending list, even after a couple of reboots of the device and a restart of the Server (just to make sure that there wasn’t a blocking issue). We then decommissioned another device and that also didn’t show up in the Pending list.

After we switched on an Accepted device and that was detected by the Server, all of a sudden both decommissioned devices showed up in the Pending list.

Not sure if there is an issue in the Server where it will not see decommissioned devices some of the time (in the past, before server 2.4 we never saw this issue) or it was a fluke. We tested network connectivity on both decom devices when they were not recognized, but that worked fine. They could even see and ping the Server IP.

8

I’ve had similar issues, and had to run the following to get it working for me again after decommissioning.

docker-compose exec mender-device-auth /usr/bin/deviceauth maintenance --decommissioning-cleanup

or

docker exec menderproduction_mender-device-auth_1 /usr/bin/deviceauth maintenance --decommissioning-cleanup

2 Likes