Hi
We’ve encountered a problem with the connectivity of one of our devices to the Mender Bitcortex server. One of our devices had been reported offline for three months.
According to our investigation, the MAC address of the ethernet interface had been changed, after a reboot. It can be considered a sufficient condition, however, according to our mender configuration, the inventory data must be sent every minute, so the necessary condition isn’t met. In other words, this situation must be handled by Mender, automatically. The following is part of our mender configuration file. It should be noted that this configuration is share among all of our devices.
...
"InventoryPollIntervalSeconds": 60,
...
Our tests for a test device show that changing the MAC address is reported successfully to the server, and the test device is shown online in the Mender Bitcortex server, while the device has been reported disconnected for 3 months. The following are error messages that we’ve found in the journaling logs:
# journalctl -u mender-client.service
...
level=error msg="Failed to authorize with \"https://XXXXXXXXXXX\": authentication request rejected server error message: dev auth: unauthorized"
...
level=error msg="Failed to submit inventory data: transient error: authorization request failed"
...
level=error msg="inventory submit failed: transient error: authorization request failed"
...
We’ve tried to manually send inventory data to the server using mender send-inventory command upon a remote ssh connection, and a day after our last try to send the inventory data, manually, the device is magically shown online and no more complaints and error is found in the journaling log data.
In our opinion, it would be a server issue. Please guide us to find the source of the problem.
Regards
Hi @sap1359,
I think there are a number of conditions overlapping here.
First, unless you have changed it, the MAC address is used as the device identity. Hence if it changes, the device shows up as a new device and requires re-authentication. This matches the error log that you have posted: the device is not authenticated.
Second is the different polling cycles of inventory and updates, and the interaction with the device online timestamp. How those interact with the identity change, I can’t answer right ad hoc, you’d probably have to check the logs. Also, there have been recent improvements in specifically that area.
As this is not the Hosted Mender environment, I can’t comment on which behaviour would be expected and/or explainable, as we obviously don’t know about the exact version and patch state.
In a nutshell, I would guess that these might be unintended side effects of the device changing its identity, and my advice would be to make sure this stays fixed. You might consider using something other than the MAC address, for more details, please see Identity | Mender documentation
Greets,
Josef
Hi Josef,
Thank you for the quick reply. I think you are 100 percent right on the first point.
Actually, there is somehow ambiguity about the second point you mentioned, because in our investigation on a test device, even after changing the MC address using ifconfig command, everything is detected by the server at the next updating cycle (i.e. about 1 minute). But for the device that was offline for three months, we couldn’t even update the inventory, manually, but after a day we could do that and the device became online, again.
We are eliminating the MAC address from the identification process, however, I appreciate it if you could help us resolve this mysterious puzzle.
Have a nice weekend,
Best regards,
Reza
Hi @sap1359, I guess that the device is just a “virtual dupe” now that it showed up under a new identity, and hence not being marked as online. But like said - just guessing.
Please note that changing the device identity in-flight is not a supported scenario, so I really can’t tell what is happening here. The inventory is obviously obtained on every cycle, but I don’t think there are any guarantees on the identity being dynamically updated while the client is running. So, here be dragons.
Additionally, if you share your client and server versions there might be some additional input by the devs, but again - no guarantees, and it might be slow. PTO season is upon us 
Greets,
Josef
Hi @TheYoctoJester,
I hope you’ve had a nice weekend.
Thank you so much for answering our question in a fast and nice descriptive manner.
We’ve decided to remove the MAC address from the identification process and use the serial number, instead.
As I understand from your response, there might be some unexplored issues when -one of- the identification parameter(s) is changed, and we have to keep the identification parameters, unchangeable, to avoid these issues.
Thank you again for sharing valuable data with us.
I’ve changed the status of this thread to Solved because I think you describe everything, in detail, and it is nearly impossible to re-trigger that bug.
Regards,
Reza