Client can not update inventory

General Discussions
1

Hi,

2 clients (on different locations) can not update inventory list.
You can find client log below.
Any comment?

Client Version 2.4
Server Version 2.6

Sep 03 07:26:25 node01 systemd[1]: Started Mender OTA update service.
Sep 03 07:26:25 node01 mender[24503]: time="2021-09-03T07:26:25Z" level=info msg="Loaded configuration file: /etc/mender/mender.conf"
Sep 03 07:26:25 node01 mender[24503]: time="2021-09-03T07:26:25Z" level=info msg="Mender running on partition: /dev/mmcblk0p2"
Sep 03 07:26:27 node01 mender[24503]: time="2021-09-03T07:26:27Z" level=info msg="State transition: init [none] -> init [none]"
Sep 03 07:26:27 node01 mender[24503]: time="2021-09-03T07:26:27Z" level=info msg="State transition: init [none] -> idle [Idle]"
Sep 03 07:26:27 node01 mender[24503]: time="2021-09-03T07:26:27Z" level=info msg="authorization data present and valid"
Sep 03 07:26:27 node01 mender[24503]: time="2021-09-03T07:26:27Z" level=info msg="State transition: idle [Idle] -> check-wait [Idle]"
Sep 03 07:26:27 node01 mender[24503]: time="2021-09-03T07:26:27Z" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
2

Hi @doganmurat

We’re gonna need a bit more information here, the log you posted is cut off, and it looks just fine up until this point for me at least.

3

Yes you are right, it seems ok. But never logs anything else.
I guess it can not make the update, so waiting.

4

Ok, I see.

Let’s start off digging a little bit then.

  1. Could you enable the client with debug logging?
  2. Can you list which inventory scripts are run on the device?
5

Debug output

DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/conf/config.go:175 github.com/mendersoftware/mender/conf.loadConfigFile() Configuration file does not exist: /var/lib/mender/mender.conf 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/conf/config.go:192 github.com/mendersoftware/mender/conf.readConfigFile() Reading Mender configuration from file /etc/mender/mender.conf 
INFO[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/conf/config.go:185 github.com/mendersoftware/mender/conf.loadConfigFile() Loaded configuration file: /etc/mender/mender.conf 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/conf/config.go:124 github.com/mendersoftware/mender/conf.LoadConfig() Loaded 1 configuration file(s)               
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/conf/config.go:131 github.com/mendersoftware/mender/conf.LoadConfig() Loaded configuration = &conf.MenderConfig{MenderConfigFromFile:conf.MenderConfigFromFile{ClientProtocol:"", ArtifactVerifyKey:"/etc/mender/artifact-verify-key.pem", HttpsClient:client.HttpsClient{Certificate:"", Key:"", SSLEngine:""}, RootfsPartA:"/dev/mmcblk0p2", RootfsPartB:"/dev/mmcblk0p3", DeviceTypeFile:"/var/lib/mender/device_type", UpdatePollIntervalSeconds:60, InventoryPollIntervalSeconds:60, SkipVerify:false, RetryPollIntervalSeconds:300, StateScriptTimeoutSeconds:0, StateScriptRetryTimeoutSeconds:0, StateScriptRetryIntervalSeconds:0, ModuleTimeoutSeconds:0, ServerCertificate:"/etc/mender/server.crt", ServerURL:"https://XXX.com/", UpdateLogPath:"", TenantToken:"dummy", Servers:[]client.MenderServer(nil)}, ModulesPath:"/usr/share/mender/modules/v3", ModulesWorkPath:"/var/lib/mender/modules/v3", ArtifactInfoFile:"/etc/mender/artifact_info", ArtifactScriptsPath:"/var/lib/mender/scripts", RootfsScriptsPath:"/etc/mender/scripts"} 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/conf/config.go:166 github.com/mendersoftware/mender/conf.(*MenderConfig).Validate() Verified configuration = &conf.MenderConfig{MenderConfigFromFile:conf.MenderConfigFromFile{ClientProtocol:"", ArtifactVerifyKey:"/etc/mender/artifact-verify-key.pem", HttpsClient:client.HttpsClient{Certificate:"", Key:"", SSLEngine:""}, RootfsPartA:"/dev/mmcblk0p2", RootfsPartB:"/dev/mmcblk0p3", DeviceTypeFile:"/var/lib/mender/device_type", UpdatePollIntervalSeconds:60, InventoryPollIntervalSeconds:60, SkipVerify:false, RetryPollIntervalSeconds:300, StateScriptTimeoutSeconds:0, StateScriptRetryTimeoutSeconds:0, StateScriptRetryIntervalSeconds:0, ModuleTimeoutSeconds:0, ServerCertificate:"/etc/mender/server.crt", ServerURL:"https://XXX.com/", UpdateLogPath:"", TenantToken:"dummy", Servers:[]client.MenderServer{client.MenderServer{ServerURL:"https://XXX.com"}}}, ModulesPath:"/usr/share/mender/modules/v3", ModulesWorkPath:"/var/lib/mender/modules/v3", ArtifactInfoFile:"/etc/mender/artifact_info", ArtifactScriptsPath:"/var/lib/mender/scripts", RootfsScriptsPath:"/etc/mender/scripts"} 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:153 github.com/mendersoftware/mender/installer.getEnvironmentVariable() Have U-Boot variable: mender_check_saveenv_canary=1 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:176 github.com/mendersoftware/mender/installer.getEnvironmentVariable() List of U-Boot variables:map[mender_check_saveenv_canary:1] 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:153 github.com/mendersoftware/mender/installer.getEnvironmentVariable() Have U-Boot variable: mender_saveenv_canary=1 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:176 github.com/mendersoftware/mender/installer.getEnvironmentVariable() List of U-Boot variables:map[mender_saveenv_canary:1] 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:153 github.com/mendersoftware/mender/installer.getEnvironmentVariable() Have U-Boot variable: mender_boot_part=2     
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:176 github.com/mendersoftware/mender/installer.getEnvironmentVariable() List of U-Boot variables:map[mender_boot_part:2] 
DEBU[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/partitions.go:175 github.com/mendersoftware/mender/installer.(*partitions).getAndCacheActivePartition() Setting active partition from mount candidate: /dev/mmcblk0p2 
INFO[0000]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/cli/cli.go:419 github.com/mendersoftware/mender/cli.(*runOptionsType).commonCLIHandler() Mender running on partition: /dev/mmcblk0p2  
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/installer/modules.go:881 github.com/mendersoftware/mender/installer.NewModuleInstallerFactory() ModuleTimeoutSeconds not set. Defaulting to 14400 seconds 
INFO[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/mender.go:591 github.com/mendersoftware/mender/app.transitionState() State transition: init [none] -> init [none] 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:269 github.com/mendersoftware/mender/app.(*initState).Handle() No state data stored                         
INFO[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/mender.go:591 github.com/mendersoftware/mender/app.transitionState() State transition: init [none] -> idle [Idle] 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/statescript/executor.go:82 github.com/mendersoftware/mender/statescript.Launcher.getTimeout() statescript: The timeout for executing scripts is not defined; using default of 1h0m0s seconds 
INFO[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/mender.go:198 github.com/mendersoftware/mender/app.(*Mender).IsAuthorized() authorization data present and valid         
INFO[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/mender.go:591 github.com/mendersoftware/mender/app.transitionState() State transition: idle [Idle] -> check-wait [Idle] 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:1120 github.com/mendersoftware/mender/app.(*checkWaitState).Handle() Handle check wait state                      
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:1131 github.com/mendersoftware/mender/app.(*checkWaitState).Handle() Check wait state; next checks: (update: 0001-01-01 00:01:00 +0000 UTC) (inventory: 0001-01-01 00:00:00 +0000 UTC) 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:1149 github.com/mendersoftware/mender/app.(*checkWaitState).Handle() Next check: 0001-01-01 00:00:00 +0000 UTC:inventory-update, (2021-09-03 07:51:17.861085567 +0000 UTC m=+1.084216884) 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:1181 github.com/mendersoftware/mender/app.(*checkWaitState).Handle() Check wait returned: inventory-update        
INFO[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/app/mender.go:591 github.com/mendersoftware/mender/app.transitionState() State transition: check-wait [Idle] -> inventory-update [Sync] 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/statescript/executor.go:82 github.com/mendersoftware/mender/statescript.Launcher.getTimeout() statescript: The timeout for executing scripts is not defined; using default of 1h0m0s seconds 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/statescript/executor.go:82 github.com/mendersoftware/mender/statescript.Launcher.getTimeout() statescript: The timeout for executing scripts is not defined; using default of 1h0m0s seconds 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:134 github.com/mendersoftware/mender/device.(*DeviceManager).GetCurrentArtifactName() Returning artifact name from /etc/mender/artifact_info file. 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:80 github.com/mendersoftware/mender/device.GetManifestData() Reading data from the device manifest file: /etc/mender/artifact_info 
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:91 github.com/mendersoftware/mender/device.GetManifestData() artifact_name=3.4.0                          
DEBU[0001]/home/nokta/yocto/node-os/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/mender-client/2.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:98 github.com/mendersoftware/mender/device.GetManifestData() Current manifest data: 3.4.0

Inventory scripts

mender-inventory-bootloader-integration  mender-inventory-hostinfo                mender-inventory-os                     
mender-inventory-geo                     mender-inventory-network                 mender-inventory-rootfs-type
6

As I understood mender client uses PATCH method to update inventory.
Maybe firewall is blocking PATCH method?

7

I would rather suspect that one of your inventory scripts hangs.
To begin with, could you move the mender-inventory-geo script somewhere else, so that it is not run?

8

Yes you are right.
Moving mender-inventory-geo solved the problem.

1 Like
9

Glad to hear it!

We had some issues on older clients, with the geo script, but it’s now fixed, in 2.5 and newer, also if you’d like, you can get the new and improved version from master, here

10

Good to hear that you solved it for new versions.
Thank you for helping.

11

No worries :slight_smile:

Glad it worked out so quick