Mender client hangs after network change

Hi,

I want to use mender for updating a Yocto based embedded device (Jetson Nano). Somehow the mender-client gets stuck after the network configuration changes. These are the steps:

  1. Update the network configuration using networkd:
> cat <<EOT >> /run/systemd/network/20-user.network
[Match]
Name=eth0

[Network]
DHCP=yes
EOT
> networkctl reload
  1. Check for updates:
> mender check-update

Somehow it seems to hang inside the proxy.go. Here is the log:

...
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="SIGUSR1 signal received." func=github.com/mendersoftware/mender/cli.runDaemon.func1 file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/cli/commands.go:344"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Sent wake up!" func=github.com/mendersoftware/mender/cli.runDaemon.func1 file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/cli/commands.go:351"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=info msg="Forced wake-up from sleep" func="github.com/mendersoftware/mender/app.(*waitState).Wait" file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:233"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=info msg="Forcing state machine to: update-check" func="github.com/mendersoftware/mender/app.(*MenderDaemon).Run" file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/app/daemon.go:118"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" func=github.com/mendersoftware/mender/app.transitionState file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/app/mender.go:511"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="statescript: The timeout for executing scripts is not defined; using default of 1h0m0s seconds" func=github.com/mendersoftware/mender/statescript.Launcher.getTimeout file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/statescript/executor.go:90"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="statescript: The timeout for executing scripts is not defined; using default of 1h0m0s seconds" func=github.com/mendersoftware/mender/statescript.Launcher.getTimeout file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/statescript/executor.go:90"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Handle update check state" func="github.com/mendersoftware/mender/app.(*updateCheckState).Handle" file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/app/state.go:659"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Returning artifact name 3dvl-image_0.1.31-3766f64-dirty-hesmar from database." func="github.com/mendersoftware/mender/device.(*DeviceManager).GetCurrentArtifactName" file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:133"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Reading data from the device manifest file: /var/lib/mender/device_type" func=github.com/mendersoftware/mender/device.GetManifestData file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:85"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="device_type=jetson-nano-devkit-emmc" func=github.com/mendersoftware/mender/device.GetManifestData file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:96"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Current manifest data: jetson-nano-devkit-emmc" func=github.com/mendersoftware/mender/device.GetManifestData file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/device/device.go:103"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Connecting to server http://127.0.0.1:40017" func="github.com/mendersoftware/mender/client.(*ReauthorizingClient).reconstructRequest" file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/client/client.go:186"
Dec 15 12:36:22 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:36:22Z" level=debug msg="Request: \"\" \"\" \"https\" \"mender.3dvisionlabs.com\" \"/api/devices/v2/deployments/device/deployments/next\"" func="github.com/mendersoftware/mender/app/proxy.(*proxyControllerInner).doHttpRequest" file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/app/proxy/proxy.go:307"

Calling mender check-update again gives the following output:

Dec 15 12:37:40 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:37:40Z" level=debug msg="SIGUSR1 signal received." func=github.com/mendersoftware/mender/cli.runDaemon.func1 file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/cli/commands.go:344"
Dec 15 12:37:40 jetson-nano-devkit-emmc mender[5053]: time="2022-12-15T12:37:40Z" level=debug msg="Sent wake up!" func=github.com/mendersoftware/mender/cli.runDaemon.func1 file="/home/hesmar/Development/3dvisionlabs/embedded/yocto2/tegra-demo-distro/build-3dvl-mender/tmp/work/jetson_nano_devkit_emmc-3dvl-linux/mender-client/3.4.0-r0/build/src/github.com/mendersoftware/mender/cli/commands.go:351"

After restarting the mender-client service, everything works again, even with the new network configuration.

Is this a bug or am I doing something wrong?

Thanks,
Markus

Interesting.

This does sound like a bug to me actually.

Is it 100% consistent for you @hesmar ?

Ok, I did some further investigation. It only happens when the IP address was changed. So for example, when I change my network configuration from DHCP to static IP, the issue occurs only if the new IP address is different from the previous one. If it is the same address, mender-client keeps working.

https://tracker.mender.io/browse/MEN-6198

Ticket for you to follow @hesmar

Out of curiosity, was this reproduced internally? We may be experiencing a similar issue and would like to narrow down if it is related to this bug or not. Thanks!