Connect: network is unreachable

General Discussions
1

Hello everybody !

I try to update Raspbian and Raspberry pi 3 with rootfs artifact, I also try to update raspbian demo image with demo artifact and hosted Mender and I always encounter this same error :

`2019-09-18 08:22:03 +0000 UTC info: Running Mender version 2.0.0
2019-09-18 08:22:03 +0000 UTC debug: handle update fetch state
2019-09-18 08:22:03 +0000 UTC debug: status reported, response 204 No Content
2019-09-18 08:22:04 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[X-Amz-Request-Id:[0212490A95C7D52B] Last-Modified:[Wed, 18 Sep 2019 08:21:18 GMT] Etag:["6eb227f9888b960aafe77ddb793c96e5"] X-Amz-Tagging-Count:[1] Accept-Ranges:[bytes] Content-Type:[application/vnd.mender-artifact] X-Amz-Id-2:[7k13U2T7PanLrAC/CM85SjNPRw/w8HXEjTgAx1rT0fLTgMEFhslQ0bUl5GdnDFNaHOQiUUOFycE=] Date:[Wed, 18 Sep 2019 08:22:05 GMT] Content-Length:[60779520] Server:[AmazonS3]] 0xb20040 60779520 [] false false map[] 0xb8a400 0x9d4f60}+
2019-09-18 08:22:04 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2019-09-18 08:22:04 +0000 UTC debug: handle update install state
2019-09-18 08:22:04 +0000 UTC debug: status reported, response 204 No Content
2019-09-18 08:22:04 +0000 UTC debug: Read data from device manifest file: device_type=raspberrypi3
2019-09-18 08:22:04 +0000 UTC debug: Current manifest data: raspberrypi3
2019-09-18 08:22:04 +0000 UTC info: no public key was provided for authenticating the artifact
2019-09-18 08:22:04 +0000 UTC debug: checking if device [raspberrypi3] is on compatibile device list: [raspberrypi3]
2019-09-18 08:22:04 +0000 UTC debug: installer: successfully read artifact [name: release-1_2.0.1; version: 3; compatible devices: [raspberrypi3]]
2019-09-18 08:22:04 +0000 UTC debug: Trying to install update of size: 222298112
2019-09-18 08:22:04 +0000 UTC debug: Inactive partition: /dev/mmcblk0p3
2019-09-18 08:22:04 +0000 UTC info: native sector size of block device /dev/mmcblk0p3 is 512, we will write in chunks of 1048576
2019-09-18 08:22:04 +0000 UTC info: opening device /dev/mmcblk0p3 for writing
2019-09-18 08:22:04 +0000 UTC info: partition /dev/mmcblk0p3 size: 4097835008
2019-09-18 08:23:13 +0000 UTC info: wrote 222298112/222298112 bytes of update to device /dev/mmcblk0p3
2019-09-18 08:23:13 +0000 UTC debug: status reported, response 204 No Content
2019-09-18 08:23:14 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2019-09-18 08:23:14 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2019-09-18 08:23:14 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:14 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:14 +0000 UTC debug: status reported, response 204 No Content
2019-09-18 08:23:14 +0000 UTC debug: Inactive partition: /dev/mmcblk0p3
2019-09-18 08:23:14 +0000 UTC debug: Marking inactive partition (/dev/mmcblk0p3) as the new boot candidate.
2019-09-18 08:23:14 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2019-09-18 08:23:14 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-09-18 08:23:14 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-09-18 08:23:14 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-09-18 08:23:14 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-09-18 08:23:14 +0000 UTC debug: Marking inactive partition as a boot candidate successful.
2019-09-18 08:23:14 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2019-09-18 08:23:14 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:14 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:14 +0000 UTC debug: handling reboot state
2019-09-18 08:23:14 +0000 UTC debug: status reported, response 204 No Content
2019-09-18 08:23:14 +0000 UTC info: rebooting device(s)
2019-09-18 08:23:14 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2019-09-18 08:23:22 +0000 UTC info: Running Mender version 2.0.0
2019-09-18 08:23:22 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2019-09-18 08:23:22 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-09-18 08:23:22 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-09-18 08:23:22 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-09-18 08:23:22 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-09-18 08:23:22 +0000 UTC debug: Have U-Boot variable: upgrade_available=0
2019-09-18 08:23:22 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:0]
2019-09-18 08:23:22 +0000 UTC error: transient error: Reboot to new update failed. Expected "upgrade_available" flag to be true but it was false
2019-09-18 08:23:22 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2019-09-18 08:23:22 +0000 UTC debug: transitioning to error state
2019-09-18 08:23:22 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:22 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:22 +0000 UTC info: performing rollback
2019-09-18 08:23:22 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-09-18 08:23:22 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-09-18 08:23:22 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-09-18 08:23:22 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-09-18 08:23:22 +0000 UTC debug: Have U-Boot variable: upgrade_available=0
2019-09-18 08:23:22 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:0]
2019-09-18 08:23:22 +0000 UTC debug: will try to rollback reboot the device
2019-09-18 08:23:22 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2019-09-18 08:23:22 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:22 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:22 +0000 UTC info: rebooting device(s) after rollback
2019-09-18 08:23:22 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p2
2019-09-18 08:23:30 +0000 UTC info: Running Mender version 2.0.0
2019-09-18 08:23:30 +0000 UTC info: Update was interrupted in state: rollback-reboot
2019-09-18 08:23:30 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2019-09-18 08:23:30 +0000 UTC debug: transitioning to error state
2019-09-18 08:23:30 +0000 UTC debug: Have U-Boot variable: mender_check_saveenv_canary=1
2019-09-18 08:23:30 +0000 UTC debug: List of U-Boot variables:map[mender_check_saveenv_canary:1]
2019-09-18 08:23:30 +0000 UTC debug: Have U-Boot variable: mender_saveenv_canary=1
2019-09-18 08:23:30 +0000 UTC debug: List of U-Boot variables:map[mender_saveenv_canary:1]
2019-09-18 08:23:30 +0000 UTC debug: Have U-Boot variable: upgrade_available=0
2019-09-18 08:23:30 +0000 UTC debug: List of U-Boot variables:map[upgrade_available:0]
2019-09-18 08:23:30 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2019-09-18 08:23:30 +0000 UTC debug: handling state after rollback reboot
2019-09-18 08:23:30 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2019-09-18 08:23:30 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:30 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:30 +0000 UTC debug: handle update error state
2019-09-18 08:23:30 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2019-09-18 08:23:30 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:30 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:30 +0000 UTC debug: Handling Cleanup state
2019-09-18 08:23:30 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2019-09-18 08:23:30 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds
2019-09-18 08:23:30 +0000 UTC debug: handle update status report state
2019-09-18 08:23:30 +0000 UTC error: failed to report status: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/cfcb974b-3bb5-45b3-b455-1ef5b5f90f42/status: dial tcp 3.226.176.242:443: connect: network is unreachable
2019-09-18 08:23:30 +0000 UTC error: error reporting update status: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/cfcb974b-3bb5-45b3-b455-1ef5b5f90f42/status: dial tcp 3.226.176.242:443: connect: network is unreachable
2019-09-18 08:23:30 +0000 UTC error: failed to send status to server: transient error: reporting status failed: Put https://hosted.mender.io/api/devices/v1/deployments/device/deployments/cfcb974b-3bb5-45b3-b455-1ef5b5f90f42/status: dial tcp 3.226.176.242:443: connect: network is unreachable
2019-09-18 08:23:30 +0000 UTC info: State transition: update-status-report [none] -> update-retry-report [none]
2019-09-18 08:24:23 +0000 UTC debug: wait complete
2019-09-18 08:24:23 +0000 UTC info: State transition: update-retry-report [none] -> update-status-report [none]
2019-09-18 08:24:23 +0000 UTC debug: handle update status report state
2019-09-18 08:24:24 +0000 UTC debug: status reported, response 204 No Content
2019-09-18 08:24:24 +0000 UTC debug: attempting to upload deployment logs for failed update`

I modified /etc/hosts and mender.conf to change poll intervals but nothing resolve my problem.
How can I fix this ?

Cordialy

Edit: @mirzak: formatting

2

The error seems to be:

2019-09-18 08:23:22 +0000 UTC error: transient error: Reboot to new update failed. Expected "upgrade_available" flag to be true but it was false
2019-09-18 08:23:22 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2019-09-18 08:23:22 +0000 UTC debug: transitioning to error state

Could you clarify which mender-convert version you are using and which image you are converting (link would be helpful)?

3

I create an image with mender-converter version 1.2.0b1 from git branch 1.2.X and Raspbian Buster (I fix HDMI trouble). I tried also to update last demo Raspbian image with artifacts provided on tutorials, but same error and same logs

4

About Raspbian it was lite version released in july

5

hi @jujax @mirzak,
I think i encounter the same problem. And i have tried to use different SD card. I am using raspberry pi 3 compute module lite, 2019-07-10-raspbian-buster-lite, mender-convert 1.2.0 and mender server 2.1.0. Is there any ways to resolve it? Thanks in advance for your help.

6

I was not able to reproduce using 2019-07-10-raspbian-buster-lite.img and following the tutorial , Raspberry Pi 3 Model B/B+ Raspbian on Raspberry Pi 3 B+ device.

Deployment from Hosted Mender was successful.

7

Thank you for the testing. In my situation i follow the guide to create the image bases on 2019-07-10-raspbian-buster-lite.
And I have installed some package (e.g openjdk-jre), did the apt upgrade and insert a jar file(which append text to a file at /data). The jar is executed by crontab job(@reboot java -jar myjar.jar) at startup. I create different artifacts for each changes .
The mender deployments are success until i enable the crontab job. I encountered the following error at most of the cases.

“Expected “upgrade_available” flag to be true but it was false”

I would like to know:

  1. What reason may causes the above error?
  2. If two artifacts A & B(modified from A), Is it allowed to deploy from one to another (i.e. A → B, B → A) ?

Thank you for your assistance.

8

@kacf can you comment on the possible reasons for the above error?

As for what versions are allowed to be deployed, Mender only uses the artifact version to check if you are trying to install the same version in which case it will not install. As long as the version strings are different, then it can be installed.

Drew

9 
“Expected “upgrade_available” flag to be true but it was false”

What reason may causes the above error?

This is normally a result of the boot loader configuration being wrong in the new update. However, the canary variables seem to indicate that the configuration is correct.

Can you post the U-Boot log from when it boots? Maybe it prints something that is related.

Another thing you could try is to stop at the U-Boot prompt during the reboot, and then run printenv to see the environment from the boot loaders perspective. Post that here also if you can.

10

Here is log of u-boot. The “bad CRC” occurs since i flash the .sdimg into the SD card, and i tried remove “resize.sh” before first boot from this thread and it doesn’t help.

U-Boot 2018.07-g884893e54a (Oct 31 2019 - 07:37:13 +0000)

DRAM: 948 MiB
RPI Compute Module 3 (0xa220a0)
MMC: mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT… *** Warning - bad CRC, using default environment

Failed (-5)
Loading Environment from MMC… OK
In: serial
Out: serial
Err: serial
Net: No ethernet found.
Saving Environment to MMC… Writing to redundant MMC(0)… OK
Hit any key to stop autoboot: 2 1 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1…
Found U-Boot script /boot.scr
466 bytes read in 1 ms (455.1 KiB/s)
Executing script at 02400000
switch to partitions #0, OK
mmc0 is current device
5310624 bytes read in 225 ms (22.5 MiB/s)
Kernel image @ 0x080000 [ 0x000000 - 0x5108a0 ]

11

This is from the very first boot after flashing the SD card, is that correct? Can you also provide the U-Boot log which is produced while doing a Mender update? It is presumably during this stage that something is going wrong.

12

Here are the logs of reboot during mender update, and the update failed with “Expected “upgrade_available” flag to be true but it was false”. I login to the U-Boot prompt at first reboot, the upgrade_available is 1.

U-Boot 2018.07-g884893e54a (Oct 31 2019 - 07:37:13 +0000)

DRAM:  948 MiB
RPI Compute Module 3 (0xa220a0)
MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

Failed (-5)
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Saving Environment to MMC... Writing to MMC(0)... OK
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
466 bytes read in 1 ms (455.1 KiB/s)
## Executing script at 02400000
switch to partitions #0, OK
mmc0 is current device
5310624 bytes read in 224 ms (22.6 MiB/s)
Kernel image @ 0x080000 [ 0x000000 - 0x5108a0 ]
## Flattened Device Tree blob at 2eff9400
   Booting using the fdt blob at 0x2eff9400
   reserving fdt memory region: addr=0 size=1000
   Using Device Tree in place at 2eff9400, end 2f002fbc

Starting kernel ...



U-Boot 2018.07-g884893e54a (Oct 31 2019 - 07:37:13 +0000)

DRAM:  948 MiB
RPI Compute Module 3 (0xa220a0)
MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

Failed (-5)
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Saving Environment to MMC... Writing to redundant MMC(0)... OK
Warning: Bootlimit (1) exceeded. Using altbootcmd.
Hit any key to stop autoboot:  0
Saving Environment to MMC... Writing to MMC(0)... OK
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
466 bytes read in 1 ms (455.1 KiB/s)
## Executing script at 02400000
switch to partitions #0, OK
mmc0 is current device
5310624 bytes read in 223 ms (22.7 MiB/s)
Kernel image @ 0x080000 [ 0x000000 - 0x5108a0 ]
## Flattened Device Tree blob at 2eff9400
   Booting using the fdt blob at 0x2eff9400
   reserving fdt memory region: addr=0 size=1000
   Using Device Tree in place at 2eff9400, end 2f002fbc

Starting kernel ...



U-Boot 2018.07-g884893e54a (Oct 31 2019 - 07:37:13 +0000)

DRAM:  948 MiB
RPI Compute Module 3 (0xa220a0)
MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

Failed (-5)
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
466 bytes read in 1 ms (455.1 KiB/s)
## Executing script at 02400000
switch to partitions #0, OK
mmc0 is current device
5310624 bytes read in 224 ms (22.6 MiB/s)
Kernel image @ 0x080000 [ 0x000000 - 0x5108a0 ]
## Flattened Device Tree blob at 2eff9400
   Booting using the fdt blob at 0x2eff9400
   reserving fdt memory region: addr=0 size=1000
   Using Device Tree in place at 2eff9400, end 2f002fbc

Starting kernel ...
13

To me it looks like the kernel crashes or fails to load. You see “Starting kernel” and then it goes straight back to U-Boot again. So in essence it’s rolling back, which is correct, assuming that there indeed is a kernel problem.

What are you updating to? Are the two images (SD card flash and the update) the same?

14

The different between the two images is enable a crontab job at start up - @reboot java -jar xxx.jar. The jar program execute the system command “e.g. ping” and keep printing text to a file at /data. From the log of the jar, the jar is killed by reboot after execute the “ping”. It seems my java program causes a reboot in new partition, turns the “upgrade_available” from 1 to 0 and causes a rollback. I will review the java program further.

In addition, I would like to know is there any reminders about using a program to manage the network connection of system in cooperate with the mender ?

Thank you for your kind assistance.

15

Note sure if I understood the question completely, so here’s two answers:

  • If the network does not work after an update, this will cause Mender to roll back
  • Mender itself does not manage the network, that is the job of the operating system, and requires that it is configured correctly according to that platform. This configuration must be included in update files as well