Mender install is locking up board

Board Integrations
,
1

Hi,

A mender install is locking up a new custom board based on an i.MX 8M Nano. This design is based on the EVK and we are not seeing the same issue with the EVK so it may be board related or related to a misconfiguration with the Mender for this board, our board has less memory in case this could be a cause. It can occur anywhere between 30 and 60% of the update and is reproducible every time. With --log-level defined as debug we get the following

DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:200 github.com/mendersoftware/mender/conf.readConfigFile() Reading Mender configuration from file /var/lib/mender/mender.conf 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:193 github.com/mendersoftware/mender/conf.loadConfigFile() Loaded configuration file: /var/lib/mender/mender.conf 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:200 github.com/mendersoftware/mender/conf.readConfigFile() Reading Mender configuration from file /etc/mender/mender.conf 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:193 github.com/mendersoftware/mender/conf.loadConfigFile() Loaded configuration file: /etc/mender/mender.conf 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:126 github.com/mendersoftware/mender/conf.LoadConfig() Loaded 2 configuration file(s)               
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:133 github.com/mendersoftware/mender/conf.LoadConfig() Loaded configuration = &conf.MenderConfig{MenderConfigFromFile:conf.MenderConfigFromFile{ClientProtocol:"", ArtifactVerifyKey:"", HttpsClient:client.HttpsClient{Certificate:"", Key:"", SSLEngine:""}, Security:client.Security{AuthPrivateKey:"", SSLEngine:""}, RootfsPartA:"/dev/mmcblk2p1", RootfsPartB:"/dev/mmcblk2p2", DeviceTypeFile:"/var/lib/mender/device_type", UpdatePollIntervalSeconds:1800, InventoryPollIntervalSeconds:28800, SkipVerify:false, RetryPollIntervalSeconds:300, StateScriptTimeoutSeconds:0, StateScriptRetryTimeoutSeconds:0, StateScriptRetryIntervalSeconds:0, ModuleTimeoutSeconds:0, ServerCertificate:"", ServerURL:"https://mender-server.octopushome.net/", UpdateLogPath:"", TenantToken:"dummy", Servers:[]client.MenderServer(nil)}, ModulesPath:"/usr/share/mender/modules/v3", ModulesWorkPath:"/var/lib/mender/modules/v3", ArtifactInfoFile:"/etc/mender/artifact_info", ArtifactScriptsPath:"/var/lib/mender/scripts", RootfsScriptsPath:"/etc/mender/scripts"} 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/conf/config.go:174 github.com/mendersoftware/mender/conf.(*MenderConfig).Validate() Verified configuration = &conf.MenderConfig{MenderConfigFromFile:conf.MenderConfigFromFile{ClientProtocol:"", ArtifactVerifyKey:"", HttpsClient:client.HttpsClient{Certificate:"", Key:"", SSLEngine:""}, Security:client.Security{AuthPrivateKey:"", SSLEngine:""}, RootfsPartA:"/dev/mmcblk2p1", RootfsPartB:"/dev/mmcblk2p2", DeviceTypeFile:"/var/lib/mender/device_type", UpdatePollIntervalSeconds:1800, InventoryPollIntervalSeconds:28800, SkipVerify:false, RetryPollIntervalSeconds:300, StateScriptTimeoutSeconds:0, StateScriptRetryTimeoutSeconds:0, StateScriptRetryIntervalSeconds:0, ModuleTimeoutSeconds:0, ServerCertificate:"", ServerURL:"https://mender-server.octopushome.net/", UpdateLogPath:"", TenantToken:"dummy", Servers:[]client.MenderServer{client.MenderServer{ServerURL:"https://mender-server.octopushome.net"}}}, ModulesPath:"/usr/share/mender/modules/v3", ModulesWorkPath:"/var/lib/mender/modules/v3", ArtifactInfoFile:"/etc/mender/artifact_info", ArtifactScriptsPath:"/var/lib/mender/scripts", RootfsScriptsPath:"/etc/mender/scripts"} 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:205 github.com/mendersoftware/mender/installer.getEnvironmentVariable() Have U-Boot variable: mender_check_saveenv_canary=1 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:228 github.com/mendersoftware/mender/installer.getEnvironmentVariable() List of U-Boot variables:map[mender_check_saveenv_canary:1] 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:205 github.com/mendersoftware/mender/installer.getEnvironmentVariable() Have U-Boot variable: mender_saveenv_canary=1 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:228 github.com/mendersoftware/mender/installer.getEnvironmentVariable() List of U-Boot variables:map[mender_saveenv_canary:1] 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:205 github.com/mendersoftware/mender/installer.getEnvironmentVariable() Have U-Boot variable: mender_boot_part=1     
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/bootenv.go:228 github.com/mendersoftware/mender/installer.getEnvironmentVariable() List of U-Boot variables:map[mender_boot_part:1] 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/partitions.go:175 github.com/mendersoftware/mender/installer.(*partitions).getAndCacheActivePartition() Setting active partition from mount candidate: /dev/mmcblk2p1 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/cli/cli.go:419 github.com/mendersoftware/mender/cli.(*runOptionsType).commonCLIHandler() Mender running on partition: /dev/mmcblk2p1  
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/modules.go:881 github.com/mendersoftware/mender/installer.NewModuleInstallerFactory() ModuleTimeoutSeconds not set. Defaulting to 14400 seconds 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/app/standalone.go:54 github.com/mendersoftware/mender/app.DoStandaloneInstall() Starting device update.                      
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/app/standalone.go:74 github.com/mendersoftware/mender/app.DoStandaloneInstall() Start updating from local image file: [/data/octave-cad-image-octave-imx8mn.mender] 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/app/standalone.go:77 github.com/mendersoftware/mender/app.DoStandaloneInstall() Fetching update from file results: [&{0x400005d500}], 98145280, <nil> 
Installing Artifact of size 98145280...
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/device/device.go:80 github.com/mendersoftware/mender/device.GetManifestData() Reading data from the device manifest file: /var/lib/mender/device_type 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/device/device.go:91 github.com/mendersoftware/mender/device.GetManifestData() device_type=octave-imx8mn                    
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/device/device.go:98 github.com/mendersoftware/mender/device.GetManifestData() Current manifest data: octave-imx8mn         
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/statescript/executor.go:82 github.com/mendersoftware/mender/statescript.Launcher.getTimeout() statescript: The timeout for executing scripts is not defined; using default of 1h0m0s seconds 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/installer.go:109 github.com/mendersoftware/mender/installer.ReadHeaders() No public key was provided for authenticating the artifact 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/modules.go:914 github.com/mendersoftware/mender/installer.(*ModuleInstallerFactory).GetModuleTypes() Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/installer.go:120 github.com/mendersoftware/mender/installer.ReadHeaders.func1() Checking if device [octave-imx8mn] is on compatible device list: [octave-imx8mn] 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/installer.go:196 github.com/mendersoftware/mender/installer.ReadHeaders() Installer: Successfully read artifact [name: release-2021-03-22; version: 3; compatible devices: [octave-imx8mn]] 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/partitions.go:55 github.com/mendersoftware/mender/installer.(*partitions).GetActive() Active partition: /dev/mmcblk2p1             
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/partitions.go:82 github.com/mendersoftware/mender/installer.(*partitions).getAndCacheInactivePartition() Detected inactive partition /dev/mmcblk2p2, based on active partition /dev/mmcblk2p1 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/block_device.go:66 github.com/mendersoftware/mender/installer.bdevice.Open() Opening device "/dev/mmcblk2p2" for writing  
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/block_device.go:71 github.com/mendersoftware/mender/installer.bdevice.Open() Installing update of size: 1744830464        
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/block_device.go:94 github.com/mendersoftware/mender/installer.bdevice.Open() Device: /dev/mmcblk2p2 is a ubi device: false 
INFO[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/block_device.go:146 github.com/mendersoftware/mender/installer.bdevice.Open() Native sector size of block device /dev/mmcblk2p2 is 512 bytes. Mender will write in chunks of 1048576 bytes 
DEBU[0000]/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/block_device.go:152 github.com/mendersoftware/mender/installer.bdevice.Open() Opening device: /dev/mmcblk2p2 for writing with flag: 2 
................................   1% 1024 KiB
................................   2% 2048 KiB
................................   3% 3072 KiB
................................   4% 4096 KiB
................................   5% 5120 KiB
................................   6% 6144 KiB
................................   7% 7168 KiB
................................   8% 8192 KiB
................................   9% 9216 KiB
................................  10% 10240 KiB
................................  11% 11264 KiB
................................  12% 12288 KiB
................................  13% 13312 KiB
................................  14% 14336 KiB
................................  16% 15360 KiB
................................  17% 16384 KiB
................................  18% 17408 KiB
................................  19% 18432 KiB
................................  20% 19456 KiB
................................  21% 20480 KiB
................................  22% 21504 KiB
................................  23% 22528 KiB
................................  24% 23552 KiB
................................  25% 24576 KiB
................................  26% 25600 KiB
................................  27% 26624 KiB
................................  28% 27648 KiB
................................  29% 28672 KiB
................................  30% 29696 KiB
................................  32% 30720 KiB
................................  33% 31744 KiB
................................  34% 32768 KiB
................................  35% 33792 KiB
................................  36% 34816 KiB
................................  37% 35840 KiB
................................  38% 36864 KiB
................................  39% 37888 KiB
................................  40% 38912 KiB
................................  41% 39936 KiB
................................  42% 40960 KiB
................................  43% 41984 KiB
................................  44% 43008 KiB
................................  45% 44032 KiB
................................  47% 45056 KiB
................................  48% 46080 KiB
..............[   92.075489][   T17] audit: type=1701 audit(2466.608:3): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=373 comm="CADIhdHost" exe="/usr/sbin/CADIhdHost" sig=6 res=1
..................  49% 47104 KiB
................................  50% 48128 KiB
................................  51% 49152 KiB
................................  52% 50176 KiB
................................  53% 51200 KiB
................................  54% 52224 KiB
................................  55% 53248 KiB
................................  56% 54272 KiB
................................  57% 55296 KiB
................................  58% 56320 KiB
............runtime: bad span s.state=149 s.sweepgen=2903394988 sweepgen=162
fatal error: non in-us

The interesting part looks like the very last message runtime: bad span s.state=149 s.sweepgen=2903394988 sweepgen=162
fatal error: non in-us

I’m taking a look but was wondering if anyone would have any idea as to what is going on as I’m not really familiar with go.

Many Thanks,
Martin.

2

Should have mentioned I’m building the client from the dunfell branch and the version is

mender -v
2.4.1	runtime: go1.14.15
1 Like
3

Managed to get a stack trace on one attempt:

................................  55% 53248 KiB
..............unexpected fault address 0x56c112cbf69a6a
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x56c112cbf69a6a pc=0x5ea570]

goroutine 1 [running]:
runtime.throw(0x80849a, 0x5)
	/usr/lib/go/src/runtime/panic.go:1116 +0x54 fp=0x40000e2a30 sp=0x40000e2a00 pc=0x43f73c
runtime.sigpanic()
	/usr/lib/go/src/runtime/signal_unix.go:724 +0x3bc fp=0x40000e2a60 sp=0x40000e2a30 pc=0x456124
hash/crc32.Update(0x40e21f32e0, 0xdaf563d8e9bc752, 0x4000178000, 0x8000, 0x8000, 0x0)
	/usr/lib/go/src/hash/crc32/crc32.go:218 +0x88 fp=0x40000e2ab0 sp=0x40000e2a70 pc=0x5ea570
compress/gzip.(*Reader).Read(0x40000af080, 0x4000178000, 0x8000, 0x8000, 0x8000, 0x8000, 0x40000e2bb8)
	/usr/lib/go/src/compress/gzip/gunzip.go:252 +0xc4 fp=0x40000e2b50 sp=0x40000e2ab0 pc=0x5eb9fc
archive/tar.(*regFileReader).Read(0x400012f1a0, 0x4000178000, 0x8000, 0x8000, 0x8000, 0x8000, 0x8000)
	/usr/lib/go/src/archive/tar/reader.go:665 +0x160 fp=0x40000e2bb0 sp=0x40000e2b50 pc=0x6b9248
archive/tar.(*Reader).Read(0x40000998c0, 0x4000178000, 0x8000, 0x8000, 0x8000, 0x8000, 0x8000)
	/usr/lib/go/src/archive/tar/reader.go:626 +0x58 fp=0x40000e2c10 sp=0x40000e2bb0 pc=0x6b9010
io.(*teeReader).Read(0x400012f200, 0x4000178000, 0x8000, 0x8000, 0x8000, 0x8000, 0x8000)
	/usr/lib/go/src/io/io.go:538 +0x4c fp=0x40000e2c70 sp=0x40000e2c10 pc=0x47a184
github.com/mendersoftware/mender-artifact/artifact.(*Checksum).Read(0x4000067090, 0x4000178000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/vendor/github.com/mendersoftware/mender-artifact/artifact/checksum.go:77 +0x50 fp=0x40000e2cd0 sp=0x40000e2c70 pc=0x6bcaf8
io.copyBuffer(0x8a78c0, 0x400012f2e0, 0x8a76e0, 0x4000067090, 0x4000178000, 0x8000, 0x8000, 0xffffaf778688, 0x0, 0x0)
	/usr/lib/go/src/io/io.go:405 +0xd4 fp=0x40000e2d50 sp=0x40000e2cd0 pc=0x479e0c
io.Copy(...)
	/usr/lib/go/src/io/io.go:364
github.com/mendersoftware/mender/installer.(*dualRootfsDeviceImpl).StoreUpdate(0x400010e3c0, 0x8a76e0, 0x4000067090, 0x8b4a80, 0x400014e700, 0x4000067090, 0x0)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/installer/dual_rootfs_device.go:198 +0xd8 fp=0x40000e2e50 sp=0x40000e2d50 pc=0x6d3340
github.com/mendersoftware/mender-artifact/areader.(*Reader).readAndInstallDataFiles(0x40000a0580, 0x40000998c0, 0x8b80e0, 0x4000066f50, 0x0, 0x8b0a00, 0x400004f1c0, 0x8b2140, 0x400010e3c0, 0x2, ...)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/vendor/github.com/mendersoftware/mender-artifact/areader/reader.go:1013 +0x1c4 fp=0x40000e2f40 sp=0x40000e2e50 pc=0x6cbbac
github.com/mendersoftware/mender-artifact/areader.(*Reader).readAndInstall(0x40000a0580, 0x8a73c0, 0x4000098d80, 0x8b80e0, 0x4000066f50, 0x0, 0x8b0a00, 0x400004f1c0, 0x0, 0x0)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/vendor/github.com/mendersoftware/mender-artifact/areader/reader.go:950 +0x1f0 fp=0x40000e3030 sp=0x40000e2f40 pc=0x6cb7d8
github.com/mendersoftware/mender-artifact/areader.(*Reader).readNextDataFile(0x40000a0580, 0x4000098d80, 0x10, 0x6000103)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/vendor/github.com/mendersoftware/mender-artifact/areader/reader.go:856 +0x2cc fp=0x40000e30f0 sp=0x40000e3030 pc=0x6cae24
github.com/mendersoftware/mender-artif[  287.372463][   T17] audit: type=1701 audit(3444.904:6): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=461 comm="systemd" exe="/lib/systemd/systemd" sig=6 res=1
act/areader.(*Reader).readData(0x40000a0580, 0x4000098d80, 0x0, 0x400015a900)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.com/mendersoftware/mender/vendor/github.com/mendersoftware/mender-artifact/areader/reader.go:861 +0x30 fp=0x40000e3130 sp=0x40000e30f0 pc=0x6cb008
github.com/mendersoftware/mender-artifact/areader.(*Reader).ReadArtifactData(0x40000a0580, 0x4, 0x0)
	/ws/rufilla/octopus/octave-dunfell/build-devel/tmp/work/aarch64-poky-linux/mender-client/2.4.1-r0/build/src/github.:588 +0x48 fp=0x40000e3190 sp=0 pc=0x6c8c20
github.com/mender(...)
	/ws/rufilla/octopus/oct:204
github.com/mendersoftware(0x8ae900, 0x4000118f60, 0x0, 0, 0x0, 0x4000098b40, 0x8ae840, 0x4000066aa0, 0x4000118f60, 0x1, ...)
	/ws/rufilla/octopus/oc:156 +0x214 fp=0x40000e32f0 sp=0x40000e3190 pc=0x6eaa1c
githu(0x8ae900, 0x4000118f60, 0x0, 0, 0x0, 0x4000098b40, 0x8ae840, 0x4000066aa0, 0x0, 0x0)
	/ws/r:177 +0x68 fp=0x40000e3430 sp=0 pc=0x6eb210
github.com/mender(0x4000098b40, 0xffffcf083ed9, 0x2b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/ws/rufilla/oc:92 +0x568 fp=0x40000e35f0 sp=0 pc=0x6ea590
github.com/mender(0x400005ac00, 0x4000028640, 0x, 0x4000028660, 0x1b, 0x80b791,0xf, 0xffffcf083ed9, 0x2b, 0x0,)
	/ws/rufilla/octopus/octave-:186 +0x36c fp=0x40000e37a0 sp=0x40000e35f0 pc=0x71b8d4
githu(0x40000aac60, 0x400005ac00, 0x, 0x0)
	/ws/rufilla/octopus/oc:440 +0x3cc fp=0x40000e3af0 sp=0x40000e37a0 pc=0x719c24
githu(0x400005ac00, 0x1, 0x1)
	/ws/:173 +0xb4 fp=0x40000e3b30 sp=0 pc=0x72202c
github.com/urfave(0x4000096b40, 0x400005ab40, 0x, 0x0)
	/ws/rufilla/octopus/oc:161 +0x3ec fp=0x40000e3c30 sp=0x40000e3b30 pc=0x709254
githu(0x4000001980, 0x8b1ec0, 0x4000, 0x4000020070, 0x7, 0x7, 0x0, 0x0)
	/ws/rufilla/octopus/octa:302 +0x620 fp=0x40000e3d70 sp=0x40000e3c30 pc=0x707008
githu(...)
	/ws/rufilla/octopus/oct:211
github.com/mendersoftware(0x4000020070, 0x7, 0x7, 0x4123, 0x40000e3f28)
	/ws/rufilla/o:376 +0x1cb0 fp=0x40000e3ee0 sp0x40000e3d70 pc=0x7189c8
main.(0x3)
	/ws/rufilla/octopus/oct:26 +0x50 fp=0x40000e3f50 sp=0x pc=0x723aa8
main.main()
	/ws:39 +0x20 fp=0x40000e3f70 sp=0x pc=0x723c48
runtime.main()
	/usr/lib/go/src/runtime/proc.go:203 +0x20c fp=0x40000e3fd0 sp=0x40000e3f70 pc=0x441c64
runti()
	/usr/lib/go/src/runtime/as:1148 +0x4 fp=0x40000e3fd0 sp=0 pc=0x46ee6c
fatal error: unex
panic during panic
[signal S code=0x1 addr=0xf7c0aad21bb144 pc=0x43fdb8]

runtime stack:runtime.throw(0x81ab8f, 0x2a)
	/usr/lib/go/src/runtime/panic.:1116 +0x54
runtime.sigpanic()	/usr/lib/go/src/runtime/signal:701 +0x448
runtime.dopanic_m(0x4000000180, 0x43f73c, 0x40000, 0x46c598)
	/usr/lib/go/src/r:1339 +0x250
runtime.fatalthro()
	/usr/lib/go/src/runtime/pa:1171 +0x54
runtime.systemstac(0xffff876f6940)
	/usr/lib/go/:237 +0xa0
runtime.mstart()
	/usr/lib/go/src/runtime/proc.go:1056
fatal error: unlock of u
stack trace unavailable

and then on the console afterward I see the following

[  319.778938][   T17] audit: type=1701 audit(3477.308:7): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=463 comm="mender-device-i" exe="/bin/bash.bash" sig=11 res=1
[  391.511869][  T274] systemd-journald[274]: Failed to send WATCHDOG=1 notification message: Connection refused
[  491.511590][  T274] systemd-journald[274]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected
[  571.511684][  T274] systemd-journald[274]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected
[  619.778953][   T17] audit: type=1701 audit(3777.308:8): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=465 comm="mender-device-i" exe="/bin/bash.bash" sig=11 res=1
[  671.511580][  T274] systemd-journald[274]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected
[  751.511767][  T274] systemd-journald[274]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected
4

Hi,

I did some testing using dd and can cause the kernel to panic so it’s not mender. I’ll check through the mm subsystem to see if this has been fixed already.

root@octave-imx8mn:~# dd if=/dev/urandom of=/dev/mmcblk2p2 bs=1048576
[ 1165.874929][  T472] Unable to handle kernel paging request at virtual address 00333657737acd5f
[ 1165.883575][  T472] Mem abort info:
[ 1165.887062][  T472]   ESR = 0x96000004
[ 1165.890828][  T472]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1165.896843][  T472]   SET = 0, FnV = 0
[ 1165.900598][  T472]   EA = 0, S1PTW = 0
[ 1165.904442][  T472] Data abort info:
[ 1165.908022][  T472]   ISV = 0, ISS = 0x00000004
[ 1165.912556][  T472]   CM = 0, WnR = 0
[ 1165.916227][  T472] [00333657737acd5f] address between user and kernel address ranges
[ 1165.924063][  T472] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 1165.930332][  T472] CPU: 0 PID: 472 Comm: dd Not tainted 5.4.24-yocto-octave+g4d1dd2e34534 #1
[ 1165.938853][  T472] Hardware name: Octopus i.MX8MNano Octave board (DT)
[ 1165.945465][  T472] pstate: 20000005 (nzCv daif -PAN -UAO)
[ 1165.950957][  T472] pc : mem_cgroup_page_lruvec+0x24/0x58
[ 1165.956356][  T472] lr : pagevec_lru_move_fn+0x94/0x118
[ 1165.961578][  T472] sp : ffff800011793a40
[ 1165.965584][  T472] x29: ffff800011793a40 x28: 000000000000000e
[ 1165.971589][  T472] x27: 00000000000016c0 x26: ffff80001143a098
[ 1165.977594][  T472] x25: 0000000000000000 x24: ffff8000101cfa08
[ 1165.983598][  T472] x23: 0000000000000000 x22: ffff00007dbd0080
[ 1165.989603][  T472] x21: fffffe0001504c80 x20: 0000000000000001
[ 1165.995607][  T472] x19: 0000000000000000 x18: 0000000000000000
[ 1166.001611][  T472] x17: 0000000000000000 x16: 0000000000000000
[ 1166.007615][  T472] x15: 0000000000000000 x14: 0000000000000000
[ 1166.013619][  T472] x13: 0000000000000000 x12: 0000000000000000
[ 1166.019624][  T472] x11: 0000000000000000 x10: ffff000079647000
[ 1166.025628][  T472] x9 : fffffe0001505000 x8 : 00000000000757f9
[ 1166.031632][  T472] x7 : ffff80006c8ee000 x6 : 0000000000000000
[ 1166.037636][  T472] x5 : ffff80006c8ee000 x4 : ffff000078038000
[ 1166.043640][  T472] x3 : 24333657737ac24f x2 : dd3e03351d5943a4
[ 1166.049645][  T472] x1 : 0000000000000000 x0 : 24333657737ac267
[ 1166.055649][  T472] Call trace:
[ 1166.058790][  T472]  mem_cgroup_page_lruvec+0x24/0x58
[ 1166.063840][  T472]  __lru_cache_add+0x88/0x108
[ 1166.068368][  T472]  lru_cache_add+0xc/0x18
[ 1166.072552][  T472]  add_to_page_cache_lru+0x7c/0xe8
[ 1166.077514][  T472]  pagecache_get_page+0x194/0x300
[ 1166.082390][  T472]  grab_cache_page_write_begin+0x24/0x40
[ 1166.087877][  T472]  block_write_begin+0x30/0xd8
[ 1166.092492][  T472]  blkdev_write_begin+0x28/0x30
[ 1166.097194][  T472]  generic_perform_write+0xa8/0x1c0
[ 1166.102243][  T472]  __generic_file_write_iter+0x12c/0x1b0
[ 1166.107726][  T472]  blkdev_write_iter+0xa0/0x138
[ 1166.112428][  T472]  new_sync_write+0xe4/0x180
[ 1166.116870][  T472]  __vfs_write+0x2c/0x40
[ 1166.120964][  T472]  vfs_write+0xdc/0x1c8
[ 1166.124971][  T472]  ksys_write+0x68/0xf0
[ 1166.128978][  T472]  __arm64_sys_write+0x18/0x20
[ 1166.133597][  T472]  el0_svc_handler+0x7c/0x1a8
[ 1166.138126][  T472]  el0_svc+0x8/0xc
[ 1166.141704][  T472] Code: f100007f d37efc40 9a830083 8b000c60 (f9457c00)
[ 1166.148490][  T472] ---[ end trace 3651b8efc91af0bc ]---
[ 1166.153799][  T472] Kernel panic - not syncing: Fatal exception
[ 1166.159718][  T472] Kernel Offset: disabled
[ 1166.163900][  T472] CPU features: 0x0002,20002008
[ 1166.168600][  T472] Memory Limit: none
[ 1166.172352][  T472] Rebooting in 10 seconds..
5

Hi,

I found the problem. For those who may find themselves here and are using an NXP i.MX based board it seems that the memory node in the the kernel’s device tree (which was correct in my setup) is fixed up by U-Boot and I had the incorrect memory configuration in U-Boot which meant the kernel then thought it had more memory than it did. The Mender Client was the first application to discover this, due to probably memory mapping large areas for the Mender update file :slight_smile: I think the reason NXP do it this way is that the Arm Trusted Firmware reserves some of the memory for TEE if configured so it makes sense to fixup the device tree so the kernel doesn’t try to use this.

1 Like
6

That’s great that you found the solution. Thanks for reporting the details for others.

Drew

7

Nice investigation