Initial Mender server update issues for Arria10 target

Hi,

I have been attempting to get my arria10 iWave board to work with the server.

As can be seen in the following picture I have successfully managed to get my board to register with our demo server:

However I am finding that I am unable to get the server to download a new artefact completely, it gets stuck at 69%. I’m forced to abort the deployment and when rebooting the target device it goes into a constant reset after booting the Mender OTA service.

I think this is a probably due to a target configuration problem due to the partition layout that I need to use & the setup of fw_printenv.

Target SD partition: (seen from card reader on server)

Device Boot Start End Blocks Id System
/dev/sdf1 2048 1026048 512000+ b W95 FAT32
/dev/sdf2 1026049 4098049 1536000+ 83 Linux
/dev/sdf3 4098050 4118530 10240+ a2 Unknown
/dev/sdf4 4120576 30449663 13164544 5 Extended
/dev/sdf5 4122624 7194623 1536000 83 Linux
/dev/sdf6 7196672 7458815 131072 b W95 FAT32

Sectors Type Description
0 - 511 Raw Bootloader
512 – 2047 Raw Environment Variables
2048 – 1026048 FAT32 Boot Partition
1026049 – 4098049 Ext3 Linux RootFS Bank 1
4098050 – 4118530 Raw U-boot Binary Image
4122624 - 7194623 Ext3 Linux RootFS Bank 2
7196672 - 7458815 FAT32 Mender Data Partition

The Bootloader is supplied by Intel for the Arria10. It requires that there is an FPGA image in the Boot FAT32 partition (if this is not there the bootloader will hang), once this is loaded it runs the u-boot binary located at sector 4098050. This is why I have created the second Linux RootFS partition and Data partition after the current structure of the SD card.
I have tried to reflect this in Mender variables that I added to local.conf:

############################################################
# Mender Changes

# The name of the disk image and Artifact that will be built.
# This is what the device will report that it is running, and different updates must have different names
# because Mender will skip installation of an Artifact if it is already installed.
MENDER_ARTIFACT_NAME = “release-2”

INHERIT += “mender-full”

# A MACHINE integrated with Mender.
MACHINE = “iwg24m”

MENDER_FEATURES_ENABLE_append = " mender-uboot mender-image-sd"
MENDER_FEATURES_DISABLE_append = " mender-grub mender-image-uefi"

MENDER_STORAGE_DEVICE = “/dev/mmcblk0”
MENDER_STORAGE_DEVICE_BASE = “${MENDER_STORAGE_DEVICE}p”

# Memory card with 16GiB of storage.
MENDER_STORAGE_TOTAL_SIZE_MB = “16384”
MENDER_BOOT_PART_SIZE_MB = “500”
MENDER_DATA_PART_SIZE_MB = “128”

MENDER_BOOT_PART = “{MENDER_STORAGE_DEVICE_BASE}1"* *MENDER_DATA_PART = "{MENDER_STORAGE_DEVICE_BASE}6”
MENDER_ROOTFS_PART_A = “{MENDER_STORAGE_DEVICE_BASE}2"* *MENDER_ROOTFS_PART_B = "{MENDER_STORAGE_DEVICE_BASE}5”

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = “systemd”
DISTRO_FEATURES_BACKFILL_CONSIDERED = “sysvinit”
VIRTUAL-RUNTIME_initscripts = “”

ARTIFACTIMG_FSTYPE = “ext4”

As my bootloader is not physically located in a partition, I set this variable to the FAT32 partition that is used by the bootloader. Will this cause an issue?

I do see these parameters reflected in my u-boot environment variable and my mender.conf in the rootfs:

root@iwg24m:/etc/mender# cat mender.conf
{

  • “InventoryPollIntervalSeconds”: 1800,*
  • “RetryPollIntervalSeconds”: 300,*
  • “RootfsPartA”: “/dev/mmcblk0p2”,*
  • “RootfsPartB”: “/dev/mmcblk0p5”,*
  • “ServerURL”: “https://docker.mender.io”,*
  • “ServerCertificate”: “/etc/mender/server.crt”,*
  • “UpdatePollIntervalSeconds”: 1800*
    }

However what concerns me is that the partition layout that is generated from my Yocto build (Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.sdimg) is as follows:

   *Start               End                 Blocks      Id      System*
*49152           1073151         512000     c       W95 FAT32 (LBA)*
*1081344      17154047       8036352  83      Linux*
*17154048   33226751       8036352  83      Linux*
*33226752   33488895      131072     83      Linux*

This doesn’t seem to be based on the Mender variables.
Can I used my own partition layout which matches the Mender variables I have set, or will this cause an issue?

fw_printenv:

Running mender directly from the command line I see the following error:

root@iwg24m:~# /usr/bin/mender
ERRO[0000] Failed to read the current active partition: No match between boot and root partitions.: exit status 1 module=main
ERRO[0000] Must give one of -rootfs, -commit, -bootstrap or -daemon arguments module=main

Reading some forum posts this indicated that there was an error with fw_printenv, so I ran this command:

root@iwg24m:~# fw_printenv
Cannot parse config file: No such file or directory

I found that my environment variables were being written to an incorrect area of the SD card and corrected this by adding the following setting in my local.conf.

MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 = “0x4000”
MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 = “0x9D800”

I can now see that my environment variables are being written to the correct area of my SD card. When running setenv in u-boot the environment variables are being alternatively written to the main and redundant banks.

This however is not fixing my issues with fw_printenv.

I checked /etc/fw_env.conf and can see it’s a symbolic link:

0 lrwxrwxrwx 1 root root 26 Feb 28 12:52 fw_env.config -> /data/u-boot/fw_env.config

However I do not have a u-boot directory in /data.

How does this get created?

Thanks

Hi @GaryNewman I’m not following completely here but it sounds like you are using a UBoot image provided by Intel and not one with the Mender customizations. Am I understanding correctly?

Generally the details of the partition mapping are not a problem if they are standard partitions and both UBoot and Linux find them properly. The Uboot binary does not need to be in a partition; that is very dependent on the hardware architecture in question.

It looks like your UBoot environment is not set up properly to support Mender which would be the case if you are using a UBoot recipe/binary without the Mender bbclasses.

Similarly, the /data/u-boot directory is created by meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender.inc. If you are not building the U-Boot fw-utils with that include file in your recipe you won’t have that directory.

As a first step, I recommend having a look through https://docs.mender.io/devices/yocto-project/bootloader-support/u-boot/integration-checklist. That should help you narrow in on the specific issues with your setup. Assuming you want to use the Intel provided U-Boot without Mender build time integration, you may need to manually setup some things.

Drew

Hi Drew, thanks for your quick response.

I have a Yocto build which I have followed the integration steps for a Manual U-boot integration.

https://docs.mender.io/1.6/devices/integrating-with-u-boot/manual-u-boot-integration

I have mender integrated classes for both u-boot and u-boot-fw-utils.
I’ve tried to attach my bb files, but am not allowed.

I can see that both are being built and patched for Mender.

I don’t know much about u-boot-fw-utils, I’ve been assuming that this gets automatically included in the Yocto build.

I’ve just listed the build dir for fw-utils :

gnewman@server-1:~/yocto/iWave/iWave-mender-yocto/build/tmp-angstrom-glibc/work/iwg24m-angstrom-linux-gnueabi/u-boot-fw-utils-socfpga/v2014.10-r1$ ls

  • 4 drwxrwxr-x 23 gnewman gnewman 4096 Feb 4 11:20 .*
  • 4 drwxrwxr-x 3 gnewman gnewman 4096 Jan 20 10:27 …*
  • 4 -rw-rw-r-- 1 gnewman gnewman 594 Feb 20 2019 0001-Add-missing-header-which-fails-on-recent-GCC.patch*
    12 -rw-rw-r-- 1 gnewman gnewman 10297 Feb 20 2019 0002-Generic-boot-code-for-Mender.patch
  • 4 -rw-r–r-- 1 gnewman gnewman 656 Feb 20 2019 0002-Mender_socfpga_arria10.h.patch*
  • 4 -rw-rw-r-- 1 gnewman gnewman 1526 Feb 20 2019 0003-Integration-of-Mender-boot-code-into-U-Boot.patch*
  • 4 -rw-r–r-- 1 gnewman gnewman 1439 Feb 20 2019 0005-default-gcc.patch*
  • 4 -rw-rw-r-- 1 gnewman gnewman 247 Feb 26 2019 0008-Fix-env_flags.patch*
  • 4 -rw-r–r-- 1 gnewman gnewman 33 Feb 4 11:20 configure.sstate*
  • 4 drwxr-xr-x 3 gnewman gnewman 4096 Jan 20 11:42 data*
    12 -rw-rw-r-- 1 gnewman gnewman 10240 Feb 4 11:20 data.tar
  • 4 drwxr-xr-x 3 gnewman gnewman 4096 Feb 4 11:20 deploy-ipks*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 20 11:42 deploy-u-boot-fw-utils-socfpga*
  • 4 -rw-rw-r-- 1 gnewman gnewman 3307 Feb 19 2019 fix-build-error-under-gcc6.patch*
  • 4 -rw-r–r-- 1 gnewman gnewman 55 Feb 4 11:20 fw_env.config*
  • 4 drwxr-xr-x 24 gnewman gnewman 4096 Feb 4 11:20 git*
  • 4 drwxr-xr-x 4 gnewman gnewman 4096 Feb 4 11:20 image*
  • 4 drwxrwxr-x 3 gnewman gnewman 4096 Jan 20 10:28 license-destdir*
  • 4 drwxr-xr-x 4 gnewman gnewman 4096 Feb 4 11:20 package*
  • 4 drwxr-xr-x 8 gnewman gnewman 4096 Feb 4 11:20 packages-split*
    64 -rw-rw-r-- 1 gnewman gnewman 64857 Feb 19 2019 PATCH003-iW-PRFAZ-SC-01-R2.2-REL0.1-Linux4.9.78-UBoot_basic_customization.patch
  • 4 drwxr-xr-x 6 gnewman gnewman 4096 Feb 4 11:20 pkgdata*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Feb 4 11:20 pseudo*
  • 4 drwxrwxr-x 5 gnewman gnewman 4096 Feb 4 11:20 recipe-sysroot*
  • 4 drwxrwxr-x 10 gnewman gnewman 4096 Feb 4 09:03 recipe-sysroot-native*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 30 08:58 sstate-install-deploy*
  • 4 drwxr-xr-x 2 gnewman gnewman 4096 Jan 20 11:38 sstate-install-package*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 30 08:58 sstate-install-packagedata*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 30 08:58 sstate-install-package_qa*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 30 08:58 sstate-install-package_write_ipk*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 30 08:58 sstate-install-populate_lic*
  • 4 drwxrwxr-x 2 gnewman gnewman 4096 Jan 30 08:58 sstate-install-populate_sysroot*
  • 4 drwxr-xr-x 3 gnewman gnewman 4096 Feb 4 11:20 sysroot-destdir*
    32 drwxrwxr-x 2 gnewman gnewman 28672 Feb 4 11:20 temp

I can see the missing files that I was expecting to see in the rootfs.
I’ve been assuming that this would automatically be included in the rootfs produced by yocto:
Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.tar.gz

I’m guessing I’ve been using the wrong rootfs to include in my image.

Looking at my deploy dir, which file should I be using for my initial rootfs deployment?

179884 -rw-r–r-- 2 gnewman gnewman 184195584 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.cpio
268884 -rw-r–r-- 2 gnewman gnewman 8229224448 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.ext3
201384 -rw-r–r-- 2 gnewman gnewman 8229224448 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.ext4
177440 -rw-r–r-- 2 gnewman gnewman 181691716 Feb 4 11:21 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.jffs2
79392 -rw-rw-r-- 2 gnewman gnewman 81294848 Feb 4 11:21 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.mender
4 -rw-r–r-- 2 gnewman gnewman 2278 Feb 4 11:21 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.mender.bmap
12 -rw-r–r-- 2 gnewman gnewman 12008 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.rootfs.manifest
420800 -rw-r–r-- 2 gnewman gnewman 17146314752 Feb 4 11:21 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.sdimg
8 -rw-r–r-- 2 gnewman gnewman 7514 Feb 4 11:21 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.sdimg.bmap
69676 -rw-r–r-- 2 gnewman gnewman 71341474 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.tar.gz
56360 -rw-r–r-- 2 gnewman gnewman 57709348 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.tar.xz
140 -rw-r–r-- 1 gnewman gnewman 140504 Feb 4 11:20 Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.testdata.json
328 -rw-r–r-- 1 gnewman gnewman 134217728 Feb 4 11:20 data.ext4
12 -rw-rw-r-- 2 gnewman gnewman 10240 Feb 4 11:20 data.tar

Gary

The .sdimg file is the one that Mender is expecting. It is the multiple partition image needed for full rootfs based updates.

Hi,

SD Image Format:

The format I need mt SD card to be in is dictated by the Intel bootloader that is supplied for my board.
This mean I need the .sdimg to be in this format:

Enersyn_Partition

However the ‘Angstrom-gsrd-console-image-glibc-ipk-v2017.06-iwg24m.sdimg’ format produced by the Mender Yocto build is in a different format:

Mender_Partition

I am unable to use this as its in the wrong format and doesn’t mach the mender variables I am using.
(the Yocto build I integrated Mender onto did not produce a .sdimg)
If you know how to edit this i.e. change the partition mapping, I’d be happy to edit it.

fw_printenv:

I managed to add the fw_env.config file from the uboot_fw_utils build onto my rootfs (/data/u-boot/fw_env.config).

/dev/mmcblk0 0x4000 0x1000
/dev/mmcblk0 0x9D800 0x1000

fw_printenv is now working.

Starting mender from the command line (./mender) now show that it is accessing the environment variables:

INFO[0000] Mender running on partition: /dev/mmcblk0p2 module=main

However now I have made this change, I can see from top that on boot my Mender daemon task is now dying.

How do I get error data from the Mender daemon, I need an indication of whats wrong.
Is there an error log?

Ok I seem to have it registering with the server:

mender -daemon
INFO[0000] Mender running on partition: /dev/mmcblk0p2 module=main
INFO[0000] API Gateway certificate (in PEM format):
-----BEGIN CERTIFICATE-----
MIIBfTCCASOgAwIBAgIJAJOS76a0qWuZMAoGCCqGSM49BAMCMBsxGTAXBgNVBAMM
EGRvY2tlci5tZW5kZXIuaW8wHhcNMTYxMjE0MTk1MjQ2WhcNMjYxMjEyMTk1MjQ2
WjAbMRkwFwYDVQQDDBBkb2NrZXIubWVuZGVyLmlvMFkwEwYHKoZIzj0CAQYIKoZI
zj0DAQcDQgAE7AVYis6MWGPGQYU1/tlLEnskRifDIhvkRb8Y4nQPekRkLkiBYYT3
iJ46wHrnejbHaLstU9GRdKWOmOuU6HGdO6NQME4wHQYDVR0OBBYEFGOIU4q++Vz8
9HuT1jg9V+wFeJcyMB8GA1UdIwQYMBaAFGOIU4q++Vz89HuT1jg9V+wFeJcyMAwG
A1UdEwQFMAMBAf8wCgYIKoZIzj0EAwIDSAAwRQIhAPLnEeWPNeN7eDCEYRitBfyO
X1yf2kzOm4ohBE5GY9gzAiBCq7HOSkzQDkelmQCCCpGXf/UwYNgQJjSoeGfk0j1a
TQ==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIBhDCCASmgAwIBAgIJALQrf4QDot4IMAoGCCqGSM49BAMCMB4xHDAaBgNVBAMM
E3MzLmRvY2tlci5tZW5kZXIuaW8wHhcNMTYxMjE0MTk1MjQ2WhcNMjYxMjEyMTk1
MjQ2WjAeMRwwGgYDVQQDDBNzMy5kb2NrZXIubWVuZGVyLmlvMFkwEwYHKoZIzj0C
AQYIKoZIzj0DAQcDQgAEEc/Y3T+l3DvINePkpvVZORMIdHVs29jgsl48ia7z/NRX
HlKtKxVGJyFN5Y7sBZeLgBYH3F4Bo3KfmxI7ad0tI6NQME4wHQYDVR0OBBYEFIUm
cip00QZYpe4ULflbGNJan+Y9MB8GA1UdIwQYMBaAFIUmcip00QZYpe4ULflbGNJa
n+Y9MAwGA1UdEwQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIhANHij9VZBDHOUPaC
pFiagnWnYL2HBR72W1xTKQbrLLTXAiEAvpwA4HzSnGmLd3010+jqQuMRHArN5WaX
h0fy7niBbIQ=
-----END CERTIFICATE-----
module=client
INFO[0000] Issuer: [], Valid from: 2016-12-14 19:52:46 +0000 UTC, Valid to: 2026-12-12 19:52:46 +0000 UTC module=client
INFO[0000] State transition: init [none] -> init [none] module=mender
INFO[0000] State transition: init [none] -> idle [Idle] module=mender
INFO[0000] authorization data present and valid module=mender
INFO[0000] State transition: idle [Idle] -> check-wait [Idle] module=mender
INFO[0000] State transition: check-wait [Idle] -> inventory-update [Sync] module=mender
INFO[0000] State transition: inventory-update [Sync] -> check-wait [Idle] module=mender
INFO[0000] State transition: check-wait [Idle] -> update-check [Sync] module=mender
INFO[0000] State transition: update-check [Sync] -> check-wait [Idle] module=mender

Although the mender daemon seems to crash after boot, I can restart it from the cmd line & see it registering with the server.

I’ll start a new thread with my next set of questions.

Gary