NanoPi Neo2 Porting Mender to Openwrt

Now I am just making stuff up :), but would this work,

config global automount
        option from_fstab       1
        option anon_mount       1

config global autoswap
        option from_fstab       1
        option anon_swap        0

config mount
        option target           '/'
        option options          'rw'
        option device           '/dev/root'

config mount
        option target           '/data'
        option options          'rw'
        option device           '/dev/mmcblk0p4'

config mount
        option target           '/mnt'
        option options          'rw'
        option device           '/dev/mmcblk0p2'
        option	                enabled	'0'

config mount
        option target           '/mnt'
        option options          'rw'
        option device           '/dev/mmcblk0p3'
        option	                enabled	'0'

been there, done that. Appears kernel is balking on /etc/fstab, long B4 openwrt mount proc gets to /etc/config/fstab

gonna try this /etc/fstab:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/root       /               auto    defaults        1       1
/dev/mmcblk0p1  /mnt/mmcblk0p1  auto    defaults,noauto 0       0
/dev/mmcblk0p2  /mnt/mmcblk0p2  auto    defaults,noauto 0       0
/dev/mmcblk0p3  /mnt/mmcblk0p3  auto    defaults,noauto 0       0
/dev/mmcblk0p4  /mnt/mmcblk0p4  auto    defaults        0       0

blacklist by noauto option

boots with above /etc/fstab. No “Invalid” in log. Have to add file, re-image, re-create artifacts, try again

also fixed /dev/mmcblk0p4 to mount on /data

Update failures have nothing to do with mount errors. Am running afoul of bootlimit (=1), and a crucial missing piece in mender docs: linux mainline MUST clear bootcount to signal successful boot, for u-boot, next boot, else:

image

I now do “fw_setenv bootcount 0” in rc.local and verify it is actually cleared.

but, next boot uboot: bootcount=2, Warning: Bootlimit (1) exceeded. Using altbootcmd, which swaps boot partition, effectively reverting update.

The desired event chain breaks between “fw_setenv bootcount 0” in rc.local and u-boot seeing bootcount as 2, despite clearing in linux mainline.

Any idea what’s going on?

Regards;
Bill

This is done by the Mender client when it marks the update as successful or when you run mender -commit if you are running in stand-alone mode. Manually clearing this is a hack and not recommended as it will interfere with the Mender state-machine.

It is alos mentioned in the docs for the stand-alone mode,

https://docs.mender.io/1.7/architecture/standalone-deployments#make-the-deployment-permanent

But this is handled “auto magically” when running the Mender client in daemon mode, and if the daemon is not doing it something has failed during the update

Ah; my high level boot sequence is issue:

firstboot: setup N/W, package repo, create dirs, links, clear bootcount, re-boot
secondboot: install apps (mender included, start it), configure generic openwrt to our dist, clear boot count, reboot
third+ boot: Be our dist

So, key is to NOT re-boot until mender installed, started and has time to clear bootcount (how long, done during mender init?, don’t have to wait?)

Sill doesn’t explain why my “hack” fails.

Regards;
Bill

firstboot: setup N/W, package repo, create dirs, links, clear bootcount, re-boot
secondboot: install apps (mender included, start it), configure generic openwrt to our dist, clear boot count, reboot
third+ boot: Be our dist

Do you mean that you do this on each boot or that you do this after each Mender update?

If you are trying to do tree reboots while trying to do an Mender update, that will definitely fail. Mender only expects to reboot once and anything else will be considered as a failure and it will roll-back.

re-org’ing so only one boot. rc.local (at firstboot, whether initial flash or booting new rootfs) does all package installation, configuration then replaces itself with normal rc.local for subsequent boots

thus, mender running during first and subsequent boots, no reboots unless watchdog, power, or mender managed. Thus, do not have to worry about post-update timing issues as mender client completes handshake with server.

whohooo!

Fri Mar 29 16:50:18 2019 user.notice kernel: [ 5.281036] mount_root: mounting /dev/root

Fri Mar 29 16:50:18 2019 kern.info kernel: [ 5.287942] EXT4-fs (mmcblk0p3): re-mounted. Opts: (null)

First successful mender update from partition 2 to partition 3 -> our distribution running, all unattended, apart from scheduling update

Will take me at least a week to cleanup, document, get files ready for Mender. Perhaps a bit longer, since preparing for beta trials.

THANK YOU VERY MUCH for your patience and assistance in this.

Regards;
Bill

Looking forward to seeing the result of this very long thread :smiley:.

THANK YOU VERY MUCH for your patience and assistance in this.

Thank you as well for persisting.

strange problem: first boot startup (install mender client) of mender client terminates. Last /var/log/mender entry:

time=“2019-04-05T10:10:00Z” level=info msg=“device keys not present or bootstrap forced, generating” module=mender

after this mender exits. No message, no zombie. In addition, procd does not restart. Startup script /etc/init.d/mender:

#!/bin/sh /etc/rc.common

START=99
STOP=11
USE_PROCD=1

# If want to use syslog, remove log-file, re-direct stdout, stderr
ARGS="-config /etc/mender/mender.conf -log-file /var/log/mender.log -daemon"
#ARGS="-config /etc/mender/mender.conf -daemon"

boot()
{
        /bin/mkdir -p /var/lib
        /bin/mkdir -p -m 0700 /data/mender
        /bin/ln -sf /data/mender /var/lib/mender
        rc_procd start_service
}

start_service()
{
        pidof mender
        if [ $? -eq 0 ]; then
                return 0
        fi
        #while [ ! -f /var/log/LAST_BOOT ]; do sleep 1; done
        procd_open_instance
        procd_set_param command /usr/bin/mender $ARGS
        procd_set_param respawn 3600 60 0
        #procd_set_param stdout 1
        #procd_set_param stderr 1
        procd_set_param pidfile /var/run/mender.pid
        procd_close_instance
}

Thoughts?
Regards;
Bill

If not private key is present on the device it will generate a new one, which eventually ends up in /var/lib/mender/mender-agent.pem. This is what above log means and this can take a while, minutes if your device is “slowish”.

Are you sure that the process exists?

You can generate a key pair and provision the device with it (just put the private.key it at /var/lib/mender/mender-agent.pem ),

https://docs.mender.io/1.7/server-integration/preauthorizing-devices#generate-a-client-keypair

and this

https://docs.mender.io/1.7/server-integration/preauthorizing-devices#copy-generated-device-key-to-disk-image

Can I use a global private key for all devices?

No, the public key from the key pair is part of the device identity.

Since mender appears to behave thusly (post install)
first start: log: generating keys, exit (process no exist)
wait indeterminate time
second start (manual). Key appears after some time,mender keeps running

So I need a reliable firstboot startup algorithm. This does not work (sleeps forever):

/etc/init.d/mender enable; /etc/init.d/mender start

First startup of Mender will create device key and exit

echo “Creating mender client key” >> $LOGFILE
while [ ! -f /var/lib/mender/mender-agent.pem ]; do sleep 1; done
/etc/init.d/mender restart

End Mender stuff

I need this:
/etc/init.d/mender enable; /etc/init.d/mender start
sleep X or some trigger that next mender run will be successful and create key
/etc/init.d/mender restart

Then, will find a way to get key to server for preauth

…Bill

Given unreliable mender startup; solved (hope) by this:

create device key, your script
install and start mender (still stops, first run)
do a buncha time consuming package install / config
restart mender (this time, stays running)

I still get the same lastlog entry as above, mender first run, then stop, despite key existence

Regards;
Bill

I still get the same lastlog entry as above, mender first run, then stop, despite key existence

Hm, this is not something that we have seen before so hard to comment. Could be something procd related.

Can you try running the the client “manually” from a terminal on first boot, e.g

mender -daemon -debug

And if still exits without any clear error maybe try running strace on it.

Thanks @rossbcan and @mirzak for all your work, the discussion and insight this thread has provided. Its really helpful as addition to the formal documentation to the https://mender.io/blog/porting-mender-to-a-non-yocto-build-systemU post too. I am also trying to achieve something similar. I just wanted to ask if you had any update? Interested to hear after there seemed to be such momentum built!

Hi @beruti,

I just wanted to highlight that we have an updated post which is based on the linked blog post,

@rossbcan has gotten this integration working and there are plans to publish the integration here,

Great, thanks again @mirzak!