I am following the latest documentation “Building a Mender Debian Image” and it seems I’m running into problems every step of the way. I loaded a clean raspbian image on a raspberry pi, customized a few things, and followed this documentation to create a snapshot “golden image”, and now I’m trying to run it through mender-convert. Here’s a list of what I’ve worked through and where I am stuck now:
The “dd” command example has syntax that isn’t Mac friendly. Here is the command that I ended up having success with: dd if=<DEVICE> of=golden-image-1.img bs=1m conv=sync
The git clone command for mender-convert seems to check out the code in a “detached head” state, and trying to run ./docker-build fails. Changing to 2.0.x or master allows the command to work.
Running the actual docker-mender-convert command seems to work at first, and takes several minutes to eventually fail saying :
The “dd” command example has syntax that isn’t Mac friendly. Here is the command that I ended up having success with: dd if=<DEVICE> of=golden-image-1.img bs=1m conv=sync
I do not really see what is the difference? That said, we do not test on Mac so might some quirks.
he git clone command for mender-convert seems to check out the code in a “detached head” state, and trying to run ./docker-build fails. Changing to 2.0.x or master allows the command to work.
Yes, this was an issue with an Docker baseimage being removed which was fixed in this commit. This is only available in the 2.0.x branch and occurred after the 2.0.0 tag which is referenced in the docs.
I’m assuming the docker container doesn’t support hfsplus? Is there a simple switch that needs to be added to avoid this?
Never tested mender-convert on hfsplus. But are you using hfsplus on the golden image? What is the target device that would support that?
I do not really see what is the difference? That said, we do not test on Mac so might some quirks.
Fair enough, consider this your first Mac test then, and perhaps add a note in the docs. The differences are “bs=1M” has to be a lowercase ‘m’, and conv=sync instead conv=fdatasync.
Never tested mender-convert on hfsplus . But are you using hfsplus on the golden image? What is the target device that would support that?
The target is for a raspberrypi 3b+. I’m not sure how or why the hfsplus error is occurring, perhaps it is another effect of using a Mac. I followed the docs.mender.io instructions on “Building a Mender Debian image”, and I’m reporting the results. I initially started using an Ubuntu VM as my host, but I kept running into Virtual Box related problems, so I decided to just use my macbook. I started with an official raspbian-lite image, booted the pi, made some changes, and powered it down and ran the ‘dd’ command to dump the new version of the image.
My guess is somehow the hfsplus error I am seeing is Mac specific. Perhaps I need another switch in the “dd” command to avoid the hfsplus problem.
I suspect that is true (about hfsprogs not being installed), although the rpi image that I made with dd probably shouldn’t have an hfsplus partition to begin with.
As a sanity check, I ran mender-convert (from macosx) on the stock rasbian-lite image, and it finished successfully on the first try.
Next, I’m running dd again on the sdcard to make sure I didn’t do anything stupid the first time. It is taking longer than I remember, which already leads me to think perhaps I closed the terminal window accidentally, or even worse picked the wrong drive as my input. I will report back with the results, thanks again for the replies.
I could be wrong, but I don’t think this is to do with your partitions in your dd image, it looks more like an error from within the docker build environment where its trying to bind-mount the rootfs it extracted from your image so it can use it to overlay/finalize it before adding it to the newly created mender disk image.
Because you are on a mac the extracted rootfs directory is backed by your computers hfsplus internal drive.
Note that conv=sync has a completely different meaning from conv=fdatasync, and is irrelevant in this context. The latter is only there to ensure that the whole cache is written before the command returns. If you instead just execute sync manually after the dd, it’s better to just omit conv altogether.
So the latest attempt was basically a redo, just to try and reproduce the hfs+ problem. I’m fairly certain something was different this time, as I ended up with a 25GB image from the dd command (sd card is 32 GB). My first attempt produced a 1.7GB image, which is the same as the stock raspbian image. I can’t explain what is different, I’ve lost the history from the terminal…perhaps it was human error.
Either way, I attempted to run mender-convert on the 25GB image, and threw a different error this time.
(base) mender-convert % MENDER_ARTIFACT_NAME=release-2 ./docker-mender-convert \
--disk-image input/golden-image-2.img \
--config configs/raspberrypi3_config \
--overlay rootfs_overlay_demo/
Running mender-convert --disk-image input/golden-image-2.img --config configs/raspberrypi3_config --overlay rootfs_overlay_demo/
Running mender-convert-extract: --disk-image input/golden-image-2.img --config configs/raspberrypi3_config --overlay rootfs_overlay_demo/
2020-05-26 20:47:57 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2020-05-26 20:47:57 [INFO] [mender-convert-extract] Using configuration file: configs/raspberrypi3_config
2020-05-26 20:47:57 [INFO] [mender-convert-extract] Validating disk image
2020-05-26 20:47:57 [INFO] [mender-convert-extract] Disk parsed successfully
2020-05-26 20:47:57 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: dos
2020-05-26 20:47:57 [INFO] [mender-convert-extract] PART 1: SIZE: 256M TYPE: 0xc
2020-05-26 20:47:57 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2020-05-26 20:48:37 [INFO] [mender-convert-extract] PART 2: SIZE: 29.5G TYPE: 0x83
2020-05-26 20:48:37 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
mender-convert-extract has finished. Cleaning up...
Running mender-convert-modify: --disk-image input/golden-image-2.img --config configs/raspberrypi3_config --overlay rootfs_overlay_demo/
2020-05-26 21:16:06 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2020-05-26 21:16:06 [INFO] [mender-convert-modify] Using configuration file: configs/raspberrypi3_config
mount: /mender-convert/work/rootfs: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.
mender-convert-modify has finished. Cleaning up...
umount: work/rootfs: not mounted.
2020-05-26 21:16:06 [ERROR] [mender-convert] mender-convert failed
2020-05-26 21:16:06 [DEBUG] [mender-convert-extract] When running: (modules/disk.sh:61): run_and_log_cmd():
dd if=input/golden-image-2.img of=work/part-2.fs skip=532480 bs=512 count=61801472 conv=sparse status=none
2020-05-26 21:16:06 [ERROR] [mender-convert] mender-convert exit code: 32
Interesting the wording of the error is different this time but tring to mount the same drectory, however I have seen this error before when I’m trying to manually mount disk images using loopback devices, and i was because i hadnt installed a driver to handle the file system type i was trying to mount. you could probably try execing into the docker image. I would be looking into how to exec into and instance of the mender docker container and install hfsprogs in it and commit the changes back to the image some how and rerun. (I know very little about docker so the actual how to do this can most likely be answered by the mender team)
Good news, the latest test to remove the conv parameter from the dd command seems to have worked! Thanks @kacf!
The mender-convert process finished with no errors. I uploaded the .mender file that was created to my demo mender server, and I’m deploying it to a raspberry pi now. Fingers crossed that it works, although I think the thread could be closed.
