When debugging embedded systems, few things are more frustrating than a crash that leaves no trace. By default, Yocto images using systemd store logs in volatile memory and don’t capture coredumps at all. After a reboot, the evidence is gone.
This tutorial shows how to configure persistent storage for both journald logs and coredumps. We use a direct file-based approach for coredumps that reliably captures crashes even during system shutdown - a scenario where the standard systemd-coredump service may already be stopped.
Step 0: Prerequisites
Before we start, make sure you have:
- A working Yocto build environment with systemd as the init system
- A custom meta layer where we can add our modifications
- Basic familiarity with bbappend files
If you don’t have a custom layer yet, create one using the standard bitbake command:
bitbake-layers create-layer meta-custom
bitbake-layers add-layer meta-custom
The bitbake-layers tool is available after initializing your build environment (e.g., sourcing oe-init-build-env). The create-layer verb generates the layer directory structure with a proper layer.conf, while add-layer registers the layer in your build’s bblayers.conf.
Step 1: Set up the directory structure
First, create the recipe directory structure in your meta layer:
cd meta-custom
mkdir -p recipes-core/systemd/systemd
Your layer should now look like:
meta-custom/
└── recipes-core/
└── systemd/
└── systemd/ # config files go here
Step 2: Create the configuration files
We need three configuration files. Let’s create them in the systemd/ directory.
journald-persistent.conf - enables persistent log storage:
[Journal]
Storage=persistent
SystemMaxUse=64M
This configures journald to store logs on disk rather than in volatile memory. The SystemMaxUse limit prevents logs from consuming too much storage - adjust this value based on your available space.
60-coredump-file.conf - configures direct file-based coredumps:
kernel.core_pattern=/var/lib/systemd/coredump/core.%e.%p.%t
The standard approach for coredumps in systemd is to pipe them through systemd-coredump. However, during system shutdown, the coredump socket may already be stopped when a service crashes. Writing coredumps directly to files is more reliable.
The filename prefix 60- is important: it’s higher than systemd’s default 50-coredump.conf, so our setting takes precedence.
The pattern variables are:
%e- executable name%p- process ID%t- timestamp
50-coredump-ulimit.conf - allows coredumps to be written:
* soft core unlimited
* hard core unlimited
By default, the core file size limit may be zero, which prevents coredumps from being written.
Step 3: Configure persistent /var/log
By default, Yocto makes /var/log volatile (stored on tmpfs). For journald persistence to work, we need to change this. Add to your local.conf or distro configuration:
VOLATILE_LOG_DIR = "no"
This ensures /var/log is on persistent storage, allowing journald to create and maintain /var/log/journal.
Step 4: Create the bbappend
Now create the file recipes-core/systemd/systemd_%.bbappend:
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://journald-persistent.conf"
SRC_URI += "file://60-coredump-file.conf"
SRC_URI += "file://50-coredump-ulimit.conf"
do_install:append() {
# Persistent journald logging
install -d ${D}${sysconfdir}/systemd/journald.conf.d
install -m 0644 ${WORKDIR}/journald-persistent.conf ${D}${sysconfdir}/systemd/journald.conf.d/
# Direct file-based coredumps
install -d ${D}${sysconfdir}/sysctl.d
install -m 0644 ${WORKDIR}/60-coredump-file.conf ${D}${sysconfdir}/sysctl.d/
# Ulimit configuration
install -d ${D}${sysconfdir}/security/limits.d
install -m 0644 ${WORKDIR}/50-coredump-ulimit.conf ${D}${sysconfdir}/security/limits.d/
}
FILES:${PN} += "${sysconfdir}/systemd/journald.conf.d/journald-persistent.conf"
FILES:${PN} += "${sysconfdir}/sysctl.d/60-coredump-file.conf"
FILES:${PN} += "${sysconfdir}/security/limits.d/50-coredump-ulimit.conf"
Note that we don’t create /var/log/journal or /var/lib/systemd/coredump in the recipe. In Yocto, /var is typically a symlink to volatile storage, and these directories should not be pre-created in the rootfs. systemd creates /var/log/journal at runtime when it detects persistent storage is configured, and the kernel creates the coredump directory as needed.
The FILESEXTRAPATHS line tells bitbake to look for files in the systemd/ subdirectory next to the bbappend. The do_install:append function installs our configuration files to their target locations, and the FILES entries ensure they get packaged.
Step 5: Build and verify
Build your image:
bitbake core-image-minimal
After booting the image, verify the configurations are active:
# Check the coredump pattern
cat /proc/sys/kernel/core_pattern
# Expected: /var/lib/systemd/coredump/core.%e.%p.%t
# Check ulimit
ulimit -c
# Expected: unlimited
# Check journal directory exists
ls -la /var/log/journal/
# Should show a machine-id directory
Step 6: Test with a synthetic crash
Let’s verify coredumps are captured correctly:
# Start a background process and crash it
sleep 1000 &
kill -SEGV $!
Check that a coredump was created:
ls -la /var/lib/systemd/coredump/
You should see a file like core.sleep.1234.1702654321.
To verify persistence, reboot the system and check that both the coredump and journal logs survived:
reboot
# ... after reboot ...
ls /var/lib/systemd/coredump/
journalctl -b -1 # logs from previous boot
Analyzing coredumps with gdb
To analyze a coredump, you need gdb (the GNU Debugger). Standard Yocto images don’t include it - you need to add it explicitly:
IMAGE_INSTALL:append = " gdb"
For meaningful backtraces, you also need debug symbols. Yocto provides the dbg-pkgs image feature which installs debug symbol packages for all installed software:
EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
Be aware that debug symbols significantly increase image size.
With gdb installed, analyze a coredump like this:
gdb /path/to/binary /var/lib/systemd/coredump/core.name.pid.timestamp
Essential gdb commands for coredump analysis:
bt- show backtrace (call stack at crash time)bt full- backtrace with local variablesinfo registers- show CPU register statelist- show source code around crash location (requires debug symbols)
For more details on debugging with Yocto, see the Yocto Project documentation:
Conclusion
We now have an image that preserves both system logs and coredumps across reboots. The direct file-based approach for coredumps ensures crashes are captured even during system shutdown, when the standard systemd-coredump service might no longer be available.
Note that coredumps captured this way won’t appear in coredumpctl - that tool works with systemd-coredump’s journal integration. Use gdb directly on the core files as shown above.
Version compatibility: Tested with Yocto Scarthgap (5.0) and systemd 255.