How to Monitor Device Storage with a Custom Mender Inventory Script

[TheYoctoJester]

Overview

When managing a fleet of IoT devices with Mender, understanding storage usage across your devices is crucial for maintaining operational health and planning updates. This tutorial shows you how to create a custom inventory script that reports storage information in a clean, human-readable format.

By the end of this tutorial, your Mender dashboard will show storage data like this for each device:

storage_root=484MB/1.6GB (28%)
storage_data=73KB/108GB (0%)
storage_uboot=48.5MB/99.7MB (48%)

Prerequisites

• Mender client installed and connected to your Mender server
• SSH access to your device
• Basic shell scripting knowledge

Understanding Mender Inventory Scripts

Mender inventory scripts are executable programs that collect device information and report it to the Mender server. They must:

1. Be located in /usr/share/mender/inventory/
2. Have names starting with mender-inventory-
3. Be executable (chmod +x)
4. Output data in key=value format, one entry per line
5. Exit with code 0 on success, non-zero on failure

Creating the Storage Inventory Script

The steps assume that you are logged in as root on the device. Alternatively you can create the script locally and then copy it to the device.

Step 1: create the script

cat >> /usr/share/mender/inventory/mender-inventory-storage << 'EOF'
#!/bin/sh
#
# Mender inventory script to collect storage information for devices.
# This script reports free and used storage for all mounted filesystems.
#
# The script needs to be located in $(datadir)/mender and its name shall start
# with `mender-inventory-` prefix. The script shall exit with non-0 status on
# errors. In this case the agent will discard any output the script may have
# produced.
#
# The script outputs inventory data in <key>=<value> format, one entry per line.
# Entries appearing multiple times will be joined in a list under the same key.
#
# Example output:
# storage_root=484MB/1.6GB (28%)
# storage_uboot=48.5MB/99.7MB (48%)
# storage_data=73KB/108GB (0%)
#

set -ue

# Function to convert bytes to human-readable format
bytes_to_human() {
    local bytes="$1"
    local units="B KB MB GB TB"
    local unit_index=0
    local size="$bytes"
    
    # Convert to appropriate unit
    while [ "$size" -ge 1024 ] && [ "$unit_index" -lt 4 ]; do
        size=$((size / 1024))
        unit_index=$((unit_index + 1))
    done
    
    # Get the unit name
    local unit=$(echo "$units" | cut -d' ' -f$((unit_index + 1)))
    
    # For values >= 1024, show one decimal place if meaningful
    if [ "$size" -ge 100 ] || [ "$unit_index" -eq 0 ]; then
        echo "${size}${unit}"
    else
        # Calculate with decimal for smaller values
        local decimal_size
        case "$unit_index" in
            1) decimal_size=$((bytes * 10 / 1024));;
            2) decimal_size=$((bytes * 10 / 1048576));;
            3) decimal_size=$((bytes * 10 / 1073741824));;
            4) decimal_size=$((bytes * 10 / 1099511627776));;
        esac
        local integer_part=$((decimal_size / 10))
        local decimal_part=$((decimal_size % 10))
        if [ "$decimal_part" -eq 0 ]; then
            echo "${integer_part}${unit}"
        else
            echo "${integer_part}.${decimal_part}${unit}"
        fi
    fi
}

# Check if df command is available
if ! command -v df >/dev/null 2>&1; then
    echo "Error: df command not found" >&2
    exit 1
fi

# Create temporary file for processing
temp_file=$(mktemp)
trap 'rm -f "$temp_file"' EXIT

# Get filesystem information using df with POSIX output
# Use -P flag for portable output format, -B1 for bytes
df -P -B1 > "$temp_file"

# Process each line from df output
while read filesystem blocks used available capacity mounted_on; do
    # Skip header line
    if [ "$filesystem" = "Filesystem" ]; then
        continue
    fi
    
    # Skip special filesystems (tmpfs, devtmpfs, proc, sys, etc.)
    case "$filesystem" in
        tmpfs|devtmpfs|udev|proc|sysfs|devpts|cgroup*|securityfs|debugfs|tracefs|fusectl|configfs|selinuxfs|mqueue|hugetlbfs|autofs|binfmt_misc|pstore|efivarfs|systemd-1|none)
            continue
            ;;
        /dev/loop*|/dev/ram*|/snap/*)
            continue
            ;;
    esac
    
    # Skip if mounted_on is empty or not a real mount point
    if [ -z "$mounted_on" ] || [ ! -d "$mounted_on" ]; then
        continue
    fi
    
    # Convert mount point to safe variable name (replace / and - with _)
    if [ "$mounted_on" = "/" ]; then
        safe_mount="root"
    else
        safe_mount=$(echo "$mounted_on" | sed 's|/||g; s|-|_|g')
    fi
    
    # Calculate percentage used (avoiding division by zero)
    if [ "$blocks" -gt 0 ]; then
        percent_used=$((used * 100 / blocks))
    else
        percent_used=0
    fi
    
    # Report single storage summary for this filesystem
    echo "storage_${safe_mount}=$(bytes_to_human "$used")/$(bytes_to_human "$blocks") (${percent_used}%)"
    
done < "$temp_file"
EOF

Step 2: Make the Script Executable

chmod +x /usr/share/mender/inventory/mender-inventory-storage

Step 3: Test the Script

Test your script locally to ensure it works correctly:

/usr/share/mender/inventory/mender-inventory-storage

You should see output similar to:

storage_root=484MB/1.6GB (28%)
storage_uboot=48.5MB/99.7MB (48%)
storage_data=73KB/108GB (0%)

Step 4: Trigger Inventory Update

Force the Mender client to send updated inventory data:

mender-update send-inventory

How It Works

Filesystem Detection

The script uses the df command with POSIX-compliant flags:

• -P: Ensures portable output format across different systems
• -B1: Reports sizes in bytes for precise calculations

Smart Filtering

The script automatically excludes:

• Virtual filesystems (tmpfs, proc, sysfs)
• Loop devices and snap packages
• Docker and container-related mounts

Human-Readable Output

The bytes_to_human() function dynamically converts byte values to appropriate units (B, KB, MB, GB, TB) with decimal precision when needed.

Naming Convention

• Root filesystem (/) becomes storage_root
• Other mounts strip leading slashes: /data becomes storage_data, /boot becomes storage_boot

Verification

After a few minutes, check your Mender dashboard or use the API to verify the storage data appears in your device inventory. You’ll see entries like:

• storage_root: 484MB/1.6GB (28%)
• storage_data: 73KB/108GB (0%)
• storage_uboot: 48.5MB/99.7MB (48%)

Benefits

This storage inventory script provides a starting point to collect custom data from your fleet, for use cases such as:

1. Fleet-wide visibility: Monitor storage usage across all devices from your Mender dashboard
2. Proactive maintenance: Identify devices approaching storage limits before they cause issues
3. Update planning: Ensure devices have sufficient space for OTA updates
4. Troubleshooting: Quickly identify storage-related problems across your fleet

Customization

You can modify the script to:

• Add alert thresholds (e.g., flag devices >90% full)
• Include specific filesystem types only
• Add additional metadata like filesystem type or mount options
• Change the output format to match your monitoring needs

Troubleshooting

If your script doesn’t appear in inventory:

1. Check permissions: Ensure the script is executable
2. Verify location: Script must be in /usr/share/mender/inventory/
3. Check naming: Filename must start with mender-inventory-
4. Test manually: Run the script directly to check for errors
5. Check logs: Look at Mender client logs for error messages

Conclusion

Custom inventory scripts are a powerful way to extend Mender’s device monitoring capabilities. This storage monitoring script gives you the visibility needed to maintain a healthy device fleet and prevent storage-related issues before they impact your deployments.

For more inventory script examples, check the Mender GitHub repository.