STM32 Zephyr Kernel Panic Debugging: Causes and Fixes

+------------------------------------------------------------------------+
|                   ZEPHYR THREAD STACK MEMORY LAYOUT                    |
|              Stack Grows Downward (High Addr -> Low Addr)              |
+------------------------------------------------------------------------+
|                              HIGH ADDRESS                              |
|                                                                        |
|                           Thread Stack Base                            |
|                           (Highest Address)                            |
|                         ┌────────────────────┐                         |
|                         │  Thread Stack      │                         |
|                         │  Frame (RTOS)      │                         |
|                         │  k_thread struct   │                         |
|                         └────────────────────┘                         |
|                                                                        |
|                         ┌────────────────────┐                         |
|                         │  Guard Region      │                         |
|                         │  (MPU / Sentinel)  │                         |
|                         └────────────────────┘                         |
|                                                                        |
|                         ┌────────────────────┐                         |
|                         │  Thread Local      │                         |
|                         │  Variables         │                         |
|                         │  & Call Frames     │                         |
|                         └────────────────────┘                         |
|                                                                        |
|                         ▼  STACK GROWS DOWN  ▼                         |
|                                                                        |
|                         ┌────────────────────┐                         |
|                         │  ISR Context       │                         |
|                         │  (if interrupt)    │                         |
|                         └────────────────────┘                         |
|                                                                        |
|                         ┌────────────────────┐                         |
|                         │  Nested ISR        │                         |
|                         │  Context           │                         |
|                         └────────────────────┘                         |
|                                                                        |
|                     !!! !!! OVERFLOW ZONE !!! !!!                      |
|                         ┌────────────────────┐                         |
|                         │  Overflow corrupts │                         |
|                         │  guard / adjacent  │                         |
|                         │  memory            │                         |
|                         └────────────────────┘                         |
|                                                                        |
|                              LOW ADDRESS                               |
|                             (Stack Limit)                              |
|                                                                        |
|                    Detection: CONFIG_STACK_SENTINEL                    |
|                   checks guard pattern at stack end                    |
|                  Runtime: k_thread_stack_space_get()                   |
+------------------------------------------------------------------------+

Kernel panics in Zephyr on STM32 microcontrollers can halt your embedded system unexpectedly, leaving you with a cryptic error message and a dead board. This guide walks you through diagnosing the root causes—from stack overflows to ISR missteps—and provides actionable fixes to restore stability.

Problem Statement: The Dreaded Kernel Panic

You’ve deployed your Zephyr-based STM32 application, and after running for hours or days, the system suddenly resets. The console output shows something like:

*** Fatal fault! ***
Current thread: 0x20001234 (ID: 0x1)
Faulting instruction address: 0x08001234
Error code: 0x0000000E

This is a kernel panic—a fault the Zephyr kernel cannot recover from. For senior firmware engineers, this translates to downtime, frustrated users, and urgent debugging sessions. Understanding why this happens and how to fix it is critical for reliable embedded systems.

Root Cause Analysis: Why Zephyr Panics

Zephyr’s kernel panics are typically triggered by hardware faults that the CPU traps and the kernel handles as fatal. Common causes on STM32 include:

1. Stack Overflow

When a thread’s stack grows beyond its allocated space, it corrupts adjacent memory, often leading to a hard fault. Zephyr provides stack sentinels (CONFIG_STACK_SENTINEL) to detect this, but without them, the overflow may go unnoticed until it corrupts critical data.

2. Null Pointer Dereference

Accessing memory via a null pointer (e.g., due to an uninitialized pointer or failed memory allocation) triggers a memory management fault. In ISRs or threaded code, this can happen if a peripheral driver returns an error that isn’t checked.

3. Invalid ISR Operations

Interrupt Service Routists must execute quickly and avoid blocking calls. Calling a logging function that takes a mutex, or performing a lengthy computation, can corrupt the kernel state if it interrupts a critical section.

4. Memory Protection Unit (MPU) Misconfiguration

The STM32’s MPU, when enabled, restricts memory access to prevent corruption. An incorrectly configured MPU region can cause a fault when legitimate code tries to access protected memory.

5. Stack Corruption from Buffer Overflows

Local arrays or structs on the stack that are written beyond their bounds (e.g., via sprintf without bounds checking) can overwrite return addresses or other thread state.

Solution Approaches: From Diagnosis to Fix

Enable Zephyr’s Debugging Features

Start by turning on build-time diagnostics:

• CONFIG_STACK_SENTINEL: Places a known value at the end of each stack thread; if altered, the kernel panics with a clear message.
• CONFIG_DEBUG_COREDUMP: Saves the state of the faulting thread to memory for later analysis.
• CONFIG_LOG: Enables logging to capture events leading up to the panic.
• CONFIG_ASSERT: Includes runtime checks that can catch invalid parameters early.

Add these to your prj.conf:

CONFIG_STACK_SENTINEL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_LOG=y
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_ASSERT=y

Analyze the Crash Dump

When a panic occurs with core dump enabled, Zephyr stores the faulting thread’s context. You can retrieve it via a debugger or by implementing a custom coredump reader. Look for:

• The faulting instruction address (program counter)
• The error code (from the CPU’s CFSR register)
• The stack pointer and stack limit
• The list of active threads

Use the error code to determine the fault type:

• 0x00000001: Memory management fault (MMFAR valid)
• 0x00000002: Bus fault (BFAR valid)
• 0x00000003: Usage fault (e.g., undefined instruction, unaligned access)
• 0x0000000E: Hard fault (catch-all for escalated faults)

Increase Stack Sizes

If stack sentinel triggers, increase the stack size for the offending thread. In Zephyr, define stack size in the thread definition:

K_THREAD_STACK_DEFINE(my_stack, 2048);
K_THREAD_DEFINE(my_tid, my_stack, K_THREAD_STACK_SIZEOF(my_stack), my_thread, NULL, NULL, NULL, MY_PRIORITY, 0, K_NO_WAIT);

Monitor stack usage with k_thread_stack_space_used() or via the shell plugin.

Validate ISR Safety

Ensure ISRs:

• Execute in under 10-20 microseconds (depending on your tick rate)
• Never call blocking APIs (e.g., k_sleep, k_mutex_lock)
• Use lightweight logging (e.g., LOG_DBG with deferred work)
• Clear interrupt flags promptly

Example of a safe ISR:

void gpio_callback(const struct device *dev, struct gpio_callback *cb, uint32_t pins)
{
    // Set a flag for the main thread to handle
    atomic_set_bit(&event_flags, GPIO_EVENT);
    // Exit quickly
}

Configure the MPU Correctly

If using the MPU, ensure regions cover:

• Code (flash) as read-only
• Data (RAM) as read-write
• Stack regions as no-execute (if supported)
• Peripheral regions as device memory

Use the STM32CubeMX tool or manually configure the MPU in Zephyr via CONFIG_ARM_MPU and the mpu_config API.

Check Peripheral Driver Usage

STM32 HAL drivers in Zephyr can return error codes. Always check them:

if (huart->gState != HAL_UART_STATE_READY) {
    // Handle error, do not proceed
}

Ignoring these can lead to undefined behavior, especially in DMA or interrupt modes.

Complete Code Examples

Example 1: Enabling Stack Sentinel and Logging

In prj.conf:

CONFIG_BOARD_NRF52840_PCA10056=y
CONFIG_STACK_SENTINEL=y
CONFIG_DEBUG_COREDUMP=y
CONFIG_LOG=y
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_ASSERT=y
CONFIG_SHELL=y

Example 2: Safe GPIO Interrupt Handler

#include <zephyr/kernel.h>
#include <zephyr/device.h>
#include <zephyr/drivers/gpio.h>

static struct gpio_callback gpio_cb;

void gpio_callback(const struct device *dev, struct gpio_callback *cb, uint32_t pins)
{
    // Post a semaphore or set an event for the worker thread
    k_sem_give(&gpio_sem);
}

void main(void)
{
    const struct device *gpio_dev = DEVICE_DT_GET(DT_ALIAS(gpio0));
    if (!device_is_ready(gpio_dev)) {
        return;
    }

    k_sem_init(&gpio_sem, 0, 1);

    gpio_pin_configure(gpio_dev, DT_GPIO_PIN(DT_ALIAS(gpio0), gpios), GPIO_INPUT);
    gpio_pin_interrupt_configure(gpio_dev, DT_GPIO_PIN(DT_ALIAS(gpio0), gpios),
                                 GPIO_INT_EDGE_TO_ACTIVE);

    gpio_init_callback(&gpio_cb, gpio_callback, DT_GPIO_PIN(DT_ALIAS(gpio0), gpios));
    gpio_add_callback(gpio_dev, &gpio_cb);

    while (true) {
        k_sem_take(&gpio_sem, K_FOREVER);
        // Process the event in thread context
        handle_gpio_event();
    }
}

Example 3: MPU Configuration for STM32

#include <zephyr/arm/mpu/arm_mpu.h>

static const struct arm_mpu_region mpu_regions[] = {
    /* Region 0: Flash (code) */
    MPU_REGION_ENTRY(\"FLASH_0\",\n                     DT_FLASH_ADDR,\n                     REGION_FLASH_ATTR(DT_FLASH_SIZE_K * 1024),\n                     REGION_EXECUTE_ATTR),

    /* Region 1: SRAM (data) */
    MPU_REGION_ENTRY(\"SRAM_0\",\n                     DT_SRAM_ADDR,\n                     REGION_RAM_ATTR(DT_SRAM_SIZE_K * 1024),\n                     REGION_NO_EXECUTE_ATTR),

    /* Region 2: Peripherals */
    MPU_REGION_ENTRY(\"PERIPHERAL\",\n                     0x40000000,\n                     REGION_DEVICE_ATTR(0x40000000 + 0x100000 - 1),\n                     0),
};

static const struct arm_mpu_config mpu_config = {
    .num_regions = ARRAY_SIZE(mpu_regions),
    .mpu_regions = mpu_regions,
};

void main(void)
{
    arm_mpu_configure(&mpu_config);
    // Rest of application
}

Verification and Testing Steps

Reproduce the Panic (Carefully!)

To test your fixes, you need a way to reliably trigger the panic. For stack overflow, create a thread with a tiny stack and a recursive function:

void overflow_thread(void *p1, void *p2, void *p3)
{
    char buffer[100];
    // Deliberately overflow
    for (int i = 0; i < 200; i++) {
        buffer[i] = 0xAA;
    }
    k_thread_abort(k_current_get());
}

Build without stack sentinel to see the panic, then add it and verify the kernel detects the overflow.

Validate Fixes

After applying a fix (e.g., increasing stack size), run your system under load for an extended period. Use:

• Stress testing: Exercise all peripherals and threads at peak load.
• Fault injection: Use tools like QEMU or hardware fault injectors to simulate bit flips.
• Code review: Ensure all ISRs are short and all peripheral calls check return values.

Monitor in Production

Deploy with logging enabled to a remote console or storage device. Watch for:

• Recurring panic messages (should be zero after fixes)
• Stack usage reports (if using shell plugin)
• Unexpected resets

Summary

Kernel panics in Zephyr on STM32 are diagnosable and fixable. By enabling kernel debugging features, analyzing fault contexts, increasing stack sizes, validating ISR safety, configuring the MPU correctly, and checking peripheral driver usage, you can eliminate these crashes. Remember to:

1. Turn on CONFIG_STACK_SENTINEL and CONFIG_DEBUG_COREDUMP for early detection.
2. Keep ISRs lightweight and defer work to threads.
3. Always check return values from peripheral APIs.
4. Test fixes under load and monitor in production.

With these practices, your Zephyr STM32 applications will run reliably, minimizing downtime and maximizing uptime.

Related Reading

• Zephyr Power Management: PM Subsystem Deep Dive
• Fixing Slow Boot Time on Embedded Linux
• RTOS Task Notifications vs. Queues

References

1. Zephyr Project, “Kernel Documentation,” https://docs.zephyrproject.org/latest/kernel/index.html
2. STMicroelectronics, “STM32F4xx Reference Manual (RM0090),” https://github.com/psas/gps-rf-board/blob/master/pubs/stm32f4/STM32F407-reference-manual-DM00031020.pdf
3. ARM Limited, “ARMv7-M Architecture Reference Manual (DDI 0403 E.e),” https://developer.arm.com/documentation/ddi0403/latest
4. Zephyr Project, “Thread Analyzer (Stack Analysis),” https://docs.zephyrproject.org/latest/services/debugging/thread-analyzer.html
5. Zephyr Project, “Memory Protection Unit (MPU),” https://docs.zephyrproject.org/latest/build/advanced/mpu.html
6. Zephyr Project, “Coredump,” https://docs.zephyrproject.org/latest/services/debugging/coredump.html