Zephyr thread stack overflow: debugging with runtime monitoring

Thread stack overflow is a critical issue in real-time operating systems like Zephyr that can lead to unpredictable behavior, memory corruption, and system crashes. When a thread’s stack exceeds its allocated size, it overwrites adjacent memory regions, potentially corrupting other threads’ stacks, global variables, or kernel data structures. This article provides a comprehensive guide to detecting, diagnosing, and fixing stack overflow issues in Zephyr applications using runtime monitoring techniques.

Understanding Stack Overflow in Zephyr

In Zephyr, each thread is allocated a fixed-size stack area in memory. The stack is used for function call return addresses, local variables, and register saves during context switches. The kernel initializes the stack with a known pattern (0xaa) to enable stack usage monitoring.

A stack overflow occurs when:

• A function call chain becomes too deep (deep recursion or many nested function calls)
• Local variables consume excessive stack space (large arrays or structures)
• The initial stack allocation is insufficient for the thread’s actual usage

Note: On architectures with a dedicated interrupt stack (e.g., ARM Cortex-M using MSP in Handler Mode), ISRs do not consume the thread’s stack. Zephyr configures a separate interrupt stack via CONFIG_ISR_STACK_SIZE. However, on architectures without hardware-separated stacks, ISRs may share the current thread’s stack.

Zephyr provides several mechanisms to detect and prevent stack overflows, ranging from compile-time checks to runtime monitoring features.

Zephyr Thread Stack Memory Layout

Zephyr thread stacks grow downward from high memory addresses to low memory addresses. The stack base (highest address) is stored in the thread’s stack_info structure. The kernel tracks stack usage by checking the sentinel value at the stack end.

+------------------------------------------------------------------------+
|                   ZEPHYR THREAD STACK MEMORY LAYOUT                    |
|              Stack Grows Downward (High Addr -> Low Addr)              |
+------------------------------------------------------------------------+
|                              HIGH ADDRESS                              |
|                                                                        |
|                           Thread Stack Base                            |
|                           (Highest Address)                            |
|                          +------------------+                          |
|                          |   Thread Stack   |                          |
|                          |   Frame (RTOS)   |                          |
|                          | k_thread struct  |                          |
|                          +------------------+                          |
|                                                                        |
|                          +------------------+                          |
|                          |   Guard Region   |                          |
|                          | (MPU / Sentinel) |                          |
|                          +------------------+                          |
|                                                                        |
|                          +------------------+                          |
|                          |   Thread Local   |                          |
|                          |    Variables     |                          |
|                          |  & Call Frames   |                          |
|                          +------------------+                          |
|                                                                        |
|                          v STACK GROWS DOWN v                          |
|                                                                        |
|                          +------------------+                          |
|                          | (Stack Pointer)  |                          |
|                          |  Current SP -->  |                          |
|                          +------------------+                          |
|                                                                        |
|         Note: On ARM Cortex-M, ISRs use a separate                     |
|         interrupt stack (MSP), not the thread stack.                    |
|                                                                        |
|                     !!! !!! OVERFLOW ZONE !!! !!!                      |
|                          +------------------+                          |
|                          |     Overflow     |                          |
|                          |     corrupts     |                          |
|                          | guard / adjacent |                          |
|                          |      memory      |                          |
|                          +------------------+                          |
|                                                                        |
|                              LOW ADDRESS                               |
|                             (Stack Limit)                              |
|                                                                        |
|                    Detection: CONFIG_STACK_SENTINEL                    |
|                   checks guard pattern at stack end                    |
|                  Runtime: k_thread_stack_space_get()                   |
+------------------------------------------------------------------------+

Root Cause Analysis: Common Causes of Stack Overflow

1. Insufficient Stack Size Allocation: The most common cause is allocating too little stack space for a thread’s actual needs. Zephyr’s K_THREAD_STACK_SIZEOF macro helps calculate required size, but underestimation is frequent.

2. Recursive Function Calls: Deep recursion, especially in algorithms like tree traversals or mathematical computations, can quickly consume stack space.

3. Large Local Variables: Declaring large arrays or structures as local variables (e.g., int buffer[1024];) allocates them on the stack, potentially causing overflow.

4. Interrupt Stack Sizing: On architectures without a dedicated interrupt stack, ISRs may share the thread’s stack. Even on architectures with a separate ISR stack (ARM Cortex-M), an undersized interrupt stack (CONFIG_ISR_STACK_SIZE) can cause a separate overflow in Handler Mode.

5. Function Pointer Indirection: Complex callback chains or event-driven architectures can lead to unpredictable call depths.

Stack Sentinel (CONFIG_STACK_SENTINEL)

Zephyr’s stack sentinel feature places a known value (0xaaaaaaaa) at the end of the thread’s stack area. During context switches or stack inspections, the kernel verifies this sentinel value. If modified, it indicates a stack overflow.

To enable stack sentinel:

1. Add CONFIG_STACK_SENTINEL=y to your prj.conf
2. The sentinel is checked automatically during context switches (z_swap()), k_yield(), and when returning from non-nested interrupts

When an overflow is detected, Zephyr triggers a fatal error with the reason K_ERR_STACK_CHK_FAIL, providing the thread name and fault context.

Stack Checking (CONFIG_STACK_CHECKING)

For more granular detection, Zephyr offers stack checking that verifies stack bounds on every function call. This uses either hardware features (like ARM’s MSPLIM/PSPLIM registers) or software instrumentation.

To enable stack checking:

1. Add CONFIG_STACK_CHECKING=y to your prj.conf
2. Note that CONFIG_STACK_CHECKING uses software instrumentation (compiler support) and checks stack usage at runtime (e.g. during context switch). For zero-overhead hardware bounds checking on ARM Cortex-M, use CONFIG_HW_STACK_PROTECTION instead, which utilizes the MPU or MSPLIM/PSPLIM registers.

Stack checking provides immediate detection at the point of overflow but incurs higher runtime overhead (typically 5-15% performance impact).

Runtime Stack Usage Monitoring

Zephyr provides runtime APIs to monitor actual stack usage without enabling full checking:

#include <zephyr/kernel.h>

// Variable to store unused space
size_t unused;

// Get unused stack space (returns 0 on success)
if (k_thread_stack_space_get(&thread_obj, &unused) == 0) {
    // Calculate actual usage
    size_t usage = thread_obj.stack_info.size - unused;
    
    // Print stack usage information
    printk("Thread %s: %zu/%zu bytes used\n",
           k_thread_name_get(&thread_obj),
           usage,
           thread_obj.stack_info.size);
}

These functions allow applications to log stack usage periodically or during critical operations to identify trends before overflow occurs.

Solution Approaches

1. Increase Stack Size Allocation

The simplest solution is to increase the stack size when defining the thread:

#define MY_THREAD_STACK_SIZE 1024  // Increase from default 512 or 768

K_THREAD_STACK_DEFINE(my_stack_area, MY_THREAD_STACK_SIZE);
struct k_thread my_thread_data;

k_thread_create(&my_thread_data, my_stack_area,
                K_THREAD_STACK_SIZEOF(my_stack_area),
                my_thread_func, NULL, NULL, NULL,
                MY_THREAD_PRIORITY, 0, K_NO_WAIT);

Trade-offs: Increases RAM consumption. For systems with limited memory, this may not be feasible.

2. Optimize Stack Usage

Reduce stack consumption by:

• Moving large buffers to global/static memory or heap allocation
• Converting recursive algorithms to iterative implementations
• Passing large structures by pointer instead of by value
• Using memory pools for temporary large allocations

Example: Moving a large buffer to global scope:

// Before: Local buffer on stack
void process_data(void) {
    uint8_t buffer[512];  // Consumes 512 bytes of stack
    // ...
}

// After: Global buffer
static uint8_t processing_buffer[512];

void process_data(void) {
    // Use processing_buffer instead
    // No stack consumption for buffer
}

Trade-offs: Increases global memory usage and may reduce reentrancy or thread safety if not carefully managed.

3. Enable Stack Sentinel for Runtime Detection

As mentioned, stack sentinel provides low-overhead detection. Combine it with application-level monitoring:

#include <zephyr/kernel.h>
#include <zephyr/sys/printk.h>

static void thread_stack_monitor_cb(const struct k_thread *thread, void *user_data)
{
    size_t unused;
    if (k_thread_stack_space_get(thread, &unused) == 0) {
        size_t total = thread->stack_info.size;
        size_t used = total - unused;
        
        if (used > total * 0.8) {  // Warning at 80% usage
            printk("WARNING: Thread %s stack usage: %zu/%zu (%.1f%%)\n",
                   k_thread_name_get((struct k_thread *)thread), used, total,
                   (float)used * 100.0f / total);
        }
    }
}

void stack_monitor_thread(void *arg1, void *arg2, void *arg3)
{
    while (1) {
        k_sleep(K_SECONDS(10));  // Check every 10 seconds
        k_thread_foreach(thread_stack_monitor_cb, NULL);
    }
}

Trade-offs: Minimal performance impact (<1%) but only detects overflow at check intervals, not instantly.

4. Hardware-Assisted Stack Protection

On ARM Cortex-M7 and Cortex-M33 processors, Zephyr can use the Memory Protection Unit (MPU) to create a no-access guard region at the stack boundary. Any stack overflow triggers a MemManage fault.

To enable MPU-based stack protection:

1. Add CONFIG_HW_STACK_PROTECTION=y to prj.conf
2. Configure MPU regions appropriately

Trade-offs: Requires MPU support and consumes one MPU region, but provides zero-overhead detection with immediate fault generation.

Code Example: Stack Overflow Detection and Recovery

The following example demonstrates enabling stack sentinel, monitoring stack usage, and recovering from overflow conditions:

#include <zephyr/kernel.h>
#include <zephyr/sys/printk.h>
#include <zephyr/sys/reboot.h>

#define STACK_SIZE 512
#define MONITOR_INTERVAL K_SECONDS(5)

K_THREAD_STACK_DEFINE(stack_area, STACK_SIZE);
struct k_thread thread_data;

/* Thread that intentionally causes stack overflow */
void overflowing_thread(void *arg1, void *arg2, void *arg3)
{
    /* Large local array that will overflow the stack */
    uint8_t large_buffer[600];  /* Exceeds 512-byte stack */
    
    /* Use the buffer to prevent compiler optimization */
    for (int i = 0; i < 600; i++) {
        large_buffer[i] = (uint8_t)i;
    }
    
    /* This point may never be reached due to overflow */
    printk("Overflowing thread completed (should not happen)\n");
}

/* Monitor thread to check stack usage */
void monitor_thread(void *arg1, void *arg2, void *arg3)
{
    while (1) {
        k_sleep(MONITOR_INTERVAL);
        
        size_t unused;
        if (k_thread_stack_space_get(&thread_data, &unused) == 0) {
            size_t total = STACK_SIZE;
            size_t used = total - unused;
            
            printk("Monitor: Target thread stack %zu/%zu bytes used\n",
                   used, total);
                   
            if (used > total * 0.75) {
                printk("WARNING: Approaching stack limit!\n");
            }
        }
    }
}

int main(void)
{
    /* Stack sentinel is enabled globally via CONFIG_STACK_SENTINEL=y in prj.conf */
    
    /* Create the monitoring thread */
    k_thread_create(&thread_data, stack_area,
                    K_THREAD_STACK_SIZEOF(stack_area),
                    monitor_thread, NULL, NULL, NULL,
                    7, 0, K_NO_WAIT);
    
    /* Simulate work that might lead to overflow conditions */
    while (1) {
        k_sleep(K_SECONDS(1));
        /* Application logic here */
    }
}

When stack sentinel detects an overflow, Zephyr triggers a fatal error (K_ERR_STACK_CHK_FAIL). To handle this gracefully, applications can:

1. Override k_sys_fatal_error_handler() to implement custom recovery logic (e.g., logging to persistent storage and performing a warm reboot via sys_reboot(SYS_REBOOT_WARM))
2. Enable core dumps (CONFIG_DEBUG_COREDUMP) for post-mortem analysis
3. Use watchdog timers to reset the system if overflow causes lockup

Verification and Testing

Testing Stack Overflow Detection

To verify your stack overflow detection mechanisms:

1. Intentional Overflow Test: Create a thread with a known small stack size and allocate a local buffer larger than the stack.

2. Check for Expected Fault: With stack sentinel enabled, the system should trigger a fatal error with stack check failure.

3. Validate Monitoring: Ensure your stack usage monitoring reports increasing usage before overflow.

Using Zephyr’s Thread Analyzer

Zephyr includes a built-in Thread Analyzer module that reports runtime thread stack usage. Enable it with CONFIG_THREAD_ANALYZER=y in your prj.conf. It provides APIs like thread_analyzer_run() and thread_analyzer_print() to output per-thread stack statistics.

To view stack usage at build time, use the compiler’s static stack usage analysis:

west build -t puncover

Production Monitoring

In production devices:

1. Enable stack sentinel for low-overhead protection
2. Implement periodic stack usage logging to external storage or debug port
3. Set up alerts when stack usage exceeds warning thresholds (e.g., 75% of allocated size)
4. Correlate stack usage spikes with specific events or workloads

Summary

Thread stack overflow in Zephyr is a preventable issue through proper stack sizing, runtime monitoring, and Zephyr’s built-in detection features. Key takeaways:

1. Enable Stack Sentinel: Use CONFIG_STACK_SENTINEL for minimal-overhead runtime detection.
2. Monitor Stack Usage: Regularly check stack space utilization using k_thread_stack_space_get().
3. Optimize Stack Consumption: Move large data structures off the stack and avoid deep recursion.
4. Consider Hardware Protection: Use MPU-based stack protection on supported CPUs for zero-overhead detection.
5. Test Thoroughly: Validate detection mechanisms with intentional overflow tests before deployment.

By combining these techniques, developers can create robust Zephyr applications that gracefully handle stack constraints and maintain system reliability in embedded systems.

Related Reading

• Zephyr RTOS Getting Started Guide
• Fixing UART DMA Overrun Errors on STM32
• Optimizing Cortex-M Tail Chaining for Sub-Microsecond Latency
• Zephyr Memory Protection MPU Configuration

References

1. Zephyr Project Documentation. “Threads.” https://docs.zephyrproject.org/latest/kernel/services/threads/index.html
2. ARM Limited. “ARMv7-M Architecture Reference Manual.” 2020.
3. Zephyr Project Documentation. “Thread Analyzer.” https://docs.zephyrproject.org/latest/services/debugging/thread_analyzer.html
4. J. Regehr. “Understanding Stack Overflow.” ACM SIGPLAN Notices, vol. 42, no. 6, 2007.
5. Zephyr Project Documentation. “Fatal Errors.” https://docs.zephyrproject.org/latest/kernel/services/other/fatal.html
6. STMicroelectronics. “STM32F4xx Reference Manual.” RM0090, 2022.
7. M. Barr. “Programming Embedded Systems in C and C++.” O’Reilly Media, 2006.