Zephyr thread stack overflow: debugging with runtime monitoring

Thread stack overflow is a critical issue in real-time operating systems like Zephyr that can lead to unpredictable behavior, memory corruption, and system crashes. When a thread’s stack exceeds its allocated size, it overwrites adjacent memory regions, potentially corrupting other threads’ stacks, global variables, or kernel data structures. This article provides a comprehensive guide to detecting, diagnosing, and fixing stack overflow issues in Zephyr applications using runtime monitoring techniques.

Understanding Stack Overflow in Zephyr

In Zephyr, each thread is allocated a fixed-size stack area in memory. The stack is used for function call return addresses, local variables, and register saves during context switches. The kernel initializes the stack with a known pattern (0xaa) to enable stack usage monitoring.

A stack overflow occurs when:

• A function call chain becomes too deep (deep recursion or many nested function calls)
• Local variables consume excessive stack space (large arrays or structures)
• Interrupt service routines (ISRs) use the thread’s stack and exceed its size
• The initial stack allocation is insufficient for the thread’s actual usage

Zephyr provides several mechanisms to detect and prevent stack overflows, ranging from compile-time checks to runtime monitoring features.

Zephyr Thread Stack Memory Layout

Zephyr thread stacks grow downward from high memory addresses to low memory addresses. The stack base (highest address) is stored in the thread’s stack_info structure. The kernel tracks stack usage by checking the sentinel value at the stack end.

+------------------------------------------------------------------------+
|                   ZEPHYR THREAD STACK MEMORY LAYOUT                    |
|              Stack Grows Downward (High Addr -> Low Addr)              |
+------------------------------------------------------------------------+
|                              HIGH ADDRESS                              |
|                                                                        |
|                           Thread Stack Base                            |
|                           (Highest Address)                            |
|                         +----------------+                             |
|                         |  Thread Stack   |                             |
|                         |  Frame (RTOS)    |                             |
|                         |  k_thread struct |                             |
|                         +----------------+                             |
|                                                                        |
|                         +----------------+                             |
|                         |  Guard Region   |                             |
|                         |  (MPU / Sentinel)|                            |
|                         +----------------+                             |
|                                                                        |
|                         +----------------+                             |
|                         |  Thread Local   |                             |
|                         |  Variables      |                             |
|                         |  & Call Frames  |                             |
|                         +----------------+                             |
|                                                                        |
|                         v  STACK GROWS DOWN  v                         |
|                                                                        |
|                         +----------------+                             |
|                         |  ISR Context    |                             |
|                         |  (if interrupt) |                             |
|                         +----------------+                             |
|                                                                        |
|                         +----------------+                             |
|                         |  Nested ISR     |                             |
|                         |  Context        |                             |
|                         +----------------+                             |
|                                                                        |
|                     !!! !!! OVERFLOW ZONE !!! !!!                      |
|                         +----------------+                             |
|                         |  Overflow       |                             |
|                         |  corrupts       |                             |
|                         |  guard / adjacent|                            |
|                         |  memory         |                             |
|                         +----------------+                             |
|                                                                        |
|                              LOW ADDRESS                               |
|                             (Stack Limit)                              |
|                                                                        |
|                    Detection: CONFIG_STACK_SENTINEL                    |
|                   checks guard pattern at stack end                    |
|                  Runtime: k_thread_stack_space_get()                   |
+------------------------------------------------------------------------+

Root Cause Analysis

Common Causes of Stack Overflow

1. Insufficient Stack Size Allocation: The most common cause is allocating too little stack space for a thread’s actual needs. Zephyr’s K_THREAD_STACK_SIZEOF macro helps calculate required size, but underestimation is frequent.

2. Recursive Function Calls: Deep recursion, especially in algorithms like tree traversals or mathematical computations, can quickly consume stack space.

3. Large Local Variables: Declaring large arrays or structures as local variables (e.g., int buffer[1024];) allocates them on the stack, potentially causing overflow.

4. Interrupt Service Routines: ISRs in Zephyr typically use the stack of the interrupted thread. If an ISR uses significant stack space and occurs when the thread’s stack is nearly full, overflow can occur.

5. Function Pointer Indirection: Complex callback chains or event-driven architectures can lead to unpredictable call depths.

Zephyr Stack Memory Layout

Zephyr thread stacks grow downward from high memory addresses to low memory addresses. The stack base (highest address) is stored in the thread’s stack_info structure. The kernel tracks stack usage by checking the sentinel value at the stack end.

Stack Sentinel (CONFIG_STACK_SENTINEL)

Zephyr’s stack sentinel feature places a known value (0xaaaaaaaa) at the end of the thread’s stack area. During context switches or stack inspections, the kernel verifies this sentinel value. If modified, it indicates a stack overflow.

To enable stack sentinel:

1. Add CONFIG_STACK_SENTINEL=y to your prj.conf
2. Optionally adjust CONFIG_STACK_SENTINEL_CHECK_INTERVAL for check frequency

When an overflow is detected, Zephyr triggers a fatal error with the reason K_ERR_STACK_CHK_FAIL, providing the thread name and fault context.

Stack Checking (CONFIG_STACK_CHECKING)

For more granular detection, Zephyr offers stack checking that verifies stack bounds on every function call. This uses either hardware features (like ARM’s MSPLIM/PSPLIM registers) or software instrumentation.

To enable stack checking:

1. Add CONFIG_STACK_CHECKING=y to your prj.conf
2. Select the checking method: CONFIG_STACK_CHECKING uses software, while CONFIG_STACK_CHECKING with CONFIG_ARM uses hardware bounds registers on ARM Cortex-M

Stack checking provides immediate detection at the point of overflow but incurs higher runtime overhead (typically 5-15% performance impact).

Runtime Stack Usage Monitoring

Zephyr provides runtime APIs to monitor actual stack usage without enabling full checking:

#include <zephyr/kernel.h>

// Get current stack usage for a thread
size_t usage = k_thread_stack_space_get(&thread_obj);

// Get unused stack space
size_t unused = k_thread_stack_free_space_get(&thread_obj);

// Print stack usage information
printk("Thread %s: %zu/%zu bytes used\n",
       thread_obj->base.name,
       k_thread_stack_space_get(&thread_obj),
       thread_obj->stack_info.size);

These functions allow applications to log stack usage periodically or during critical operations to identify trends before overflow occurs.

Solution Approaches

1. Increase Stack Size Allocation

The simplest solution is to increase the stack size when defining the thread:

#define MY_THREAD_STACK_SIZE 1024  // Increase from default 512 or 768

K_THREAD_STACK_DEFINE(my_stack_area, MY_THREAD_STACK_SIZE);
struct k_thread my_thread_data;

k_thread_create(&my_thread_data, my_stack_area,
                K_THREAD_STACK_SIZEOF(my_stack_area),
                my_thread_func, NULL, NULL, NULL,
                MY_THREAD_PRIORITY, 0, K_NO_WAIT);

Trade-offs: Increases RAM consumption. For systems with limited memory, this may not be feasible.

2. Optimize Stack Usage

Reduce stack consumption by:

• Moving large buffers to global/static memory or heap allocation
• Converting recursive algorithms to iterative implementations
• Passing large structures by pointer instead of by value
• Using memory pools for temporary large allocations

Example: Moving a large buffer to global scope:

// Before: Local buffer on stack
void process_data(void) {
    uint8_t buffer[512];  // Consumes 512 bytes of stack
    // ...
}

// After: Global buffer
static uint8_t processing_buffer[512];

void process_data(void) {
    // Use processing_buffer instead
    // No stack consumption for buffer
}

Trade-offs: Increases global memory usage and may reduce reentrancy or thread safety if not carefully managed.

3. Enable Stack Sentinel for Runtime Detection

As mentioned, stack sentinel provides low-overhead detection. Combine it with application-level monitoring:

#include <zephyr/kernel.h>
#include <zephyr/sys/printk.h>

void stack_monitor_thread(void *arg1, void *arg2, void *arg3)
{
    while (1) {
        k_sleep(K_SECONDS(10));  // Check every 10 seconds

        // Check all threads for stack usage
        struct k_thread *thread;
        SYS_SLIST_FOR_EACH_CONTAINER(&_kernel.threads, thread, node) {
            size_t used = k_thread_stack_space_get(thread);
            size_t total = thread->stack_info.size;
            if (used > total * 0.8) {  // Warning at 80% usage
                printk("WARNING: Thread %s stack usage: %zu/%zu (%.1f%%)\n",
                       thread->base.name, used, total,
                       (float)used * 100.0f / total);
            }
        }
    }
}

Trade-offs: Minimal performance impact (<1%) but only detects overflow at check intervals, not instantly.

4. Hardware-Assisted Stack Protection

On ARM Cortex-M7 and Cortex-M33 processors, Zephyr can use the Memory Protection Unit (MPU) to create a no-access guard region at the stack boundary. Any stack overflow triggers a MemManage fault.

To enable MPU-based stack protection:

1. Add CONFIG_HW_STACK_PROTECTION=y to prj.conf
2. Configure MPU regions appropriately

Trade-offs: Requires MPU support and consumes one MPU region, but provides zero-overhead detection with immediate fault generation.

Code Example: Stack Overflow Detection and Recovery

The following example demonstrates enabling stack sentinel, monitoring stack usage, and recovering from overflow conditions:

#include <zephyr/kernel.h>
#include <zephyr/sys/printk.h>
#include <zephyr/sys/reboot.h>

#define STACK_SIZE 512
#define MONITOR_INTERVAL K_SECONDS(5)

K_THREAD_STACK_DEFINE(stack_area, STACK_SIZE);
struct k_thread thread_data;

/* Thread that intentionally causes stack overflow */
void overflowing_thread(void *arg1, void *arg2, void *arg3)
{
    /* Large local array that will overflow the stack */
    uint8_t large_buffer[600];  /* Exceeds 512-byte stack */
    
    /* Use the buffer to prevent compiler optimization */
    for (int i = 0; i < 600; i++) {
        large_buffer[i] = (uint8_t)i;
    }
    
    /* This point may never be reached due to overflow */
    printk("Overflowing thread completed (should not happen)\n");
}

/* Monitor thread to check stack usage */
void monitor_thread(void *arg1, void *arg2, void *arg3)
{
    while (1) {
        k_sleep(MONITOR_INTERVAL);
        
        size_t used = k_thread_stack_space_get(&thread_data);
        size_t total = STACK_SIZE;
        
        printk("Monitor: Main thread stack %zu/%zu bytes used\n",
               used, total);
               
        if (used > total * 0.75) {
            printk("WARNING: Approaching stack limit!\n");
        }
    }
}

void main(void)
{
    /* Enable stack sentinel via K_THREAD_STACK_DEFINE with sentinel */
    /* Actually, stack sentinel is a global config option */
    
    /* Create the monitoring thread */
    k_thread_create(&thread_data, stack_area,
                    K_THREAD_STACK_SIZEOF(stack_area),
                    monitor_thread, NULL, NULL, NULL,
                    7, 0, K_NO_WAIT);
    
    /* Simulate work that might lead to overflow conditions */
    while (1) {
        k_sleep(K_SECONDS(1));
        /* Application logic here */
    }
}

When stack sentinel detects an overflow, Zephyr will panic. To handle this gracefully, applications can:

1. Use CONFIG_ASSERT to convert fatal errors to restartable exceptions
2. Implement a custom fatal error handler that attempts recovery
3. Use watchdog timers to reset the system if overflow causes lockup

Verification and Testing

Testing Stack Overflow Detection

To verify your stack overflow detection mechanisms:

1. Intentional Overflow Test: Create a thread with a known small stack size and allocate a local buffer larger than the stack.

2. Check for Expected Fault: With stack sentinel enabled, the system should trigger a fatal error with stack check failure.

3. Validate Monitoring: Ensure your stack usage monitoring reports increasing usage before overflow.

Using Zephyr’s Stack Analyzer

Zephyr includes a stack analyzer tool (in scripts/stack analyzer.py) that computes maximum stack usage per function call graph. While it requires linking information, it provides valuable insights during development.

Run the stack analyzer as part of your build process:

west build -t stack_usage

Production Monitoring

In production devices:

1. Enable stack sentinel for low-overhead protection
2. Implement periodic stack usage logging to external storage or debug port
3. Set up alerts when stack usage exceeds warning thresholds (e.g., 75% of allocated size)
4. Correlate stack usage spikes with specific events or workloads

Summary

Thread stack overflow in Zephyr is a preventable issue through proper stack sizing, runtime monitoring, and Zephyr’s built-in detection features. Key takeaways:

1. Enable Stack Sentinel: Use CONFIG_STACK_SENTINEL for minimal-overhead runtime detection.
2. Monitor Stack Usage: Regularly check stack space utilization using k_thread_stack_space_get().
3. Optimize Stack Consumption: Move large data structures off the stack and avoid deep recursion.
4. Consider Hardware Protection: Use MPU-based stack protection on supported CPUs for zero-overhead detection.
5. Test Thoroughly: Validate detection mechanisms with intentional overflow tests before deployment.

By combining these techniques, developers can create robust Zephyr applications that gracefully handle stack constraints and maintain system reliability in embedded systems.

Related Reading

• Zephyr RTOs Getting Started Guide
• Fixing UART DMA Overrun Errors on STM32
• Optimizing Cortex-M Tail Chaining for Sub-Microsecond Latency
• Zephyr Memory Protection MPU Configuration

References

1. Zephyr Project Documentation. “Thread Stacks.” https://docs.zephyrproject.org/latest/kernel/threads/stacks.html
2. ARM Limited. “ARMv7-M Architecture Reference Manual.” 2020.
3. Zephyr Project Documentation. “Stack Sentinel.” https://docs.zephyrproject.org/latest/kernel/threads/stack_sentinel.html
4. J. Regehr. “Understanding Stack Overflow.” ACM SIGPLAN Notices, vol. 42, no. 6, 2007.
5. Zephyr Project Documentation. “Stack Checking.” https://docs.zephyrproject.org/latest/kernel/threads/stack_checking.html
6. STMicroelectronics. “STM32F4xx Reference Manual.” RM0090, 2022.
7. M. Barr. “Programming Embedded Systems in C and C++.” O’Reilly Media, 2006.