Fixing Zephyr Workqueue Deadlock From ISR Context

Zephyr RTOS provides the workqueue subsystem as the standard architectural bridge between top-half interrupt service routines (ISRs) and bottom-half thread-level deferred processing. By offloading time-consuming operations—such as sensor calibration, protocol parsing, and bus I/O—from high-priority hardware interrupts to schedulable thread contexts, workqueues preserve system responsiveness and maintain low interrupt latency.

However, bridging hardware interrupts and thread-level work items frequently introduces fatal synchronization traps. Developers often encounter insidious system lockups where a workqueue thread halts permanently, watchdogs bite, and the RTOS appears frozen.

This guide delivers an academic-grade technical dissection of why workqueue deadlocks occur when handling ISR-submitted work in Zephyr. It investigates the mechanics of circular wait lock inversions, examines the kernel constraints of k_mutex versus k_spinlock, evaluates system workqueue starvation, and provides production-hardened patterns with verification test harnesses.

+===========================================================================+
|             ZEPHYR ISR-TO-WORKQUEUE SYNCHRONIZATION ARCHITECTURE          |
+===========================================================================+
|                                                                           |
|   HARDWARE DOMAIN                  KERNEL THREAD DOMAIN                   |
|                                                                           |
|   +-------------------+            +----------------------------------+   |
|   | Hardware Trigger  |            | Application / Driver Thread      |   |
|   | (GPIO / DMA / IRQ)|            |                                  |   |
|   +---------+---------+            | 1. Takes dev_lock (k_mutex)      |   |
|             |                      | 2. Starts hardware transaction   |   |
|             v                      | 3. Calls k_work_flush() or       |   |
|   +-------------------+            |    k_sem_take(&done, K_FOREVER)  |   |
|   | ISR Context       |            +-----------------+----------------+   |
|   | (Top-Half Handler)|                              |                    |
|   |                   |                              | BLOCKS WAITING     |
|   | Reads HW Status   |                              | ON WORK COMPLETION |
|   | Puts into RingBuf |                              v                    |
|   | k_work_submit()   | ---------> +----------------------------------+   |
|   +-------------------+  Enqueues  | Dedicated Workqueue Thread       |   |
|                            Work    | (Bottom-Half Handler)            |   |
|                                    |                                  |   |
|                                    | 4. Work dequeued by scheduler    |   |
|                                    | 5. Calls k_mutex_lock(dev_lock)  |   |
|                                    |    ==> DEADLOCK (Circular Wait)  |   |
|                                    +----------------------------------+   |
|                                                                           |
|   DEADLOCK CONDITION:                                                     |
|   - Application Thread holds dev_lock, waiting for Workqueue completion   |
|   - Workqueue Thread cannot finish because it blocks waiting on dev_lock  |
|                                                                           |
+===========================================================================+

Root Cause Analysis: How the Deadlock Forms

A workqueue deadlock involving ISR-submitted work rarely stems from the k_work_submit() call in isolation. Because k_work_submit() is an atomic, lock-free, non-blocking queue insertion, an ISR can execute it safely in a fraction of a microsecond.

Instead, deadlocks materialize through three distinct architectural flaws:

1. Circular wait lock inversion across calling threads and deferred work handlers.
2. Illegal invocation of blocking primitives (k_mutex) within interrupt contexts.
3. Head-of-line blocking and self-dependency on the shared system workqueue (k_sys_work_q).

Flaw 1: The Circular Wait (AB-BA Lock Inversion)

Consider an embedded sensor driver or communication stack where an application thread coordinates bus transfers protected by a mutual exclusion primitive (k_mutex):

1. Thread Holds Resource: The application thread acquires dev->mutex to initiate a burst read over an SPI or I2C bus.
2. Thread Waits on Completion: After triggering the transfer, the thread synchronously waits for the hardware to finish and process the incoming buffer. It blocks on a completion primitive—for example, invoking k_work_flush(&dev->work) or k_sem_take(&dev->done_sem, K_FOREVER).
3. ISR Submits Work: The peripheral triggers an interrupt upon FIFO watermark or transfer completion. The ISR runs, clears interrupt status registers, and invokes k_work_submit(&dev->work) to defer buffer decoding.
4. Work Handler Blocks on Mutex: The scheduler dispatches the workqueue thread to execute dev->work’s handler. The handler begins execution and attempts to acquire k_mutex_lock(&dev->mutex, K_FOREVER) to access device state.
5. Permanent Circular Stall:
   • The workqueue thread cannot acquire dev->mutex because the application thread owns it.
   • The application thread cannot release dev->mutex because it is blocked waiting for the workqueue thread to complete.
   • Neither execution context can make forward progress.

+-----------------------------------------------------------------------------+
|                     CIRCULAR WAIT DEPENDENCY TIMELINE                       |
+-----------------------------------------------------------------------------+
|                                                                             |
|  APP THREAD             HARDWARE / ISR          WORKQUEUE THREAD            |
|  (Holds Mutex)                                  (Wants Mutex)               |
|       |                                                |                    |
|  k_mutex_lock(M)                                       |                    |
|  Trigger Transfer                                      |                    |
|  k_work_flush(W) ----+                                 |                    |
|  (SLEEPING...)       |                                 |                    |
|       |              v                                 |                    |
|       |         [IRQ Fires]                            |                    |
|       |         sensor_isr()                           |                    |
|       |         k_work_submit(W) --------------------> |                    |
|       |         ISR Exits                              | (W Dispatched)     |
|       |                                                | work_handler()     |
|       |                                                | k_mutex_lock(M)    |
|       |                                                | (BLOCKS ON M)      |
|       v                                                v                    |
|  [Waiting for W]                                 [Waiting for M]            |
|                                                                             |
|  ================== DEADLOCK: CIRCULAR DEPENDENCY FORMED =================  |
|                                                                             |
+-----------------------------------------------------------------------------+

Flaw 2: The ISR-Context Illegal Mutex Trap

In an attempt to guard shared state between an ISR and a workqueue handler, engineers occasionally attempt to call k_mutex_lock() inside the ISR.

In Zephyr RTOS, k_mutex is strictly prohibited in ISR context:

• A Zephyr mutex implements priority inheritance, which requires an owning thread (struct k_thread *owner). Interrupt contexts have no associated thread struct.
• k_mutex_lock() can block, but an ISR runs on the interrupt stack (e.g., MSP on ARM Cortex-M) and can never yield, sleep, or be scheduled out.
• The Zephyr kernel explicitly validates the calling context via k_is_in_isr():

/* Zephyr kernel implementation excerpt: kernel/mutex.c */
int z_impl_k_mutex_lock(struct k_mutex *mutex, k_timeout_t timeout)
{
    SYS_PORT_TRACING_OBJ_FUNC_ENTER(k_mutex, lock, mutex, timeout);

    CHECKIF(k_is_in_isr()) {
        SYS_PORT_TRACING_OBJ_FUNC_EXIT(k_mutex, lock, mutex, timeout, -EADDRNOTAVAIL);
        return -EADDRNOTAVAIL;
    }
    /* ... */
}

If CONFIG_ASSERT=y is set (standard in debug builds), calling k_mutex_lock() from an ISR triggers an immediate fatal kernel panic:

ASSERTION FAIL [!arch_is_in_isr()] at zephyr/kernel/mutex.c:xxx
E: >>> ZEPHYR FATAL ERROR 4: Kernel Oops on CPU 0

If assertions are disabled in production builds (CONFIG_ASSERT=n), k_mutex_lock() returns -EADDRNOTAVAIL immediately without taking the lock. If the code does not check the return value, the ISR executes without mutual exclusion, leading to race conditions and memory corruption rather than an orderly lock.

Flaw 3: System Workqueue Starvation and Cascading Hangs

Zephyr initializes a global workqueue known as the system workqueue (k_sys_work_q), driven by a single kernel thread (CONFIG_SYSTEM_WORKQUEUE_PRIORITY).

When an ISR submits work via k_work_submit(&my_work), the work item enters k_sys_work_q. If my_work blocks—waiting on an I2C transaction, flash page erase, network packet allocation, or another thread that has been preempted—the single thread running k_sys_work_q halts.

Because Bluetooth LE controller events, network protocol state machines, and power management callbacks rely on k_sys_work_q, halting this single thread causes the entire operating system to freeze.

The Flawed Pattern vs. The Correct Pattern

The following before-and-after listings illustrate the structural bug and its production-grade remedy.

Before: Circular Wait Deadlock

In this flawed driver implementation, the application thread holds a mutex while flushing the workqueue, while the work handler attempts to claim that same mutex.

#include <zephyr/kernel.h>
#include <zephyr/drivers/sensor.h>

struct sensor_context {
    struct k_mutex lock;
    struct k_work work;
    uint16_t raw_buffer[32];
    size_t sample_count;
};

static struct sensor_context g_ctx;
void trigger_hardware_conversion(void);

/* Bottom-half work handler running in workqueue thread context */
static void bad_work_handler(struct k_work *work)
{
    struct sensor_context *ctx = CONTAINER_OF(work, struct sensor_context, work);

    /* DEADLOCK: Tries to claim mutex held by app_read_samples() */
    k_mutex_lock(&ctx->lock, K_FOREVER);
    
    for (size_t i = 0; i < ctx->sample_count; i++) {
        /* Process samples */
    }
    
    k_mutex_unlock(&ctx->lock);
}

/* Interrupt Service Routine triggered by hardware GPIO */
void sensor_gpio_isr(const struct device *port, struct gpio_callback *cb, uint32_t pins)
{
    /* Safe non-blocking submission to system workqueue */
    k_work_submit(&g_ctx.work);
}

/* Application thread requesting a sample update */
int app_read_samples(void)
{
    /* 1. Acquire driver lock */
    k_mutex_lock(&g_ctx.lock, K_FOREVER);

    /* 2. Configure hardware registers to trigger sampling */
    trigger_hardware_conversion();

    /* 
     * 3. DEADLOCK TRIGGER:
     * App thread waits synchronously for work handler to finish.
     * But the work handler needs g_ctx.lock, which this thread holds!
     */
    struct k_work_sync sync;
    k_work_flush(&g_ctx.work, &sync);

    k_mutex_unlock(&g_ctx.lock);
    return 0;
}

Architectural Remedies & Solutions

Solution 1: Break the Circular Wait via Asynchronous Decoupling

To prevent circular dependency deadlocks, never hold a mutex while synchronously waiting on workqueue completion (k_work_flush() or k_work_cancel_sync()). Instead:

• Release the lock before waiting on completion.
• Decouple the synchronization using dedicated binary semaphores (k_sem) or event objects that indicate hardware transfer completion without holding locks.

+-----------------------------------------------------------------------------+
|                      DECOUPLED ASYNCHRONOUS EXECUTION                       |
+-----------------------------------------------------------------------------+
|                                                                             |
|  APP THREAD                     ISR                     DEDICATED WORKQUEUE |
|       |                          |                               |          |
|  k_mutex_lock(M)                 |                               |          |
|  Setup Hardware                  |                               |          |
|  k_mutex_unlock(M)  <-- RELEASE  |                               |          |
|  k_sem_take(done, K_FOREVER)     |                               |          |
|  (WAITING ON SEMAPHORE)          |                               |          |
|       |                          |                               |          |
|       |                    [IRQ Fires]                           |          |
|       |                    Save Status                           |          |
|       |                    k_work_submit() --------------------> |          |
|       |                    ISR Exits                             |          |
|       |                                                k_mutex_lock(M)      |
|       |                                                (LOCK ACQUIRED!)     |
|       |                                                Process Data         |
|       |                                                k_mutex_unlock(M)    |
|       |                                                k_sem_give(done)     |
|       |<---------------------------------------------------------+          |
|  (Wakes up)                                                      |          |
|  Read Completed Data                                             v          |
|                                                            Idle / Complete  |
+-----------------------------------------------------------------------------+

Solution 2: Use k_spinlock for ISR/Thread Mutual Exclusion

When critical state must be shared directly between an ISR and a thread-level work handler, k_spinlock is the only valid primitive.

k_spinlock operates by disabling local interrupts on uniprocessor architectures (BASEPRI/PRIMASK on ARM Cortex-M) and acquiring an atomic memory lock on symmetric multiprocessing (SMP) systems. It never sleeps.

#include <zephyr/kernel.h>
#include <zephyr/spinlock.h>

struct sensor_shared_data {
    struct k_spinlock lock;
    uint32_t status_flags;
    int16_t latest_raw_reading;
};

static struct sensor_shared_data g_dev_state;
extern struct k_work_q custom_work_q;
extern struct k_work sensor_work;
void compute_dsp_filtering(int16_t sample, uint32_t flags);

void sensor_hardware_isr(const void *arg)
{
    /* Acquire spinlock: disables interrupts locally, returns interrupt key */
    k_spinlock_key_t key = k_spin_lock(&g_dev_state.lock);

    /* Ultra-low latency critical section (< 5 microseconds) */
    g_dev_state.status_flags = HW_REG_STATUS;
    g_dev_state.latest_raw_reading = (int16_t)HW_REG_DATA;

    /* Restore interrupt state */
    k_spin_unlock(&g_dev_state.lock, key);

    /* Submit deferred processing */
    k_work_submit_to_queue(&custom_work_q, &sensor_work);
}

void sensor_work_handler(struct k_work *work)
{
    uint32_t flags;
    int16_t sample;

    /* Safely extract snapshot under spinlock */
    k_spinlock_key_t key = k_spin_lock(&g_dev_state.lock);
    flags = g_dev_state.status_flags;
    sample = g_dev_state.latest_raw_reading;
    g_dev_state.status_flags = 0; /* Acknowledge */
    k_spin_unlock(&g_dev_state.lock, key);

    /* Perform heavy, un-locked floating point conversion */
    compute_dsp_filtering(sample, flags);
}

[!IMPORTANT] Keep k_spinlock critical sections bounded (under 10 us). Spinlocks disable interrupts on the active CPU core, which directly increases hardware interrupt jitter. Never invoke blocking functions, memory allocations, or logging macros inside a spinlock critical section.

Solution 3: Dedicated Workqueue Isolation

To prevent head-of-line blocking on the shared system workqueue, always allocate a dedicated workqueue for high-frequency or I/O-intensive drivers:

#define SENSOR_WQ_STACK_SIZE 2048
#define SENSOR_WQ_PRIORITY   K_PRIO_PREEMPT(5)

static struct k_work_q sensor_work_q;
K_THREAD_STACK_DEFINE(sensor_work_q_stack, SENSOR_WQ_STACK_SIZE);

void sensor_subsystem_init(void)
{
    struct k_work_queue_config cfg = {
        .name = "sensor_wq",
        .no_yield = false,
    };

    /* Initialize and start the dedicated workqueue thread */
    k_work_queue_start(
        &sensor_work_q,
        sensor_work_q_stack,
        K_THREAD_STACK_SIZEOF(sensor_work_q_stack),
        SENSOR_WQ_PRIORITY,
        &cfg
    );
}

Architectural Advantages:

• Isolation: A stall or blocking I2C transfer in sensor_work_q cannot delay Bluetooth LE, networking, or kernel timers running on k_sys_work_q.
• Configurable Priority: Allows fine-tuning thread priority (cooperative K_PRIO_COOP(x) or preemptive K_PRIO_PREEMPT(x)) relative to application threads.
• Dedicated Stack: Prevents stack overflow caused by nested driver callbacks.

Solution 4: Non-Blocking Work Scheduling with k_work_delayable

When deferred work requires back-off, debouncing, or periodic hardware polling, use k_work_delayable rather than combining hardware timers with synchronous sleep calls inside work handlers:

static struct k_work_delayable sensor_retry_work;
extern struct k_work_q sensor_work_q;
int reset_sensor_hardware(void);

void sensor_error_isr(void)
{
    /* 
     * Schedule deferred retry in 50 ms.
     * k_work_schedule_for_queue() is 100% ISR-safe and non-blocking.
     */
    k_work_schedule_for_queue(&sensor_work_q, &sensor_retry_work, K_MSEC(50));
}

void sensor_retry_handler(struct k_work *work)
{
    struct k_work_delayable *dwork = k_work_delayable_from_work(work);

    if (reset_sensor_hardware() != 0) {
        /* Exponential backoff: reschedule for 100 ms */
        k_work_schedule_for_queue(&sensor_work_q, dwork, K_MSEC(100));
    }
}

Solution 5: Lock-Free Single-Producer Single-Consumer (SPSC) Ring Buffer

For high-throughput sensors (> 10 kHz), spinlocks can create unacceptable interrupt latency. Zephyr’s struct ring_buf provides a lock-free byte and item ring buffer optimized for single-producer (ISR) and single-consumer (workqueue thread) architectures.

#include <zephyr/sys/ring_buffer.h>

#define SENSOR_QUEUE_ITEMS 32
struct sensor_event {
    uint32_t timestamp_cycles;
    int16_t axis_x;
    int16_t axis_y;
    int16_t axis_z;
};

RING_BUF_DECLARE(sensor_rb, SENSOR_QUEUE_ITEMS * sizeof(struct sensor_event));
extern struct k_work_q sensor_work_q;
extern struct k_work sensor_drain_work;
int16_t read_accel_x(void);
int16_t read_accel_y(void);
int16_t read_accel_z(void);

void high_rate_gpio_isr(const void *arg)
{
    struct sensor_event ev = {
        .timestamp_cycles = k_cycle_get_32(),
        .axis_x = read_accel_x(),
        .axis_y = read_accel_y(),
        .axis_z = read_accel_z(),
    };

    /* Lock-free enqueue: returns number of bytes written */
    uint32_t written = ring_buf_put(&sensor_rb, (const uint8_t *)&ev, sizeof(ev));
    if (written == sizeof(ev)) {
        /* Submit work item to drain the buffer in thread context */
        k_work_submit_to_queue(&sensor_work_q, &sensor_drain_work);
    } else {
        /* Buffer full: handle overflow / dropped sample */
    }
}

Under single-producer single-consumer operation, memory accesses to write and read indexes are atomic, eliminating the need for spinlocks or interrupt masking.

Complete Working Example: Production Sensor Driver

Below is an end-to-end, production-grade implementation of an interrupt-driven sensor driver. It incorporates a dedicated workqueue, a lock-free ring buffer, proper thread naming, and thread-safe deferred I2C/SPI transactions.

/* sensor_driver_safe.c - Hardened ISR-to-Workqueue Pattern */

#include <zephyr/kernel.h>
#include <zephyr/drivers/gpio.h>
#include <zephyr/sys/ring_buffer.h>
#include <zephyr/logging/log.h>
#include <inttypes.h>

LOG_MODULE_REGISTER(safe_sensor, LOG_LEVEL_INF);

#define SENSOR_WQ_STACK_SIZE 2048
#define SENSOR_WQ_PRIORITY   K_PRIO_PREEMPT(5)
#define SENSOR_RINGBUF_SIZE  512

/* Data packet transferred from ISR to Workqueue */
struct sensor_raw_event {
    int64_t timestamp_us;
    uint8_t status_reg;
};

/* Driver Context */
struct safe_sensor_dev {
    struct k_work_q work_q;
    struct k_work drain_work;
    struct ring_buf event_rb;
    uint8_t rb_storage[SENSOR_RINGBUF_SIZE];
    struct k_sem completion_sem;
    uint32_t processed_count;
    atomic_t dropped_count;
};

static struct safe_sensor_dev g_sensor;
K_THREAD_STACK_DEFINE(sensor_stack, SENSOR_WQ_STACK_SIZE);

/* Deferred work handler running in thread context */
static void sensor_work_drain_handler(struct k_work *work)
{
    struct safe_sensor_dev *dev = CONTAINER_OF(work, struct safe_sensor_dev, drain_work);
    struct sensor_raw_event ev;

    /* Report any dropped events detected by the ISR in thread context */
    if (atomic_get(&dev->dropped_count) > 0) {
        LOG_WRN("Dropped %ld sensor events due to ring buffer saturation",
                (long)atomic_set(&dev->dropped_count, 0));
    }

    /* Drain all pending events from the lock-free ring buffer */
    while (ring_buf_get(&dev->event_rb, (uint8_t *)&ev, sizeof(ev)) == sizeof(ev)) {
        dev->processed_count++;

        /* 
         * Perform blocking I/O (e.g., I2C bus transactions) safely here.
         * Thread context permits sleeping, bus arbitration, and retries.
         */
        LOG_DBG("Processing event: TS=%" PRIi64 " us, STATUS=0x%02X",
                ev.timestamp_us, ev.status_reg);

        /* Emulate I2C bus processing */
        k_busy_wait(10);
    }

    /* Signal completion to any thread waiting on a flush */
    k_sem_give(&dev->completion_sem);
}

/* Fast hardware interrupt handler */
void sensor_hardware_interrupt(const struct device *dev, struct gpio_callback *cb, uint32_t pins)
{
    struct sensor_raw_event ev;

    /* Capture timestamp and status with minimal CPU cycles */
    ev.timestamp_us = k_ticks_to_us_near64(k_uptime_ticks());
    ev.status_reg = 0x01; /* Emulate reading hardware interrupt status */

    /* Lock-free push into ring buffer */
    uint32_t written = ring_buf_put(&g_sensor.event_rb, (const uint8_t *)&ev, sizeof(ev));
    if (written == sizeof(ev)) {
        /* Submit work to dedicated queue */
        k_work_submit_to_queue(&g_sensor.work_q, &g_sensor.drain_work);
    } else {
        /* Lock-free saturation counter: avoids blocking or string formatting in ISR */
        atomic_inc(&g_sensor.dropped_count);
    }
}

/* Driver Initialization */
int safe_sensor_init(void)
{
    /* 1. Initialize synchronization objects */
    atomic_set(&g_sensor.dropped_count, 0);
    k_sem_init(&g_sensor.completion_sem, 0, 1);
    ring_buf_init(&g_sensor.event_rb, sizeof(g_sensor.rb_storage), g_sensor.rb_storage);
    k_work_init(&g_sensor.drain_work, sensor_work_drain_handler);

    /* 2. Start dedicated workqueue */
    struct k_work_queue_config cfg = {
        .name = "sensor_wq",
        .no_yield = false,
    };

    k_work_queue_start(
        &g_sensor.work_q,
        sensor_stack,
        K_THREAD_STACK_SIZEOF(sensor_stack),
        SENSOR_WQ_PRIORITY,
        &cfg
    );

    LOG_INF("Safe sensor driver initialized with dedicated workqueue");
    return 0;
}

Field Debugging & Runtime Inspection

When a workqueue stops processing items, determine whether the work thread is blocked, starved, or stuck in an infinite loop using Zephyr’s diagnostic subsystems.

Step 1: Kernel Configuration

Enable kernel thread monitoring, stack statistics, and shell inspection in your prj.conf:

# Core thread observability
CONFIG_THREAD_MONITOR=y
CONFIG_THREAD_STACK_INFO=y
CONFIG_THREAD_NAME=y
CONFIG_THREAD_RUNTIME_STATS=y

# Zephyr Shell interface
CONFIG_SHELL=y
CONFIG_KERNEL_SHELL=y

# Post-mortem coredump support
CONFIG_COREDUMP=y
CONFIG_COREDUMP_BACKEND_LOGGING_RAM=y

Step 2: Interactive Shell Diagnostics

Connect via the serial console and execute the following commands:

uart:~$ kernel threads
Scheduler: 142095 since last call
Threads:
 *0x200010c0 shell_uart       (prio -1) state: pending
  0x200021e8 sysworkq         (prio -1) state: pending
  0x200034b0 sensor_wq        (prio  5) state: pending
  0x20000a40 idle             (prio 15) state: ready

uart:~$ kernel stacks
0x200034b0 sensor_wq        (size 2048, used 416, free 1632)
0x200021e8 sysworkq         (size 1024, used 688, free 336)

Look for:

• state: pending: The thread is waiting on a kernel object (semaphore, mutex, or workqueue item).
• state: suspended: Explicitly halted via k_thread_suspend().
• free < 128: Imminent stack overflow risk.

Step 3: Programmatic Thread Traversal

To dump thread states programmatically across a fleet of devices without interactive shell access, use k_thread_foreach():

#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>

LOG_MODULE_REGISTER(diagnostics, LOG_LEVEL_INF);

static void thread_audit_callback(const struct k_thread *thread, void *user_data)
{
    char state_str[32];
    k_tid_t tid = (k_tid_t)thread;
    const char *name = k_thread_name_get(tid);

    size_t unused = 0;
    k_thread_stack_space_get(tid, &unused);

    LOG_INF("Thread: [%s] | Priority: %d | Unused Stack: %zu bytes",
            name ? name : "unnamed",
            k_thread_priority_get(tid),
            unused);
}

void audit_system_threads(void)
{
    LOG_INF("=== Running System Thread Audit ===");
    k_thread_foreach(thread_audit_callback, NULL);
}

Production Verification & Stress Test Harness

The following Zephyr Ztest unit test exercises 10,000 rapid ISR event submissions to verify that the dedicated workqueue and lock-free ring buffer remain stable under maximum throughput without deadlocks or buffer overruns.

/* test_workqueue_stress.c */

#include <zephyr/ztest.h>
#include <zephyr/kernel.h>
#include <zephyr/sys/ring_buffer.h>

#define STRESS_ITERATIONS 10000
#define TEST_WQ_STACK_SIZE 2048

static struct k_work_q stress_wq;
K_THREAD_STACK_DEFINE(stress_stack, TEST_WQ_STACK_SIZE);

static struct k_work stress_work;
static struct ring_buf stress_rb;
static uint8_t rb_buffer[1024];

static atomic_t events_submitted = ATOMIC_INIT(0);
static atomic_t events_processed = ATOMIC_INIT(0);
static struct k_sem test_done_sem;

static void stress_work_handler(struct k_work *work)
{
    uint32_t val;
    while (ring_buf_get(&stress_rb, (uint8_t *)&val, sizeof(val)) == sizeof(val)) {
        atomic_inc(&events_processed);
    }

    if (atomic_get(&events_processed) == STRESS_ITERATIONS) {
        k_sem_give(&test_done_sem);
    }
}

/* Producer simulating high-rate hardware interrupts */
static void simulate_hardware_isr(void)
{
    for (uint32_t i = 0; i < STRESS_ITERATIONS; i++) {
        /* Spin until buffer accepts entry */
        while (ring_buf_put(&stress_rb, (const uint8_t *)&i, sizeof(i)) != sizeof(i)) {
            k_busy_wait(1);
        }
        atomic_inc(&events_submitted);
        k_work_submit_to_queue(&stress_wq, &stress_work);
    }
}

ZTEST_SUITE(workqueue_deadlock_suite, NULL, NULL, NULL, NULL, NULL);

ZTEST(workqueue_deadlock_suite, test_high_throughput_no_deadlock)
{
    k_sem_init(&test_done_sem, 0, 1);
    ring_buf_init(&stress_rb, sizeof(rb_buffer), rb_buffer);
    k_work_init(&stress_work, stress_work_handler);

    struct k_work_queue_config cfg = {
        .name = "stress_wq",
        .no_yield = false,
    };

    k_work_queue_start(
        &stress_wq,
        stress_stack,
        K_THREAD_STACK_SIZEOF(stress_stack),
        K_PRIO_PREEMPT(4),
        &cfg
    );

    /* Run stress test producer */
    simulate_hardware_isr();

    /* Await completion with 5-second timeout */
    int ret = k_sem_take(&test_done_sem, K_MSEC(5000));
    zassert_equal(ret, 0, "Stress test timed out! Workqueue deadlocked.");

    zassert_equal(atomic_get(&events_processed), STRESS_ITERATIONS,
                  "Mismatched processed event count.");
}

Architectural Comparison Matrix

Summary & Core Rules

Eliminating workqueue deadlocks in Zephyr RTOS requires strict compliance with four fundamental engineering principles:

1. Never Hold Mutexes While Waiting on Work: When an application thread calls k_work_flush() or k_work_cancel_sync(), it must never hold locks that the deferred work handler requires.
2. k_mutex is Illegal in Interrupts: Calling k_mutex_lock() from an ISR triggers a fatal oops (ASSERTION FAIL [!arch_is_in_isr()]) or silently fails with -EADDRNOTAVAIL. Protect shared ISR data using k_spinlock or atomic variables.
3. Isolate High-Frequency Tasks in Dedicated Workqueues: Avoid overloading the global k_sys_work_q. Spin up an isolated k_work_q with tailored stack sizing and preemptive priority to prevent head-of-line blocking.
4. Prefer Lock-Free Queues: Use single-producer single-consumer ring buffers (struct ring_buf) to decouple top-half interrupt capture from bottom-half bus transactions.

Related Reading

• Zephyr Workqueue API for Deferred Work in Embedded Systems
• Lock-Free Ring Buffers for ISR to Task Communication
• Debugging Zephyr Kernel Panic on STM32
• Fixing Zephyr Bluetooth LE Connection Timeout Issues
• Zephyr PM Subsystem Deep Sleep Device Runtime PM
• Zephyr Dynamic Memory Allocation Failures Debugging
• Mutex vs Semaphore in RTOS Choosing the Right Primitive
• Context Switching and Scheduling in RTOS A Deep Dive

References

1. Zephyr Project Documentation — Workqueue Threads. https://docs.zephyrproject.org/latest/kernel/services/threads/workqueue.html (accessed 2026-10-25)
2. Zephyr Project Documentation — Synchronization: Mutexes. https://docs.zephyrproject.org/latest/kernel/services/synchronization/mutexes.html (accessed 2026-10-25)
3. Zephyr Project Documentation — Synchronization: Spinlocks. https://docs.zephyrproject.org/latest/kernel/services/synchronization/spinlocks.html (accessed 2026-10-25)
4. Zephyr Project Documentation — Data Passing: Ring Buffers. https://docs.zephyrproject.org/latest/kernel/services/data_passing/ring_buffers.html (accessed 2026-10-25)
5. Zephyr Project Documentation — Threads: Runtime Statistics. https://docs.zephyrproject.org/latest/kernel/services/threads/index.html (accessed 2026-10-25)
6. ARM Cortex-M Programming Guide — Interrupt Handling and Context Switching. https://support.arm.com/documentation/100067/0618/ (accessed 2026-10-25)