Zephyr Workqueue API for Deferred Work in Embedded Systems

Problem: Blocking Threads in Interrupt Handlers

In embedded systems, interrupt service routines (ISRs) must execute quickly to avoid missing subsequent interrupts. Performing lengthy processing inside an ISR can lead to increased interrupt latency, missed events, and system instability. Similarly, high-priority threads should not be blocked by lengthy computations that could be deferred to lower-priority contexts.

Consider a real-world scenario: an STM32-based sensor hub receiving data from an accelerometer via I2C at 1 kHz. When the sensor’s Data Ready (DRDY) pin asserts, an external GPIO interrupt fires. In Zephyr, synchronous bus transfer APIs such as i2c_burst_read() or i2c_write_read_dt() are blocking calls that sleep on an internal synchronization semaphore while the peripheral transfer takes place. Attempting to execute synchronous I2C read functions directly inside an ISR violates kernel invariants and triggers an immediate fatal assertion (z_impl_k_sem_take() cannot be called from ISR context).

Furthermore, even if the ISR only reads hardware registers without blocking, attempting to execute sensor fusion algorithms (e.g., AHRS Kalman filtering, matrix multiplications, coordinate transformations) directly in the ISR context creates severe timing violations. On an STM32F4 (ARM Cortex-M4 at 168 MHz), servicing the GPIO interrupt and queueing an event takes under 250 ns, whereas executing floating-point sensor fusion calculations can consume over 1.2 ms. On a 1 kHz sensor stream (1.0 ms period), the computation time overruns the interrupt period, causing unhandled interrupt accumulation, missed sensor packets, and starvation of all lower-priority interrupts.

Another common case involves high-priority threads handling time-critical control loops. If such a thread performs lengthy file system operations or complex protocol processing, it can miss deadlines for lower-priority but still important tasks like user interface updates or diagnostic logging. The priority inversion problem emerges when a medium-priority task preempts the high-priority thread during its lengthy operation, indirectly affecting system responsiveness.

These scenarios illustrate why deferring non-urgent work is essential: ISRs must remain short to guarantee interrupt latency bounds, and high-priority threads must yield the CPU when possible to maintain system schedulability. The Zephyr Workqueue API provides a standardized mechanism to shift processing from time-critical contexts to regular thread contexts where preemption is allowed and longer-running tasks are acceptable.

Root Cause: Need for Deferred Execution

The Zephyr kernel offers multiple deferred execution mechanisms, but choosing the wrong API exacerbates problems. Developers often default to busy-wait loops or inappropriate synchronization primitives, wasting CPU cycles and potentially causing priority inversions. For example, using a semaphore to wake a thread from an ISR works, but if the ISR gives the semaphore repeatedly before the thread processes the first signal, semaphore overflow occurs or interrupts get lost.

Busy-wait loops inside ISRs are particularly dangerous. They block all interrupts of equal or lower priority, defeating the purpose of interrupt-driven design. A developer might poll a hardware register inside an ISR waiting for a conversion to complete, not realizing this prevents other interrupts from servicing. On a Cortex-M4, a 10-microsecond busy-wait loop can delay up to 100 pending interrupts if they share the same priority level.

The root cause lies in misunderstanding Zephyr’s deferred execution options:

• k_timer: Best for periodic or one-shot time-based events. Crucially, the expiry callback executes in ISR context (the system clock tick interrupt), meaning it cannot sleep, wait on mutexes, or make blocking driver calls.
• k_work: Ideal for event-driven deferral where an ISR or thread offloads work to run in a dedicated thread context, enabling blocking I/O, mutex acquisition, and preemption.
• k_work_delayable: Combines a kernel timer with work deferral (e.g., debouncing a button press after 50 ms or managing communication timeouts), running the handler in thread context once the timer expires.
• Kernel Queues (k_msgq, k_fifo): Ideal for high-throughput producer-consumer pipelines that require passing full data payloads between ISRs and processing threads without dropping samples.

Using k_timer for sensor data processing creates unnecessary complexity—you’d need to manage timer start/stop cycles and handle potential overruns. Conversely, using k_work for simple periodic tasks over-engineers the solution when k_timer suffices. The Workqueue API (k_work/k_work_delayable) strikes the right balance for event-driven deferred processing, offering flexibility without timing overhead when exact scheduling isn’t required.

Solution: Using the Workqueue API

Zephyr’s Workqueue API allows you to queue a work item (a function pointer and associated metadata) to be processed by a dedicated workqueue thread. This shifts the execution from a time-critical context (such as a hardware ISR or hard real-time thread) to a cooperative or preemptive thread context where blocking system calls and longer-running processing are permissible.

Workqueue Basics

A standard work item is encapsulated by struct k_work. It is initialized with a callback handler function and subsequently submitted to a workqueue using k_work_submit() or k_work_submit_to_queue().

+-------------------------+
  | Hardware ISR / Thread   |
  | (sensor_drdy_isr)       |
  +-------------------------+
               |
               | k_work_submit(&sensor_work)
               v
  +-------------------------+
  | Workqueue Wait Queue    |  <-- Thread-safe intrusive singly-linked list
  | [work_item 1]->[work_2] |
  +-------------------------+
               |
               | Dispatched by RTOS Scheduler (Thread Context)
               v
  +-------------------------+
  | Workqueue Thread        |
  | (process_sensor_data)   |  <-- Blocking I2C read & calibration executed here
  +-------------------------+

Key Workqueue API Functions:

• k_work_init(): Initializes a struct k_work item with its designated callback handler.
• k_work_submit(): Submits a work item to the global system workqueue (k_sys_work_q).
• k_work_submit_to_queue(): Submits a work item to a user-defined dedicated workqueue.
• k_work_cancel(): Cancels a pending work item if execution has not yet started.
• k_work_is_pending(): Checks if a work item is currently queued or in flight.
• k_work_init_delayable(): Initializes a struct k_work_delayable item with an integrated timer.
• k_work_schedule(): Submits a delayable work item to the system workqueue after a specified delay (k_timeout_t).
• k_work_schedule_for_queue(): Submits a delayable work item to a custom workqueue after a specified delay.
• k_work_reschedule(): Atomically updates or restarts the deadline for a scheduled delayable work item.
• k_work_cancel_delayable(): Cancels a scheduled delayable work item and its associated timer.

The system workqueue shares a single thread with other system subsystems. For many general tasks, this is sufficient. However, the system workqueue thread runs at a configurable priority (CONFIG_SYSTEM_WORKQUEUE_PRIORITY, default: -1, i.e., the lowest cooperative priority) and has a fixed stack size (CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE, default: 1024 bytes). Submitting compute-heavy or long-blocking work to the system workqueue will stall other critical kernel services.

Example: Deferring Sensor Processing from a GPIO ISR

Consider a hardware accelerometer that pulses an interrupt pin when a new sample is ready. The ISR cannot issue synchronous I2C read requests directly because i2c_burst_read() blocks on a semaphore. Instead, the ISR immediately queues a struct k_work item, leaving the bus transaction and data calibration to the workqueue thread.

To avoid dangerous global state, idiomatic Zephyr code embeds struct k_work inside a custom container struct. The work handler recovers the parent data using the standard CONTAINER_OF macro:

#include <zephyr/kernel.h>
#include <zephyr/device.h>
#include <zephyr/drivers/gpio.h>
#include <zephyr/drivers/i2c.h>

#define SENSOR_I2C_ADDR 0x68
#define REG_ACCEL_XOUT_H 0x3B

struct sensor_device {
    const struct device *i2c_dev;
    struct gpio_callback gpio_cb;
    struct k_work work;
    int16_t x, y, z;
};

static struct sensor_device sensor_dev;

/* Workqueue handler: Runs in Thread context */
void sensor_work_handler(struct k_work *item)
{
    /* Recover enclosing structure using CONTAINER_OF */
    struct sensor_device *dev = CONTAINER_OF(item, struct sensor_device, work);
    uint8_t raw_buf[6];

    /* Safe to call blocking I2C transactions in thread context */
    int ret = i2c_burst_read(dev->i2c_dev, SENSOR_I2C_ADDR, 
                             REG_ACCEL_XOUT_H, raw_buf, sizeof(raw_buf));
    if (ret != 0) {
        printk("I2C read failed with error: %d\n", ret);
        return;
    }

    dev->x = (int16_t)((raw_buf[0] << 8) | raw_buf[1]);
    dev->y = (int16_t)((raw_buf[2] << 8) | raw_buf[3]);
    dev->z = (int16_t)((raw_buf[4] << 8) | raw_buf[5]);

    /* Perform compute-intensive sensor calibration */
    apply_sensor_calibration(&dev->x, &dev->y, &dev->z);
}

/* GPIO ISR callback: Runs in ISR context */
void sensor_drdy_isr(const struct device *port, struct gpio_callback *cb, 
                     gpio_port_pins_t pins)
{
    struct sensor_device *dev = CONTAINER_OF(cb, struct sensor_device, gpio_cb);

    /* Submit work item to the system workqueue without blocking */
    k_work_submit(&dev->work);
}

/* Initialization */
void sensor_subsystem_init(const struct device *i2c_handle,
                           const struct gpio_dt_spec *drdy_gpio)
{
    sensor_dev.i2c_dev = i2c_handle;

    /* Initialize work item */
    k_work_init(&sensor_dev.work, sensor_work_handler);

    /* Configure GPIO Data-Ready interrupt callback */
    gpio_init_callback(&sensor_dev.gpio_cb, sensor_drdy_isr, BIT(drdy_gpio->pin));
    gpio_add_callback(drdy_gpio->port, &sensor_dev.gpio_cb);
}

In this architecture, sensor_drdy_isr() completes in less than 200 nanoseconds, avoiding bus contention and interrupt overruns. The actual I2C transaction occurs safely inside the system workqueue thread.

Workqueue Thread Configuration

When multiple modules submit tasks to the system workqueue, a long-running calculation can starve other drivers. Applications requiring dedicated latency bounds, custom thread priorities, or distinct stack sizes should allocate a dedicated workqueue:

#define CUSTOM_WQ_STACK_SIZE 2048
#define CUSTOM_WQ_PRIORITY   5

K_THREAD_STACK_DEFINE(custom_wq_stack, CUSTOM_WQ_STACK_SIZE);
struct k_work_q custom_work_q;

void custom_workqueue_init(void)
{
    /* Initialize the work queue structure */
    k_work_queue_init(&custom_work_q);

    /* Start the dedicated workqueue thread */
    k_work_queue_start(&custom_work_q,
                       custom_wq_stack,
                       K_THREAD_STACK_SIZEOF(custom_wq_stack),
                       CUSTOM_WQ_PRIORITY,
                       NULL);
}

/* Submit work to dedicated queue */
void submit_sensor_work(void)
{
    k_work_submit_to_queue(&custom_work_q, &sensor_dev.work);
}

This allows you to tune the stack size and priority of the workqueue thread to match your application’s needs.

Configuration Guidelines:

1. Stack Size: Allocate enough for the worst-case call stack of your work handler plus Zephyr’s internal overhead. Start with 2048 bytes and adjust based on stack usage analysis.
2. Priority: Set relative to other threads in your system. In Zephyr, cooperative priorities (negative values) are always higher-priority than preemptive priorities (non-negative values). Within each class, a larger numerical value means lower priority. For sensor processing that shouldn’t block control loops, use a preemptive priority lower than your control thread but higher than background tasks.
3. Number of Workqueues: Avoid creating excessive workqueues—each consumes a thread. Group related work items into fewer workqueues when possible.

Using k_work_delayable for Timed Deferral

If work must occur after a designated delay rather than immediately, use struct k_work_delayable. This is common for contact switch debouncing, communication keep-alives, or retry backoff loops:

static struct k_work_delayable debounce_work;

void button_pressed_isr(const struct device *port, struct gpio_callback *cb,
                        gpio_port_pins_t pins)
{
    /* Schedule work to execute after 50 ms debounce stabilization */
    k_work_schedule(&debounce_work, K_MSEC(50));
}

void debounce_work_handler(struct k_work *item)
{
    /* Runs in thread context after 50 ms */
    check_stable_button_state();
}

void button_subsystem_init(void)
{
    k_work_init_delayable(&debounce_work, debounce_work_handler);
}

Cancellation and Rescheduling:

• k_work_reschedule(): Atomically resets the deadline of an existing delayable work item. In watchdog patterns or incoming packet streams, calling k_work_reschedule(&timeout_work, K_MSEC(200)) safely resets the timer without race conditions.
• k_work_cancel_delayable(): Cancels both the underlying kernel timer and any pending submission of the work item. If the work item handler is already running, use k_work_cancel_delayable_sync() to wait until execution completes before deallocating resources.

Verification: Testing the Workqueue Behavior

To verify that work is indeed deferred and not executed in the ISR context, you can check the current interrupt nesting level or use kernel tracing.

Simple Verification with k_is_in_isr()

void process_sensor_data(struct k_work *work)
{
    __ASSERT(!k_is_in_isr(), "Work handler called from ISR!");
    /* ... processing ... */
}

If the assertion fails, the work was incorrectly executed in an ISR context.

Kernel Tracing

Enable Zephyr’s tracing subsystem to monitor workqueue submissions and executions, providing visibility into timing and thread stepping.

/* In prj.conf */
CONFIG_TRACING=y
CONFIG_TRACING_SYSLOG=y

Then use a log backend or a UART logger to see traces like:

[00:00:00.123456] workqueue: work submitted to system workqueue
[00:00:00.124000] workqueue: work handler started (thread: workqueue_thread)
[00:00:00.124001] isr_exit: exiting ISR context

This confirms the work handler executes in thread context after ISR exit.

Stack Usage Validation

Workqueue threads can overflow their allocated stacks if work handlers invoke deep call graphs or declare large local buffers. Zephyr provides runtime stack monitoring via k_thread_stack_space_get():

/* Ensure CONFIG_INIT_STACKS=y and CONFIG_THREAD_STACK_INFO=y in prj.conf */
void process_sensor_data(struct k_work *work)
{
    size_t unused_bytes;
    int ret = k_thread_stack_space_get(k_current_get(), &unused_bytes);

    if (ret == 0 && unused_bytes < 256) {
        printk("WARNING: Workqueue thread stack low: %zu bytes remaining\n", unused_bytes);
    }

    /* ... remaining processing ... */
}

Summary

The Zephyr Workqueue API is an essential primitive for architecting responsive, deterministic embedded systems:

1. Keep ISRs Minimal: Only acknowledge hardware flags and submit work; never execute blocking bus reads (i2c_read, spi_transceive) inside ISR context.
2. Thread Context Advantages: Workqueue handlers run as threads, unlocking full access to mutexes, semaphores, delays, and blocking device drivers.
3. Use Dedicated Workqueues for Compute-Heavy Tasks: Avoid overloading the system workqueue (k_sys_work_q) to prevent starving core RTOS subsystems.
4. Encapsulate Data Safely: Use CONTAINER_OF to associate contextual state with struct k_work instead of relying on unsynchronized global buffers.
5. Differentiate Immediate vs Delayable Work: Use k_work for immediate ISR offloading and k_work_delayable for timed deferral and timeouts.

Related Reading

• Zephyr Thread Stack Overflow Debugging with Runtime Monitoring
• Fixing Zephyr Deep Sleep Backup SRAM Retention on STM32
• Optimizing Cortex-M Tail Chaining for Sub-Microsecond Latency
• ARM Cortex-M NVIC Interrupt Latency Optimization Techniques
• Debugging Intermittent I2C Bus Hangs in Embedded Systems

References

1. Zephyr Project Documentation. “Workqueue Threads.” https://docs.zephyrproject.org/latest/kernel/services/threads/workqueue.html
2. Zephyr Project Documentation. “Work Queue APIs Reference.” https://docs.zephyrproject.org/latest/doxygen/html/group__workqueue__apis.html
3. Zephyr Project Documentation. “Interrupt Service Routines (ISRs).” https://docs.zephyrproject.org/latest/kernel/services/interrupts.html
4. ARM Limited. “Cortex-M4 Devices Generic User Guide.” https://developer.arm.com/documentation/dui0553/a