
Under CONFIG_PREEMPT_RT=y, the Linux kernel converts standard spinlocks into sleeping synchronization primitives. This fundamental transformation breaks a foundational assumption in traditional driver design: that spin_lock_irqsave() is non-blocking and safe to call anywhere. When an embedded driver attempts to acquire a spinlock_t while holding a hardware spinlock, running in a non-threaded hardirq, or executing with preemption disabled, the runtime validation subsystem immediately triggers the fatal diagnostic: BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:48.
This guide examines the internal mechanics of PREEMPT_RT sleeping locks, analyzes the exact root cause of atomic sleep splats, demonstrates lockdep and ftrace diagnosis workflows, and establishes production-ready fix patterns that guarantee safety on both vanilla and real-time Linux kernels.
On standard (vanilla) Linux kernels configured with CONFIG_PREEMPT_NONE, CONFIG_PREEMPT_VOLUNTARY, or CONFIG_PREEMPT, spinlock_t is a hardware spinlock. Acquiring it disables preemption (and disables local interrupts if using the _irqsave variant). The CPU busy-waits until the lock is released. Because preemption is disabled throughout the critical section, high-priority real-time tasks (SCHED_FIFO or SCHED_DEADLINE) cannot preempt the lock holder, creating unbounded scheduling latencies.
To eliminate latency spikes and provide deterministic execution guarantees, PREEMPT_RT replaces non-raw spinlocks with rt_spinlock, an abstraction backed by struct rt_mutex_base.
+------------------+-----------------------+-----------------------+| Lock Type | Vanilla Linux Kernel | PREEMPT_RT Kernel |+------------------+-----------------------+-----------------------+| spinlock_t | Hardware spinlock | rt_spinlock (sleeping)|| raw_spinlock_t | Hardware spinlock | Hardware spinlock || rwlock_t | Hardware rwlock | rt_rwlock (sleeping) || raw_rwlock_t | Hardware rwlock | Hardware rwlock || mutex | Sleeping mutex | rtmutex (sleeping) |+------------------+-----------------------+-----------------------+
Under PREEMPT_RT, replacing spinlocks with sleeping locks achieves two vital real-time properties:
spinlock_t critical section remain preemptible by higher-priority tasks.spinlock_t held by a lower-priority task, the high-priority task blocks on the underlying rtmutex. The scheduler immediately boosts the effective priority of the lock holder to match the priority of the waiting task. Once the holder releases the lock, its original priority is restored, preventing priority inversion.Because rt_spin_lock() can block, its implementation invokes might_sleep():
+---------------------------+ +---------------------------+| Driver calls: | | Under PREEMPT_RT: || spin_lock_irqsave(&lock) | | spinlock_t -> rt_spinlock || inside atomic context | | contains might_sleep() |+-------------+-------------+ +-------------+-------------+| || Invokes rt_spin_lock() |+------------------------------------------------->+|v+---------------------------+| in_atomic() || irqs_off() || triggers __might_resched || BUG: sleeping in invalid || context! |+---------------------------+
A common misconception is that the kernel only panics if the lock is contested and actually attempts to sleep. This is incorrect. The might_sleep() macro evaluates:
if (in_atomic() || irqs_disabled())__might_resched(__FILE__, __LINE__, 0);
When CONFIG_DEBUG_ATOMIC_SLEEP is active, might_sleep() asserts that the caller is in a valid schedulable context unconditionally on lock entry, even if the lock is completely unheld and acquired without waiting.
In include/linux/spinlock_rt.h, the PREEMPT_RT implementation of spin_lock_irqsave() is defined as:
#define spin_lock_irqsave(lock, flags) \do { \typecheck(unsigned long, flags); \flags = 0; \spin_lock(lock); \} while (0)#define spin_unlock_irqrestore(lock, flags) \do { \typecheck(unsigned long, flags); \(void)flags; \spin_unlock(lock); \} while (0)
spin_lock_irqsave() on a regular spinlock_t does not disable CPU interrupts. Disabling hardware interrupts before acquiring a sleeping lock would break kernel rules, as tasks cannot yield the processor with hardware interrupts disabled.
Consequently, if code is executed inside a genuine hardirq context where the hardware interrupt line is already masked by the CPU, might_sleep() detects irqs_disabled() != 0 and triggers the BUG.
On PREEMPT_RT, standard device drivers registered via request_irq() have their handlers automatically converted into preemptible kernel threads (irq/X-name). Inside forced-threaded handlers, spinlock_t is safe to use.
The failure occurs when an interrupt handler executes in genuine hardirq context:
IRQF_NO_THREAD (e.g. system timers, inter-processor interrupts, cascading interrupt controllers, and ultra-low-latency triggers).request_threaded_irq(irq, top_half, bottom_half, ...).// ILLEGAL on PREEMPT_RT if registered with IRQF_NO_THREADstatic irqreturn_t sample_hardirq_handler(int irq, void *dev_id){struct sample_device *dev = dev_id;unsigned long flags;/* BUG: dev->lock is spinlock_t (sleeping rt_spinlock) in hardirq! */spin_lock_irqsave(&dev->lock, flags);sample_ack_hardware(dev);spin_unlock_irqrestore(&dev->lock, flags);return IRQ_HANDLED;}
Acquiring a raw_spinlock_t disables local CPU preemption (preempt_count > 0). If the critical section subsequently invokes any function that acquires a regular spinlock_t, might_sleep() detects in_atomic() == 1.
// ILLEGAL NESTING on PREEMPT_RTraw_spin_lock_irqsave(&dev->raw_hw_lock, flags);/* dev->data_lock is a regular spinlock_t */spin_lock(&dev->data_lock); // BUG: sleeping lock inside raw_spinlock!dev->packet_count++;spin_unlock(&dev->data_lock);raw_spin_unlock_irqrestore(&dev->raw_hw_lock, flags);
Kernel configuration CONFIG_PROVE_RAW_LOCK_NESTING=y exists specifically to catch this violation.
Standard timers (struct timer_list) execute in softirq context, which PREEMPT_RT runs inside preemptible ktimersoftd daemon threads. However, high-resolution timers (struct hrtimer) configured with HRTIMER_MODE_HARD (mandatory for high-frequency control loops, PWM generation, and hardware clock synchronization) execute in hardirq context.
// ILLEGAL on PREEMPT_RTstatic enum hrtimer_restart sample_hrtimer_callback(struct hrtimer *timer){struct sample_device *dev = container_of(timer, struct sample_device, hr_timer);/* hr_timer is initialized with HRTIMER_MODE_HARD -> hardirq context! */spin_lock(&dev->lock); // BUG: rt_spin_lock called in hardirq!sample_poll_sensor(dev);spin_unlock(&dev->lock);return HRTIMER_NORESTART;}
Driver routines that invoke preempt_disable(), local_irq_disable(), or local_irq_save() create an atomic region. Calling any regular spinlock_t function inside that block triggers might_sleep().
// ILLEGAL on PREEMPT_RTpreempt_disable();spin_lock(&dev->lock); // BUG: preempt_count > 0/* critical section */spin_unlock(&dev->lock);preempt_enable();
Ensure your development and test kernels enable comprehensive lockdep and atomic context validation options in .config:
CONFIG_PROVE_LOCKING=yCONFIG_DEBUG_ATOMIC_SLEEP=yCONFIG_PROVE_RAW_LOCK_NESTING=yCONFIG_DEBUG_SPINLOCK=yCONFIG_DEBUG_PREEMPT=yCONFIG_LOCKDEP=yCONFIG_DEBUG_LOCK_ALLOC=y
When an invalid context call occurs, the kernel prints a diagnostic backtrace:
BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:48in_atomic(): 1, irqs_disabled(): 128, non_block: 0, pid: 142, name: irq/42-samplepreempt_count: 1, expected: 0RCU nest depth: 0, expected: 0CPU: 2 PID: 142 Comm: irq/42-sample Not tainted 6.6.0-rt15 #1Hardware name: Embedded Board Rev B (DT)Call trace:dump_backtrace+0x94/0xecshow_stack+0x18/0x24dump_stack_lvl+0x48/0x60dump_stack+0x18/0x24__might_resched+0x138/0x168rt_spin_lock+0x2c/0x7csample_irq_handler+0x34/0x88 [sample_driver]handle_irq_event_percpu+0x58/0xb4handle_irq_event+0x48/0x88handle_edge_irq+0x98/0x1c4generic_handle_irq+0x34/0x4cgic_handle_irq+0x78/0x9c
Key fields in the diagnostic output:
kernel/locking/spinlock_rt.c:48: The location within the PREEMPT_RT sleeping spinlock implementation where might_sleep() fired.in_atomic(): 1: The current execution context is atomic (preemption is disabled or the CPU is in an interrupt handler).irqs_disabled(): 128: Hardware interrupts are disabled. On ARM/ARM64 architectures, 128 corresponds to bit 7 (I bit in DAIF / CPSR), confirming that the CPU core has local IRQs masked. On x86 architectures, non-zero indicates the IF flag in EFLAGS is cleared.preempt_count: 1, expected: 0: The per-task preemption counter indicates an active non-preemptible section.Call trace: Demonstrates the exact path: sample_irq_handler called rt_spin_lock, which called __might_resched.If a complex driver triggers atomic sleep violations indirectly through deep subsystem callbacks, use the kernel function_graph tracer to capture the call graph leading into rt_spin_lock:
# Set up ftrace filtersecho 0 > /sys/kernel/debug/tracing/tracing_onecho function_graph > /sys/kernel/debug/tracing/current_tracerecho rt_spin_lock > /sys/kernel/debug/tracing/set_ftrace_filterecho 1 > /sys/kernel/debug/tracing/tracing_on# Trigger the workload, then capture the tracecat /sys/kernel/debug/tracing/trace | head -n 30
Use when: The lock protects registers or state shared between non-threaded hardirq context (IRQF_NO_THREAD, primary ISRs, or hrtimer HARD) and process context.
raw_spinlock_t remains a genuine hardware spinlock on all kernel configurations (vanilla, PREEMPT, and PREEMPT_RT). It disables preemption and local interrupts without invoking might_sleep().
#include <linux/spinlock.h>#include <linux/interrupt.h>struct sample_device {raw_spinlock_t raw_lock; /* Protects hardware register access */void __iomem *base;u32 cached_status;};static irqreturn_t sample_hardirq_handler(int irq, void *dev_id){struct sample_device *dev = dev_id;unsigned long flags;/* Safe in hardirq on both vanilla and PREEMPT_RT */raw_spin_lock_irqsave(&dev->raw_lock, flags);dev->cached_status = readl_relaxed(dev->base + STATUS_REG);writel_relaxed(CLEAR_ACK, dev->base + ACK_REG);raw_spin_unlock_irqrestore(&dev->raw_lock, flags);return IRQ_HANDLED;}static int sample_device_init(struct sample_device *dev){raw_spin_lock_init(&dev->raw_lock);return 0;}
[!WARNING] Keep
raw_spinlock_tcritical sections as short as possible (sub-microsecond execution time). While holding araw_spinlock_t, preemption and hardware interrupts are disabled on the local CPU, which directly increases worst-case scheduling latency.
Use when: The data structure is accessed exclusively from process/task context (e.g. file_operations like read, write, ioctl, workqueues, or kernel threads).
Do not convert process-context locks to raw_spinlock_t. Retaining spinlock_t preserves real-time preemption and priority inheritance.
struct sample_device {spinlock_t data_lock; /* Thread-context data protection */struct list_head record_list;};static ssize_t sample_read(struct file *filp, char __user *buf,size_t count, loff_t *fpos){struct sample_device *dev = filp->private_data;/* On PREEMPT_RT, this is a sleeping rt_spinlock.* Process context is allowed to sleep; no BUG is triggered. */spin_lock(&dev->data_lock);/* copy buffered records */spin_unlock(&dev->data_lock);return count;}
Use when: Handling high-throughput peripherals where top-half latency must be minimal, but post-processing requires extensive parsing, memory allocation (GFP_KERNEL), or mutex acquisition.
This is the standard architectural design pattern for modern Linux drivers:
raw_spinlock_t to acknowledge hardware, mask the peripheral interrupt line, and returns IRQ_WAKE_THREAD.irq/X-name). Uses standard spinlock_t or struct mutex, can allocate memory, and can sleep without triggering warnings.+-------------------------------------------------------------------------+| Top-Half Primary ISR (Hardirq Context) || - Runs with interrupts disabled || - Uses raw_spinlock_t || - Reads & clears hardware interrupt register || - Returns IRQ_WAKE_THREAD |+------------------------------------+------------------------------------+|v+------------------------------------+------------------------------------+| Bottom-Half Threaded ISR (Preemptible Task Context) || - Runs in kernel thread context (irq/X-dev) || - Can sleep, allocate memory, acquire regular spinlock_t or mutex || - Preemptible by higher-priority real-time tasks |+-------------------------------------------------------------------------+
#include <linux/interrupt.h>#include <linux/mutex.h>struct sample_device {raw_spinlock_t hw_raw_lock; /* Top-half lock */struct mutex worker_mutex; /* Bottom-half sleeping lock */void __iomem *io_base;u32 pending_events;};/* Top-half: Executed in hardirq context */static irqreturn_t sample_primary_irq(int irq, void *dev_id){struct sample_device *dev = dev_id;u32 status;raw_spin_lock(&dev->hw_raw_lock);status = readl_relaxed(dev->io_base + IRQ_STATUS_REG);if (!status) {raw_spin_unlock(&dev->hw_raw_lock);return IRQ_NONE;}/* Mask peripheral interrupt & stash status */writel_relaxed(0, dev->io_base + IRQ_ENABLE_REG);dev->pending_events |= status;raw_spin_unlock(&dev->hw_raw_lock);return IRQ_WAKE_THREAD;}/* Bottom-half: Executed in threaded IRQ task context */static irqreturn_t sample_threaded_irq(int irq, void *dev_id){struct sample_device *dev = dev_id;u32 events;mutex_lock(&dev->worker_mutex);raw_spin_lock_irq(&dev->hw_raw_lock);events = dev->pending_events;dev->pending_events = 0;/* Re-enable peripheral interrupt */writel_relaxed(IRQ_ALL_ENABLED, dev->io_base + IRQ_ENABLE_REG);raw_spin_unlock_irq(&dev->hw_raw_lock);/* Process events using sleeping allocations or subsystem calls */sample_process_events(dev, events);mutex_unlock(&dev->worker_mutex);return IRQ_HANDLED;}int sample_setup_interrupts(struct platform_device *pdev, struct sample_device *dev){int irq = platform_get_irq(pdev, 0);raw_spin_lock_init(&dev->hw_raw_lock);mutex_init(&dev->worker_mutex);return devm_request_threaded_irq(&pdev->dev, irq,sample_primary_irq,sample_threaded_irq,IRQF_SHARED,"sample_device", dev);}
Use when: Transferring data streams between a hardirq producer and a process-context consumer.
Lockless single-producer single-consumer ring buffers (kfifo) eliminate lock acquisition entirely between contexts:
#include <linux/kfifo.h>#define FIFO_SIZE 1024struct sample_device {DECLARE_KFIFO(rx_fifo, u8, FIFO_SIZE);wait_queue_head_t read_wait;};/* Top-half hardirq producer */static irqreturn_t sample_rx_hardirq(int irq, void *dev_id){struct sample_device *dev = dev_id;u8 byte = readb_relaxed(dev->io_base + RX_DATA_REG);/* Lockless put into single-producer FIFO */kfifo_put(&dev->rx_fifo, byte);wake_up_interruptible(&dev->read_wait);return IRQ_HANDLED;}/* Process-context consumer */static ssize_t sample_user_read(struct file *filp, char __user *buf,size_t count, loff_t *fpos){struct sample_device *dev = filp->private_data;unsigned int copied;int ret;ret = wait_event_interruptible(dev->read_wait, !kfifo_is_empty(&dev->rx_fifo));if (ret)return ret;ret = kfifo_to_user(&dev->rx_fifo, buf, count, &copied);return ret ? ret : copied;}
Under PREEMPT_RT, lock acquisition hierarchies must adhere strictly to raw versus sleeping boundary rules:
Legal Nesting Order:+----------------------------+| Outer Lock: spinlock_t | (Preemptible sleeping lock)| +------------------------+| | Inner Lock: | (Hardware non-sleeping spinlock)| | raw_spinlock_t || +------------------------++----------------------------+Illegal Nesting Order (Triggers Splat / Deadlock):+----------------------------+| Outer Lock: raw_spinlock_t | (Preemption disabled!)| +------------------------+| | Inner Lock: spinlock_t | <--- BUG: might_sleep() called with| +------------------------+ preempt_count > 0!+----------------------------+
Kernel config CONFIG_PROVE_RAW_LOCK_NESTING=y tracks lock classes and asserts that:
raw_spinlock_t must never attempt to acquire a spinlock_t, rwlock_t, or mutex.spinlock_t may safely acquire a raw_spinlock_t, provided no inversions occur across CPUs.If an invalid nesting sequence occurs, lockdep emits an exhaustive lock hierarchy report detailing the offending lock chain, lock class names, and stack traces for both acquisition sites.
After applying lock conversions, benchmark the system under synthetic load to verify that replacing spinlocks does not degrade real-time performance.
# 1. Verify kernel dmesg for atomic context warningsdmesg | grep -E "BUG: sleeping|might_sleep|raw_lock_nesting"# 2. Run cyclictest under load to verify RT scheduling latency# Target worst-case latency: < 50 us on industrial embedded targetscyclictest --mlockall --smp --priority=99 --interval=1000 --distance=0 --duration=15m# 3. Inject lock contention and memory bus stressstress-ng --lockbus 4 --vm 2 --vm-bytes 128M --timeout 60s
T: 0 ( 1842) P:99 I:1000 C: 900000 Min: 3 Act: 8 Avg: 7 Max: 28T: 1 ( 1843) P:99 I:1000 C: 900000 Min: 3 Act: 9 Avg: 8 Max: 34
raw_spinlock_t critical sections are strictly bounded and not executing heavy register polling loops or memory copies while interrupts are disabled.The BUG: sleeping function called from invalid context diagnostic under PREEMPT_RT is an explicit design invariant enforced by the Linux kernel. When real-time preemption transforms spinlock_t into an rtmutex-backed sleeping primitive, acquiring it inside any atomic context triggers might_sleep().
Decision Workflow:Is the lock acquired from hardirq, NMI, or hrtimer HARD context?YES -> Use raw_spinlock_t (keep critical section < 5 us)NO -> Is the lock held inside a raw_spinlock critical section?YES -> Violates CONFIG_PROVE_RAW_LOCK_NESTING: refactor lock hierarchyNO -> Is execution strictly in task/thread context?YES -> Retain spinlock_t (allows preemption and PI boosting)NO -> For mixed IRQ/thread workloads, decouple with request_threaded_irq()
By decoupling top-half register acknowledgment with raw_spinlock_t from bottom-half deferred processing with request_threaded_irq(), embedded drivers achieve optimal real-time determinism while remaining completely free of atomic sleep bugs.
Quick Links
Legal Stuff





