Debugging Linux Kernel Oops: Using Serial Console and KGDB

When an embedded Linux system encounters a kernel oops, the console fills with cryptic register dumps and stack traces that demand immediate interpretation. Unlike application crashes, kernel oopses threaten system stability and require low-level debugging techniques that many engineers rarely practice. This guide walks through capturing and analyzing kernel oopses using two essential tools: serial console for reliable log collection and KGDB for live kernel debugging.

+-------------------+      +-------------------+      +-------------------+
|   KERNEL OOPS     |      |   SERIAL CONSOLE  |      |      KGDB         |
|   OCCURS          | ---> |   CAPTURES        | ---> |   LIVE DEBUG      |
|  [Registers,      | UART |  [Full oops dump, | GDB  |  [Breakpoints,    |
|   Stack, PC]      |      |   Backtrace]      |      |   Regs, Memory]   |
+-------------------+      +-------------------+      +-------------------+
        |                         |                        |
        v                         v                        v
+-------------------+      +-------------------+      +-------------------+
|  UNSTABLE STATE   |      |  PERSISTENT LOG   |      |  ROOT CAUSE       |
|  SYSTEM MAY RESET |      |  ON HOST          |      |  IDENTIFIED       |
+-------------------+      +-------------------+      +-------------------+

Problem Statement: Kernel Oops Halts Development Progress

Consider an STM32MP1-based industrial controller that sporadically reboots during operation. The bootloader logs show no errors, and application logs cut off abruptly. Engineers suspect a kernel-space issue but lack visibility into what happens after the oops message flashes on the display before the system resets. Without reliable access to kernel debug information, root cause analysis becomes guesswork.

Root Cause Analysis: Why Kernel Oopses Are Elusive

Kernel oopses occur when the kernel detects an internal inconsistency but attempts to continue execution rather than immediately panicking. This behavior distinguishes an oops from a kernel panic: the system remains running, but in a potentially corrupted state. Common causes include:

• Null pointer dereferences in device drivers, often from missing validation before pointer use
• Stack overflows in interrupt handlers where limited kernel stack space is exceeded
• Invalid memory accesses from corrupted kernel data structures or use-after-free errors
• Synchronization bugs such as race conditions between interrupt context and process context
• Division by zero or other arithmetic exceptions in kernel code paths

The challenge lies in the ephemeral nature of oops information. On embedded systems with limited display output or graphical consoles that reset during crashes, the oops message may disappear before engineers can record it. Even when captured, the raw register dump requires translation into meaningful source code locations through tools like addr2line or kernel symbol tables. Without persistent logging, the evidence vanishes with the next reboot, leaving engineers with only the symptom (unexpected reset) but not the cause.

Solution Approach: Dual-Tool Debugging Strategy

Effective kernel oops debugging combines two complementary techniques that address different phases of the debugging lifecycle:

1. Serial Console: Captures kernel log output to a persistent host-side log, immune to display resets or GPU failures. This is the post-mortem foundation — without it, the oops message is lost.
2. KGDB: Provides live kernel debugging via gdb, enabling register inspection, stack tracing, and memory examination at the oops point. This transforms debugging from log analysis to interactive investigation.

These tools are not alternatives — they are sequential. Serial console captures what happened; KGDB lets you reproduce and inspect why it happened. Together they reduce oops debugging from hours of guesswork to minutes of focused analysis.

Serial Console Configuration

Kernel oops messages appear in the kernel ring buffer, accessible via dmesg or console drivers. Configuring a serial console ensures these messages route to a UART port independent of the display pipeline.

Device Tree Configuration (STM32MP1 example):

aliases {
    serial0 = &uart4;
};

chosen {
    stdout-path = "serial0:115200n8";
};

&uart4 {
    pinctrl-names = "default";
    pinctrl-0 = <&uart4_pins_a>;
    status = "okay";
};

Kernel Boot Arguments:

console=ttySTM0,115200n8

This configuration routes kernel output to the dedicated UART console, ensuring oops messages survive display server or GPU pipeline failures.

KGDB Setup for Live Debugging

KGDB requires compiling the kernel with debug symbols and enabling the KGDB over Console (kgdboc) driver. For ARM Cortex-A cores common in embedded MPUs:

Kernel Configuration:

CONFIG_DEBUG_INFO=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y

For standard UARTs, there is no need to write a custom low-level driver. The kgdboc driver will automatically use the serial port provided the underlying UART driver implements the poll_get_char and poll_put_char callbacks—which most modern serial drivers do out-of-the-box.

Kernel Boot Arguments:

kgdboc=ttySTM0,115200

Alternatively, configure or re-bind KGDB dynamically at runtime via sysfs:

echo ttySTM0 > /sys/module/kgdboc/parameters/kgdboc

To halt early boot and wait for GDB to attach, append kgdbwait to the bootargs. Connect a host machine running cross-GDB with the uncompressed kernel image (vmlinux) containing debug symbols:

# On host
arm-linux-gnueabihf-gdb vmlinux
(gdb) target remote /dev/ttyUSB0

Complete Working Code Examples

Capturing Oops Messages with Serial Console

On the host side, use screen or minicom to log serial output:

screen /dev/ttyUSB0 115200,cs8
# Press Ctrl+A then H to start logging to screenlog.0

When an oops occurs, the full register dump and backtrace appear in the log:

[   12.345678] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[   12.345789] pgd = (ptrval)
[   12.345901] [00000000] *pgd=00000000
[   12.346012] Internal error: Oops: 17 [#1] SMP ARM
[   12.346123] Modules linked in: ...
[   12.346234] CPU: 0 PID: 123 Comm: kworker/0:1 Not tainted 5.15.0-rc3-custom #1
[   12.346345] Hardware name: STM32MP157C-DK2 (DT)
[   12.346456] PC is at func_ptr_call+0x10/0x20
[   12.346567] LR is at worker_func+0x4c/0x80
[   12.346678] pc : [<c010a0b0>]    lr : [<c010a0d0>]    psr: 20000013
[   12.346789] sp : c020ffe0  ip : c010a0e0  fp : c010a0f0
[   12.346901] r10: 00000000  r9 : 00000000  r8 : 00000000
[   12.347012] r7 : 00000000  r6 : 00000000  r5 : 00000000  r4 : 00000000
[   12.347123] r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 00000000
[   12.347234] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   12.347345] Process kworker/0:1 (pid: 123, stack limit = 0xc020e000)
[   12.347456] Stack: (0xc020ffe0 to 0xc0210000)
[   12.347567] ffe0:                   00000000 00000000 00000000 00000000
[   12.347678] ff00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   12.347789] ff20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   12.347901] ff40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   12.348012] ff60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   12.348123] ff80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   12.348234] [<c010a0b0>] (func_ptr_call) from [<c010a0d0>] (worker_func+0x4c/0x80)
[   12.348345] [<c010a0d0>] (worker_func) from [<c010a0f0>] (worker_thread+0x58/0x70)
[   12.348456] [<c010a0f0>] (worker_thread) from [<c010a110>] (kthread+0xd0/0xf0)
[   12.348567] [<c010a110>] (kthread) from [<c010a130>] (ret_from_fork+0x14/0x24)
[   12.348678] Code: e1a00000 e59f3010 e5932000 e0822003 (e5903000)
[   12.348789] ---[ end trace 0000000000000000 ]---

Using KGDB to Inspect Oops State

With KGDB connected, halt the kernel at the oops point by setting a breakpoint on oops_enter:

(gdb) break oops_enter
Breakpoint 1 at 0xc0142b80: file kernel/panic.c, line 630.
(gdb) continue

When the oops occurs, KGDB stops execution and allows inspection:

(gdb) info registers
r0             0x0         0
r1             0x0         0
r2             0x0         0
r3             0x0         0
r4             0x0         0
r5             0x0         0
r6             0x0         0
r7             0x0         0
r8             0x0         0
r9             0x0         0
r10            0x0         0
fp             0xc010a0f0  3222315248
ip             0xc010a0e0  3222315232
sp             0xc020ffe0  3223388128
lr             0xc010a0d0  3222315216
pc             0xc010a0b0  3222315184
cpsr           0x20000013  536870931

(gdb) x/i $pc
=> 0xc010a0b0 <func_ptr_call+16>:   ldr r3, [r0, #0]

(gdb) print/x $r0
$1 = 0x0

(gdb) bt
#0  func_ptr_call (ptr=0x0) at drivers/example.c:45
#1  0xc010a0d0 in worker_func (data=0x0) at drivers/example.c:60
#2  0xc010a0f0 in worker_thread (data=0x0) at drivers/example.c:75
#3  0xc010a110 in kthread (thread=0xc120e000) at kernel/kthread.c:220
#4  0xc010a130 in ret_from_fork () at arch/arm/kernel/entry-common.S:120

The backtrace reveals a null pointer dereference in func_ptr_call at line 45 of drivers/example.c, where ptr is used without validation.

Verification and Testing Steps

Serial Console Verification

1. Confirm kernel output appears on serial port during normal boot:

   dmesg | grep -i console

   Should show: console [ttySTM0] enabled

2. Test oops capture by triggering a controlled oops (null pointer dereference):

   # On target (use with caution!)
   echo c > /proc/sysrq-trigger

   Verify the oops message appears in the serial log.

KGDB Verification

1. Confirm KGDB connection from the host:

   arm-linux-gnueabihf-gdb vmlinux
   (gdb) target remote /dev/ttyUSB0

   Should connect successfully to the waiting kernel.

2. Trigger an immediate drop into KGDB:

   # On target:
   echo g > /proc/sysrq-trigger

   KGDB halts execution and returns control to the (gdb) prompt. You can now set breakpoints, inspect variables, or continue (continue).

3. Trigger an oops to hit a breakpoint:

   (gdb) break oops_enter
   (gdb) continue

   Then on target:

   echo c > /proc/sysrq-trigger

   KGDB will trap at oops_enter, allowing pre-panic inspection.

Post-Debugging Validation

After applying a fix (e.g., adding null pointer check):

1. Reproduce the original scenario that triggered the oops.
2. Confirm no oops messages appear in serial console logs.
3. Verify system stability under stress testing for 24+ hours.

Summary

Kernel oopses on embedded Linux systems require specialized debugging techniques beyond application-level tools. By configuring serial console for reliable log capture and KGDB for live kernel inspection, engineers can transform cryptic oops dumps into actionable debugging sessions. Key takeaways:

• Serial console provides indispensable crash log persistence when displays fail.
• KGDB enables register-level inspection and stack tracing at the oops point.
• Combined, these tools reduce oops debugging from hours of guesswork to minutes of focused analysis.
• Always validate fixes by reproducing the original failure condition and confirming absence of oops messages.

Related Reading

• Fixing I2C Clock Stretching Timeouts on STM32
• Zephyr Thread Stack Overflow Debugging with Runtime Monitoring
• Debugging Zephyr Kernel Panic on STM32
• Cortex-M Cache Maintenance for DMA Coherency
• Watchdog Timer Design for Reliable Embedded Systems

References

1. Linux Kernel Documentation: Kernel Oopses - https://www.kernel.org/doc/html/latest/admin-guide/bug-hunting.html
2. Linux Kernel Documentation: KGDB - https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html
3. Linux Kernel Documentation: Serial Console - https://www.kernel.org/doc/html/latest/admin-guide/serial-console.html
4. “Linux Device Drivers, 3rd Edition” - Jonathan Corbet, Alessandro Rubini, Greg Kroah-Hartman
5. ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition - ARM DDI 0406C