Fixing Flaky CI Pipelines in Embedded Firmware

Flaky CI pipelines waste engineering hours, erode trust in automation, and delay releases. In embedded firmware, flakiness is amplified by hardware dependencies, timing constraints, and environmental differences between local development and CI runners. This article dissects the root causes, shows how to systematically detect flaky tests, and provides battle-tested stabilization patterns for GitHub Actions.

The Problem: Why Embedded CI Flakiness Is Different

Embedded firmware CI fails in ways pure software CI does not. A test that passes 99 times locally may fail on the 100th run in CI because:

• Hardware timing variance: Flash write cycles, ADC settling times, and interrupt latencies vary across runs
• Shared runner state: GitHub Actions hosted runners reuse VMs; leftover USB device claims, GPIO states, or memory maps poison subsequent runs
• Non-deterministic HIL: Hardware-in-the-loop setups introduce cable jitter, power supply noise, and probe connection flakiness
• Toolchain non-determinism: Linker script memory layout, compiler optimization artifacts, and binary reproducibility issues
• Resource contention: Parallel test jobs competing for limited USB ports, JTAG adapters, or target boards

The search query engineers type at 2 AM: “GitHub Actions embedded tests flaky”, “firmware CI intermittent failures”, “hardware-in-the-loop test unreliable”.

Root Cause Taxonomy

Before fixing, classify each flaky failure into one of five categories. This taxonomy drives the remediation strategy.

+----------------------------------------------------------------------+
|                        FLAKY FAILURE TAXONOMY                        |
+-------------------+---------------------------------+----------------+
| CLASS             | SYMPTOMS                        | FIX PATH       |
+-------------------+---------------------------------+----------------+
| TIMING SENSITIVE  | Random timeouts, race cond.,    | Mocks, bounded |
|                   | scheduling jitter               | timeout loops  |
+-------------------+---------------------------------+----------------+
| HARDWARE STATE    | USB claim fails, GPIO stuck,    | Host runner    |
| LEAKAGE           | stale flash sector              | cleanup hook   |
+-------------------+---------------------------------+----------------+
| HIL ENVIRONMENT   | Probe disconnect, power dip,    | Dedicated HIL  |
| NOISE             | cable/connector jitter          | queue + retry  |
+-------------------+---------------------------------+----------------+
| TOOLCHAIN         | Binary differs, link address    | Reproducible   |
| NON-DETERMINISM   | shifts across runner VMs        | build pipeline |
+-------------------+---------------------------------+----------------+
| RESOURCE          | "Port busy", "Target lost",     | Serialized pool|
| CONTENTION        | parallel debug job collisions   | or concurrency |
+-------------------+---------------------------------+----------------+

A test is formally classified as flaky when it yields non-deterministic outcomes across identical Git commits without code or toolchain modifications. The metric governing flakiness severity is defined as:

N_inconsistent_runs
Flake Rate (%) = --------------------- * 100
                    N_total_executions

Deep Dive: Timing-Sensitive Failures

Timing-sensitive flakiness is the most insidious category because it masquerades as genuine race conditions in the production code. The failure manifests as intermittent assertion violations on cycle counts, interrupt latencies, or RTOS scheduling boundaries. The root cause is almost always environmental: CI runner load shifts thread scheduling, thermal throttling changes CPU frequency, or container noise introduces microsecond-scale jitter.

Consider a test verifying that an ISR completes within 15 microseconds. On a bare-metal development board, this passes consistently. In a GitHub Actions Ubuntu runner sharing a physical host with other jobs, the same test fails when a noisy neighbor triggers CPU frequency scaling. The fix is not to relax the timing requirement — that masks real regressions. Instead, decouple the timing measurement from the test assertion. Mock the timer peripheral in unit tests, and gate hardware-verified timing tests behind a separate HIL pipeline with dedicated runners.

Deep Dive: Hardware State Leakage

GitHub Actions hosted runners reuse VM instances across workflow runs. A test that opens /dev/ttyACM0 to communicate with an STM32 via USB and crashes mid-execution leaves the device claimed. The subsequent workflow run fails with “Resource temporarily unavailable” — a classic flaky signal that has nothing to do with the code under test.

The remediation requires three layers: runner-level cleanup in always() blocks, test-process-level resource tracking, and CI-level isolation. The runner cleanup resets USB subsystem state and kills stray debug probe processes. The test framework tracks every opened resource and ensures deterministic teardown. The CI configuration uses concurrency groups to serialize hardware-dependent jobs.

Deep Dive: HIL Environment Noise

Hardware-in-the-loop setups introduce physical-world variability that pure software CI never encounters. A probe disconnects because a cable shifted. The target board browns out because the USB power delivery negotiated 500mA but the test draws 800mA during flash programming. The JTAG adapter enters an undefined state because the previous test left the TAP controller in an unexpected state.

These failures are not code bugs — they are infrastructure reliability problems. The solution is a dedicated HIL queue with mandatory health checks before each test run, automatic retry with full hardware reset between attempts, and physical infrastructure monitoring (probe connection status, target power rail voltage, environmental temperature).

Deep Dive: Toolchain Non-Determinism

A test passes on Monday and fails on Tuesday with no code changes. The binary size differs by 12 bytes. The linker placed a critical section at a different address. The compiler emitted a different instruction sequence for the same source due to a minor version bump in the hosted runner’s toolchain.

Non-reproducible builds are a silent flakiness amplifier. The fix is enforcing reproducible builds: strip build timestamps, normalize debug prefix maps, pin exact toolchain versions in CI, and verify binary equivalence across clean builds as a CI gate.

Deep Dive: Resource Contention

Parallel test jobs compete for scarce physical resources: USB ports, JTAG adapters, target boards, license servers for proprietary tools. Job A claims the only J-Link; Job B times out waiting. Job C starts flashing while Job D is mid-debug-session. The failure is non-deterministic because it depends on scheduler whims.

The fix is a resource pool with explicit acquisition and release semantics, integrated with GitHub Actions concurrency controls. Each job acquires a lease before touching hardware, the lease has a timeout, and the always() block guarantees release even on cancellation.

Systematic Flaky Test Detection

You cannot fix what you cannot measure. Implement a three-layer detection strategy.

Layer 1: CI-Native Flaky Detection (GitHub Actions)

In embedded firmware CI, host-based test harnesses (compiled via CMake/Ninja using Ceedling, Unity, or GoogleTest) should emit standard JUnit XML. CMake/CTest natively produces JUnit XML via the --output-junit flag. GitHub Actions can then parse this XML to surface annotations, highlight timing outliers, and track flaky executions without external SaaS dependencies.

# .github/workflows/test.yml
name: Embedded Firmware Host Tests

on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  host-unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Toolchain and Test Dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y cmake ninja-build gcc-multilib

      - name: Configure CMake Host Build
        run: |
          cmake -B build -G Ninja \
            -DCMAKE_BUILD_TYPE=Debug \
            -DBUILD_TESTING=ON \
            -DENABLE_MOCK_DRIVERS=ON

      - name: Compile Unit Tests
        run: cmake --build build

      - name: Execute CTest with JUnit Reporting
        run: |
          # CTest natively exports structured JUnit XML for all executed test suites
          ctest --test-dir build --output-on-failure --output-junit ctest-report.xml

      - name: Publish Test Annotations
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Firmware Unit Test Report
          path: ctest-report.xml
          reporter: java-junit
          fail-on-error: true

      - name: Upload Test Report Artifact
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: junit-test-results
          path: ctest-report.xml

Layer 2: Automated Repeated Runs

Intermittent race conditions and subtle memory corruption often pass on single runs. Schedule a nightly workflow that executes the test suite 50 consecutive times. Because GitHub Actions steps execute with set -e by default, any test failure would prematurely abort the evaluation loop; use set +e to ensure all 50 iterations run to completion.

# .github/workflows/flaky-detection.yml
name: Flaky Test Stress Detector
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC nightly
  workflow_dispatch:

jobs:
  flaky-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build Test Binaries
        run: |
          cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON
          cmake --build build

      - name: Execute 50 Stress Iterations
        run: |
          set +e
          mkdir -p test_logs
          for i in $(seq 1 50); do
            echo "Starting stress run iteration $i/50..."
            ctest --test-dir build --output-junit "test_logs/run_$i.xml" > "test_logs/console_$i.log" 2>&1
            STATUS=$?
            echo "Iteration $i finished with exit code $STATUS" >> test_logs/summary.txt
          done
          set -e

      - name: Analyze Flakiness Distribution
        run: |
          python3 scripts/analyze_flakiness.py test_logs > flaky_report.md
          cat flaky_report.md >> $GITHUB_STEP_SUMMARY

      - name: Open Issue for Flaky Regressions
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            if (fs.existsSync('flaky_report.md')) {
              const report = fs.readFileSync('flaky_report.md', 'utf8');
              if (report.includes('| FLAKY |')) {
                await github.rest.issues.create({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  title: '[Flaky CI Alert] Non-deterministic tests detected in nightly stress run',
                  body: report,
                  labels: ['flaky-test', 'ci-reliability']
                });
              }
            }

The analysis script aggregates pass/fail counts across all XML reports to calculate the exact flake rate per test:

# scripts/analyze_flakiness.py
import sys, glob, xml.etree.ElementTree as ET

def parse_runs(log_dir):
    stats = {}
    for xml_path in glob.glob(f"{log_dir}/*.xml"):
        tree = ET.parse(xml_path)
        for tc in tree.iter("testcase"):
            name = f"{tc.attrib.get('classname', '')}.{tc.attrib.get('name', '')}"
            stats.setdefault(name, {"pass": 0, "fail": 0})
            if tc.find("failure") is not None or tc.find("error") is not None:
                stats[name]["fail"] += 1
            else:
                stats[name]["pass"] += 1
    return stats

def main():
    log_dir = sys.argv[1] if len(sys.argv) > 1 else "test_logs"
    stats = parse_runs(log_dir)
    print("## Nightly Flaky Test Summary\n")
    print("| Test Case | Passes | Failures | Flake Rate | Status |")
    print("| :--- | :--- | :--- | :--- | :--- |")
    for name, counts in sorted(stats.items()):
        total = counts["pass"] + counts["fail"]
        if total == 0: continue
        rate = (counts["fail"] / total) * 100.0
        status = "FLAKY" if 0 < counts["fail"] < total else ("FAILING" if counts["pass"] == 0 else "STABLE")
        print(f"| `{name}` | {counts['pass']} | {counts['fail']} | {rate:.1f}% | **{status}** |")

if __name__ == "__main__":
    main()

Layer 3: Local Reproduction Harness

Developers must reproduce intermittent failures locally before pushing commits. A local reproduction harness stresses CPU scheduling, introduces artificial memory pressure via stress-ng, and activates AddressSanitizer (ASan) runtime instrumentation:

#!/usr/bin/env bash
# scripts/reproduce_flaky.sh
# Usage: ./scripts/reproduce_flaky.sh <test_binary_or_filter> [iterations=100]

set -euo pipefail

TEST_TARGET="${1:-all}"
ITERATIONS="${2:-100}"

echo "======================================================"
echo "Stressing test target: ${TEST_TARGET} (${ITERATIONS} iterations)"
echo "Simulating CI thread preemption and memory jitter..."
echo "======================================================"

# Enable ASan memory detection flags
export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"

FAIL_COUNT=0

for i in $(seq 1 "${ITERATIONS}"); do
  # Run background CPU load generator to trigger OS context switch jitter
  stress-ng --cpu 2 --cpu-load 75 --timeout 1s > /dev/null 2>&1 &
  STRESS_PID=$!

  if ! ./build/tests/unit_tests --filter="${TEST_TARGET}" > "run_${i}.log" 2>&1; then
    echo "[!] INCONSISTENT FAILURE on iteration ${i}"
    FAIL_COUNT=$((FAIL_COUNT + 1))
  fi

  wait "${STRESS_PID}" 2>/dev/null || true
done

FLAKE_RATE=$(awk "BEGIN {printf \"%.2f\", (${FAIL_COUNT}/${ITERATIONS})*100}")
echo "Finished ${ITERATIONS} iterations. Failures: ${FAIL_COUNT} (Flake Rate: ${FLAKE_RATE}%)"

if [ "${FAIL_COUNT}" -gt 0 ]; then
  exit 1
fi
echo "Result: STABLE"

Stabilization Patterns

Pattern 1: Hardware Abstraction Layer for Testability

Decouple hardware-dependent code behind interfaces. Test the logic with mocks; test the hardware integration separately.

+-----------------------------------------------------------------------+
|                         HAL TEST ARCHITECTURE                         |
+-----------------------------------------------------------------------+
|                                                                       |
|  [HOST CI: Fast & Deterministic]       [HIL / TARGET: Real Silicon]   |
|                                                                       |
|  +----------------------------+        +---------------------------+  |
|  |     Unit Test Runner       |        |   Automated HIL Runner    |  |
|  |   (CTest / Ceedling)       |        |  (pytest-embedded / JTAG) |  |
|  +--------------+-------------+        +-------------+-------------+  |
|                 | stimulates                         | flashes/tests  |
|                 v                                    v                |
|  +----------------------------+        +---------------------------+  |
|  |   Firmware Business Logic  |        |  Firmware Business Logic  |  |
|  |  (Algorithms, State Machine|        | (Algorithms, State Machine|  |
|  |   Protocol Decoders, etc.) |        |  Protocol Decoders, etc.) |  |
|  +--------------+-------------+        +-------------+-------------+  |
|                 | calls                              | calls          |
|                 v                                    v                |
|  +----------------------------+        +---------------------------+  |
|  |     HAL Interface API      |        |     HAL Interface API     |  |
|  |  (adc_read, gpio_set, etc) |        | (adc_read, gpio_set, etc) |  |
|  +--------------+-------------+        +-------------+-------------+  |
|                 |                                    |                |
|                 v                                    v                |
|  +----------------------------+        +---------------------------+  |
|  |     Mock HAL Driver        |        |    Hardware HAL Driver    |  |
|  |  (Synthetic test vectors,  |        | (STM32 / NXP peripheral   |  |
|  |   zero hardware reliance)  |        |  registers, DMA, NVIC)    |  |
|  +----------------------------+        +-------------+-------------+  |
|                                                      |                |
|                                                      v                |
|                                        +---------------------------+  |
|                                        |     Physical Hardware     |  |
|                                        |  (ADC Pins, Sensors, Bus) |  |
|                                        +---------------------------+  |
|                                                                       |
|  CI Pipeline: Runs on Host Runner       HIL Pipeline: Dedicated Bench |
|  Execution: ~50 ms, 100% stable         Execution: ~30 s, HW verified |
+-----------------------------------------------------------------------+

Before (flaky - direct register access with unbounded polling):

// adc_reader.c - tightly coupled to physical hardware registers
uint16_t adc_read_channel(uint8_t ch) {
    ADC1->SQR3 = ch;
    ADC1->CR2 |= ADC_CR2_SWSTART;
    // Unbounded poll: if ADC clock is disabled or runner host drops USB, CI hangs indefinitely
    while (!(ADC1->SR & ADC_SR_EOC));
    return (uint16_t)ADC1->DR;
}

After (testable - decoupled interface with timeout guards):

// adc_driver.h - hardware-agnostic interface
#ifndef ADC_DRIVER_H
#define ADC_DRIVER_H

#include <stdint.h>
#include <stdbool.h>

typedef struct {
    bool (*init)(void);
    bool (*read_channel)(uint8_t channel, uint16_t *value_out);
} AdcDriver;

/* Core business logic under test */
bool adc_compute_average(const AdcDriver *driver, uint8_t ch, uint8_t samples, uint16_t *avg_out);

#endif /* ADC_DRIVER_H */

// adc_stm32.c - physical hardware implementation with bounded polling
#include "adc_driver.h"
#include "stm32f4xx.h"

#define ADC_TIMEOUT_CYCLES  100000UL

static bool stm32_adc_init(void) {
    RCC->APB2ENR |= RCC_APB2ENR_ADC1EN;
    ADC1->CR2 |= ADC_CR2_ADON;
    return true;
}

static bool stm32_adc_read(uint8_t ch, uint16_t *val) {
    if (!val || ch > 18) return false;
    ADC1->SQR3 = ch;
    ADC1->CR2 |= ADC_CR2_SWSTART;

    uint32_t timeout = ADC_TIMEOUT_CYCLES;
    while (!(ADC1->SR & ADC_SR_EOC)) {
        if (--timeout == 0) {
            return false; /* Bounded timeout: prevents CI runner hangs */
        }
    }
    *val = (uint16_t)ADC1->DR;
    return true;
}

const AdcDriver ADC_STM32_DRIVER = {
    .init = stm32_adc_init,
    .read_channel = stm32_adc_read
};

// adc_mock.c - deterministic host mock implementation
#include "adc_driver.h"

#define MAX_CHANNELS 8
static uint16_t mock_table[MAX_CHANNELS] = {0};

void mock_adc_set_channel(uint8_t ch, uint16_t value) {
    if (ch < MAX_CHANNELS) mock_table[ch] = value;
}

static bool mock_init(void) { return true; }
static bool mock_read(uint8_t ch, uint16_t *val) {
    if (!val || ch >= MAX_CHANNELS) return false;
    *val = mock_table[ch];
    return true;
}

const AdcDriver ADC_MOCK_DRIVER = {
    .init = mock_init,
    .read_channel = mock_read
};

// test_adc_filter.c - host unit test executed in standard CI runners
#include "unity.h"
#include "adc_driver.h"

void mock_adc_set_channel(uint8_t ch, uint16_t value);

void test_moving_average_filters_out_noise(void) {
    const AdcDriver *driver = &ADC_MOCK_DRIVER;
    TEST_ASSERT_TRUE(driver->init());

    mock_adc_set_channel(0, 1024);
    uint16_t avg = 0;
    TEST_ASSERT_TRUE(adc_compute_average(driver, 0, 4, &avg));
    TEST_ASSERT_EQUAL_UINT16(1024, avg);
}

Pattern 2: Deterministic Test Data and Seeds

Eliminate randomness in tests. Use fixed seeds, deterministic generators, and explicit test vectors.

// BAD: Random test data
TEST(RandomData, BufferOverflow) {
    uint8_t buf[64];
    size_t len = rand() % 128;  // Non-deterministic!
    fill_random(buf, len);
    TEST_ASSERT_FALSE(check_overflow(buf, len));
}

// GOOD: Exhaustive test vectors
static const struct {
    size_t input_len;
    bool expect_overflow;
} test_vectors[] = {
    {0, false}, {32, false}, {63, false}, {64, true}, {65, true}, {127, true}
};

TEST(BufferOverflow, ExhaustiveVectors) {
    for (size_t i = 0; i < ARRAY_SIZE(test_vectors); i++) {
        uint8_t buf[64] = {0};
        bool result = check_overflow(buf, test_vectors[i].input_len);
        TEST_ASSERT_EQUAL(test_vectors[i].expect_overflow, result);
    }
}

Pattern 3: Test Quarantine Pipeline

Don’t let flaky tests block merges. Quarantine them in a separate non-blocking pipeline with tracking.

# .github/workflows/test.yml
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run stable tests
        run: make test STABLE=1
      - name: Run quarantined tests (non-blocking)
        run: make test QUARANTINE=1 || true
        continue-on-error: true
      - name: Report quarantined results
        if: always()
        run: |
          python3 scripts/quarantine_report.py \
            --input quarantine_results.xml \
            --output quarantine_dashboard.json
      - name: Update flaky dashboard
        uses: actions/upload-artifact@v4
        with:
          name: flaky-dashboard
          path: quarantine_dashboard.json

Quarantine dashboard tracks:

• Test name and failure rate
• First seen date
• Assigned owner
• Root cause status (investigating / fixed / wont-fix)
• Timebox deadline

Pattern 4: Reproducible Builds for Binary Comparisons

Flaky tests sometimes stem from non-reproducible builds where binary layout, interrupt vector tables, or string offsets shift across runs. Ensure absolute binary reproducibility by normalizing file paths, disallowing date/time macros, and stripping non-deterministic ELF notes:

# CMakeLists.txt - enforce reproducible embedded builds
# Normalize compiler source file paths for debug symbols and macros
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -ffile-prefix-map=${CMAKE_SOURCE_DIR}=. -Wdate-time")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ffile-prefix-map=${CMAKE_SOURCE_DIR}=. -Wdate-time")

# Strip variable ELF metadata and notes using cross-compiler objcopy
if(CMAKE_OBJCOPY)
    add_custom_command(TARGET firmware POST_BUILD
        COMMAND ${CMAKE_OBJCOPY} --remove-section .comment --remove-section .note.gnu.build-id $<TARGET_FILE:firmware>
    )
endif()

# .github/workflows/reproducible.yml
- name: Verify Reproducible Binary Output
  run: |
    # Pin build timestamp epoch to latest commit timestamp
    export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

    cmake -B build_a -G Ninja -DCMAKE_BUILD_TYPE=Release
    cmake --build build_a --target firmware

    cmake -B build_b -G Ninja -DCMAKE_BUILD_TYPE=Release
    cmake --build build_b --target firmware

    echo "Verifying bit-for-bit equivalence between builds..."
    sha256sum build_a/firmware.elf build_b/firmware.elf

    # Enforce strict bitwise equivalence; fail the CI gate immediately on divergence
    if ! cmp -s build_a/firmware.elf build_b/firmware.elf; then
      echo "::error::Binary non-determinism detected between clean builds!"
      exit 1
    fi
    echo "Bit-for-bit binary reproduction confirmed."

Pattern 5: Resource Pool for Serialized Hardware Access

When multiple parallel test jobs require access to physical target hardware (e.g. STM32 Nucleo boards, J-Link debug probes, or logic analyzers), unsynchronized access causes hard bus collisions, broken SWD connections, and erratic test failures.

Approach A: Server-Side GitHub Actions Concurrency (Recommended)

GitHub Actions provides built-in server-side serialization via concurrency groups. Assigning a concurrency group per physical test bench ensures only one workflow accesses the target hardware at a time:

# .github/workflows/hil-pool.yml
jobs:
  hil-test:
    runs-on: [self-hosted, hil-pool]
    # Serializes access across all concurrent PRs and runs for this runner bench
    concurrency:
      group: hil-hardware-pool-${{ runner.name }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v4
      - name: Execute HIL Hardware Test Suite
        run: |
          # Safe: No other runner job can touch the J-Link probe simultaneously
          pytest tests/hil/ --target=stm32f4 --probe-serial=105001234

Approach B: Host-Level Process Lock for Shared Daemons

If multiple runner processes or developers share the same host machine and USB probes, command-level locking with the Linux standard flock(1) command provides kernel-managed mutual exclusion:

# Wraps test invocation in a non-reentrant file descriptor lock held by the kernel
flock --exclusive --timeout 300 /var/lock/hil_hardware.lock make hil-test

For cross-step coordination across distinct workflow steps, use an atomic file-lock manager that tracks process IDs and automatically reclaims orphaned locks left by cancelled runs:

# scripts/hil_lock.py
import os, sys, time, argparse

LOCK_FILE = "/tmp/hil_hardware.lock"

def is_pid_alive(pid: int) -> bool:
    try:
        os.kill(pid, 0)
        return True
    except (OSError, ProcessLookupError):
        return False

def acquire(timeout: int = 300):
    start = time.time()
    while time.time() - start < timeout:
        try:
            # Atomic creation: fails with FileExistsError if locked
            fd = os.open(LOCK_FILE, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
            with os.fdopen(fd, 'w') as f:
                f.write(f"{os.getpid()}\n{time.time()}\n")
            print(f"[hil_lock] Lock successfully acquired by PID {os.getpid()}")
            return
        except FileExistsError:
            # Check for stale lock from terminated processes
            try:
                with open(LOCK_FILE, 'r') as f:
                    holder_pid = int(f.readline().strip())
                if not is_pid_alive(holder_pid):
                    print(f"[hil_lock] Reclaiming stale lock from dead PID {holder_pid}")
                    os.unlink(LOCK_FILE)
                    continue
            except (ValueError, FileNotFoundError, PermissionError):
                pass
            time.sleep(1)
    raise TimeoutError(f"Could not acquire HIL lock within {timeout} seconds")

def release():
    try:
        os.unlink(LOCK_FILE)
        print("[hil_lock] Lock successfully released")
    except FileNotFoundError:
        pass

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="HIL Physical Hardware Resource Lock")
    parser.add_argument("action", choices=["acquire", "release"])
    parser.add_argument("--timeout", type=int, default=300)
    args = parser.parse_args()

    if args.action == "acquire":
        acquire(args.timeout)
    elif args.action == "release":
        release()

GitHub Actions Specific Hardening

Cache Key Stability

Unstable cache keys cause “cache miss” rebuilds that shift timing and trigger flakiness.

# BAD: Cache key changes on every commit
- uses: actions/cache@v4
  with:
    path: build/
    key: build-${{ github.sha }}

# GOOD: Stable key based on inputs
- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/cmake
      ~/.ccache
    key: ${{ runner.os }}-cmake-${{ hashFiles('CMakeLists.txt', 'cmake/**') }}
    restore-keys: |
      ${{ runner.os }}-cmake-

Runner Warmup

Cold runners have different thermal and frequency characteristics. Warm up before timing-sensitive tests.

- name: Warm up runner
  run: |
    # Run CPU-intensive task to stabilize frequency governor
    stress-ng --cpu $(nproc) --timeout 30s
    # Wait for thermal equilibrium
    sleep 10

Matrix Strategy for Environment Coverage

Test across multiple toolchain versions to catch environment-specific flakiness early.

strategy:
  matrix:
    include:
      - gcc: arm-none-eabi-gcc-12.3
        cmake: 3.28
      - gcc: arm-none-eabi-gcc-13.2
        cmake: 3.29
      - gcc: arm-none-eabi-gcc-13.3
        cmake: 3.30
  fail-fast: false  # Collect all results

Monitoring and Prevention

Flaky Test Dashboard

Build a lightweight persistence layer to track intermittent failures across CI pipelines over time:

# scripts/flaky_dashboard.py
import sqlite3
from datetime import datetime, timezone, timedelta

DB = "flaky_history.db"

def record_result(test_name: str, passed: bool, timestamp: str = None):
    ts = timestamp or datetime.now(timezone.utc).isoformat()
    with sqlite3.connect(DB) as conn:
        conn.execute("""
            CREATE TABLE IF NOT EXISTS results (
                test_name TEXT, passed BOOLEAN, timestamp TEXT
            )
        """)
        conn.execute("INSERT INTO results VALUES (?, ?, ?)", (test_name, passed, ts))
        conn.commit()

def flakiness_rate(test_name: str, days: int = 30) -> float:
    with sqlite3.connect(DB) as conn:
        cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
        cur = conn.execute("""
            SELECT COUNT(*) as total, SUM(CASE WHEN passed THEN 1 ELSE 0 END) as passed
            FROM results WHERE test_name=? AND timestamp > ?
        """, (test_name, cutoff))
        row = cur.fetchone()
        if not row or row[0] == 0:
            return 0.0
        passed_count = row[1] or 0
        return 1.0 - (passed_count / row[0])

def top_flaky_tests(limit: int = 10, days: int = 30):
    with sqlite3.connect(DB) as conn:
        cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
        cur = conn.execute("""
            SELECT test_name, 
                   COUNT(*) as runs,
                   SUM(CASE WHEN passed THEN 1 ELSE 0 END) as passes,
                   1.0 - (SUM(CASE WHEN passed THEN 1 ELSE 0 END) * 1.0 / COUNT(*)) as flake_rate
            FROM results 
            WHERE timestamp > ?
            GROUP BY test_name
            HAVING COUNT(*) >= 5
            ORDER BY flake_rate DESC
            LIMIT ?
        """, (cutoff, limit))
        return cur.fetchall()

if __name__ == "__main__":
    print("Top Flaky Tests (Last 30 Days):")
    for name, runs, passes, rate in top_flaky_tests():
        print(f"  {name}: {rate*100:.1f}% failure rate ({passes}/{runs} passed)")

Code Review Gates

Add flakiness prevention to your PR template:

## Flakiness Prevention Checklist
- [ ] New tests use hardware abstraction (mocks for CI)
- [ ] No exact timing assertions — use bounded ranges
- [ ] Test data is deterministic (fixed seeds, explicit vectors)
- [ ] Test runs 20x locally with `./scripts/reproduce_flaky.sh`
- [ ] No shared global state between test cases
- [ ] Hardware-dependent tests gated behind `HIL=1` flag

Weekly Flaky Review Ritual

Every Monday, 15 minutes:

1. Open flaky dashboard
2. For each test >5% flakiness: assign owner, set 2-week timebox
3. Close fixed flaky tests, move to stable suite
4. Delete tests with no unique coverage after 4 weeks in quarantine

CI Metrics That Matter

Quantify flakiness impact by measuring rerun rate (run_attempt > 1), which isolates non-deterministic re-executions from genuine syntax or compilation errors:

# .github/workflows/ci-metrics.yml
- name: Collect CI Flakiness Metrics
  run: |
    # Query last 100 workflow runs to inspect retry behavior
    RUNS_JSON=$(gh run list --limit 100 --json conclusion,run_attempt)
    TOTAL_RUNS=$(echo "$RUNS_JSON" | jq 'length')
    
    if [ "$TOTAL_RUNS" -eq 0 ]; then
      echo "No workflow runs found."
      exit 0
    fi

    # Flaky runs: runs requiring more than 1 attempt, or succeeded after prior failure
    FLAKY_RUNS=$(echo "$RUNS_JSON" | jq '[.[] | select(.run_attempt > 1)] | length')
    FLAKY_RATE=$(awk "BEGIN {printf \"%.2f\", ($FLAKY_RUNS * 100.0) / $TOTAL_RUNS}")

    echo "ci_flaky_rate=$FLAKY_RATE" >> $GITHUB_OUTPUT
    echo "Total runs: $TOTAL_RUNS | Flaky retries: $FLAKY_RUNS | Flake rate: ${FLAKY_RATE}%"

Target: Keep flaky rate below 2% for stable suite, below 10% for quarantined suite.

Summary

Flaky CI in embedded firmware is not inevitable — it is a design smell. The stabilization path:

1. Classify each flaky failure using the taxonomy (Timing, Hardware State, HIL Noise, Toolchain, Resource)
2. Detect systematically with CI-native tools, nightly repeated runs, and local reproduction harnesses
3. Stabilize with hardware abstraction, deterministic test vectors, quarantine pipelines, reproducible builds, and resource pools
4. Monitor with a flaky dashboard, code review gates, and weekly review rituals

The goal is not zero flakiness — it is flakiness that is detected, tracked, and resolved within a bounded timebox. A flaky test that is quarantined, owned, and fixed in two weeks is a healthy signal. A flaky test that blocks merges for months is a process failure.

Real-World Impact: Before and After

Teams that adopt this systematic approach typically see:

The investment pays for itself within the first month: reduced CI costs, faster merges, and engineers who stop ignoring test failures. Start with the taxonomy — classify your top 10 flaky tests this week. Build the dashboard next week. Quarantine the worst offenders immediately. The compounding returns on CI reliability are real and measurable.

Related Reading

• Fixing Zephyr Build Errors from Missing Device Tree Overlays
• Automating Firmware Release Pipelines with GitHub Actions
• Reducing Firmware Size with CMake Dead Code Elimination
• Effective Embedded Firmware Code Review Checklist
• Managing Vendor SDK Updates Without Breaking Embedded Builds
• Debugging Production Firmware Issues: Field Diagnostics That Work
• Building a Home Lab for Embedded Development
• Embedded Firmware Testing Strategies for Safety-Critical Systems

References

1. GitHub Actions Documentation — “Re-running workflows and jobs” — https://docs.github.com/en/actions/how-tos/managing-workflow-runs-and-deployments/managing-workflow-runs/re-running-workflows-and-jobs
2. Google Testing Blog — “Flaky Tests at Google and How We Mitigate Them” — https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
3. Martin Fowler — “Eradicating Non-Determinism in Tests” — https://martinfowler.com/articles/nonDeterminism.html
4. Embedded Artistry — “Hardware Abstraction Layer” — https://embeddedartistry.com/fieldmanual-terms/hardware-abstraction-layer/
5. CMake Documentation — “Reproducible Builds” — https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html#reproducible-builds
6. Zephyr Project — “Twister Test Framework” — https://docs.zephyrproject.org/latest/develop/test/twister.html