Embedded Linux: Fixing Slow Boot Time

Introduction

Slow boot times in Embedded Linux systems are a common frustration for engineers, delaying product development and impacting time-to-market. Whether you’re working on a consumer IoT device, an industrial controller, or automotive electronics, users expect instant responsiveness. A slow boot not only creates a poor first impression but can also affect power consumption in battery-operated devices, as the system spends more time in a high-energy state during initialization.

This article addresses the real-world problem of slow boot times by providing a systematic approach to diagnose, analyze, and optimize the boot process. We’ll cover measurement techniques, identify common bottlenecks across the boot stack, and present practical solutions ranging from kernel configuration to distribution customization. Each solution includes trade-offs, implementation steps, and verification methods to help you achieve faster boot times without compromising functionality.

Measuring Boot Time

Before optimizing, you must measure. The first step in any performance improvement effort is establishing a baseline. For Embedded Linux systems using systemd (the default in most modern distributions), the systemd-analyze tool provides detailed insights into boot performance.

Run the following command on your target device to see a breakdown of boot time:

systemd-analyze

This outputs something like:

Startup finished in 2.5s (kernel) + 4.8s (userspace) = 7.3s 
graphical.target reached after 4.8s in userspace

To see which services are taking the longest, use:

systemd-analyze blame

Sample output:

2.1s plymouth-quit-wait.service
          1.8s dev-mmcblk0p2.device
          1.5s networking.service
          1.2s wpa_supplicant.service
          900ms systemd-modules-load.service
          800ms systemd-udevd.service
          600ms accounts-daemon.service

This blame output helps you identify specific services that are delaying boot. Additionally, systemd-analyze plot generates an SVG visualizing the boot process, highlighting parallelism and sequential dependencies.

For systems without systemd (e.g., using SysVinit or BusyBox), you can measure boot time by checking the kernel timestamp in dmesg or using a GPIO toggle with an oscilloscope. However, systemd-analyze remains the most convenient and detailed method for most Embedded Linux setups.

Visualizing the Boot Process

Here’s a simplified ASCII art representation of the typical Embedded Linux boot sequence:

+------------------------------------------------------------------+
| Power On --> Bootloader --> Kernel --> Init --> Userspace --> Ready |
+------------------------------------------------------------------+

Each stage contributes to the total boot time, and optimizations can be applied at each level.

Common Causes of Slow Boot Times

Understanding the root causes allows you to target optimizations effectively. The boot process consists of several stages, each contributing to the total time:

1. Kernel Initialization

The kernel boot time includes decompression, hardware detection, driver initialization, and mounting the root filesystem. Common culprits:

• Unnecessary drivers: Built-in drivers for hardware not present on your board (e.g., SATA controllers on a pure Ethernet device).
• Excessive debugging: Kernel debug prints (CONFIG_DEBUG_KERNEL, dynamic_debug) increase boot time and log volume.
• Slow storage initialization: Waiting for spinning hard drives or slow eMMC initialization can add seconds.

2. Device Tree Processing

The device tree describes hardware to the kernel. A bloated or inefficient device tree increases kernel boot time:

• Unused nodes: Including definitions for peripherals not present on your variant.
• Deeply nested structures: Complex hierarchies require more parsing time.
• Redundant properties: Duplicate or unnecessary properties increase processing overhead.

3. Init System Overhead

The init system (systemd, SysVinit, or BusyBox) starts user-space services. Issues include:

• Sequential startup: Services that don’t depend on each other starting one after another.
• Blocking services: Services waiting for network or hardware that isn’t ready yet.
• Excessive services: Starting daemons that aren’t needed for your application (e.g., Bluetooth daemon on a headless sensor node).

4. Service Initialization

Individual services may have slow startups due to:

• Heavy initialization: Loading large configuration files or initializing complex subsystems.
• External dependencies: Waiting for remote services or hardware that responds slowly.
• Inefficient algorithms: O(n^2) initialization routines in user-space daemons.

5. Storage Performance

The root filesystem location significantly impacts boot time:

• Slow interface: Using USB 2.0 instead of eMMC or SATA III.
• Filesystem choice: Journaling filesystems (ext4) have overhead; consider squashfs for read-only parts.
• Fragmentation: Severe fragmentation increases read times (less common with modern flash but still possible).

Solution 1: Kernel Optimization

The Linux kernel is highly configurable. By building a kernel tailored to your hardware, you can eliminate unnecessary initialization steps.

Step-by-Step Kernel Optimization

1. Start with a baseline configuration: Use your board’s defconfig as a starting point.

   make <board_defconfig>

2. Enable modularization: Build drivers as modules (m) whenever possible, so they load only when needed.

   • Navigate to Device Drivers → Generic Driver Options → [*] Support for uevent helper to cold-plug modules (optional)
   • Set individual drivers to M (module) instead of Y (built-in).

3. Disable unused subsystems: Go through each menu and disable what you don’t need.

   • Example: If you don’t use USB host, disable Device Drivers → USB support.
   • If you don’t need wireless networking, disable Networking support → Wireless.

4. Optimize kernel features:

   • Under General Setup, consider disabling:
     • [*] Kernel .config support (if you don’t need /proc/config)
     • [*] Enable access to .config through /proc/config (CONFIG_IKCONFIG)
     • [*] Enable PCI support (if your board has no PCI)
   • Under Processor type and features, disable:
     • [*] Symmetric multi-processing support (if single-core)
     • [*] SMT (Hyperthreading) scheduler support (if not needed)

5. Compile and install: Build the kernel, install it on your target, and reboot.

   make -j$(nproc)
   make modules_install
   cp arch/arm/boot/zImage /boot/
   cp arch/arm/boot/dts/<your-dtb>.dtb /boot/

Trade-offs

• Pros: Significant reduction in kernel boot time (often 20-50% faster), smaller kernel image size.
• Cons: Requires careful configuration; enabling a driver later requires rebuilding the kernel. Modular drivers add slight indirection when loaded.

Example: .config Snippet

Here’s a snippet showing disabled unused features:

# General setup
# CONFIG_IKCONFIG is not set
# CONFIG_IKCONFIG_PROC is not set

# Processor type and features
# CONFIG_SMP is not set

# Bus options (PCI etc.)
# CONFIG_PCI is not set

# Device Drivers
# CONFIG_USB_SUPPORT is not set
# CONFIG_SERIAL_8250 is not set
# CONFIG_INPUT_EVDEV is not set

Solution 2: Device Tree Optimization

The device tree is a data structure describing hardware. Optimizing it reduces kernel boot time by minimizing the data the kernel must parse.

Techniques for Device Tree Optimization

1. Remove unused nodes: If your board variant doesn’t use a peripheral, delete its node or comment it out.

   • Example: If you don’t use the second CAN controller, remove the &can1 node or disable it via status = “disabled”.

2. Use __overrides__ for flexibility: Instead of creating multiple device tree files for minor variations, use overrides to enable/disable features at runtime.

   • Define an override node that changes properties based on a bootloader parameter.

3. Simplify property values: Avoid long strings or unnecessary properties.

   • Instead of a custom string for label, use a simple identifier if possible.

4. Combine similar nodes: If multiple instances of a peripheral have identical configurations, use a single node with reg array and status properties to enable/disable instances.

Example: Before and After

Before (inefficient):

&can0 {
    status = "okay";
    /* ... 20 lines of configuration ... */
};

&can1 {
    status = "disabled";  /* Not used on this variant */
    /* ... 20 lines of identical configuration ... */
};

&can2 {
    status = "disabled";  /* Not used */
    /* ... 20 lines of identical configuration ... */
};

After (optimized):

/* Only enable what we use */
&can0 {
    status = "okay";
    /* ... configuration ... */
};

/* Disable unused controllers to save parsing time */
&can1 {
    status = "disabled";
};

&can2 {
    status = "disabled";
};

Even better, if you never use CAN1 and CAN2 on any variant, remove their nodes entirely from the base .dtsi and only enable CAN0 in your board file.

Trade-offs

• Pros: Direct reduction in kernel boot time (measurable with systemd-analyze), cleaner device tree source.
• Cons: Requires maintaining board-specific device tree files; changes must be synchronized with hardware revisions.

Solution 3: Init System Optimization

Most modern Embedded Linux distributions use systemd. Optimizing service startup can yield substantial userspace boot time improvements.

Using systemd-analyze for Optimization

1. Identify blocking services: Run systemd-analyze blame and look for services with high times that aren’t critical for early boot.
2. Check dependencies: Use systemd-analyze critical-chain to see the boot chain and dependencies.

   systemd-analyze critical-chain

3. Disable unnecessary services: Mask services you don’t need.

   systemctl mask bluetooth.service  # Example: disable Bluetooth if not used
   systemctl mask avahi-daemon.service

4. Parallelize startup: Ensure services that don’t depend on each other can start in parallel.
   • Check service units for After= and Before= dependencies that might be overly restrictive.
   • Use systemd-analyze plot to visualize concurrency.

Example: Optimizing a Service

Suppose wpa_supplicant.service takes 1.5 seconds to start because it waits for the network interface to be ready. You can:

• Make it start earlier by adjusting After= dependencies.
• If Wi-Fi isn’t needed immediately, delay its startup until after user login (using Wants= in a target that starts later).

Trade-offs

• Pros: Significant reduction in userspace boot time (often 30-60% faster), better resource utilization.
• Cons: Masking services might break functionality if dependencies are misunderstood; requires testing to ensure required services still start.

Solution 4: Storage Optimization

The storage subsystem affects both kernel initialization (rootfs mount) and userspace service startup (reading binaries, libraries, and data).

Strategies for Faster Storage

1. Choose faster hardware: If possible, upgrade from SD card to eMMC or UFS for faster sequential and random access.
2. Optimize filesystem type:
   • For read-only root filesystem: Use squashfs with lz4 compression (fast decompression).
   • For read-write: Consider ext4 with data=writeback journaling (less safe but faster) or disable journaling entirely if you have a power-loss tolerance mechanism.
   • Example fstab entry for optimized ext4:

     /dev/mmcblk0p2  /  ext4  defaults,noatime,nodiratime,data=writeback  0  1

3. Reduce filesystem checks: Adjust fsck frequency or disable it for fast boot (with caution).

   tune2fs -i 0 -c 0 /dev/mmcblk0p2  # Disable interval and count-based checks

4. Use initramfs effectively: Keep the initramfs small and only include essential modules and binaries.
   • Example: Use mkinitfs (Buildroot) or dracut with a minimal module list.

Trade-offs

• Pros: Faster read times reduce both kernel and userspace boot stages.
• Cons: Some optimizations (like disabling journaling) increase risk of corruption on power loss; always validate with your reliability requirements.

Solution 5: Minimal Distribution with Yocto

Yocto Project allows you to create a custom Linux distribution tailored exactly to your needs, eliminating unnecessary packages and services.

Building a Minimal Image with Yocto

1. Set up Yocto environment: Follow the Yocto Project Quick Start to set up your build directory.
2. Start with a minimal base: Use core-image-minimal as your base image.
   • Edit conf/local.conf to set:

     IMAGE_INSTALL:append = " packagegroup-core-boot"
     IMAGE_FSTYPES:append = " ext3"

3. Remove unnecessary packages: Use IMAGE_INSTALL:remove to exclude packages you don’t need.
   • Example: Remove Bluetooth, Wi-Fi, X11, and GTK if not required.

     IMAGE_INSTALL:remove = "bluez5 bluez5-utils wpa-supplicant gtk+3"

4. Enable only essential services: In your image recipe, add only the services you need.
   • Example: Add systemd and enable sshd and your application service.

     IMAGE_INSTALL:append = " systemd sshd"
     SYSTEMD_AUTO_ENABLE:append = " sshd"

5. Build the image:

   bitbake core-image-minimal

Advanced Optimization

• Use packagegroup-core-boot: This group includes only the bare essentials for booting.
• Optimize kernel configuration: Create a custom kernel .config file in your Yocto layer and assign it to linux-yocto.
• Compress root filesystem: Use IMAGE_FSTYPE:append = " squashfs" for a compressed, read-only root filesystem.

Trade-offs

• Pros: Complete control over what’s in the image; often results in 50-70% smaller root filesystem and faster boot times due to less data to load.
• Cons: Increased build complexity; requires maintaining Yocto layers and recipes; build times can be long.

Solution 6: Bootloader Optimization

The bootloader (typically U-Boot in Embedded Linux) runs before the kernel. Optimizing it saves time in the earliest stage.

U-Boot Optimization Techniques

1. Reduce bootdelay: Set CONFIG_BOOTDELAY to 0 or 1 second (or use environment variable to interrupt only on keypress).
   • In your board config file:

     #define CONFIG_BOOTDELAY 0

2. Skip unnecessary scans: Disable scanning for devices you don’t use.
   • Example: If you don’t use USB, disable CONFIG_USB_STORAGE.
3. Optimize environment: Keep the environment small and store it in fast memory (e.g., SPI NOR flash instead of eMMC if appropriate).
4. Use Falcon mode or secure boot: For bootloaders that support it, enable verified boot to reduce authentication time (if your hardware supports it).

Trade-offs

• Pros: Saves time in the first stage of boot; every millisecond counts in cold-boot scenarios.
• Cons: Aggressive reduction of bootdelay makes recovery harder if the system fails to boot; ensure you have a recovery mechanism (e.g., failsafe bootloader).

Verification and Testing

After implementing optimizations, verify the improvements to ensure you’ve actually reduced boot time without breaking functionality.

Measurement Procedure

1. Establish baseline: Measure boot time before changes using systemd-analyze (or oscilloscope method).
2. Apply one optimization at a time: This helps you isolate the impact of each change.
3. Measure after each change: Run systemd-analyze again and compare the numbers.
4. Check functionality: Verify that all required services start and your application works correctly.
5. Use automated testing: For production environments, create a test script that boots the system, checks for readiness, and records boot time.

Example: Before and After Optimization

Baseline:

Startup finished in 3.2s (kernel) + 6.5s (userspace) = 9.7s

After kernel optimization:

Startup finished in 2.1s (kernel) + 6.5s (userspace) = 8.6s

(Kernel time reduced by 1.1s)

After disabling unnecessary services:

Startup finished in 2.1s (kernel) + 4.0s (userspace) = 6.1s

(Userspace time reduced by 2.5s)

After storage optimization (faster eMMC):

Startup finished in 1.5s (kernel) + 3.0s (userspace) = 4.5s

(Both kernel and userspace improved due to faster storage access)

Conclusion

Slow boot times in Embedded Linux are rarely due to a single issue; they result from inefficiencies across the entire boot stack. By systematically measuring, analyzing, and optimizing each stage—from bootloader to kernel, device tree, init system, services, and storage—you can achieve dramatic improvements.

The key takeaways are:

1. Measure first: Use systemd-analyze to understand where time is spent.
2. Target the biggest wins: Focus on optimizations that reduce the largest bottlenecks (often kernel initialization and userspace services).
3. Apply changes incrementally: This allows you to verify correctness and isolate the impact of each optimization.
4. Consider the trade-offs: Balance boot time improvements against reliability, flexibility, and development effort.
5. Validate thoroughly: Ensure that your optimizations don’t break required functionality under all operating conditions.

With these strategies, reducing boot time from seconds to under a second is achievable for many Embedded Linux applications, leading to better user experiences, lower power consumption, and faster time-to-market.

Related Reading

For further optimization techniques specific to embedded systems, see these embeddedSoft articles:

• Fixing UART DMA Overrun Errors on STM32
• Memory Pool Allocation for Deterministic Embedded Systems
• Linker Scripts and Memory Layout in Embedded C

Note: Always verify internal links against the live sitemap before publishing to ensure they use the correct URL slugs.

References

1. The Linux Kernel Documentation: https://www.kernel.org/doc/html/latest/admin-guide/boot-time.html
2. systemd-analyze Man Page: https://www.freedesktop.org/software/systemd/man/systemd-analyze.html
3. Yocto Project Documentation: https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html
4. U-Boot Documentation: https://docs.denx.de/u-boot/latest/
5. Device Tree Usage: https://www.devicetree.org/Usage/
6. Embedded Linux Boot Time Optimization: Linux Journal, “Boot Time Optimization for Embedded Linux Systems”, 2023.
7. Analyzing Boot Time with systemd: Andreas Wiedemann, “systemd-analyze: A Tool for Boot Time Analysis”, 2019. EOF