Understanding the OS Image Composer Build Process#

The OS Image Composer tool creates customized OS images through a well-defined build pipeline. Understanding this process helps you optimize your image builds and troubleshoot issues effectively.

Table of Contents#

Overview of the Build Pipeline#

The OS Image Composer uses a unified build command that processes an image template through a series of well-defined stages. This staged approach provides several advantages:

  • Modularity: Each stage performs a specific function with clear inputs and outputs

  • Caching: Intermediate results (packages, chroot environments) are cached for performance

  • Reliability: Each stage validates its inputs and outputs, catching errors early

  • Extensibility: New capabilities can be added to each stage independently

  • Transparency: Detailed logging at each stage helps with debugging and understanding

The build process takes a single YAML image template file as input and produces a bootable OS image in your specified format (raw or ISO). All stages are executed automatically in sequence by the build command.

Build Command Workflow#

The build command orchestrates the entire image creation process:

sudo -E os-image-composer build my-image-template.yml

Internal Workflow:

  1. Load Configuration - Load global configuration and command-line overrides

  2. Load and Merge Templates - Parse user image template and merge with OS-specific default template for the image type and architecture

  3. Initialize Provider - Select the appropriate provider (Azure Linux, EMT, eLxr) based on template target (OS, distribution, architecture)

  4. PreProcess - Provider validates template and prepares the build environment

  5. BuildImage - Provider executes the complete image build pipeline

  6. PostProcess - Provider performs cleanup and generates SBOM

The provider’s BuildImage method orchestrates all build stages internally. There is only one build command that manages the entire process from template to final image.

Build Stages in Detail#

The BuildImage process executes several internal operations to create the final image.

1. Template Loading and Validation#

Purpose: Ensure that the build template is syntactically correct and valid for the target provider before starting the build.

Key Tasks:

  • Parse and validate YAML template syntax

  • Merge user image template with OS-specific default template (e.g., default-raw-x86_64.yml or default-iso-x86_64.yml from config/osv/{os}/{dist}/imageconfigs/defaultconfigs/)

  • Validate merged configuration against JSON schema

  • Verify that the referenced provider exists and is properly configured

  • Check for existence of required files (custom scripts, configuration files, SSH keys)

  • Validate that the combination of settings is compatible

  • Ensure that any specified custom files exist and are accessible

  • Verify bootloader configuration is valid for the target architecture

Failure Handling: If validation fails, the build is aborted immediately with detailed error messages to help fix the issue.

Note: You can validate a template without building by using the validate command:

os-image-composer validate my-image-template.yml

2. Packages Stage#

Purpose: Collect all required packages and their dependencies, download them, verify integrity, and prepare them for installation.

Key Tasks:

  1. Determine all packages specified in the merged template

  2. Fetch repository metadata from remote package repositories

  3. Resolve all direct and indirect dependencies

  4. Check the package cache (cache/pkgCache/{provider-id}/) for previously downloaded packages

  5. Download missing packages from remote repositories

  6. Verify package integrity (GPG signatures, checksums)

  7. Store packages in the local cache for future builds

  8. Generate dependency graph (chrootpkgs.dot) for visualization

Package Cache Benefits:

The package cache stores downloaded packages (.rpm or .deb files) in cache/pkgCache/{provider-id}/. This cache:

  • Dramatically reduces build time for similar images

  • Decreases network bandwidth usage

  • Enables offline builds if packages were previously cached

  • Works even when templates change (shared packages are reused)

See Understanding Caching for detailed information about how the package cache works.

Dependency Resolution:

Direct dependencies (packages explicitly listed in the template) are processed first, then indirect dependencies (packages required by direct dependencies) are resolved automatically. Provider-specific libraries handle dependency resolution for RPM and DEB package systems.

Package Resolution Sequence:

        sequenceDiagram
    autonumber
    participant IC as os-image-composer
    participant P as provider
    participant RR as remote repository
    participant LC as local cache

    IC->>IC: Validate and merge templates
    IC->>P: Init(dist, arch)
    P->>RR: Fetch repository metadata (primary.xml.gz, GPG keys)
    RR-->>P: Return manifest & keys
    P-->>IC: Initialized

    IC->>P: FetchPackages()
    P->>RR: Parse repository metadata
    RR-->>P: Return full package list
    P->>P: Build in-memory package index
    P-->>IC: All available packages

    IC->>IC: Match requested packages from template
    IC->>P: Resolve dependencies
    P->>P: Resolve all direct and indirect dependencies
    P-->>IC: Complete package list with dependencies

    IC->>P: Download packages
    P->>LC: Check package cache (cache/pkgCache/)
    LC-->>P: Return cached packages
    P->>RR: Download missing packages
    RR-->>LC: Store in local cache
    P-->>IC: Downloads complete

    IC->>P: Verify packages
    P->>LC: Verify GPG signatures and checksums
    P->>LC: Generate chrootpkgs.dot
    P-->>IC: Verification complete

Multiple Repository Support:

You can add multiple package repositories to include proprietary packages or upstream packages pending integration. See Multiple Package Repository Support for details.

3. Compose Stage#

Purpose: Create the disk image, install all packages, and configure the operating system.

Key Tasks:

The compose stage has two main phases:

Phase 1: Disk/ISO Preparation

For raw images:

  • Create an empty raw disk image file in workspace/{provider-id}/imagebuild/{systemConfigName}/

  • Create partitions according to the disk configuration (boot, root, swap, data, etc.)

  • Format partition filesystems (ext4, xfs, vfat, etc.)

  • Set up loop devices for partition access

For ISO images:

  • Create ISO directory structure

  • Prepare bootable ISO layout

  • Set up isolinux/grub configuration

Phase 2: OS Installation and Configuration

  • Create or reuse chroot environment from workspace/{provider-id}/chrootenv/

    • First build: Creates new chroot environment

    • Subsequent builds: Reuses existing chroot (significant time savings)

  • Mount necessary pseudo-filesystems (proc, sys, dev)

  • Execute pre-installation scripts (if specified in template)

  • Install all packages from the packages stage in correct dependency order

  • Configure base OS environment:

    • Set hostname and timezone

    • Configure network settings (interfaces, DNS, routes)

    • Create user accounts and set passwords

    • Install and configure SSH keys

    • Copy custom files to their destinations

    • Enable or disable system services

    • Configure kernel parameters

  • Install and configure bootloader (GRUB or systemd-boot)

  • Execute post-installation scripts (if specified in template)

  • Apply security configurations:

    • SELinux/AppArmor policies

    • dm-verity for read-only filesystem integrity

    • Secure boot configuration

    • Filesystem permissions and hardening

  • Clean up temporary files and caches

  • Unmount filesystems and detach loop devices

Chroot Environment Reuse:

The compose stage works within an isolated chroot environment. Key caching behavior:

  • First build for a provider: Creates chroot environment in workspace/{provider-id}/chrootenv/ and tarball in workspace/{provider-id}/chrootbuild/chrootenv.tar.gz

  • Subsequent builds: Reuses existing chroot environment, avoiding the overhead of recreating the base system

  • Image build directory: workspace/{provider-id}/imagebuild/{systemConfigName}/ is rebuilt for each build to ensure clean output

This reuse significantly improves build performance, especially when building multiple images for the same OS/distribution/architecture combination.

4. Image Signing#

Purpose: Apply digital signatures to the image for integrity verification and secure boot.

Key Tasks:

  • Generate or use existing signing keys

  • Sign bootloader components (if secure boot is enabled)

  • Sign kernel and initrd images

  • Generate verification metadata

  • Store signature information in image metadata

Image signing is optional and is only performed if configured in the template. See Understanding Templates for signing configuration examples.

5. Finalize Stage#

Purpose: Prepare the final output format, apply compression, and generate metadata.

Key Tasks:

  • Finalize image format:

    • raw images: Copy from workspace to final output location

    • ISO images: Create bootable ISO file using genisoimage/mkisofs

  • Apply compression if specified (gzip, xz, zstd)

  • Generate image manifest file (JSON) containing:

    • Image metadata (name, version, size, format)

    • Package list with versions

    • Build timestamp and configuration hash

  • Generate Software Bill of Materials (SBOM) in SPDX format (tmp/spdx_manifest.json)

  • Copy final image to output location

  • Clean up temporary build artifacts from workspace/{provider-id}/imagebuild/{systemConfigName}/

Output Formats:

The tool supports two primary image types:

Format

Use Case

Additional Formats

Notes

raw

Bare metal, VMs, cloud deployment

Can be converted to vhd, vhdx, qcow2, vmdk, vdi

Bootable disk image with partitions. Supports compression (gz, xz, zstd)

iso

Installation media, live systems

ISO format only

Bootable ISO image

Build Configuration Options#

The build process can be customized through multiple configuration layers.

Global Configuration Options#

Global configuration default stored in root of the repository os-image-composer.yml`:

# Worker configuration
workers: 16                                       # Number of concurrent package download workers

# Directory configuration
cache_dir: /var/cache/os-image-composer          # Package cache location
work_dir: /var/tmp/os-image-composer             # Working directory for builds
temp_dir: /tmp                                    # Temporary files location

# Logging configuration
logging:
  level: info                                     # Log level: debug, info, warn, error
  file: /var/log/os-image-composer.log           # Optional log file path

Default Templates vs Image Templates#

The OS Image Composer uses a layered configuration approach:

Default Templates (config/osv/{os}/{dist}/imageconfigs/defaultconfigs/)

These provide OS-specific base configurations that are merged with your image templates:

  • default-raw-{arch}.yml - Default settings for raw images for specific architecture

  • default-iso-{arch}.yml - Default settings for ISO images for specific architecture

  • default-initrd-{arch}.yml - Default settings for initrd images for specific architecture

Located under each OS vendor path, for example:

  • config/osv/azure-linux/azl3/imageconfigs/defaultconfigs/default-raw-x86_64.yml

  • config/osv/edge-microvisor-toolkit/emt3/imageconfigs/defaultconfigs/default-iso-x86_64.yml

  • config/osv/wind-river-elxr/elxr12/imageconfigs/defaultconfigs/default-raw-x86_64.yml

Default templates contain common settings like partition layouts, essential packages, and bootloader configurations. They ensure consistent base configurations across all images of the same OS, distribution, and image type.

Image Templates (user-provided YAML files in image-templates/)

These are the templates you create to define your specific image requirements. They specify what differs from the defaults:

image:
  name: my-custom-image
  version: "1.0.0"

target:
  os: azure-linux      # Operating system provider
  dist: azl3           # Distribution version
  arch: x86_64         # Target architecture
  imageType: raw       # Output format: raw or iso

disk:
  name: Custom_Name    # Maps to systemConfig name
  size: 4GiB

systemConfig:
  name: Custom_Name    # Unique identifier for this configuration
  description: Production server configuration
  packages:
    - openssh-server
    - docker-ce
    - vim
  
  kernel:
    version: "6.12"
    cmdline: "console=ttyS0,115200 console=tty0 loglevel=7"
    packages:
      - kernel-azure
      - kernel-modules-azure

Template Merging Process:

  1. Load OS-specific default template based on imageType and arch from config/osv/{os}/{dist}/imageconfigs/defaultconfigs/

  2. Load user-provided image template from image-templates/

  3. Merge configurations (user image template values override defaults)

  4. Validate merged configuration against JSON schema

This layering allows you to specify only what differs from the defaults, making templates concise and maintainable.

See Understanding Templates for complete template documentation and schema reference.

Command-Line Overrides#

Command-line flags override configuration file settings:

# Override worker count and cache directory
sudo -E os-image-composer build \
  --workers 32 \
  --cache-dir /mnt/fast-ssd/cache \
  my-template.yml

# Enable verbose logging for debugging
sudo -E os-image-composer build --verbose my-template.yml

# Generate dependency graph visualization
sudo -E os-image-composer build --dotfile deps.dot my-template.yml

# Generate a graph that only shows SystemConfig roots
sudo -E os-image-composer build --dotfile system.dot --system-packages-only my-template.yml

Common Build Patterns#

The image-templates/ directory contains example templates for different use cases. Each template is named following the pattern {os}{dist}-{arch}-{purpose}-{imageType}.yml, for example:

  • azl3-x86_64-minimal-raw.yml - Minimal raw disk image

  • elxr12-x86_64-edge-raw.yml - Edge device optimized image

  • emt3-x86_64-minimal-iso.yml - Minimal ISO image

These templates demonstrate:

  • Different package selections for various use cases

  • Disk partitioning schemes (minimal, standard, secure)

  • Bootloader configurations (GRUB, systemd-boot)

  • Security features (dm-verity, SELinux, immutable root)

  • Image artifacts and compression options

Refer to the schema documentation and example templates in image-templates/ for creating your own custom templates.

Build Performance Optimization#

Improving Build Speed#

1. Use Package Caching

Package caching is enabled by default and provides significant performance improvements:

  • First build: Downloads all packages (~5-10 minutes depending on package count)

  • Subsequent builds: Reuses cached packages (< 1 minute for package stage)

The cache is stored in cache/pkgCache/{provider-id}/ and is automatically managed.

2. Leverage Chroot Environment Reuse

The chroot environment in workspace/{provider-id}/chrootenv/ is automatically reused:

  • First build for a provider: Creates chroot environment (~2-3 minutes)

  • Subsequent builds for same provider: Reuses chroot (< 30 seconds)

3. Optimize Worker Count

Adjust the worker count based on your system. For example:

# For systems with fast network and many CPU cores
--workers 32

# For systems with limited network bandwidth
--workers 8

4. Use Local Package Mirrors

Configure your template to use local package mirrors:

repositories:
  - name: local-mirror
    baseurl: http://local-mirror.example.com/packages/
    gpgcheck: true
    gpgkey: http://local-mirror.example.com/RPM-GPG-KEY

5. Use Fast Storage for Cache and Workspace

Point cache and workspace directories to fast storage (SSD/NVMe):

--cache-dir /mnt/nvme/cache --work-dir /mnt/nvme/workspace

Reducing Build Time for Development#

During development, you can speed up iteration:

1. Start with Minimal Package Set

Test your changes with a minimal package list first:

packages:
  - filesystem
  - kernel
  - systemd
  # Add only packages you're actively testing

2. Use Verbose Logging Only When Needed

Normal logging is sufficient for most builds. Use verbose mode only for debugging:

# Normal build
sudo -E os-image-composer build template.yml

# Verbose only when investigating issues
sudo -E os-image-composer build --verbose template.yml

3. Leverage Reusable Artifacts

Keep your workspace directory across builds to benefit from chroot reuse:

# Use consistent work directory
--work-dir /var/tmp/os-image-composer

Troubleshooting Build Issues#

Common Problems and Solutions#

Problem: Build fails during package download

Error: Failed to download package openssh-server

Solutions:

  • Check network connectivity to package repositories

  • Verify repository URLs in template are correct

  • Check if package name is spelled correctly

  • Ensure GPG keys are properly configured

  • Try using a different package mirror

Problem: Insufficient disk space

Error: No space left on device

Solutions:

  • Check available space in work directory: df -h /var/tmp

  • Check available space in cache directory: df -h /var/cache

  • Clean up old builds: rm -rf workspace/*/imagebuild/

  • Clear package cache if needed: rm -rf cache/pkgCache/

  • Use a different work directory with more space: --work-dir /mnt/large-disk/workspace

Problem: Permission denied errors

Error: Permission denied when creating loop device

Solutions:

  • Run with sudo: sudo -E os-image-composer build template.yml

  • Ensure user has appropriate permissions

  • Check that work directory and cache directory are writable

  • Verify SELinux is not blocking operations (if applicable)

Problem: Package dependency conflicts

Error: Package conflict: package-a requires version 1.0, but version 2.0 is already installed

Solutions:

  • Review your package list for conflicting requirements

  • Check if you’re mixing packages from incompatible repositories

  • Use specific package versions if needed

  • Review dependency resolution with --dotfile to visualize conflicts

Problem: Chroot environment corruption

Error: Failed to enter chroot environment

Solutions:

  • Remove corrupted chroot: rm -rf workspace/{provider-id}/chrootenv/

  • Remove chroot tarball: rm -rf workspace/{provider-id}/chrootbuild/

  • Next build will recreate the chroot environment from scratch

Detailed Debugging#

Enable Verbose Logging:

sudo -E os-image-composer build --verbose my-template.yml

See the build command documentation for all available flags.

This provides detailed output for each step:

  • Package download progress

  • Chroot environment setup

  • Command execution within chroot

  • Filesystem operations

  • Mount/unmount operations

Generate Dependency Graph:

sudo -E os-image-composer build --dotfile deps.dot my-template.yml
dot -Tpng deps.dot -o deps.png

The dependency graph is also automatically saved to cache/pkgCache/{provider-id}/chrootpkgs.dot.

This visualization helps with:

  • Understanding why certain packages are included

  • Identifying dependency conflicts

  • Optimizing package selection

Check Build Artifacts:

Inspect the workspace and cache directories:

# View chroot environment
ls -la workspace/{provider-id}/chrootenv/

# Check package cache
ls -la cache/pkgCache/{provider-id}/

# Check dependency graph
cat cache/pkgCache/{provider-id}/chrootpkgs.dot

# Examine build directory
ls -la workspace/{provider-id}/imagebuild/{systemConfigName}/

# Check SBOM
cat tmp/spdx_manifest.json

Build Log Analysis#

The build process logs important milestones:

INFO: Using configuration from: /etc/os-image-composer/config.yml
INFO: Loading template: my-template.yml
INFO: Merging with default template: config/general/default-raw.yml
INFO: Initializing provider: azure-linux-azl3-x86_64
INFO: Starting build process
INFO: [Validate] Validating template against schema
INFO: [Packages] Fetching repository metadata
INFO: [Packages] Resolving dependencies
INFO: [Packages] Downloading packages (0/245 cached)
INFO: [Packages] Verifying package integrity
INFO: [Packages] Generating dependency graph: cache/pkgCache/.../chrootpkgs.dot
INFO: [Compose] Reusing existing chroot environment
INFO: [Compose] Creating raw disk image (10G)
INFO: [Compose] Creating partitions
INFO: [Compose] Formatting filesystems
INFO: [Compose] Installing packages (245 total)
INFO: [Compose] Configuring system
INFO: [Compose] Installing bootloader
INFO: [Finalize] Generating manifest
INFO: [Finalize] Generating SBOM: tmp/spdx_manifest.json
INFO: Build completed successfully
INFO: Output image: workspace/.../my-custom-image-1.0.0.raw

Look for ERROR or WARN messages in the log to identify issues.

Conclusion#

The OS Image Composer build process uses a single unified build command that internally executes multiple stages to create customized OS images. By understanding the internal build process, you can:

  • Create efficient build specifications

  • Troubleshoot issues more effectively

  • Optimize the build process for your needs

  • Leverage caching to improve performance

Key Takeaways:

  1. OS-Specific Configuration: Default templates are organized by OS vendor, distribution, and architecture in config/osv/

  2. Template Merging: OS-specific default templates provide base configuration, user image templates (in image-templates/) customize

  3. Caching System: Package caching and chroot reuse dramatically improve performance

  4. Two Image Types: raw (with format conversion to vhd, vhdx, qcow2, vmdk, vdi) and iso for different deployment scenarios

  5. Troubleshooting Tools: Verbose logging and dependency graphs facilitate debugging