Why Does My Job Show High IO Wait?

Answer

A GitHub Actions job shows high IO wait when the processor sits idle while the kernel waits for a storage device to answer, so the job spends wall clock without spending cores. On a runner that pattern almost always traces back to large writes landing on a network-attached volume, or to a cache restore unpacking a big archive into the work directory.

There is no chart named iowait in the runner metrics view. The shape that identifies the problem is disk I/O climbing toward its ceiling while CPU utilization stays low or moderate over the same window. WarpBuild records disk I/O as the maximum rolling average of read plus write throughput over the last 30 seconds and marks an instance High Disk IO once max disk I/O reaches 80 percent of supported throughput (observability documentation, checked on 2026-08-13).

Two collection rules decide whether the charts exist. Metrics and logs are collected only for jobs longer than about 1 minute, and collection can be paused for an organization, in which case the Usage view shows the telemetry agent initializing with no data behind it (observability documentation). The fleet-wide version of this reading, aggregated by repository, workflow, job, and instance type, is on the GitHub Actions observability page.

The fixes run in one order: local storage first, a smaller cache second, prepared machine state third.

Detail

Confirm it inside the job

Sample the wa column while the job runs, then print the worst samples at the end. This works on any Linux runner and costs one background process.

jobs:
  build:
    runs-on: warp-ubuntu-latest-x64-4x
    steps:
      - uses: actions/checkout@v4
      - name: Start sampling
        run: nohup vmstat 5 120 > /tmp/vmstat.log 2>&1 &
      - run: npm ci
      - run: npm run build
      - name: Highest IO wait samples
        if: always()
        run: |
          head -2 /tmp/vmstat.log
          tail -n +3 /tmp/vmstat.log | sort -k16 -n | tail -5

The wa column is the share of CPU time spent waiting on storage. Line the timestamps of the worst samples up against step boundaries in the job log: if they sit inside the cache restore step or a docker pull, that step is the source, and no amount of extra vCPU changes it. The guide to runner level metrics covers the same reading from the dashboard and from the API rather than from inside the job.

Three sources produce almost all of it

Cache unpacking. A restore reads a compressed archive and writes every file in it. Decompression runs on one core while the writes queue behind it, which is the textbook low CPU with pinned disk shape. Caches that outgrow their budget make this worse on both ends: GitHub caps a repository at 10 GB across all entries and evicts entries that have not been accessed for seven days (GitHub dependency caching docs, checked on 2026-08-13), so a fat cache both takes longer to unpack and misses more often.

Container layer writes. Every pulled layer is decompressed and written into the overlay filesystem before the container starts. A base image with a few hundred megabytes of dependencies writes those bytes on every job.

A build tool writing many small files. A dependency install creating 200,000 files, a compiler emitting object files, or a test runner writing coverage fragments spends its time in per-file syscalls. Throughput can look unremarkable while the job still waits, which is why file count matters as much as byte count.

Fix one: keep the work on local disk

The highest IO wait numbers come from work directories that are not local disk at all. On BYOC runners, pick an instance type with local NVMe and the runner does the rest: it detects the local SSDs at boot, creates a RAID-0 array when there is more than one, formats with ext4, and mounts the result at the runner work directory such as /home/runner/work, all before the job starts and with no configuration (local SSD documentation, checked on 2026-08-13). Candidate families are c5d, m5d, r5d, i3, c6id, m6id, and r6id on AWS, the -lssd machine types on GCP, and Lsv3 or Lasv3 on Azure.

Two limits belong in the same breath. Local SSD auto-mount is available on Linux images only, so Windows and macOS runner types keep the regular boot disk. The mounted storage is ephemeral and dies with the instance, which is the expected contract for a runner. The local SSD runner page lists the setup end to end.

On hosted runners the work directory is already local: every Linux x64 size in the catalog ships 150GB SSD, from warp-ubuntu-latest-x64-2x at $0.004 per minute to warp-ubuntu-latest-x64-32x at $0.064 per minute (pricing page, checked on 2026-08-13). Storage capacity does not change with the label, so sizing up to solve a storage problem buys cores that stay idle.

Fix two: write fewer bytes and fewer files

Narrow the cached paths to the ones that pay back the unpack. Leave build output directories out, split one large cache into two so a job restores only what it needs, and stop archiving directories that rebuild in seconds. Work the arithmetic on a 6 GB cache restored by 400 jobs a month, using WarpBuild list prices checked on 2026-08-13:

The storage line is rounding error and the minutes line is the bill. Substitute the restore duration from your own job timing for the 1.5 minutes above; the ratio holds. Trimming that cache from 6 GB to 2 GB removes bytes from the disk queue and minutes from the invoice at the same time. The answer on slow cache restores covers key design, compression, and restore-key fallbacks in detail.

Fix three: prepared machine state

The fastest restore is the one that never runs. Snapshot runners capture a runner VM mid-workflow and boot later jobs from that snapshot, so dependencies are already on disk when the job starts. Add snapshot.enabled=true to create from the base image, or snapshot.key=<alias> to boot from an existing snapshot. Snapshots are supported on WarpBuild Cloud Ubuntu runners, are deleted after 15 days, and do not persist /tmp (snapshot runner documentation, checked on 2026-08-13).

Price it against the setup it replaces, at list prices checked on 2026-08-13. A job whose setup steps take 8 minutes on warp-ubuntu-latest-x64-4x spends 8 x $0.008 = $0.064 per job, or $128.00 across 2,000 jobs a month. Snapshot restore bills $0.04 per job, or $80.00 across the same 2,000 jobs, plus snapshot storage at $0.025 per snapshot-hour, which is $18.00 for one snapshot held through a 720 hour month. That is $98.00 against $128.00, and the wall clock saved is whatever those setup steps show in your job timing today.

Image layer writes have their own version of this. Remote Docker builders keep the layer cache on the builder machine's own disk, so the runner never unpacks layers to prove that a build is repeatable.

What acting on this costs

Full rates by runner type are on the pricing page.

Related Questions

Does the runner metrics view have an IO wait chart?

No. The five recorded metrics are CPU, memory, filesystem, disk I/O, and network utilization. Disk I/O is the maximum rolling average of read plus write throughput over the last 30 seconds, and IO wait shows up as that series climbing while CPU utilization stays low across the same window. The Recommendations view labels an instance High Disk IO once max disk I/O reaches 80 percent of supported throughput, per the observability documentation.

Disk I/O looks moderate but the step still waits on storage. What is happening?

The job is bound by operations rather than by bytes. A dependency install or an artifact unpack that writes 200,000 small files spends its time on per-file syscalls, so throughput never plateaus while the wa column stays high. Cut the file count first: narrower cache paths, fewer extracted layers, and build output directories left out of the archive. The guide to runner level metrics shows how to read those two cases apart per job.

Will a bigger runner label fix high IO wait?

Only when the cores are also busy. A larger label buys vCPU and RAM, and every Linux x64 size in the WarpBuild catalog carries the same 150GB SSD, so a job that idles at 20 percent CPU while writing a 6 GB cache to disk idles the same way on a wider machine. Move the bytes to local disk with a local SSD runner, then remove bytes, then remove the restore entirely with prepared machine state.