A Developer's Guide to Speeding Up GitHub Actions

GitHub Actions offer a powerful tool for automating CI/CD pipelines. However, slow runs can be frustrating. This guide provides an exhaustive evaluation of GitHub Actions workflow performance and suggests effective mitigation strategies.

If you are just starting off on GitHub Actions performance optimization, here are the first 3 things you should do:

1. Cache Dependencies: Reduce build times by avoiding repetitive dependency downloads.
2. Parallelize Jobs: Accelerate workflows by running jobs concurrently.
3. Use Powerful Runners: For intensive tasks, custom runners with high-spec hardware are beneficial.

Here is a comprehensive list of factors that can affect GitHub Action performance, along with examples and mitigation strategies.

Workflow and Script Configuration

Complex Workflows

• Description: Workflows with many steps or intricate logic can take longer to execute.
• Example: A workflow that includes multiple build processes, testing across different environments, and deployment steps can become time-consuming.
• Mitigation: Simplify workflows by breaking them into smaller, more focused jobs. Use conditional steps to avoid unnecessary runs.

Inefficient Build Scripts

• Description: Scripts not optimized for performance can slow down the workflow.
• Example: A build script that compiles code without leveraging caching or incremental builds.
• Mitigation: Refactor scripts for efficiency, use caching where possible, and remove redundant operations.

Misconfigured Workflow Triggers

• Description: Triggers that initiate workflows unnecessarily or too frequently can cause delays.
• Example: A workflow set to trigger on every push, including documentation updates, can lead to unneeded builds.
• Mitigation: Configure triggers carefully, such as triggering workflows only on specific branches or for specific types of changes.

Poorly Managed Artifacts/Logs

• Description: Large or numerous build artifacts and logs can slow down operations.
• Example: Saving extensive logs or large binary files after every build can consume significant storage and bandwidth.
• Mitigation: Implement log rotation, compress artifacts, and retain only essential artifacts. If using self-hosted runners, the default GitHub cache can be extremely slow. Consider hosting your own cache server and/or artifact registry that is closer to compute.

Repository and Codebase Factors

High Repository Size

• Description: Large repositories with extensive histories or large files take longer to clone and process.
• Example: A repository with gigabytes of historical data and large binary files will be slow to clone.
• Mitigation: Use Git LFS for large files, prune old branches, and compact repository history.

Build Complexity

• Description: Complex applications with many dependencies and modules require more time to build.
• Example: A project with dozens of dependencies and a multi-module structure.
• Mitigation: Optimize build processes, use dependency caching, and modularize the codebase.

Dependency Management

• Description: Fetching large or numerous external dependencies can be time-consuming.
• Example: A build process that downloads hundreds of npm packages or large Docker images.
• Mitigation: Cache dependencies and consider using a package manager that supports efficient dependency resolution.

GitHub Hosted Runner Specifications

Computational Resources

• Description: Limited CPU and memory on runners can slow down intensive tasks.
• Example: Compiling a large application on a runner with only 2 CPU cores and limited RAM.
• Mitigation: Optimize code for resource efficiency or use self-hosted runners with better specs.

Disk I/O Performance

• Description: Disk speed and capacity affect tasks involving significant read/write operations.
• Example: A workflow that involves processing large datasets stored on disk.
• Mitigation: Optimize disk usage, consider storing less data on disk, or use self-hosted runners with faster disks.

Network Bandwidth

• Description: Limited network speed can delay tasks that require data transfer.
• Example: Uploading large build artifacts or pulling large Docker images from a registry.
• Mitigation: Compress data before transfer, reduce the size of artifacts, and use Docker layer caching.

External and Environmental Factors

External Dependencies

• Description: Reliance on external services or APIs can introduce delays.
• Example: A workflow that makes numerous calls to external APIs, which may be slow or rate-limited.
• Mitigation: Minimize external API calls, implement retry logic with backoff, and use mocking for testing.

Network Issues

• Description: General internet connectivity problems or specific network issues can cause delays.
• Example: Slowdowns due to poor network performance in the runner's region.
• Mitigation: If persistent, consider using self-hosted runners in a different region with better connectivity.

Service Outages/Degradation

• Description: Issues with GitHub or third-party services can impact workflow performance.
• Example: Delays or failures due to GitHub service outages.
• Mitigation: Implement error handling and retry mechanisms, stay informed about service statuses.

Rate Limiting

• Description: Hitting rate limits on external APIs or services used in workflows.
• Example: Exceeding the API rate limit of a third-party service used in a build process.
• Mitigation: Optimize API usage, cache responses where possible, and handle rate limit errors gracefully.

Resource Management and Optimization Strategies

Caching Strategies

• Description: Effective use of caching can significantly reduce build times.
• Example: Rebuilding dependencies in every run can be time-consuming. Using cached dependencies speeds up the process.
• Mitigation: Implement caching for dependencies, build outputs, and other reusable data to avoid unnecessary repetition.

Parallelization

• Description: Running tasks in parallel can reduce overall workflow duration.
• Example: Sequential execution of tests can be slow. Running tests in parallel across different runners can speed it up.
• Mitigation: Break down workflows into parallelizable jobs and use matrix strategies for testing across environments.

Resource Allocation Policies

• Description: GitHub's internal resource allocation can affect performance.
• Example: During peak times, resource contention might slow down workflows.
• Mitigation: Optimize workflows to run efficiently with available resources and consider off-peak scheduling.

Use of Self-Hosted Runners

• Description: Custom hardware specifications can meet specific performance needs.
• Example: A workflow requiring high CPU and RAM might be slow on GitHub-hosted runners. This is especially true of disk intensive workloads and workflows that require GPUs.
• Mitigation: Set up self-hosted runners with tailored resources for intensive workflows.

Platform-Specific Constraints and Policies

Concurrent Usage Limits

• Description: Limits on the number of concurrent workflows can delay execution.
• Example: If multiple workflows are queued, subsequent ones will wait, causing delays.
• Mitigation: Optimize workflow triggers and durations to minimize queuing delays.

Resource Throttling Policies

• Description: GitHub may limit resources to manage platform stability.
• Example: Throttling can occur during times of high demand, affecting performance.
• Mitigation: Design workflows to be efficient under varied resource availability conditions.

Underlying Infrastructure and Platform Changes

Updates in GitHub's Infrastructure

• Description: GitHub's infrastructure changes can temporarily affect performance.
• Example: Upgrades or maintenance activities can slow down or disrupt services.
• Mitigation: Stay informed about GitHub updates, and plan for alternative strategies during scheduled maintenance.

Environmental Variables Configuration

• Description: Management of environment variables affects workflow efficiency.
• Example: Misconfigured environment variables, like the CI, DEBUG, and LOGLEVEL variables, can lead to errors or inefficiencies.
• Mitigation: Regularly review and optimize the use of environment variables, ensuring they are used effectively and securely.

By addressing these factors, one can effectively manage resources, adapt to platform constraints, and stay responsive to changes in GitHub's infrastructure, all contributing to improved GitHub Action performance.

A Note on GitHub-hosted Runner Processors

The processors used in GitHub Actions runners are not optimized for CI and build workloads. They are server-grade processors designed for high-performance computing and data center workloads. The following table compares the specifications of the processors used in GitHub Actions runners (on 2023-11-13). From an anecdotal sampling of ~100 runs, the AMD EPYC 7763 64-Core Processor appears to be the most common processor used in GitHub Actions runners.

For comparison, here are the specifications of some popular desktop processors:

Source: PassMark Note: ChatGPT was used to scrape processor information and structuring the article.

They are not equivalent comparisons however, since desktop and mobile processors can have heterogeneous cores, which are optimized for different workloads. WarpBuild provides runners with high performance processors, which are optimized for CI and build workloads with fast disk IO and improved caching (coming soon). You can learn more about our hosted runners here.

Conclusion

Improving the speed of your GitHub Actions isn’t just about tweaking a few settings; it’s about a holistic approach to workflow management, and resource optimization. By understanding the underlying factors, you'll be well on your way to more efficient, faster GitHub Actions.

Share & Feedback: Found this guide useful? Share it with fellow engineers. Feedback and additional factors are welcome. I'm [email protected].

Keywords:

• GitHub Actions Performance
• CI/CD Pipeline Optimization
• Workflow Efficiency in GitHub
• GitHub Actions Speed Tips
• Developer Guide to GitHub Actions