Concurrent tests in GitHub Actions

18 April 20248 minute read

Running builds and tests are probably the most common use-cases for GitHub Actions. Modern test frameworks and build systems have built-in support for sharding tests and remote execution. Sharding is essentially running your tests or builds in parallel, not only on different threads or cores but distributing them across different machines altogether.

This guide details how to set up concurrent tests for some of the popular test frameworks. We will explore how to set up test sharding in GitHub Actions for Jest, Playwright and Pytest.

Jest

Jest is a popular JavaScript testing framework. It provides native support for test sharding using the --shard option to run your tests in parallel simultaneously across multiple machines. The option takes an argument in the form of shardIdx/shardCount, where shardIdx is a number representing index of the shard and shardCount is the total number of shards.

A simple benchmark on a dummy test suite run showed that sharding improves the run time from 3 minutes to 30 seconds.

Here's how you can set it up in GitHub Actions:

1name: CI
2on: push
3
4jobs:
5  test:
6    name: Tests
7    runs-on: warp-ubuntu-latest-x64-4x
8
9    strategy:
10      fail-fast: false
11      matrix:
12        shardCount: [10]
13        shardIdx: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
14
15    steps:
16      - uses: actions/checkout@v4
17      - uses: actions/setup-node@v4
18      - run: npm ci
19
20      - name: Run Jest tests
21        run: npx jest --shard=${{ matrix.shardIdx }}/${{ matrix.shardCount }}

Tip

Jest parallelizes test runs across workers to maximize performance. You could optimize the performance of each shard by using the --maxWorkers option to specify the number of workers to use.

Playwright

Playwright is a popular end-to-end automation and testing framework for web applications. Native support for test sharding is available using the --shard option. Just like we saw with Jest earlier, the --shard option takes an argument in the form of shardIdx/shardCount, where shardIdx is the index of the shard and shardCount is the total number of shards.

Sharding the tests improve the time from around 5 minutes 26 seconds to 1 minute 25 seconds for a dummy test suite run.

The setup is very similar to that of Jest:

1name: CI
2on: push
3
4jobs:
5  tests:
6    name: Tests
7    runs-on: warp-ubuntu-latest-x64-4x
8
9    strategy:
10      fail-fast: false
11      matrix:
12        shardCount: [10]
13        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
14
15    steps:
16      - uses: actions/checkout@v4
17      - uses: actions/setup-node@v4
18      - run: npm ci
19
20      - name: Install Playwright browsers
21        run: npx playwright install --with-deps
22
23      - name: Run Playwright tests
24        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardCount }}

Tip

  1. Playwright runs your tests in parallel by default using worker processes. To further optimize the tests in each shard, you can use the --workers option to specify the number of workers to use.

  2. Consider using container-based jobs to potentially speed up tests by reducing the overhead of browser installation for each shard.

Pytest

Pytest is a widely used testing framework for Python. Although Pytest doesn't natively support test sharding across machines, the third-party plugin pytest-split can distribute tests based on duration or name.

The performance gain after sharding was really significant since Pytest doesn't run tests in parallel by default. The dummy test runs showed a 10x improvement in test run times – from 500 to 50 seconds.

Setting up the workflow involves installing the pytest-split package and running pytest with the --splits and --group options:

1name: CI
2on: push
3
4jobs:
5  test:
6    name: Tests
7    runs-on: warp-ubuntu-latest-x64-4x
8
9    strategy:
10      fail-fast: false
11      matrix:
12        splitCount: [10]
13        group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
14
15    steps:
16      - uses: actions/checkout@v4
17      - uses: actions/setup-python@v5
18      - run: pip install pytest pytest-split
19
20      - name: Run pytest
21        run: pytest --splits ${{ matrix.splitCount }} --group ${{ matrix.group }}

Tip

You can also optimize the tests to run faster by parallelizing the tests within each shard by using the pytest-xdist plugin.

Comparison and Notes

All of the test suites used, workflows and their runs can be found in our concurrent-tests repository. Here is a comparison the test run performance before and after sharding for the three test frameworks:

Test FrameworkDefaultSharded x10ImprovementWorkflow Run
Jest3m30s6xLink
Playwright5m 26s1m 25s3.8xLink
Pytest8m 20s50s10xLink

Note

An important thing to keep in mind before sharding your tests with GitHub Actions is – all the steps inside the job are run again for each shard. This includes dependency installs, fetching third party actions, etc. For example, as stated earlier in the Playwright setup, we need to install the browsers for each shard.

Steps like these could add significant overhead and the performance gain might not be worth it or even inexistent. In worst cases, it might even increase the overall run time. It makes sense to benchmark your tests before and after sharding to see the benefits. Usually, large test suites with long run times benefit the most from sharding.

Limitations

While GitHub Actions' matrix strategy is really handy for parallelizing our tests and builds, a constraint that can't be ignored for performance is the imposed limit on number of concurrent jobs. GitHub has a limit of 20 concurrent jobs per account. You can increase this limit to 40 on a pro account or 60 on a team account but it is still very limiting for large teams.

Further improvements

Overall CI workflow times with GitHub actions can be reduced by parallelizing tests. Run times can be further improved by making the tests within each shard run faster by using the native parallelization features of the test frameworks or by using third-party packages.

WarpBuild provides blazing fast runners, optimized for CPU, network, and disk performance that are 30% faster at half the cost. At WarpBuild, there are no limits on the number of concurrent jobs. You can run as many jobs as you want in parallel and minimize the wait times on GitHub Actions workflows. It takes only ~2 minutes to get started.

Note

Even if WarpBuild doesn't impose a concurrency limitation, there is an upstream constraint by GitHub which limits the maximum number of jobs generated by a job matrix to 256. But that limit is seldom the problem in practice.

Previous post

Docker registry mirror setup

17 April 2024
DockerGuideEngineering
Next post

A Complete Guide to Self-hosting GitHub Actions Runners

29 April 2024
GitHub ActionsGitHubGuideEngineering