Split tests by timing | Depot CI | Depot Docs

Depot CI

Cut the wall-clock time of a large test suite by splitting it across jobs into balanced shards based on historical timing data.

How it works

Sharding tests across parallel jobs can reduce a suite’s wall-clock time, but a shard with disproportionately slow tests becomes the bottleneck that determines when the suite finishes. Using historical timing data to distribute tests evenly across shards helps reduce the suite’s total run time.

Equal file counts versus balanced durations

Six files take 8, 4, 3, 2, 2, and 1 seconds. Equal file counts put A, C, and F in one shard and B, D, and E in the other, lasting 12 and 8 seconds. Balancing by duration puts A and E in one shard and B, C, D, and F in the other. Both shards take 10 seconds. Each shard runs one file at a time.

Equal file counts

Finishes in 12s

Balanced by duration

Finishes in 10s

The example runs one file at a time in each shard. Bar widths show illustrative file durations. Your test runner can also run files in parallel within a shard.

Depot splits your tests by how long they take. First you give Depot a list of candidates. A candidate is one thing your test command can run, such as a test file or a package name. You also tell Depot how many shards to split into. Depot gives each shard a different, non-overlapping subset of the candidates, and it uses the time each candidate took in past runs to balance how long the shards take to run.

Each job runs its shard and uploads a JUnit XML report. Depot uses the times in those reports to balance the next run.

Test splitting and timing feedback

Every shard receives the same candidate list. Depot uses historical timings to assign a different subset to each shard. Each shard runs its subset and uploads JUnit XML. Those reports provide timing history for later runs.

• Candidate list
• The same list goes to every shard
• Depot assigns candidates
• Historical timings balance the split

Timing history

From JUnit reports

If a suite has no past timing data, filename candidates fall back to file-size splitting, and other candidates use fallback weights. Each later run uses the times from previous runs, so the split gets more balanced over time.

Authentication

depot tests split, depot tests run, and depot tests report are supported only in Depot CI and Depot GitHub Actions runner jobs. Depot CI authenticates automatically. For Depot GitHub Actions runners, grant the workflow id-token: write permission, as shown in the examples.

Split, run, and report in a matrix job

depot/tests-run-action wraps depot tests run: it selects the list assigned to the current shard, runs your command, and uploads the JUnit XML report it creates.

jobs:
  test:
    runs-on: depot-ubuntu-24.04
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1, 2, 3]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - name: Install gotestsum
        run: |
          go install gotest.tools/gotestsum@latest
          echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
      - name: Run test shard
        uses: depot/tests-run-action@v1
        with:
          candidates-command: go list ./...
          command: mkdir -p reports && xargs gotestsum --junitfile reports/junit.xml --
          report-path: reports/junit.xml

Copy code

The action installs no test-runner integration of its own. Replace go list and gotestsum with the candidate discovery and command your test framework needs.

In Depot CI, DEPOT_MATRIX_JOB_INDEX and DEPOT_MATRIX_JOB_TOTAL supply this matrix job's shard position. When shard inputs are omitted, the CLI uses those values. Non-matrix jobs receive 0 and 1, so they remain unsplit.

On Depot GitHub Actions runners, pass the shard inputs explicitly instead:

permissions:
  contents: read
  id-token: write

# ...inside depot/tests-run-action@v1 with:
index: ${{ strategy.job-index }}
total: ${{ strategy.job-total }}

Copy code

Explicit shard inputs override the corresponding Depot CI values. For a direct depot tests run call, --total 1 runs the complete candidate list, even inside a matrix.

In a multi-axis matrix, set index and total explicitly when tests should split on only one axis instead of the flattened matrix. The following example splits only on shard.

Choose a split scope in a multi-axis matrix

Automatic Depot CI values describe the immediate strategy's flattened matrix. A runtime × usage × shard matrix with 2 runtimes, 2 usages, and 4 shards has 16 jobs. With the same candidate list and split identity across all jobs, those values produce one 16-way split. Different candidate lists within the same split cause a candidate list changed for this test split error.

Flattened matrix versus splitting on the shard axis

A matrix with two runtimes, two suites, and four shards creates sixteen jobs. Automatic matrix values produce one sixteen-way split when all jobs share the same candidate list and split identity. Different candidate lists within one split cause an error. Explicit shard index and total values, with a distinct split-key for each runtime and suite combination, create four independent four-way splits.

Automatic matrix values

One 16-way split across the flattened matrix

Requires the same candidate list and split identity across all 16 jobs

01456789

Explicit shard index, total, and split-key

Four independent 4-way splits, one for each runtime and suite

When only the shard axis should split tests, pass its index and total explicitly. Set a distinct split-key for each runtime and usage combination, and use the same candidate list within each group. In this example, each combination receives its own four-way split:

strategy:
  matrix:
    runtime: [node-20, node-22]
    usage: [unit, integration]
    shard: [0, 1, 2, 3]

steps:
  - uses: depot/tests-run-action@v1
    with:
      candidates-command: ./discover-tests ${{ matrix.runtime }} ${{ matrix.usage }}
      # Override the flattened matrix values: split only by the shard axis.
      index: ${{ matrix.shard }}
      total: 4
      split-key: ${{ matrix.runtime }}-${{ matrix.usage }}
      # ...command and report path

Copy code

In reusable workflows, use explicit values whenever the intended split scope differs from the called job's immediate strategy.

Choose a candidate source

A candidate is one runnable unit that your test command accepts. It must map to either a filename or classname emitted by your runner's JUnit report. The Depot CI matrix example uses go list ./... to produce Go package names, which the JUnit report treats as classnames.

When the effective shard total is greater than one, provide exactly one candidate source: a candidate list, file, or command. When the effective total is 1, a candidate source is optional and depot tests run runs the command with empty standard input when none is provided.

• candidates-command runs dynamic discovery in the Depot CLI. Use it for commands such as go list ./...; its standard output must contain only one candidate per line, while diagnostics belong on standard error.
• candidates-file reads an existing or shared newline-delimited list. Use it for a large list or one another step has already generated.
• candidates provides a small, fixed newline-delimited list directly in the workflow.

Choose a candidate identity

candidate-type is optional in both the action and CLI. When it is omitted, Depot infers filename when every candidate looks like a recognized source or test-file path; otherwise, it infers classname. The inferred or explicit type must match the identity in the uploaded JUnit report that Depot uses to look up timings.

For the following validated test runners, use the recommended candidate type. This is not an exhaustive list.

You can omit candidate-type when the candidate strings' apparent file or class identity matches the identity recorded by the test report. Set it explicitly when they differ—especially when a runner accepts file paths but writes those paths to the JUnit classname field. Vitest and Playwright are examples: pass candidate-type: classname so Depot identifies their timings correctly.

Choose timing granularity

timings-type controls the granularity of the timing data Depot uses for each candidate. filename and classname use timings recorded for the whole candidate and must match the resolved candidate-type; testname uses individual JUnit test cases and is the only timing identity that can differ.

Use the default when the JUnit report records a timing for each candidate. Set timings-type: testname when the report has individual test cases and you want Depot to calculate each candidate's duration from those tests. For example, the direct Go/gotestsum commands pass package candidates but use timings-type: testname to calculate package durations from their individual test names.

Use keys for advanced workflows

Basic matrices need neither split-key nor report-key: the split identity defaults to the job and action identity, and the report identity remains the action default. Use split-key only when distinct logical suites in one job would otherwise share split scope, such as when they have different candidate lists, timing identities, or granularity. Give every shard of one suite the same split-key.

For example, when one job runs browser and API test suites, keep their timing assignments separate:

steps:
  - name: Run browser tests
    uses: depot/tests-run-action@v1
    with:
      split-key: browser-tests
      report-key: browser-tests
      # ...browser candidates and command

- name: Run API tests
    uses: depot/tests-run-action@v1
    with:
      split-key: api-tests
      report-key: api-tests
      # ...API candidates and command

Copy code

report-key identifies a report upload. Use a distinct value only when one job uploads more than one report; ordinary matrix jobs need no per-shard report-key.

Run the CLI directly

Use depot tests split when you only need Depot to print a shard's candidates:

depot tests split \
  --candidates-command 'go list ./...' \
  --timings-type testname \
  --index 0 \
  --total 4

Copy code

Use depot tests run to select a shard, run a command, and upload its reports in one command:

mkdir -p reports
depot tests run \
  --candidates-command 'go list ./...' \
  --timings-type testname \
  --index 0 \
  --total 4 \
  --command 'xargs gotestsum --junitfile reports/junit.xml --' \
  --report-path reports/junit.xml

Copy code

You can also pass candidates through standard input or an existing list with --candidates-file.

For either command, Depot uses historical timings when it has them. A first run still works: filename candidates fall back to file-size splitting, and other candidates use deterministic fallback weights. After a run uploads JUnit XML, those durations are available to balance later runs.

See Depot CI test results or GitHub Actions test results for JUnit reporting, result views, and analytics.

Ask AI