Advanced Usage of GitHub Actions Matrix Strategy (Dynamic Fan Out)
GitHub Actions has a feature called a "matrix strategy" that lets you run multiple copies of a job in parallel across different configurations.
Most people use it for static combinations like OS and language versions, but here’s the fun part: you can also use it to fan out over a dynamically generated set of job configurations.
This article was written as part of my preparations for the upcoming "GitHub Actions: Beginner to Pro" course I am working on.
See: https://github.com/sidpalas/devops-directive-github-actions-course for more info!
Standard Usage
The canonical example is to handle testing or building for multiple language versions and operating system combinations:
jobs:
example_matrix:
strategy:
matrix:
version: [10, 12, 14]
os: [ubuntu-latest, windows-latest]
By default, a job will be created for EACH combination of the input arrays. The above example would produce six jobs (3 versions
x 2 os
options).
You can exclude specific combinations that are covered by the input arrays:
jobs:
example_matrix_with_exclude:
strategy:
matrix:
version: [10, 12, 14]
os: [ubuntu-latest, windows-latest]
exclude:
# Here we SKIP the version 14 windows configuration
- version: 14
os: windows-latest
You can include additional combinations which are not covered by the input arrays:
jobs:
example_matrix_with_include:
strategy:
matrix:
version: [10, 12, 14]
os: [ubuntu-latest, windows-latest]
include:
# Here we test version 16 ONLY for ubuntu
- version: 16
os: ubuntu-latest
If your desired job configurations aren't shaped like a combinatorial set, you can skip passing any input arrays and instead pass ONLY an include
list:
jobs:
example_matrix_only_include:
strategy:
matrix:
include:
# Here we run two specific configurations
# which have no common elements
- version: 12
os: ubuntu-latest
- version: 14
os: windows-latest
Generic Job Fan Out
While many CI tasks do make sense to run across combinations of individual configuration sets, others do not. One pattern in particular which I have found to be extremely useful is to have one upstream job which determines what set of downstream jobs need to be run.
The upstream job may check which file paths have changed, or use a build system determinator to determine which set of downstream jobs should run and with what configurations.
For simplicity, let's use the names "filter"
and "execute"
to refer to the the upstream and downstream jobs respectively.
Non-matrix Solution
One way to implement this pattern is to have a fixed set of jobs, and to have named outputs from the filter
job which can be passed to each of the execute
jobs explicitly.
An example of this pattern is highlighted in this recent Reddit Engineering blog post: https://www.reddit.com/r/RedditEng/comments/1megwf1/our_buildkite_brings_all_the_devs_to_the_yard/
unit-tests-1:
uses: ./.github/workflows/unit-test-shard.yml
secrets: inherit
needs: [build-selector]
if: ${{ needs.build-selector.outputs.unit-test-shard-1 != '' }}
with:
shard-index: 1
gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-1 }}
total-shards: ${{ needs.build-selector.outputs.total-test-shards }}
unit-tests-2:
uses: ./.github/workflows/unit-test-shard.yml
secrets: inherit
needs: [build-selector]
if: ${{ needs.build-selector.outputs.unit-test-shard-2 != '' }}
with:
shard-index: 2
gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-2 }}
total-shards: ${{ needs.build-selector.outputs.total-test-shards }}
unit-tests-3:
uses: ./.github/workflows/unit-test-shard.yml
secrets: inherit
needs: [build-selector]
if: ${{ needs.build-selector.outputs.unit-test-shard-3 != '' }}
with:
shard-index: 3
gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-3 }}
total-shards: ${{ needs.build-selector.outputs.total-test-shards }}
This solution, while functional, has two major downsides. Namely, you must:
- define a fixed set of jobs
- copy/paste lots of yaml
In fact, these developer experience pain points were one of the big drawbacks the Reddit Engineering team cited that ultimately led them to choose Buildkite over GitHub Actions.
Matrix Solution
Fortunately, there is a way to achieve this type of job fan out which solves both of those downsides. The key enabler is the fromJSON()
function supported within GitHub Action Expressions!
We can combine fromJSON
with the ability to specify a matrix strategy containing only an include
key to dynamically generate and execute an arbitrary set of job configurations!
Using the reddit example above, we could refactor our upstream filtering job (aka build-selector
) to output something like:
[
{
"shard-index": 1,
"unit-test-this-shard": true,
"gradle-task": "test-shard-1",
"total-shards": 3
},
{
"shard-index": 2,
"unit-test-this-shard": false,
"gradle-task": "test-shard-2",
"total-shards": 3
},
{
"shard-index": 3,
"unit-test-this-shard": true,
"gradle-task": "test-shard-3",
"total-shards": 3
}
]
This would enable refactoring the downstream jobs into one using a matrix strategy:
unit-tests:
uses: ./.github/workflows/unit-test-shard.yml
secrets: inherit
needs: [build-selector]
if: ${{ matrix.unit-test-this-shard == 'true' }}
strategy:
matrix:
include: ${{ fromJson(needs.build-selector.outputs.matrix) }}
with:
shard-index: ${{ matrix.shard-index }}
gradle-task: ${{ matrix.gradle-task }}
total-shards: ${{ matrix.total-shards }}
It is important to call out this doesn’t actually work because you can’t use an if
conditional in a matrix job.
Luckily, because we control the JSON, we can simplify further and omit any entries we don't want to execute test jobs for:
[
{
"shard-index": 1,
"gradle-task": "test-shard-1",
"total-shards": 3
},
{
"shard-index": 3,
"gradle-task": "test-shard-3",
"total-shards": 3
}
]
This allows us to reduce the json array AND remove the now unnecessary conditional:
unit-tests:
uses: ./.github/workflows/unit-test-shard.yml
secrets: inherit
needs: [build-selector]
strategy:
matrix:
include: ${{ fromJson(needs.build-selector.outputs.matrix) }}
with:
shard-index: ${{ matrix.shard-index }}
gradle-task: ${{ matrix.gradle-task }}
total-shards: ${{ matrix.total-shards }}
Working Example
A full workflow demonstrating this pattern can be seen here: https://github.com/sidpalas/devops-directive-github-actions-course/blob/3e5401e1d8ea113bf1f9f4e2372fa8f753324fd0/.github/workflows/04-advanced-features--07-dynamic-matrix.yaml
The generate-matrix
job outputs a JSON array of length N (provided as input to the job) with each item in the array having random values for foo
and bar
:
[
{
foo: 35,
bar: 29
},
{
foo: 37,
bar: 41
},
{
foo: 77,
bar: 66
}
]
This is then passed into the execute
job using fromJSON
:
execute:
needs: generate-matrix
runs-on: ubuntu-latest
strategy:
matrix:
include: ${{ fromJSON(needs.generate-matrix.outputs.include) }}
Prior Art
Conditional Matrix Elements (Stack Overflow)
In writing this post, I also came across this Stack Overflow post “How do I make a GitHub Action matrix element conditional? and the GitHub action it inspired: conditional-build-matrix.
These use matrix + fromJSON
, but takes a fixed JSON file as its input and filters down to a subset of its contents to generate a the matrix configuration.
fromJSON Documentation
The fromJSON
docs show an example like this, with an upstream job returning a json object which the downstream job uses for its matrix input: https://docs.github.com/en/actions/reference/workflows-and-actions/expressions#example-returning-a-json-object
Closing Thoughts
Combining GitHub Actions’ matrix strategy with the fromJSON
function unlocks a powerful way to dynamically fan out jobs to any set of configurations without hardcoding every possibility.
This approach is especially valuable in large codebases or monorepos, where selective test sharding can save substantial CI time and cost.
The recent Reddit Engineering post reminded me how often this pattern is overlooked. While the fromJSON docs technically show an example, it’s buried and easy to miss unless you already know to look for it. Hopefully, this write-up makes the pattern more discoverable and gives you a practical starting point for your own workflows.