Advanced Usage of GitHub Actions Matrix Strategy (Dynamic Fan Out)

GitHub Actions has a feature called a "matrix strategy" that lets you run multiple copies of a job in parallel across different configurations.

Most people use it for static combinations like operating system and language versions, but here’s the fun part: you can also use it to fan out over a set of job configurations dynamically generated at runtime.

Standard Usage

The canonical use case for the matrix strategy feature is to handle testing or building for multiple language versions and operating system combinations:

jobs:
  example_matrix:
    strategy:
      matrix:
        version: [10, 12, 14]
        os: [ubuntu-latest, windows-latest]

By default, a job will be created for EACH combination of the input arrays. The above example would produce six jobs (3 versions x 2 os options).

You can exclude specific combinations that are covered by the input arrays:

jobs:
  example_matrix_with_exclude:
	strategy:
	  matrix:
		version: [10, 12, 14]
		os: [ubuntu-latest, windows-latest]
		exclude:
			# Here we SKIP the version 14 windows configuration
			- version: 14
			  os: windows-latest

You can include additional combinations which are not covered by the input arrays:

jobs:
  example_matrix_with_include:
	strategy:
	  matrix:
		version: [10, 12, 14]
		os: [ubuntu-latest, windows-latest]
		include:
			# Here we test version 16 ONLY for ubuntu
			- version: 16
			  os: ubuntu-latest

If your desired job configurations aren't shaped like a combinatorial set, you can skip passing any input arrays and instead pass ONLY an include list:

jobs:
  example_matrix_only_include:
	strategy:
	  matrix:
		include:
			# Here we run two specific configurations
			# which have no common elements
			- version: 12
			  os: ubuntu-latest
			- version: 14
			  os: windows-latest

Dynamic Job Fan Out

While many CI tasks do make sense to run across static combinations of individual configuration sets, others do not. One pattern in particular which I have found to be extremely useful is to have one upstream job which determines what set of downstream jobs need to be run.

The upstream job may check which file paths have changed, or use a build system determinator to determine which set of downstream jobs should run and with what configurations.

For simplicity, let's use the names "filter" and "execute" to refer to the the upstream and downstream jobs respectively.

Non-matrix Solution

One way to implement this pattern is to have a fixed set of jobs, and to have named outputs from the filter job which can be passed to each of the execute jobs explicitly.

An example of this approach is highlighted in this recent Reddit Engineering blog post: https://www.reddit.com/r/RedditEng/comments/1megwf1/our_buildkite_brings_all_the_devs_to_the_yard/

unit-tests-1:    
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit    
  needs: [build-selector]    
  if: ${{ needs.build-selector.outputs.unit-test-shard-1 != '' }}    
  with:      
    shard-index: 1      
    gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-1 }}      
    total-shards: ${{ needs.build-selector.outputs.total-test-shards }}  
    
unit-tests-2:    
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit
  needs: [build-selector]
  if: ${{ needs.build-selector.outputs.unit-test-shard-2 != '' }}    
  with:      
     shard-index: 2      
     gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-2 }}      
     total-shards: ${{ needs.build-selector.outputs.total-test-shards }}  
     
unit-tests-3: 
   uses: ./.github/workflows/unit-test-shard.yml    
   secrets: inherit    
   needs: [build-selector]    
   if: ${{ needs.build-selector.outputs.unit-test-shard-3 != '' }}    
   with:      
      shard-index: 3
      gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-3 }}     
      total-shards: ${{ needs.build-selector.outputs.total-test-shards }}

This solution, while functional, has two major downsides. Namely, you must:

1. define a fixed set of jobs
2. copy/paste lots of yaml

In fact, these developer experience pain points were one of the big drawbacks the Reddit Engineering team cited that ultimately led them to choose Buildkite over GitHub Actions.

Matrix Solution

Fortunately, there is a way to achieve this type of job fan out which solves both of those downsides. The key enabler is the fromJSON() function supported within GitHub Action Expressions!

We can combine fromJSON with the ability to specify a matrix strategy containing only an include key to dynamically generate and execute an arbitrary set of job configurations!

Using the reddit example above, we could refactor our upstream filtering job (aka build-selector) to output something like:

[
  {
    "shard-index": 1,
    "unit-test-this-shard": true,
    "gradle-task": "test-shard-1",
    "total-shards": 3
  },
  {
    "shard-index": 2,
    "unit-test-this-shard": false,
    "gradle-task": "test-shard-2",
    "total-shards": 3
  },
  {
    "shard-index": 3,
    "unit-test-this-shard": true,
    "gradle-task": "test-shard-3",
    "total-shards": 3
  }
]

This would enable refactoring the downstream jobs into one using a matrix strategy:

unit-tests:
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit    
  needs: [build-selector]    
  if: ${{ matrix.unit-test-this-shard == 'true' }}    
  strategy:
	  matrix:
		  include: ${{ fromJson(needs.build-selector.outputs.matrix) }}
  with:      
    shard-index: ${{ matrix.shard-index }}      
    gradle-task: ${{ matrix.gradle-task }}
    total-shards: ${{ matrix.total-shards }}

🚨 NOTE: It is important to call out that this doesn’t actually work because you can’t use an if conditional in a matrix job.

Luckily, because we control the JSON, we can simplify further and omit any entries we don't want to execute test jobs for:

[
  {
    "shard-index": 1,
    "gradle-task": "test-shard-1",
    "total-shards": 3
  },
  {
    "shard-index": 3,
    "gradle-task": "test-shard-3",
    "total-shards": 3
  }
]

This allows us to reduce the JSON array AND remove the now unnecessary conditional:

unit-tests:
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit    
  needs: [build-selector]    
  strategy:
	  matrix:
		  include: ${{ fromJson(needs.build-selector.outputs.matrix) }}
  with:      
    shard-index: ${{ matrix.shard-index }}      
    gradle-task: ${{ matrix.gradle-task }}
    total-shards: ${{ matrix.total-shards }}

Working Example

A full workflow demonstrating this pattern can be seen here: https://github.com/sidpalas/devops-directive-github-actions-course/blob/3e5401e1d8ea113bf1f9f4e2372fa8f753324fd0/.github/workflows/04-advanced-features--07-dynamic-matrix.yaml

The generate-matrix job outputs a JSON array of length N (provided as input to the job) with each item in the array having random values for foo and bar:

[
	{
		foo: 35,
		bar: 29
	},
	{
		foo: 37,
		bar: 41
	},
	{
		foo: 77,
		bar: 66
	}
]

This is then passed into the execute job using fromJSON:

  execute:
    needs: generate-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix:
        include: ${{ fromJSON(needs.generate-matrix.outputs.include) }}

Prior Art

Conditional Matrix Elements (Stack Overflow)

In writing this post, I also came across this Stack Overflow post “How do I make a GitHub Action matrix element conditional? and the GitHub action it inspired: conditional-build-matrix.

These use matrix + fromJSON, but takes a fixed JSON file as its input and filters down to a subset of its contents to generate a the matrix configuration.

fromJSON Documentation

The fromJSON docs show an example like this, with an upstream job returning a json object which the downstream job uses for its matrix input: https://docs.github.com/en/actions/reference/workflows-and-actions/expressions#example-returning-a-json-object

Closing Thoughts

Combining GitHub Actions’ matrix strategy with the fromJSON function unlocks a powerful way to dynamically fan out jobs to any set of configurations without hardcoding every possibility.

This approach is especially valuable in large codebases or monorepos, where selective test sharding can save substantial CI time and cost.

The recent Reddit Engineering post reminded me how often this pattern is overlooked. While the fromJSON docs technically show an example, it’s buried and easy to miss unless you already know to look for it. Hopefully, this write-up makes the pattern more discoverable and gives you a practical starting point for your own workflows.