The Rendered Manifests Pattern

As I built out Kube Starter Kit, I considered a variety of different approaches for implementing GitOps.

I considered the tradeoffs between ArgoCD, Flux, and Kluctl, eventually deciding on ArgoCD based on its large community and mature UI. During this process I came across this article from Akuity (the creators of ArgoCD) "The Rendered Manifests Pattern" and ultimately decided to implement the pattern (with a few tweaks).

What is the rendered manifests pattern?

The rendered manifests pattern avoids runtime abstractions or mutations between the source of truth (in Git) and the resources deployed to Kubernetes (etcd). Abstractions such as Helm or Kustomize are still used, but only during a rendering step that produces fully hydrated manifests, which are then stored in Git and applied to clusters.

Pros

Full visibility and reviewability

All changes to resources are visible as plain YAML in Git and can be reviewed directly, without needing to reason about how a change to values or overlays might affect the final rendered output.

Tooling agnostic

Any tooling can be used to generate the final manifests, including newer or more expressive tools such as Timoni or CUE. By decoupling rendering from reconciliation, you are no longer constrained by the set of tools supported by your GitOps engine, and clusters no longer need access to those tools.

Simpler break glass and safer recovery

In an emergency, it is possible to modify the deployed manifests directly to apply a hotfix. Because future changes must still go through the rendering process, folding those fixes back into the source configuration is a safe and explicit operation rather than an implicit state reconstruction.

Improved GitOps engine performance

Moving rendering out of the GitOps reconciliation loop reduces the amount of work the GitOps engine needs to perform, improving performance and making it easier to scale across many applications or clusters.

Cons

Requires additional CI logic and validation

The primary disadvantage of the rendered manifests pattern is the need for additional CI logic to either:

1. render and commit manifests, or
2. validate that pre-rendered manifests are up to date and correct

This adds extra pipeline code and conventions that must be maintained.

Inability to use helm hooks

Because Helm is not executed at apply time, Helm lifecycle features such as hooks are unavailable. Some GitOps tools (e.g. ArgoCD) offer similar mechanisms, but they are not fully equivalent and may require chart-specific adjustments.

Rendered Secrets would contain sensitive values

Because rendered manifests are stored in Git, embedding raw Kubernetes Secret resources is problematic. Instead should be moved to a dedicated secrets manager and injected at runtime using tools such as External Secrets Operator. While this removes some convenience, it enforces a more secure approach.

Larger git repositories

Rendering all Kubernetes resources results in additional files being stored in the repository. Over time this can lead to larger repos, increased diff noise, and slower checkouts if not managed carefully.

Comparing Implementations

Unrendered (not using the rendered manifests pattern)

Before implementing the rendered manifests pattern, it’s useful to understand how a system without it typically works and how it fails to benefit from the advantages described above.

Popular GitOps engines such as ArgoCD and Flux support Kustomize and Helm directly. Because of this, you can store a reference to a Helm chart along with a values.yaml file, and the GitOps engine will render and apply the resources to the cluster.

This approach keeps configuration minimal and DRY. However, it makes it difficult to review or reason about how a change in Git will impact deployed resources. Upgrading a Helm chart version or modifying a single value can result in large and unexpected changes to the rendered manifests. A reviewer may not realize the scope of those changes, leading to potentially dangerous deployments.

Akuity Suggested Implementation

The Akuity article describes the following implementation of the pattern:

1. A user commits configuration changes to the main branch (typically via a PR)
2. A continuous integration pipeline renders the manifests and commits them to an environment-specific branch
3. Updates to environment-specific branches deploy the configuration automatically or via manual sync (with ArgoCD configured to use those branches as its source of truth)

This is a reasonable improvement, but it still limits reviewability: the rendered output is produced after the primary review step unless an additional manual gate is introduced.

Kube Starter Kit's implementation

My goal was to maximize reviewability before merge while still enforcing correctness after merge.

To achieve this, I chose to implement the pattern differently:

1. A user makes configuration changes on a branch
2. Before creating a PR, the user renders the manifests locally and commits the changes to an environment-specific path in the repository
3. PRs against the main branch trigger a CI pipeline that validates all manifests are properly rendered (i.e. re-rendering produces no diff)
4. Merging to main deploys the new configuration (with ArgoCD configured to use environment-specific paths on the main branch)

This approach provides full visibility during review while maintaining safety by enforcing correct rendering through CI.

One downside is that users must have the required tooling installed locally and remember to render manifests before opening a PR. Tooling such as mise and pre-push Git hooks can reduce this burden. In practice, this has proven to be a worthwhile trade-off given the improvements to reviewability and operational safety.

In practice, the difference between these approaches is not where rendering happens, but when correctness and visibility are enforced.

Conclusion

The rendered manifests pattern makes the full configuration of resources being deployed explicit before it reaches a cluster. Rendering manifests ahead of time makes changes easier to review, reason about, and recover from.

It does require some additional continuous integration work, but the benefits in visibility, safety, and operational clarity make it worthwhile.