DevOps Directive

🤖 Deputies

2026-07-28T16:51:47.677Z

Deputies

Self-Hosted Background Agents for Engineering Teams

The Problem

Coding agents are powerful when a developer is actively supervising them from a terminal. But using agents across an engineering organization requires much more than an LLM and a command-line tool.

Teams need a way to assign work, run agents in isolated environments, preserve progress, share results, manage access, integrate with existing workflows, and review exactly what happened. Building that control plane requires queues, sandboxes, authentication, event streaming, artifact storage, integrations, and reliable recovery.

Without that infrastructure, agent usage remains tied to individual laptops, difficult to audit, and disconnected from the systems where engineering work begins.

What is Deputies?

Deputies is an open-source control plane for delegating engineering work to background agents.

Give it a task through the web UI, Slack, GitHub, or pretty much any system that can send a webhook. Deputies prepares an isolated sandbox with the appropriate repositories and tools, runs the agent asynchronously, and preserves its progress, diagnostics, artifacts, and final response in a durable session.

It gives engineering teams the shared infrastructure needed to move coding agents out of individual terminals and into repeatable, observable workflows.

How It Works

Assign work. Start a session from the web UI or trigger one from Slack, GitHub, a webhook, or a schedule.
Run in isolation. Deputies prepares a sandbox with the selected repositories, environment, tools, and context.
Follow progress. Watch messages, tool calls, diagnostics, generated artifacts, and application previews as the agent works.
Review and iterate. Inspect the result, queue follow-up instructions, download artifacts, or have the agent prepare a pull request.

What's Included

Component	Description
Durable Agent Sessions	Persistent sessions with ordered messages, searchable history, replayable events, artifacts, and follow-ups
Agent Web UI	Assign work, follow live progress, inspect diagnostics, preview applications, and review results
Isolated Sandboxes	Run agents with Daytona, Docker, Tensorlake, Kubernetes Agent Sandbox, or AWS Lambda MicroVM environments
Repository Environments	Reusable, revisioned work contexts that can include multiple repositories
GitHub Integration	Trigger work from GitHub, provide scoped repository access, and let agents prepare branches and pull requests
Slack Integration	Delegate work from Slack threads and deliver completion responses back to the originating conversation
Webhooks and Automations	Create agent work from external events or scheduled workflows
Artifacts and Previews	Preserve generated files, reports, logs, and links to applications running inside agent sandboxes
Queued Follow-Ups	Add instructions while work is running or return later and resume the same durable session
Skills and Prompt Snippets	Package repeatable team procedures and quickly reuse common instructions
Team Access Control	Tenant-wide viewer, member, and admin roles, with support for private sessions
Multiple Model Providers	Use standard LLM API keys or supported subscriptions through the Pi agent runtime
Portable Deployment	Deploy with Railway, Docker Compose, Kubernetes, or the AWS reference architecture
Observability	OpenTelemetry support for control-plane traces and operational metrics

Deployment Options

Deployment	Sandbox Provider	Best For
Railway Template	Daytona or another hosted provider	Getting a hosted instance running with minimal infrastructure work
Docker Compose	Docker	Small teams, a dedicated server, or production-style local deployments
Kubernetes Helm Chart	Kubernetes Agent Sandbox	Organizations that want agent execution inside an existing Kubernetes environment
AWS Reference Architecture	Lambda MicroVM sandboxes	Teams standardizing on ECS, RDS, S3, and AWS-native isolation
Custom Containers	Any supported provider	Organizations with specific networking, security, or infrastructure requirements

Key Design Decisions

Deployability is a core feature. Deputies uses portable infrastructure primitives—Node.js, Postgres, HTTP, containers, and optional S3-compatible storage—rather than depending on a single cloud platform.
Agent work is durable. Sessions, messages, events, artifacts, sandbox state, and callback deliveries survive beyond an individual browser tab or developer laptop.
Sandboxes are provider-neutral. The control plane owns the workflow while execution environments plug in through a stable provider interface. Teams can choose managed sandboxes, Docker, Kubernetes, or AWS infrastructure based on their requirements.
Integrations create work; they do not own it. Slack, GitHub, webhooks, and schedules all feed the same durable session model. Work can be reviewed and continued in Deputies regardless of where it started.
You control the trust boundary. Deputies can run alongside private repositories, internal tools, CI systems, and observability platforms. Operators decide where the control plane, sandboxes, credentials, and model access live.
Start simple and scale when needed. Run the API and worker together for a small deployment, then split and scale web, API, worker, storage, and sandbox orchestration independently as usage grows.
The agent runtime is replaceable infrastructure. Deputies currently uses Pi for real agent work while keeping the durable control-plane concerns (sessions, queues, integrations, artifacts, and sandbox lifecycle) separate from the runner.

Example Workflows

Assign an issue from GitHub and have an agent investigate it, make a change, run tests, and prepare a pull request.
Ask for a CI failure investigation from Slack and receive the result back in the same thread.
Schedule recurring dependency, security, or maintenance checks.
Launch an agent against multiple related repositories with a preconfigured development environment.
Generate reports, migration plans, screenshots, or other artifacts and preserve them with the session.
Run and review an application inside the sandbox before accepting the agent's changes.
Give agents controlled access to internal tools and data through remote MCP servers such as Executor.

Who Is This For?

Engineering and platform teams who:

Want to delegate real engineering work without tying execution to individual developer machines
Need shared visibility into what agents are doing and how they reached their conclusions
Work with private repositories, internal services, or custom development environments
Want agents to participate in GitHub, Slack, CI, incident response, and scheduled workflows
Need control over where agent infrastructure and persisted data are hosted
Would rather customize an open-source control plane than build agent orchestration from scratch
Want to start with a small deployment and retain a path to production-scale infrastructure

[!NOTE]
Deputies is still early-stage and evolving quickly. Expect new capabilities and occasional breaking changes as the project develops.

🚀 Kube Starter Kit

2026-07-28T16:48:56.979Z

Kube Starter Kit

A Production-Ready Kubernetes Platform Starter

The Problem

Every engineering team that adopts Kubernetes faces the same challenge: spinning up a cluster is easy, but building a production-ready platform is not. You need infrastructure-as-code, GitOps pipelines, secrets management, observability, compliance controls, and CI/CD automation, all wired together correctly across multiple environments.

This takes months to build properly, and most teams end up with inconsistent, hard-to-maintain infrastructure because they're racing to ship features.

What is Kube Starter Kit?

Kube Starter Kit is an opinionated, production-ready Kubernetes platform for AWS and EKS. It provides everything needed to run applications on Kubernetes: infrastructure, deployment pipelines, and baseline components, ready to deploy and customize.

It's the platform you'd build yourself if you had the time (and had done it a few times before).

What's Included

Component	Description
Terraform Infrastructure	Modular Terraform for AWS with multi-environment support
Terramate Orchestration	Stack-based orchestration with change detection and CI/CD
AWS Architecture	Multi-account setup with VPC, EKS, and secure boundaries
User Management	GitHub + AWS IAM Identity Center integration
GitOps (ArgoCD)	Declarative, auditable deployments via app-of-apps pattern
K8s Baseline	Ingress, cert-manager, external-dns, external-secrets, observability
CI/CD Pipelines	Automated container builds and staging deployments on merge
Image CVE Scanning	Automated vulnerability scanning with daily scheduled scans
Release Management	Automated release PRs with release-please
Demo Applications	Fully functional Go services demonstrating end-to-end patterns
Local Development	KinD + Tilt + mirrord for fast local iteration

Repository Structure

kube-starter-kit/
├── .github/workflows/    # CI/CD pipelines (build, deploy, drift detection)
├── kubernetes/
│   ├── src/              # Source manifests (Kustomize, Helm, Timoni)
│   └── rendered/         # Generated manifests by environment
├── terraform/
│   ├── modules/          # Reusable modules (eks, networking, etc.)
│   └── live/             # Environment configs (staging, production)
├── services/             # Application source code
└── local/                # KinD cluster config + Tiltfile

Key Design Decisions

You own everything. No managed service dependency. Infrastructure runs in your AWS account with full control.
Standard tools, no abstraction layer. Unlike OpenShift or Tanzu, there's no proprietary UI. It's standard Kubernetes with standard open-source tools (ArgoCD, cert-manager, external-secrets, etc.).
GitOps from day one. All changes flow through Git. ArgoCD handles deployments. Infrastructure changes go through Terramate-orchestrated Terraform with PR previews.
Multi-environment by default. Staging and production environments with proper separation, automated staging deploys, and controlled production releases.
Compliance-ready. Features map directly to SOC2 controls; audit trails, access management, vulnerability scanning, and secrets management are built in.

Who Is This For?

Engineering teams at early-stage companies who:

Know Kubernetes is the right choice for their infrastructure
Want to reach production quickly without cutting corners
Would rather customize a working system than build from scratch
Need to meet compliance requirements (SOC2, etc.) without a dedicated platform team

📚 Courses

2026-07-28T16:45:59.314Z

Courses

I create top tier DevOps tutorials and release them on YouTube, GitHub, and at https://courses.devopsdirective.com/

Take your DevOps skills to the next level!

💼 Consulting

2026-01-05T17:20:35.432Z

Consulting

I help companies build application platforms that enable them to ship quickly and safely.

Much of my recent work has been with Kubernetes + AWS, but I also have experience with GCP, and deploying to airgapped on-prem networks.

Interested in taking your DevOps to the next level? Reach out to sid@devopsdirective.com!

Some Previous Engagements:

Nominal

https://nominal.io/

Sid created the foundation for our infrastructure, setting up continuous deployment using best practices and state of the art open source tools, working with us 2023-2025 as we grew from 6 to 50 engineers. Even in the very early days of our engineering organization, developers had the leverage of master-crafted foundational tools and automation, something that we’ve seen larger and mature organizations take teams of people to do. We gave him the nickname of “Gandalf” because he so frequently made magic happen. Although he might be too humble to admit it, I suspect some of his work would still be crucial even if we hired 10x as many engineers.

Despite only working part time, Sid was a clear owner when it came to devops and infrastructure; he was always incredibly responsive to any questions from developers and very often jumped in to help with incident response, even if unrelated to his domain.

Sid is also an excellent teacher. He is passionate about spreading his knowledge in a scalable and repeatable way. He left a wake of fantastic documentation for everything he did.

Sid very often sought and incorporated feedback from developers so that he was always pointed on the most important infrastructure and devops issues. He was also an essential thought partner for developers for when we had new expansion opportunities such as deploying on premises or in customer clouds.

Jason Hoch, Co-founder

Gauntlet Networks

https://gauntlet.xyz/

Sid started working with Gauntlet in April 2020 when the company was less than 10 people. We had a basic Kubernetes cluster set up to handle some manually triggered simulation jobs, and brought Sid on board to help take our DevOps and infrastructure to the next level. Over the following two years, Sid led a number of significant projects including:

Rolling out and upgrading Kubeflow for better management and automation of ETL, simulation, and optimization workflows on Kubernetes

Leveraging cluster autoscaling and pre-emptible instances to efficiently serve bursty workloads at scale

Designing and implementing a CI/CD workflow that fit the needs of our organization, including setting up IaaS via Terraform, cluster deployment and maintenance across multiple development environments, as well as observability and monitoring solutions

Sid was able to independently own most of these projects over their entire lifecycle as a part-time contractor, while still collaborating effectively with others on the team to navigate complex and changing business requirements. Sid is always meticulous about code quality, documentation, and knowledge sharing, which resulted in a smooth transition at the end of the project. Much of Gauntlet’s infrastructure continues to rely on Sid’s work, which now supports a team of 50, running hundreds of daily jobs with tricky compute requirements and dependency management. I would recommend working with Sid in a heartbeat for any similar projects."

Rei Chiang, Chief Technology Officer

Ursa Space Systems

https://ursaspace.com/

Sid worked with Ursa Space Systems from November 2021 through December 2022. We hired Sid to support our Systems Team as a DevOps Consultant while we were expanding our DevOps team. His impact on our operations cannot be overstated! Sid delivered great value, even as a part-time employee.

His expertise with Kubernetes, AWS, and Gitlab CI is top-notch. He collaborated effectively with our development team and earned our highest trust as an external contractor. For all of these reasons, I would highly recommend Sid to any company looking for DevOps expertise."

Julie Baker, Chief Operating Officer

👋 About

2026-01-05T17:20:39.242Z

About

Hi, I'm Sid.

I am a technologist, engineer, athlete, beard grower, and tiny living enthusiast.

My goal with DevOps Directive is to provide educational and entertaining (hopefully?...) content to enable engineers to level up their skills and keep up with the continuously evolving DevOps and Cloud Infrastructure landscape.

I will be exploring many different types of media, including:

Topics include:

Infrastructure as Code
Kubernetes
Databases
Networking
Security
Testing
Continuous Integration
Continuous Deployment
Monitoring
Cloud Providers
Career Advancement

I am not an expert in all of these topics. That being said, I am good at Reading the Docs, asking the right questions, and explaining what I learn along the way.

Join me!

-Sid

forking-the-planetscale-terraform-provider

2026-03-10T19:21:50.370Z

I forked the PlanetScale Terraform Provider

Disclaimer: I am bullish on PlanetScale. I also have strong opinions about Infrastructure as Code.

Last week, I was excited to see the PlanetScale Terraform provider v1 (LINK) was released and now includes support for their Postgres product! 🎉

Unfortunately, it only includes resources for managing branches (not the databases themselves). See: https://planetscale.com/docs/terraform#conceptual-model

[!WARNING] PlanetScale databases themselves (the logical database containers) are not managed directly as Terraform resources in v1.0. Instead:

Create databases in PlanetScale first (via the PlanetScale UI or API).

Use Terraform to manage branches and related resources (roles, passwords, etc.) within those databases.

In my opinion, this makes the provider totally unusable. Having to diverge from Infrastructure as Code to click-ops the database and then import the default branch breaks any fully automated workflow and results in a less than ideal solution.

After chatting with some folks from PlanetScale (shoutout to @josh and @Josh 😅) they helped me understand the reasoning behind this design choice:

TL;DR: The PlanetScale data model and lifecycle for databases and branches doesn't map cleanly onto the Terraform resource model and they didn't want to lock in an approach until they were confident it was the right one.

Challenges

There are three main challenges with how PlanetScale databases/branches behave when designing the Terraform provider:

A default branch is auto-created when a database is created
A database cannot exist without at least one branch
The default branch can be modified after creation

[!INFO] Note:

For the sake of this article I will use postgres for all examples. Analogous resources would exist for PlanetScale vitess databases/branches.

Together, these raise questions about how to design the Terraform provider:

Who owns the default branch in Terraform?
How is the auto-created branch adopted without import?
How is a change in default branch represented?
What should happen during terraform destroy?

A naive implementation would either lead to resource contention between the database and branch resources or require a two-step create-then-import process for the default branch.

Prior Art

To understand how to best approach this, I looked at existing Terraform providers that face similar challenges.

The AWS, GitHub , and Kubernetes providers contain a handful of resources which must account for pre-existing and default lifecycle behaviors:

aws_default_vpc and aws_default_subnet: These resources have different behaviors depending if the defaults they represent exist within AWS. If they do not exist, Terraform creates them, but if they do exist, Terraform adopts them.
github_branch_default: This resource models the default branch as its own terraform resource which acts as a pointer to the repo/branch.
kubernetes_default_service_account_v1: The default service account for each namespace is always created by kubernetes, so this resource is always used to adopt/manage that service account (e.g. to add labels/annotations)

User Stories

To capture my goals I came up with the following user stories:

As a Terraform user, I want to manage PlanetScale database resources directly, not just branches, so that I can model the full database lifecycle in Terraform.
As a Terraform user, I want to manage a database’s default branch properties without a separate two-step import workflow, so that I can bring new and existing databases under management more smoothly.
As a Terraform user, I want to change a database’s default branch with clear, predictable lifecycle behavior, so that repointing the default branch works cleanly during apply and does not create confusing destroy or drift semantics.
As a Terraform user, I want terraform destroy to cleanly remove both database and branch resources, so that ephemeral environments can be created and torn down without manual cleanup.

My Solution

I decided to fork the provider repo and experiment with implementing a proposed solution. Here is what I ended up with: PULL REQUEST

ADD planetscale_database_postgres
ADD planetscale_database_default_branch
MODIFY planetscale_postgres_branch
- New field: adopt_if_exists (default false) so Terraform can adopt a pre-existing branch instead of failing creation, which is especially useful for auto-created branches like main.
- New field: on_default_delete to define behavior when a branch being destroyed is still the database’s current default branch:
  - error (default): fail with an actionable message explaining that the default branch must be repointed first
  - remove_from_state: if the API refuses deletion because the branch is still the default, Terraform removes the branch from state and continues

Together, these enable all of the desired user stories and I was able to use the modified provider to:

Provision a new database, adopt the default branch, create a new branch, and modify the default branch to point to that new branch in a single apply command
Cleanly tear down all of those resources with a single destroy command

Here is my config:

terraform {
  required_providers {
    planetscale = {
      source  = "sidpalas/planetscale"
      version = "1.1.0-postgres-db"
    }
  }
}

provider "planetscale" {

}

resource "planetscale_database_postgres" "db" {
  organization = "sid-devopsdirective"
  name         = "with-branch"
  cluster_size = "PS_5_AWS_ARM"
}

resource "planetscale_postgres_branch" "main" {
  organization      = "sid-devopsdirective"
  database          = planetscale_database_postgres.db.id
  name              = "main"
  cluster_size      = "PS_5_AWS_ARM"
  adopt_if_exists   = true
}

resource "planetscale_postgres_branch" "someotherbranch" {
  organization      = "sid-devopsdirective"
  database          = planetscale_database_postgres.db.id
  name              = "someotherbranch"
  cluster_size      = "PS_5_AWS_ARM"
  on_default_delete = "remove_from_state"
}

resource "planetscale_database_default_branch" "this" {
  organization = "sid-devopsdirective"
  database     = planetscale_database_postgres.db.id
  branch       = planetscale_postgres_branch.someotherbranch.name
}

terraform apply 🚀

$ terraform apply  

...

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

planetscale_database_postgres.db: Creating...
planetscale_database_postgres.db: Still creating... [00m10s elapsed]
...
planetscale_database_postgres.db: Still creating... [01m50s elapsed]
planetscale_database_postgres.db: Creation complete after 1m51s [id=p4wvn79boc9j]
planetscale_postgres_branch.someotherbranch: Creating...
planetscale_postgres_branch.main: Creating...
planetscale_postgres_branch.main: Creation complete after 3s [id=2bqz7zmvhm33]
planetscale_postgres_branch.someotherbranch: Still creating... [00m10s elapsed]
...
planetscale_postgres_branch.someotherbranch: Still creating... [02m10s elapsed]
planetscale_postgres_branch.someotherbranch: Creation complete after 2m20s [id=xgi6gwxpmafo]
planetscale_database_default_branch.this: Creating...
planetscale_database_default_branch.this: Creation complete after 0s [id=p4wvn79boc9j]
╷
│ Warning: existing branch adopted
│ 
│   with planetscale_postgres_branch.main,
│   on main.tf line 21, in resource "planetscale_postgres_branch" "main":
│   21: resource "planetscale_postgres_branch" "main" {
│ 
│ The PostgreSQL branch already exists and was adopted because adopt_if_exists is true.
╵

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

terraform destroy 🧨

terraform destroy

...

Plan: 0 to add, 0 to change, 4 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

planetscale_database_default_branch.this: Destroying... [id=p4wvn79boc9j]
planetscale_postgres_branch.main: Destroying... [id=2bqz7zmvhm33]
planetscale_database_default_branch.this: Destruction complete after 0s
planetscale_postgres_branch.someotherbranch: Destroying... [id=xgi6gwxpmafo]
planetscale_postgres_branch.someotherbranch: Destruction complete after 0s
planetscale_postgres_branch.main: Destruction complete after 1s
planetscale_database_postgres.db: Destroying... [id=p4wvn79boc9j]
planetscale_database_postgres.db: Destruction complete after 0s
╷
│ Warning: default branch not deleted remotely
│ 
│ The branch is currently the database default branch and cannot be deleted by PlanetScale. The resource was
│ removed from Terraform state because on_default_delete is set to remove_from_state.
╵

Destroy complete! Resources: 4 destroyed.

Limitations

There are some important limitations in the current implementation:

This provider is generated from the OpenAPI API specification using Speakeasy, but adopt_if_exists plus on_default_delete are currently implemented as manual edits in generated code in internal/provider/postgresbranch_resource.go. As a result, regenerating the provider will overwrite those changes.
The current solution only covers PostgreSQL resources. Equivalent lifecycle support would need to be added for Vitess databases and branches.
This implementation was produced largely with OpenCode and GPT-assisted development. It should be treated as a prototype and should receive additional engineering review, testing, and validation before being considered production-ready.
(Unrelated to the specific challenges addressed here) Modifying the cluster configuration (e.g. # of replicas) for the database currently forces it to be recreated. This behavior is different from the PlanetScale UI where the cluster configuration can be modified in-place, but appears to be a limitation of the public API (based on the OpenAPI spec).

Input Requested! 🙏

[!INFO] NOTE

The PlanetScale team wants to get to a clean long-term solution here, but wanted to avoid locking in a pattern they would eventually come to regret! My goal with the fork + article was to help make progress towards such a solution!

If you have experience working with PlanetScale and using (or authoring) Terraform providers, I’d love your input on the lifecycle tradeoffs discussed here, especially around default branch management, adoption of pre-existing resources, and destroy behavior for ephemeral environments.

Please share your feedback, edge cases, or preferred UX on the GitHub issue: https://github.com/planetscale/terraform-provider-planetscale/issues/260

[!TIP] Try it out!

I published my version of the provider here if you want to take a look and experiment with its ergonomics! https://registry.terraform.io/providers/sidpalas/planetscale/latest

The source for my fork can be seen here: https://github.com/sidpalas/terraform-provider-planetscale

rendered-manifests-pattern

2026-01-15T14:13:57.381Z

The Rendered Manifests Pattern

As I built out Kube Starter Kit, I considered a variety of different approaches for implementing GitOps.

I considered the tradeoffs between ArgoCD, Flux, and Kluctl, eventually deciding on ArgoCD based on its large community and mature UI. During this process I came across this article from Akuity (the creators of ArgoCD) "The Rendered Manifests Pattern" and ultimately decided to implement the pattern (with a few tweaks).

What is the rendered manifests pattern?

The rendered manifests pattern avoids runtime abstractions or mutations between the source of truth (in Git) and the resources deployed to Kubernetes (etcd). Abstractions such as Helm or Kustomize are still used, but only during a rendering step that produces fully hydrated manifests, which are then stored in Git and applied to clusters.

Pros

Full visibility and reviewability

All changes to resources are visible as plain YAML in Git and can be reviewed directly, without needing to reason about how a change to values or overlays might affect the final rendered output.

Tooling agnostic

Any tooling can be used to generate the final manifests, including newer or more expressive tools such as Timoni or CUE. By decoupling rendering from reconciliation, you are no longer constrained by the set of tools supported by your GitOps engine, and clusters no longer need access to those tools.

Simpler break glass and safer recovery

In an emergency, it is possible to modify the deployed manifests directly to apply a hotfix. Because future changes must still go through the rendering process, folding those fixes back into the source configuration is a safe and explicit operation rather than an implicit state reconstruction.

Improved GitOps engine performance

Moving rendering out of the GitOps reconciliation loop reduces the amount of work the GitOps engine needs to perform, improving performance and making it easier to scale across many applications or clusters.

Cons

Requires additional CI logic and validation

The primary disadvantage of the rendered manifests pattern is the need for additional CI logic to either:

render and commit manifests, or
validate that pre-rendered manifests are up to date and correct

This adds extra pipeline code and conventions that must be maintained.

Inability to use helm hooks

Because Helm is not executed at apply time, Helm lifecycle features such as hooks are unavailable. Some GitOps tools (e.g. ArgoCD) offer similar mechanisms, but they are not fully equivalent and may require chart-specific adjustments.

Rendered Secrets would contain sensitive values

Because rendered manifests are stored in Git, embedding raw Kubernetes Secret resources is problematic. Instead should be moved to a dedicated secrets manager and injected at runtime using tools such as External Secrets Operator. While this removes some convenience, it enforces a more secure approach.

Larger git repositories

Rendering all Kubernetes resources results in additional files being stored in the repository. Over time this can lead to larger repos, increased diff noise, and slower checkouts if not managed carefully.

Comparing Implementations

Unrendered (not using the rendered manifests pattern)

Before implementing the rendered manifests pattern, it’s useful to understand how a system without it typically works and how it fails to benefit from the advantages described above.

Popular GitOps engines such as ArgoCD and Flux support Kustomize and Helm directly. Because of this, you can store a reference to a Helm chart along with a values.yaml file, and the GitOps engine will render and apply the resources to the cluster.

This approach keeps configuration minimal and DRY. However, it makes it difficult to review or reason about how a change in Git will impact deployed resources. Upgrading a Helm chart version or modifying a single value can result in large and unexpected changes to the rendered manifests. A reviewer may not realize the scope of those changes, leading to potentially dangerous deployments.

Akuity Suggested Implementation

The Akuity article describes the following implementation of the pattern:

A user commits configuration changes to the main branch (typically via a PR)
A continuous integration pipeline renders the manifests and commits them to an environment-specific branch
Updates to environment-specific branches deploy the configuration automatically or via manual sync (with ArgoCD configured to use those branches as its source of truth)

This is a reasonable improvement, but it still limits reviewability: the rendered output is produced after the primary review step unless an additional manual gate is introduced.

Kube Starter Kit's implementation

My goal was to maximize reviewability before merge while still enforcing correctness after merge.

To achieve this, I chose to implement the pattern differently:

A user makes configuration changes on a branch
Before creating a PR, the user renders the manifests locally and commits the changes to an environment-specific path in the repository
PRs against the main branch trigger a CI pipeline that validates all manifests are properly rendered (i.e. re-rendering produces no diff)
Merging to main deploys the new configuration (with ArgoCD configured to use environment-specific paths on the main branch)

This approach provides full visibility during review while maintaining safety by enforcing correct rendering through CI.

One downside is that users must have the required tooling installed locally and remember to render manifests before opening a PR. Tooling such as mise and pre-push Git hooks can reduce this burden. This has proven to be a worthwhile trade-off given the improvements to reviewability and operational safety.

In practice, the difference between these approaches is not where rendering happens, but when correctness and visibility are enforced.

Conclusion

The rendered manifests pattern makes the full configuration of resources being deployed explicit before it reaches a cluster. Rendering manifests ahead of time makes changes easier to review, reason about, and recover from.

It does require some additional continuous integration work, but the benefits in visibility, safety, and operational clarity make it worthwhile.

github-actions-required-checks-for-conditional-jobs

2025-08-20T22:02:46.000Z

GitHub Actions Required Checks for Conditional Jobs

GitHub has a feature called rulesets which allows you to define rulesets to control various behaviors within a repo.

One common use case for rulesets is to define required status checks which must pass before a PR can be merged into the default branch.

For standard workflows, this is pretty straightforward. You choose the GitHub Actions workflow step you want to validate, and select it. For a workflow with a conditional job, or one which uses a matrix strategy to fan out over a variety of configurations, things get more complex.

[!TIP] Upcoming Course
This article was written as part of my preparations for the upcoming "GitHub Actions: Beginner to Pro" course I am working on.

See: https://github.com/sidpalas/devops-directive-github-actions-course for more info!

For example, this repo conditionally runs tests for services based on which changed in the PR using a dynamic matrix (see: advanced-github-actions-matrix). For any given PR, from 0 to N run-tests steps will have executed and we want to validate any that ran passed.

Which step should I choose?! 😱

Before we continue it is also important to highlight the following (somewhat counterintuitive) behavior:

[!WARNING]
A job that is skipped will report its status as "Success". It will not prevent a pull request from merging, even if it is a required check.

(See: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks).

This GitHub Issue contains lots of useful discussion about this behavior and how to work with/around it and was the inspiration for Option 3 below.

Option 1: Require ALL THE CHECKS!

We could select each of the run-tests steps as required.

As noted, if a check is skipped, it reports its status as "success", so this would work for ensuring all checks that ran passed.

Pros:

Does not require modifying the workflow

Cons:

If a new services is added, you must remember to update the status checks
The UI will have lots of required checks, many of which are skipped

Option 2: Add Downstream Check to Compile the Result

We could also add a job to the workflow which depends on our dynamic matrix but runs even if it fails or is skipped by specifying job.<job_id>.if: always():

  run-tests-status-check:
    runs-on: ubuntu-24.04
    if: always()
    needs: [filter, run-tests]
    steps:
      - name: Run Tests Status Check
        run: |
          svcs='${{ needs.filter.outputs.services }}'
          echo "services=${svcs}"
          echo "run-tests.result=${{ needs.run-tests.result }}"
          
          # If no services changed, treat as success (no tests to run)
          if  -z "$svcs" ; then
            echo "No matching services changed; passing status check."
            exit 0
          fi
          
          # Otherwise, require the matrix job to have succeeded
          if [[ "${{ needs.run-tests.result }}" == "success" ]]; then
            echo "All test matrix jobs passed."
            exit 0
          else
            echo "::error One or more test matrix jobs failed or were cancelled."
            exit 1
          fi

We can then use this single status check as the required check for our workflow:

Pros:

A single status check keeps PR UI clean
Continues working with no modifications when adding services

Cons:

Requires modifying the workflow
Adds a few seconds to the critical path (even in the success case)

Option 3: Add Downstream Check Which Raises Failures

A third option is to take advantage of the "skipped = success" behavior and add a job which is skipped UNLESS the upstream jobs fail or are cancelled.

  run-tests-failure-alert:
    runs-on: ubuntu-latest
    needs: [filter, run-tests]
    if: ${{ cancelled() || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'failure') }}
    steps:
      - name: Conditionally Fail Required Check
        shell: bash
        run: |
          echo "::error Some required job has failed!"
          exit 1

Successful Run (failure alert step is skipped)

Failed Run (failure alert runs and fails)

Pros:

A single status check keeps PR UI clean
Continues working with no modifications when adding services
Doesn't add time to the critical path in the success case

Cons:

Requires modifying the workflow
The if: ${{ cancelled() || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'failure') }} is relatively complex
Relies on understanding the skipped = success behavior (job naming + comments can help)

Final Thoughts

The fact that option 1 requires remembering to make a manual update as services are added/removed makes it inherently unsafe.

In my opinion, options 2 & 3 are both reasonable approaches and choosing one over the other is a matter of personal preference. I like option 3 slightly more as implemented here because it avoids inline bash and doesn't extend the the critical path runtime.

advanced-github-actions-matrix

2025-08-20T15:19:02.336Z

Advanced Usage of GitHub Actions Matrix Strategy (Dynamic Fan Out)

GitHub Actions has a feature called a "matrix strategy" that lets you run multiple copies of a job in parallel across different configurations.

Most people use it for static combinations like operating system and language versions, but here’s the fun part: you can also use it to fan out over a set of job configurations dynamically generated at runtime.

[!TIP] Upcoming Course
This article was written as part of my preparations for the upcoming "GitHub Actions: Beginner to Pro" course I am working on.

See: https://github.com/sidpalas/devops-directive-github-actions-course for more info!

Standard Usage

The canonical use case for the matrix strategy feature is to handle testing or building for multiple language versions and operating system combinations:

jobs:
  example_matrix:
    strategy:
      matrix:
        version: [10, 12, 14]
        os: [ubuntu-latest, windows-latest]

By default, a job will be created for EACH combination of the input arrays. The above example would produce six jobs (3 versions x 2 os options).

You can exclude specific combinations that are covered by the input arrays:

jobs:
  example_matrix_with_exclude:
	strategy:
	  matrix:
		version: [10, 12, 14]
		os: [ubuntu-latest, windows-latest]
		exclude:
			# Here we SKIP the version 14 windows configuration
			- version: 14
			  os: windows-latest

You can include additional combinations which are not covered by the input arrays:

jobs:
  example_matrix_with_include:
	strategy:
	  matrix:
		version: [10, 12, 14]
		os: [ubuntu-latest, windows-latest]
		include:
			# Here we test version 16 ONLY for ubuntu
			- version: 16
			  os: ubuntu-latest

If your desired job configurations aren't shaped like a combinatorial set, you can skip passing any input arrays and instead pass ONLY an include list:

jobs:
  example_matrix_only_include:
	strategy:
	  matrix:
		include:
			# Here we run two specific configurations
			# which have no common elements
			- version: 12
			  os: ubuntu-latest
			- version: 14
			  os: windows-latest

Dynamic Job Fan Out

While many CI tasks do make sense to run across static combinations of individual configuration sets, others do not. One pattern in particular which I have found to be extremely useful is to have one upstream job which determines what set of downstream jobs need to be run.

The upstream job may check which file paths have changed, or use a build system determinator to determine which set of downstream jobs should run and with what configurations.

For simplicity, let's use the names "filter" and "execute" to refer to the the upstream and downstream jobs respectively.

Non-matrix Solution

One way to implement this pattern is to have a fixed set of jobs, and to have named outputs from the filter job which can be passed to each of the execute jobs explicitly.

An example of this approach is highlighted in this recent Reddit Engineering blog post: https://www.reddit.com/r/RedditEng/comments/1megwf1/our_buildkite_brings_all_the_devs_to_the_yard/

unit-tests-1:    
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit    
  needs: [build-selector]    
  if: ${{ needs.build-selector.outputs.unit-test-shard-1 != '' }}    
  with:      
    shard-index: 1      
    gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-1 }}      
    total-shards: ${{ needs.build-selector.outputs.total-test-shards }}  
    
unit-tests-2:    
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit
  needs: [build-selector]
  if: ${{ needs.build-selector.outputs.unit-test-shard-2 != '' }}    
  with:      
     shard-index: 2      
     gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-2 }}      
     total-shards: ${{ needs.build-selector.outputs.total-test-shards }}  
     
unit-tests-3: 
   uses: ./.github/workflows/unit-test-shard.yml    
   secrets: inherit    
   needs: [build-selector]    
   if: ${{ needs.build-selector.outputs.unit-test-shard-3 != '' }}    
   with:      
      shard-index: 3
      gradle-task: ${{ needs.build-selector.outputs.unit-test-shard-3 }}     
      total-shards: ${{ needs.build-selector.outputs.total-test-shards }}

This solution, while functional, has two major downsides. Namely, you must:

define a fixed set of jobs
copy/paste lots of yaml

In fact, these developer experience pain points were one of the big drawbacks the Reddit Engineering team cited that ultimately led them to choose Buildkite over GitHub Actions.

Matrix Solution

Fortunately, there is a way to achieve this type of job fan out which solves both of those downsides. The key enabler is the fromJSON() function supported within GitHub Action Expressions!

We can combine fromJSON with the ability to specify a matrix strategy containing only an include key to dynamically generate and execute an arbitrary set of job configurations!

Using the reddit example above, we could refactor our upstream filtering job (aka build-selector) to output something like:

[
  {
    "shard-index": 1,
    "unit-test-this-shard": true,
    "gradle-task": "test-shard-1",
    "total-shards": 3
  },
  {
    "shard-index": 2,
    "unit-test-this-shard": false,
    "gradle-task": "test-shard-2",
    "total-shards": 3
  },
  {
    "shard-index": 3,
    "unit-test-this-shard": true,
    "gradle-task": "test-shard-3",
    "total-shards": 3
  }
]

This would enable refactoring the downstream jobs into one using a matrix strategy:

unit-tests:
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit    
  needs: [build-selector]    
  if: ${{ matrix.unit-test-this-shard == 'true' }}    
  strategy:
	  matrix:
		  include: ${{ fromJson(needs.build-selector.outputs.matrix) }}
  with:      
    shard-index: ${{ matrix.shard-index }}      
    gradle-task: ${{ matrix.gradle-task }}
    total-shards: ${{ matrix.total-shards }}

🚨 NOTE: It is important to call out that this doesn’t actually work because you can’t use an if conditional in a matrix job.

Luckily, because we control the JSON, we can simplify further and omit any entries we don't want to execute test jobs for:

[
  {
    "shard-index": 1,
    "gradle-task": "test-shard-1",
    "total-shards": 3
  },
  {
    "shard-index": 3,
    "gradle-task": "test-shard-3",
    "total-shards": 3
  }
]

This allows us to reduce the JSON array AND remove the now unnecessary conditional:

unit-tests:
  uses: ./.github/workflows/unit-test-shard.yml    
  secrets: inherit    
  needs: [build-selector]    
  strategy:
	  matrix:
		  include: ${{ fromJson(needs.build-selector.outputs.matrix) }}
  with:      
    shard-index: ${{ matrix.shard-index }}      
    gradle-task: ${{ matrix.gradle-task }}
    total-shards: ${{ matrix.total-shards }}

Working Example

A full workflow demonstrating this pattern can be seen here: https://github.com/sidpalas/devops-directive-github-actions-course/blob/3e5401e1d8ea113bf1f9f4e2372fa8f753324fd0/.github/workflows/04-advanced-features--07-dynamic-matrix.yaml

The generate-matrix job outputs a JSON array of length N (provided as input to the job) with each item in the array having random values for foo and bar:

[
	{
		foo: 35,
		bar: 29
	},
	{
		foo: 37,
		bar: 41
	},
	{
		foo: 77,
		bar: 66
	}
]

This is then passed into the execute job using fromJSON:

  execute:
    needs: generate-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix:
        include: ${{ fromJSON(needs.generate-matrix.outputs.include) }}

Prior Art

Conditional Matrix Elements (Stack Overflow)

In writing this post, I also came across this Stack Overflow post “How do I make a GitHub Action matrix element conditional? and the GitHub action it inspired: conditional-build-matrix.

These use matrix + fromJSON, but takes a fixed JSON file as its input and filters down to a subset of its contents to generate a the matrix configuration.

fromJSON Documentation

The fromJSON docs show an example like this, with an upstream job returning a json object which the downstream job uses for its matrix input: https://docs.github.com/en/actions/reference/workflows-and-actions/expressions#example-returning-a-json-object

Closing Thoughts

Combining GitHub Actions’ matrix strategy with the fromJSON function unlocks a powerful way to dynamically fan out jobs to any set of configurations without hardcoding every possibility.

This approach is especially valuable in large codebases or monorepos, where selective test sharding can save substantial CI time and cost.

The recent Reddit Engineering post reminded me how often this pattern is overlooked. While the fromJSON docs technically show an example, it’s buried and easy to miss unless you already know to look for it. Hopefully, this write-up makes the pattern more discoverable and gives you a practical starting point for your own workflows.

speed-run-new-post

2025-07-09T18:03:34.000Z

This is a test to see how fast I can create a post

See site-revamp-3.0 for more info!

Not bad!

site-revamp-3.0

2025-07-10T14:00:53.077Z

Yet Another Site Revamp (devopsdirective․com v3.0)

At the time of publishing this article (2025-07-09), I hadn't published an article to this site in 602 days! 😱

I'm not 100% sure why this is, but I think it as at least partially due to the friction involved with creating and publishing posts.

Previous Setup

For a full history of the tooling for this site see:

Previously, the site was built using https://gohugo.io/. I authored posts in VS Code, either locally or using GitPod, and committed the changes to GitHub, where a GitHub Action handled building/pushing to Google Cloud Storage to be served.

This worked fine, but didn't ✨spark joy✨

Despite my previous attempts to improve the workflow, the (small) friction of authoring in VS Code and interacting with Git directly were still annoying. I also had a couple of custom Hugo components for links and images which added additional micro-cuts to the workflow.

Should I use Ghost?

As I considered what to do, I once again revisited https://ghost.org/ as a potential option. It is slick, but requires either:

Paying them to host it (which gets quite pricey as the number of members grows)
Hosting it yourself and managing:
- The Ghost application
- MySQL database (+ backups)
- A Mailgun subscription
- A Cloudinary subscription

I played around with hosting it on Railway with this template, but ran into some wonkiness around:

Healthchecks: I was unable to get Railway's healthchecks for the Ghost application working properly on Railway (which in turn prevents zero downtime deploys) for two reasons:
- https redirect: Railway healthchecks are made without TLS and ghost returns a 301 redirect from http -> https. I think this could be worked around by setting this http header: X-Forwarded-Proto: https (but Railway does not allow setting custom headers for healthchecks).
- hostname: Railway healthchecks originate from healthcheck.railway.app. Again, if we could set custom http headers, we could use: Host: devopsdirective.com, but we can’t.
Volume Snapshots: Volume snapshots on Railway are only available to pro users. I could have implemented a separate mechanism to backup the database OR committed to this costing at least $20/month... but I'm irrationally frugal (sometimes to my own detriment)! 🤑

That canvas view is 🔥 though!

Asking for Inputs

Two days ago I decided to see if anyone had recommendations for a blog setup they loved, and Victor Buendia recommended using Obsidian!

He might be onto something 👀

I already use Obsidian for note taking, so the questions for me were:

Can I make it do what I want it to?
How seamless the publishing could be?

Obsidian Digital Garden

Turns out, a Norwegian legend named Ole Eskild Steensen has already done all of the hard work to make managing and publishing a website from Obsidian super nice with his project:

Docs: https://dg-docs.ole.dev/
Repo: https://github.com/oleeskild/obsidian-digital-garden

How Does it Work?

The setup instructions can be found here: https://dg-docs.ole.dev/getting-started/01-getting-started/, but essentially you:

Install the Digital Garden obsidian plugin
Create a GitHub repo to store the site contents
Authenticate the plugin via a GitHub personal access token
Configure Vercel to build/deploy the site
Add dg-publish: true to the frontmatter of any Obsidian note you want to publish
Use the Obsidian command pallet to publish one (or all) specified notes

The magic happens with:

The Plugin:
1. Determining which notes + images need to be published
2. Generating the appropriate file structure expected by the Eleventy configuration
3. Automatically handling the git operations to commit and push those
Vercel:
1. Building each commit with Eleventy
2. Deploying each commit

My New Workflow

I have a dedicated Obsidian vault for this website. It is synced across computers and mobile via iCloud. The daily note is configured to use the following template:

---
dg-publish: false
dg-pinned: false
dg-hide: false
tags: 
dg-published-at: {{date}}
title: foo bar baz
---

and is created at: /Posts/YYYY/MM/DD.

So, to create a post:

Use the Obsidian "Daily Notes: Open today's daily note" command
Update the note name + title
Write the actual post 😅
Toggle dg-publish: true
Use the Obsidian "Digital Garden: Publish Multiple Notes" command

...about 30 seconds later the post is live!

For images, I paste them into obsidian directly. The initially are stored in /attachments with an autogenerated name like Pasted Image 20250709113743_986, but I usually move them to a directory alongside the post note and rename to something meaningful.

Porting Everything Over

In order to make the switch, I needed the corpus of articles already published migrated over.

Luckily, Hugo mostly uses vanilla markdown for authoring posts, so I was able to copy everything over and make a few minor tweaks to:

Images: I had custom Hugo components for handling images (and images that are also links).

{{< img-link "path-to-image.png" "https://some.link" "Some caption" >}}

I updated these to be:

[![[path-to-image.png]]](https://some.link)
*Some caption*

Links: In Hugo, default markdown links opened within the same tab so I had a custom link component to open links in new tabs.

{{< link "https://some.link" "Displayed text" >}}

I updated these to be:

%% External Links %%
[Displayed text](https://some.link)

%% Internal Links %%
[some-internal-page|Displayed text]

I could have scripted this, but the number of pages was such that I just converted these manually. Let me know if you notice an image or link that seems broken! 🙏

Javascript sorting woes

Everything was looking great, when I realized that the sorting of posts was wonky.

10, 11, 02, 03,... WTH?

When constructing my directories (and permalinks), I had used left padded zeros to ensure that the months would sort correctly numerically AND alphabetically.

So what is going on here? The digital garden site has a method which sorts these recursively which can be found here: filetreeUtils.js

This CORRECTLY sorts the months as you would expect. However, this sorted tree is stored as a normal Javascript object which gets rendered by a Nunjucks template here: filetree.njk

This traverses the keys of the Javascript object, resulting in:

"10", "11", and "12" being interpreted as integer indices
"01", "02", "03", ...etc being interpreted as non-integer indices

Given that, the ES6 Rules specify:

First, the keys that are integer indices, in ascending numeric order.
Then, all other string keys, in the order in which they were added to the object.
Lastly, all symbol keys, in the order in which they were added to the object.

(See https://2ality.com/2015/10/property-traversal-order-es6.html for more detail on what "Integer Indices" actually are)

We end up with this incorrect ordering! 🤬

My solution was to strip the leading zeros when constructing the key:

folders = note.filePathStem
	.split("notes/")[1]
	.split("/")
	// Here we strip leading zeros on the month folder to avoid JS treating 10, 11, 12 as integers,
	// but 01, 02, 03 as strings and sorting incorrectly when displaying with nunjucks. This allows us to
	// keep the leading zeros so that posts from our old hugo site all have the same URL structure
	.map((str) => str.replace(/^0+(?=\d)/, ''));
}

This gives me the sorting I want, while maintaining the URL structure from my old site (with leading zeros) to avoid breaking old links.

Isn't Vercel overengineered for a static site?

The site is built with Eleventy, which generates a static site that could be hosted in many ways (including the old GCS + Cloudflare setup I had before). I could totally update my old GitHub action to use Eleventy (instead of Hugo) to build the new site and deploy to GCS as I was before (and I may do that some day).

For now, I used Vercel because the Obsidian Digital Garden project has a "Deploy to Vercel" button which creates the repo and wires up the necessary configurations. This enabled me to get up and running in just a few minutes! I will monitor if anything ends up costing me (e.g. Vercel's Image optimization or total volume of traffic served), but for now I'm happy with the setup.

Speedrun a New Post

Okay, let's now test how long it takes me to create a new post.

34 seconds!

This is a test to see how fast I can create a post

Will this revamp actually inspire me to write more often? I'm hopeful, but only time will tell... 🤞

organizing-terraform-configurations

2025-08-04T15:13:57.881Z

Organizing Terraform Configurations (Single-Instance vs. Multi-Instance Root Modules)

There are a few primary approaches when it comes to organizing terraform configurations to be deployed across multiple environments (e.g. dev/staging/prod).

I have seen a bunch of strong opinions about this on this floating around the interwebs recently, and wanted to lay out my thinking on the matter.

Terminology

Before we can have an informed discussion about this topic, first we should agree on terminology and review existing resources:

Matt Gowie (https://masterpoint.io/) uses the terms:

multi-instance root module: A root module with multiple state files associated with it. This is done either with TF workspaces or with dynamic backend configurations, aka "Dynamic Backends".
single-instance root module: A root module directory with only one state file associated with it.

Sources: Blog, LinkedIn Post

Yevgeniy "Jim" Brikman (https://www.gruntwork.io/) describes the different approaches:

Workspaces: Terraform workspaces are one method of achieving multi-instance root modules.
Branches: Git branches can be used to store different static backend configurations for different environments within the same root module. Similar to managing long lived released branches, this will lead to pain and suffering. Please don't do this 🙏.
File Layout (optionally using Terragrunt or similar): Having separate root modules allows for configuring different backends by having separate files, organized via a logical directory structure.

Sources: Terraform Up and Running (Book), Workspaces Blog, Branches Blog, Terragrunt Blog

I like the single-instance vs multi-instance terminology and will use it for the remainder of the article.

[!IMPORTANT]
There are multiple ways to implement each of these approaches and those differences are important. For example, Terraform doesn't have dynamic backends but OpenTofu does. This is a critical difference as to whether a multi-instance root module can meet your organizations security requirements.

What are we trying to achieve?

Rather than jump in and claim one approach is better than the other, we should first lay out what goals we want to achieve with our infrastructure configurations:

Safety - Guarantee the Infrastructure as Code changes execute only in their intended environment (staging ⟹ staging, production ⟹ production).
Consistency - Prevent accidental configuration drift between environments.
Upgradability - Control the flow of provider and module upgrades across environments.
Clarity - Make it obvious to readers which environments exist and what resources they contain.
Security - Isolate production state from non-production state for compliance rules and reduce the blast radius of any change.

1. Safety

It is critical that we apply Infrastructure as Code configurations to the desired environment and ONLY the desired environment. An accidental application the wrong configuration to a production environment could be catastrophic.

The proper way to achieve this type of safety is to avoid applying configurations to production manually and limit production applies to a Terraform Automation and COllaboration Software (TACOS) tool such as Atlantis, Spacelift, or Env0. Once configured, these automations will ensure applies end up targeting the right environment.

What about in a break glass scenario?! 💥

In the heat of an incident, teams will often resort to overriding existing automations. I believe having single-instance root modules (as opposed to terraform workspaces) makes it slightly more clear which environment you are targeting.

[!NOTE] Bottom Line
The difference between approaches here is small and in my opinion does not tip the scale one way or another.

2. Consistency

Another key consideration for IaC configurations is keeping environments consistent with one another.

A multi-instance root module by definition achieves this (it is a single module).

multi-instance supporters often claim that single-instance root modules will cause drift and duplication due to the additional files. In my opinion this is mostly a straw man argument.

Why? Because regardless of which approach you chose, the logic for your configurations should live in SEPARATE CHILD MODULES! In the single-instance approach, the root modules are primarily:

A backend configuration
Consuming a bunch of child modules (populating their input variables)

If the logic lives in separate modules, the difference between whether you store the input variables in environment specific .tfvars files and pass them into a multi-instance root module or store them in an environment specific root module directly is minimal from a consistency standpoint.

Let's look at set of files required by each approach for a hypothetical config:

# The multi-instance root modules maps a bunch of tfvars values into child module inputs
└── multi-instance
    ├── live
    │   ├── compute
    │   │   ├── main.tf
    │   │   ├── production.tfvars
    │   │   └── staging.tfvars
    │   ├── global
    │   │   └── main.tf
    │   └── networking
    │       ├── main.tf
    │       ├── production.tfvars
    │       └── staging.tfvars
    └── modules # ALL OF THE LOGIC LIVES HERE!
        ├── compute
        │   ├── main.tf
        │   └── variables.tf
        └── networking
            ├── main.tf
            └── variables.tf

# The single-instance root modules set the child module inputs directly
└── single-instance
    ├── live
    │   ├── global
    │   │   └── main.tf
    │   ├── production
    │   │   ├── compute
    │   │   │   └── main.tf
    │   │   └── networking
    │   │       └── main.tf
    │   └── staging
    │       ├── compute
    │       │   └── main.tf
    │       └── networking
    │           └── main.tf
    └── modules # ALL OF THE LOGIC LIVES HERE!
        ├── compute
        │   ├── main.tf
        │   └── variables.tf
        └── networking
            ├── main.tf
            └── variables.tf

Note: Tools like Terragrunt and Terramate help avoid the duplication of the backend blocks and reduce opportunities for configuration drift.

[!NOTE] Bottom Line
A multi-instance root module has an edge in this category, but it is much smaller than its proponents claim.

3. Upgradability

Within a root module, you can only specify a single version for each usage of a particular provider and module.

All workspaces share the same required_providers and child module source/version constraints.

multi-instance root modules require upgrading these versions all at once. You can still apply them separately (i.e. apply to non-prod first), but until you have applied the upgrade across all environments, the configuration in version control will be out of date with reality.
single-instance root modules can specify different version of providers and child modules. This allows for a more controlled upgrade path, where new versions can be validated in lower environments before a change is made to production.

[!NOTE] Bottom Line
single-instance root modules allow for testing upgrades of providers and child modules in a more controlled fashion than a multi-instance root module.

4. Clarity

Another consideration is the speed with which a developer can read and understand an IaC configuration. This includes things like:

What environments exist?
What resources are deployed in each environment?
How to validate my changes for a particular environment?

For a multi-instance root module these questions are answered by examining the root module and mentally mapping values from corresponding .tfvars files into it. For single-instance root modules, you navigate to the corresponding directory and everything is right there.

[!NOTE] Bottom Line
In my opinion, single-instance root modules are easier to reason about and quickly understand exactly what is deployed to a given environment. Note: This is subjective and may be due me having more experience working with them.

5. Security

For both practical and compliance reasons, it is useful to isolate the state of production environments from the state of non-production environments.

Terraform workspaces share a single backend block, so state isolation for multi-instance root modules wasn’t natively supported until the release of OpenTofu 1.8.0 (released 2024-07-29), which introduced the ability to use variables and locals within the backend block.

Prior to that release, the only ways to achieve true state isolation were to either:

Use single instance root modules.
Use a Terraform wrapper such as Terragrunt/Terramate to dynamically generate the backend at runtime.

[!NOTE] Bottom Line
For Terraform, single-instance root modules are the only native way to achieve true state isolation. For OpenTofu, a multi-instance root module with a dynamic backend configuration can achieve this as well.

Closing thoughts

Both directory‑based single‑instance roots and workspace/dynamic‑backend multi‑instance roots are viable patterns in 2025. The right choice hinges on which trade‑offs map to your team’s priorities:

Security & isolation: OpenTofu’s dynamic backends eliminate the last hard blocker for multi‑instance layouts, allowing a unique bucket and IAM role per workspace. If you are on Terraform, single‑instance still offers the most straightforward state segregation.
Upgrade cadence & visibility: Separate directories make it trivial to stage provider or module upgrades in lower environments before promoting to production, and they double as living documentation of what runs where.
Operational consistency: A multi‑instance root module guarantees every environment uses the exact same codebase, reducing drift and simplifying CI pipelines, but moving logic into child modules mostly accomplishes the same thing for single-instance configurations.

[!NOTE] Bottom Line
For an infrastructure configuration that will be repeated many times, e.g. deploying a bunch of single tenant infrastructure stacks for different customers, I would lean towards the multi-instance root module approach.

For an infrastructure configuration that has a fixed set of environments, e.g. a company with a staging and production environment, I would lean towards single-instance root modules organized by the file layout.

github-actions-tool-cache

2025-08-08T01:57:21.432Z

Installing Tools within GitHub Actions Workflows (Deep Dive: GitHub Action Tool Cache)

Two of the most common use cases for GitHub Actions are:

Running an application's test suite
Building an application for release

Both usually require a language runtime and/or build toolchain.

This article explores how to do so optimally, which requires understanding and using the GitHub Actions Tool Cache.

The code samples from this article can be found at: https://github.com/sidpalas/tool-cache-deep-dive

[!TIP] Upcoming Course
This article was written as part of my preparations for the upcoming "GitHub Actions: Beginner to Pro" course I am working on.

See: https://github.com/sidpalas/devops-directive-github-actions-course for more info!

Aside: Ephemeral vs Stateful Runners

In most cases (including default GitHub-hosted runners), the environment where jobs are executed is ephemeral and lives only as long as the job it executes. This has many useful properties related to security, consistency/reproducibility, and scalability, but introduces inefficiencies related to installing dependencies.

Specifically, any changes to the system (such as installing a tool) are not persisted across jobs. Unless you are intentional about how you install and cache tools, it is easy to end up downloading and installing things with every workflow run, wasting both time and compute resources.

It is worth mentioning that self-hosted runners can be stateful and reused across jobs. In that case, a script that performs if not installed, then install is sufficient. The rest of the article assumes you are using an ephemeral runner.

What is the Tool Cache?

The tool cache is a well-known directory inside every runner where “setup-x” actions (e.g. https://github.com/actions/setup-node) check for and install versioned tool payloads. There are two official locations:

OS	Default Path
Linux / macOS	`$RUNNER_TOOL_CACHE` → `/opt/hostedtoolcache/`
Windows	`%RUNNER_TOOL_CACHE%` → `C:\hostedtoolcache\windows\`

Within that directory, tools are stored using a directory structure like (using Node.js as an example):

$ tree /opt/hostedtoolcache/node -L 2
node/
`-- 20.19.3
    |-- x64 -> /bin/node
    `-- x64.complete # indicates that install is complete and ready for use

The path to the tool encodes the version and architecture (x64, arm64), and a .complete file indicates the tool is installed and ready for use.

[!WARNING]
Despite being named "cache", the tool cache is not automatically shared across jobs (see section about ephemeral runners above).

@actions/tool-cache npm package

The @actions/tool-cache npm package provides utilities for checking if a tool exists within the cache as well as downloading and extracting in platform specific ways.

For example, to check if a specific version of Node.js exists within the cache you can use the find function:

const tc = require('@actions/tool-cache');

const nodeDirectory = tc.find('node', '12.x', 'x64');

This package is used by the setup-x actions to ensure consistency when setting up necessary tools.

How do `actions/setup-x` actions work?

The official setup-x actions generally follow the same pattern:

Look in $RUNNER_TOOL_CACHE/<tool>/<version>/<arch> for a folder marked .complete.
If present, configure the environment to use it
If absent, it downloads the tool, unpacks it into that same path, drops the .complete sentinel, and then adds the folder to PATH — seeding the cache for any later steps or jobs that run on the same runner.

What are my options?

Option 0: Use pre-installed versions + `setup-x` actions

Many popular tools come preinstalled on the runner image (see: https://github.com/actions/runner-images). For example the ubuntu-24.04 image contains:

Language runtimes:
- Node.js 20.19.3
- Python 3.12.3
Popular CLIs:
- AWS CLI 2.27.50
- GitHub CLI 2.75.0
Lots more!

If you are willing and able to use the preinstalled version, this is ideal! You can generally use one of the official actions/setup-x actions (e.g. https://github.com/actions/setup-node) and it will:

Identify that the desired version exists
Skip the installation step

Here is an example job using the preinstalled Node.js version via actions/setup-node:

jobs:
  preinstalled:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20 # This will use any major version match (e.g. 20.x.x)

Using the pre-installed version, setup-node executes in just 1 second!

Option 1: Install with each run

If you need a different version, you can tell setup-x to use the desired version, but this will cause the tool to be re-downloaded and re-installed with each run.

Here is an example job pinned to a different version of Node.js not available in the runner image by default:

jobs:
  older-version:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20.19.2

In the event of a cache miss, the action downloads from the upstream release

Option 2: Cache manually (via `actions/cache`)

One way to reuse the tool across ephemeral runners is to cache (and restore from cache) using actions/cache.

This avoids needing to download from the upstream release, but downloading from the GitHub actions cache isn't much faster than downloading the release directly.

Here is an example job caching and restoring a non-default version:

  older-version-gha-cache:
    runs-on: ubuntu-24.04
    env:
      NODE_DIR: /opt/hostedtoolcache/node/20.19.2/
    steps:
      - name: Restore Node from cache
        id: cache-task
        uses: actions/cache@v4
        with:
          path: ${{ env.NODE_DIR }}
          key: node-20.19.2-${{ runner.os }}-${{ runner.arch }}
      - uses: actions/setup-node@v4
        with:
          node-version: 20.19.2

Restoring from the GitHub cache takes approximately as long as downloading the release directly

Option 3: Use another caching mechanism (e.g. Namespace Cache Volumes + toolchain caching)

If GitHub Action caching isn't much faster than the naive "reinstall every run" approach, is there a faster caching option we can use?

Namespace Labs provides one such option with their cache volumes. These are built on NVMe attached to their hosted runners which you can configure for their hosted runners.

One option when configuring these cache volumes is whether or not to cache toolchain downloads. When enabled, Namespace will automatically cache tools stored according to the tool cache conventions for use in future runs.

On the GitHub Actions workflow side, the only necessary change is to use the corresponding runner name in the runs-on field for the job:

jobs:
  older-version-namespace-cache-volume:
    runs-on: namespace-profile-toolchain-volume-cache
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20.19.2

[!NOTE] This post is not sponsored
Namespace Labs is sponsoring my upcoming GitHub Actions course, but this post is not sponsored.

Option 4: Use a custom Container Image or AMI

If you want to avoid dealing with setup-x altogether, you can run your jobs inside of a container or virtual machine preconfigured with the desired tool version(s).

Container

GitHub Actions provides the option to run your jobs inside of a container with an image of your choosing.

One downside to this approach is that running the container introduces its own initialization overhead.

Here is an example job using this option:

jobs:
  older-version-container:
    runs-on: ubuntu-24.04
    container:
      image: node:20.19.2-alpine
    steps:
      - run: |
          which node
          node -v

[!NOTE] Self-hosted runners with ARC
If you are self-hosting runners via the actions-runner-controller you could similarly use a container image preconfigured with the desired tools.

Virtual Machine

If you are self-hosting runners as virtual machines, such as with runs-on, you can build the virtual machine image (e.g. AMI if on AWS) with the desired tools using a tool like Packer

Option 5: Do everything within the Dockerfile

Similar to using a custom image for the runner, we can bypass the need to use the tool-cache entirely by performing all of the necessary build/test actions within the container build process.

This can be accomplished via a multi-stage build, with specific build targets which do things like execute the tests:

# Dockerfile
FROM node:20.19.2-bookworm-slim AS base
WORKDIR /usr/src/app
COPY package*.json ./

###################################
FROM base AS which-and-version
RUN which node && node -v

###################################
FROM base AS test
RUN npm ci 
COPY ./src/ .
ARG FAIL_OR_SUCCEED
RUN npm run test-${FAIL_OR_SUCCEED}

###################################
FROM base AS deploy
ENV NODE_ENV=production
RUN npm ci --production
USER node
COPY --chown=node:node ./src/ .
EXPOSE 3000
CMD [ "node", "index.js" ]

{
  "name": "tool-cache-deep-dive",
  "version": "1.0.0",
  "description": "",
  "homepage": "https://github.com/sidpalas/tool-cache-deep-dive#readme",
  "bugs": {
    "url": "https://github.com/sidpalas/tool-cache-deep-dive/issues"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/sidpalas/tool-cache-deep-dive.git"
  },
  "license": "ISC",
  "author": "",
  "type": "commonjs",
  "main": "src/index.js",
  "scripts": {
    "test-succeed": "echo \"This mimics a passing test suite... ✅\" && exit 0;",
    "test-fail": "echo \"This mimics a failing test suite... ❌\" && exit 1;"
  },
  "devDependencies": {
    "noop": "^1.0.1"
  }
}

The full example can be found at https://github.com/sidpalas/tool-cache-deep-dive.

By targeting the test stage we can then execute our tests within the container. For example:

➜  tool-cache-deep-dive git:(sp/initial-workflow-development) docker build --build-arg FAIL_OR_SUCCEED=fail . --target=test                                   ...
 => ERROR [test 3/3] RUN npm run test-fail
------                                                                           
 > [test 3/3] RUN npm run test-fail:                                             
0.168 
0.168 > tool-cache-deep-dive@1.0.0 test-fail
0.168 > echo "This mimics a failing test suite... ❌" && exit 1;
0.168 
0.170 This mimics a failing test suite... ❌
------
Dockerfile:14
--------------------
  12 |     COPY ./src/ .
  13 |     ARG FAIL_OR_SUCCEED
  14 | >>> RUN npm run test-${FAIL_OR_SUCCEED}
  15 |     
  16 |     ###################################
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c npm run test-${FAIL_OR_SUCCEED}" did not complete successfully: exit code: 1

Closing Thoughts

I wrote this article mostly because I found the existing documentation around the GitHub Actions tool cache to be lacking. Hopefully this gives you ideas for how to install tools required by your GitHub Actions workflows without incurring a significant time penalty.

Whether sophisticated caching is “worth it” comes down to the trade-off between complexity and time saved. Heavy toolchains (e.g. large Java distributions) can add significant time to every job. Every fresh download also introduces another external dependency: if the upstream mirror hiccups, your entire pipeline stalls.

I would generally take the following approach:

If the runner contains an acceptable version preinstalled, use it!
If you must pin a different version and you control self-hosted runners, a persistent cache volume (e.g. Namespace Labs cache volumes, ARC PVC, or similar) gives you the same performance with almost no extra YAML.
When you’re already packaging and deploying containers, baking the runtime into the image keeps all dependencies self-contained and guarantees reproducibility across every environment.

Start with the lightest solution, determine how long it takes, and add caching techniques when the saved time outweigh the added complexity.

hugo-and-caddy-on-gcp

2025-07-10T14:02:33.204Z

The Making of This Site (Hugo, Caddy, + GCP)

Alternate title: Just use Netlify.

TL;DR: Deploy a Hugo site to GCP for free in less time than it will take you to read this article (Github Repo with all referenced commands)

EDITS:

In October 2020 I updated the hosting of this site to use GCS + Cloudflare
In July 2025 I updated the site to use Obsidian + Eleventy + Vercel

Table of Contents:

Choosing a Site Generator

With GeoCities no longer a viable option for hosting websites in 2020 (apparently Yahoo Japan shut down the final remnants in March 2019) I needed a more modern solution to host this site.

After a brief look at the leading static site generators Jekyll, Hugo, Next.js, Gatsby, etc... I came to the conclusion that almost any of these would work just fine for my needs. I ended up choosing Hugo for two main reasons:

1) Language: It's written in Go, a language I have wanted to learn. Having this site with Hugo may provide a nudge of motivation to make some contribution to the Hugo project.

2) Speed: Site build times are near instant for this (tiny) site, but that would be true for any generator with so few pages. However, people have performed a number of benchmarks showing Hugo's performance as a site grows.

Initial Setup

The Hugo documentation is concise and they have an easy-to-follow quick start guide found here: https://gohugo.io/getting-started/quick-start/

I'm currently (as of February 2020) still using the suggested Ananke theme with a few minor styling tweaks, but eventually will probably spend some more time customizing the theme.

Choosing a Hosting Solution

With the site generator working, I then needed to decide how to host the site. In the past, I have used Github Pages to host static sites, but I noticed that they explicitly prohibit "Get-rich-quick schemes" (which learning & writing about DevOps and Cloud infrastructure is) so that was out of the running.

I also looked at hosting within an AWS / GCP](https://cloud.google.com/storage) / Azure bucket. These are all super easy to set up and scale effortlessly, but if you want to use a domain with HTTPS enabled, you end up having to jump through some hoops configuring a Content Delivery Network.

I then came across Caddy, a web server with automatic HTTPS configuring using Let's Encrypt which seemed ideal for this use case!

With a lock for a logo, it must be secure!

NOTE: If you are following along and setting up your own site, for most people, the best option at this point would be to stop here and go to https://www.netlify.com/. They have a generous free tier plan and offer direct integration with github/gitlab/bitbucket to handle automatic build/deploys triggered by Git commits. I achieved a similar end result with GCP Compute Engine + Cloud Build (a topic for another post) that provides me a bit more control/extensibility, but Netlify covers most use cases with a fantastic user experience!

Local Configuration (HTTP-only for Now)

Since my computer is running macOS, but ultimately the site would be deployed on a server running some variant of Linux, my default is to use containers to eliminate any configuration headaches with slight differences between the two environments.

This appears to be the defacto standard Caddy docker image https://hub.docker.com/r/abiosoft/caddy with over 10M pulls so I used that as the base.

There are two options for accessing the files that Caddy needs to host within the container. (1) Copy them into the container image OR (2) mount a location the host filesystem into the container with -v or --mount docker flags when running the container.

Either option would be fine, but option 1 is nice for a small site because it ensures the entire site and its dependencies are included within the container image. As the site grows, I may switch to storing the site files outside of the container image.

Bundling the site files into the container can be accomplished with a 2 line Dockerfile.

FROM abiosoft/caddy:1.0.3
COPY ./public /srv

Here ./public is the local directory where Hugo builds the site, and /srv is the directory within the container Caddy expects to find the files it is serving. The following 4 commands will build the site, build the container, and run the container:

export IMAGE_NAME=my-hugo-caddy-docker-image
hugo -D    # -D flag tells Hugo to build drafts 
docker build ./ --tag $IMAGE_NAME
docker run -p 2015:2015 $IMAGE_NAME

The -p forwards the port from the host system into the container allowing us to connect to http://localhost:2015/ and that request will be forwarded into the container on port 2015 where Caddy is listening.

Deploying to GCP

With the container image working, I was then ready to deploy it somewhere. There are a variety of options to do this, but I chose to use a GCP Compute Engine f1-micro virtual machine instance running Google's Container-Optimized OS. Container-Optimized OS provides nice security features configured by default making it a good OS option for containerized applications. While the f1-micro instance is small (0.2 vCPUs + 600MB Memory), running one is included in the GCP's always free usage limits making this deployment cost me a grand total of $0!

Since this site is about DevOps, I clearly needed to automate the entire setup process, which I did here: https://github.com/sidpalas/hugo-gcp-deploy.

I also decided to create a new GCP project for this site. Doing this makes it easy to clean things up should I decide to take the site down by simply deleting the entire project without having to worry about accidentally leaving some resources running.

If you are more comfortable working with the GCP web interface, that is perfectly fine, but the following process should do the trick

NOTE: for any of the following commands $PROJECT_ID would need to be replaced with your GCP project id. I also like to explicitly pass the project ID into the commands to ensure they are executed in the correct location (Just in case I happened to have changed my default project configuration).

export PROJECT_ID=my-awesome-project-1234

1) Enable Billing for the Project

Even though the resources used here are included in the free tier, Google requires having a payment method on file. This is the one step I recommend doing via the console as the command line command is still in alpha:

https://console.cloud.google.com/billing/linkedaccount?project=$PROJECT_ID

2) Enable the Compute Engine and Container Registry APIs

This is necessary to provision the VM and to be able to push container images.

gcloud services enable compute.googleapis.com --project=$PROJECT_ID
gcloud services enable containerregistry.googleapis.com --project=$PROJECT_ID

3) Reserve a static IP address

If the VM needs to restart (or if I wanted to move to a larger machine type), a static IP ensures it won't change unexpectedly and mess up DNS configuration.

gcloud compute addresses create my-site-external-ip \
        --project=$PROJECT_ID \
        --region=us-central1

4) Add firewall rules

By default, Compute Engine VMs do not allow HTTP or HTTPS traffic. Adding firewall rules allow those requests to make it to the web server.

gcloud compute firewall-rules create default-allow-http \
        --project=$PROJECT_ID \
        --target-tags=http-server \
        --allow tcp:80
gcloud compute firewall-rules create default-allow-https \
        --project=$PROJECT_ID \
        --target-tags=https-server \
        --allow tcp:443

The target-tags allow the VM configuration to utilize these rules.

5) Create the Virtual Machine (Finally!)

When creating the VM, the static IP and firewall rule tags are used to configure it:

gcloud compute instances create my-f1-micro-instance \
        --project=$PROJECT_ID \
        --zone=us-central1-a \
        --machine-type=f1-micro \
        --image=projects/cos-cloud/global/images/cos-69-10895-385-0 \
        --address=my-site-external-ip \
        --tags=http-server,https-server

This takes a few minutes to provision.

6) Configure Docker(s)

In order to configure my local Docker install to push images to Google Container Registry I had to run:

gcloud auth configure-docker

To set up Docker and the GCR credentials within in container optimized OS running on the VM I had to run the following:

gcloud compute ssh my-f1-micro-instance \
        --project=$PROJECT_ID \
        --zone=us-central1-a -- \
        docker-credential-gcr configure-docker

7) Deploy

Before deploying the site, I had to add a Caddyfile to the container image to configure the server. The default configuration will serve the site on port 2015 which is inaccessible on our VM (because my firewall rules only allow traffic on ports 80 and 443). The following will tell Caddy to serve on port 80 and accept requests from any domain (documentation):

# This test configuration will be replaced later when setting up HTTPS
:80 {
browse
log stdout
errors stdout
}

I also updated the Dockerfile to copy this Caddyfile into the container image:

FROM abiosoft/caddy:1.0.3
COPY ./public /srv

# This environment variable gets used later during the container start up
# to accept the Let's Encrypt subscriber agreement (without requiring user input)
ENV ACME_AGREE=true 
COPY ./Caddyfile /etc/Caddyfile

With the configuration ready, I needed to get the container image into the google container registry by building, tagging, and then pushing it:

export IMAGE_NAME=my-hugo-caddy-docker-image
export IMAGE_TAG=incrementing-tag-001 # change this with each deploy to ensure latest image is used
docker build ./ --tag $IMAGE_NAME
docker tag $IMAGE_NAME gcr.io/$PROJECT_ID/$IMAGE_NAME:$IMAGE_TAG
docker push gcr.io/$PROJECT_ID/$IMAGE_NAME:$IMAGE_TAG

If the site is already running, it needs to be cleaned up before the new container can be deployed or else it would fail when trying to bind to the same host ports. This can be accomplished using:

gcloud compute ssh my-f1-micro-instance \
        --project=$PROJECT_ID \
        --zone=us-central1-a -- \
        'docker container stop $(docker container ls -aq) && docker container rm $(docker container ls -aq)'

Having to do this step is one downside to bundling the entire site into the container image, the site has to be down for a second or two while the new version starts up. If the site was mounted into the container, the server could keep running and the files could be copied onto the host filesystem with no downtime.

Finally, we can issue a docker run to run the new container image:

gcloud compute ssh my-f1-micro-instance \
        --project=$PROJECT_ID \
        --zone=us-central1-a -- \
        'docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v $HOME/.caddy:/root/.caddy gcr.io/$PROJECT_ID/$IMAGE_NAME:$IMAGE_TAG'

NOTE: the -v $HOME/.caddy:/root/.caddy mount isn't necessary here, but later once we request the TLS certificate, this will allow the certificate files to persist across container redeploys, avoiding extra requests to Let's Encrypt which could lead to being rate limited.

8) We're Live!

At this point, I was able to visit the external IP address from step 3 and see the website.

I even scripted the whole process so that it takes less than 5 minutes:

### 9) Configuring DNS

The final element of the setup is to point a domain to the IP address which I accomplished with the following settings:

Name      Type   TTL     Data
@         A      1h      123.456.78.90
www       CNAME  1h      my-awesome-domain.com.

(The CNAME record maps the www subdomain to the primary domain without www)

10) Enabling HTTPS

The final element of the setup is to enable HTTPS within Caddy. This can be accomplished by modifying the Caddyfile to include the domains and an email address for the TLS configuration:

my-awesome-domain.com www.my-awesome-domain.com {
        tls my-email-address@domain.com
        browse
        log stdout
        errors stdout
}

After redeploying and waiting for the DNS settings to propagate I was able to access my site and bask in the glory of the HTTPS connection symbol!

Closing Thoughts

Overall I'm happy with this configuration and am amazed that all of this can be accomplished for free using mostly open-source software! It was also useful to continue gaining experience with the tools and platforms.

My hope is now that everything is set up and configured, the amount of maintenance effort required should be low. Time will tell...

In a future post, I will cover how I used Google Cloud Build to automate the deployment process. I also plan to do some benchmarking to see just what kind of load this tiny server can handle!

Home

2026-03-09T15:41:27.595Z

DevOps Directive

⚙️ A place for DevOps and cloud infrastructure content!

📚 A handful of articles to start with:

🔎 The full archive of articles can be found using the left nav or search functionality!

👀 Want more? Check out:

velero-aws-account-migration

2025-07-09T02:30:21.153Z

Velero AWS Account Migration [Case Study]

TL;DR: I recently helped an organization migrate a set of applications deployed in Kubernetes (EKS) across AWS accounts using Velero. This case study describes that process.

Please start with multiple AWS accounts so you don't have to hire someone like me to help with this! 🙏

Summary

The migration of Kubernetes clusters across different AWS accounts can be a challenging process, given the intricacies and dependencies that exist within each environment.

I recently used Velero to help an engineering organization migrate two EKS clusters (representing their staging and production environments) from a single AWS account to two separate accounts. I’ll delve into the migration process, share our experiences, and outline some lessons learned along the way.

Background

Each environment consisted of a variety of AWS resources (e.g. S3 buckets, RDS instances), some externally hosted databases (MongoDB Atlas, Neo4j Cloud), and a variety of microservices running in an EKS cluster.

These microservices can be categorized into the following groups:

Job Producers: Detecting or receiving events from data vendors about new data to be processed or initiating periodic analytics jobs. These create messages containing the necessary metadata and send them to the Message Bus
Message Bus: RabbitMQ cluster serving as a message bus for the system
Job Consumers: Worker deployments that consume messages from the Message bus and perform the corresponding processing task (storing the associated data in S3 and/or databases)
Other: Public facing web portal (nginx), various APIs, etc… These services were generally stateless and the primary consideration with respect to the migration is ensuring their configuration (loaded from ConfigMaps/Secrets were updated appropriately when started in the new clusters)

The applications were deployed into the cluster automatically using a pushed-based approach from GitLab CI/CD pipelines.

Primary Goals

Migrate clusters to new accounts
Minimize customer-facing downtime
Avoid data loss or duplication

Migration Approach

Migrating Non-Kubernetes Resources

This case study is focused on using Velero for the Kubernetes workloads, but a brief description of the migration of other resources is useful context. Key points:

Cloud Development Kit (CDK) configurations were built to provision copies of all the AWS resources in the new accounts.
S3 replication was used to copy data (historical and ongoing) to new buckets.
AWS DMS was used to copy data (historical and ongoing) to new RDS instances.
External databases (MongoDB Atlas + Neo4j Cloud) were left in place and VPC peering was used to enable connectivity from VPCs in the new accounts. Note: This was simpler than migrating to new instances but did introduce the potential for legacy and new clusters to be attempting to write to them duplicate data to them.

Migrating Kubernetes Resources

We considered three methods for migrating the Kubernetes-based workloads:

Redeploy everything from GitLab

Deployments were automated via GitLab CI/CD, so one option would be to update the GitLab environment to point to the new cluster and trigger the latest deployment pipeline for each service. This is a simple approach and doesn’t require any new tooling, but there are many repositories, and not 100% of the resources in the cluster was deployed in this way.
Backup/restore with Velero

Another option would be to take a Velero backup in the legacy cluster and then restore that backup in the new cluster. Cluster migration is one of the stated use cases for Velero, but it does represent another tool to learn/install/configure if you are not already using it for backups.
Use a GitOps Approach

Using a tool like ArgoCD or FluxCD installed into the would provide a clean mechanism to pull the latest configuration for all of the applications from GitLab into the new cluster. This would require rolling out GitOps before performing the migration and would not work for any workloads not fully described in GitLab.

We decided to use Velero as it provided the simplest approach that would work even for resources that weren’t fully represented in GitLab or if we temporarily needed certain resources to differ from Git. For example ConfigMaps with references to S3 buckets or secrets containing RDS credentials. Also, we had recently set up Velero in the existing clusters as a backup solution, so the team was already familiar with its usage.

S3 configuration for replicating backups across accounts

Note: We did consider migrating applications one at a time, but this would require adding significant complexity in temporary configurations to support cross-AWS account IAM and networking and we decided to move everything at once to avoid this and prevent a potentially long period where the system lived in a partially migrated state.

Migration Sequence

The following is the sequence we used for the migration:

Install Velero into both clusters: Each bucket used a separate S3 bucket located in the AWS account where the cluster lived.
Set up S3 replication rule to copy backups between buckets: This is one place our process differs from the described Velero migration documentation because the clusters live in different AWS accounts. Rather than set up the necessary IAM for the new cluster to access the legacy bucket, we opted to replicate the backup data from the legacy bucket into the new one.
Scale down “Job Producers” in legacy cluster (This is highly specific to our application and migration plan): In order to avoid the potential for data loss and duplication we decided that the best approach would be to scale the job producers to zero and allow time for the job consumers to finish processing active jobs. Luckily, these data processing jobs are not directly visible to end users and therefore can be disabled temporarily without incurring customer-facing downtime.
Trigger backup in legacy cluster: Nothing special here, we created a backup from a schedule.
Deploy modified ConfigMaps and Secrets to new cluster: Many of the application consume their configuration from ConfigMaps and Secrets. When running in the new cluster they needed to use the new S3 buckets, RDS instances, etc. Velero by default does not overwrite existing Kubernetes objects, so pre-populating these resources would ensure they have the appropriate configuration when the corresponding applications are restored. We had open Merge Requests (or modifications to GitLab Secrets) containing all of these differences ready to Merge after the migration was complete to avoid reverting with the next deployment.
Restore from backup in new cluster: Again nothing special, we restored from the backup.
Scale up “Job Producers” in new cluster: Scaling the job producers back to their normal replica counts and cron schedules enables them to resume where they left off in the legacy cluster
Validate workload performance: We defined a number of checks to determine that critical workloads were healthy in the new cluster
Cut over DNS for external traffic: Once we were satisfied that things were behaving as expected, we cut over external DNS for the public apps/APIs
Update GitLab to deploy to new cluster: We updated the GitLab environment such that future deploys would target the new cluster instead of the legacy one.

Lessons Learned during the Migrations

All things considered, the migration went well! That being said, there were some hiccups related to Velero configuration details that are worth highlighting.

Velero doesn't back up cluster-scoped resources for namespace backups

Our Velero backups were scoped to specific Kubernetes namespaces (for reasons that are not relevant here). However, namespace-scoped backups do not back up cluster resources by default. This seems obvious in hindsight… but we had not accounted for it.

This caused certain CRDs our applications relied on to not be migrated. We identified this issue during the staging migration and updated one of the backup configurations to include the --include-cluster-resources=true option.

Velero backs up Pods by default

The default behavior for Velero is to back up all resource types, including Pods. For most applications this behavior is fine, but for certain stateful applications, the ordering of pod creation matters. In our case, our RabbitMQ cluster came up split-brained.

Because of how we opted to scale down/up the job producers we decided it was not necessary to snapshot and migrate the persistent volumes associated with RabbitMQ. Velero backed up three pods and those pods were restored simultaneously in the new cluster. Each of them started, failed to find a leader amongst its peers, and declared itself the leader. Instead of having one cluster with three instances, we had three clusters with one instance each, behind Kubernetes Service which dutifully load-balanced requests across them.

Unfortunately, we did not catch this during the staging migration because staging had been running with a single RabbitMQ replica (keep your environments identical people!) but by manually modifying the Service selector and labels on the pods we were able to systematically shift all traffic to one instance at a time and allow active jobs to be processed, eventually removing the separate instances, and finally scaling back to the desired single cluster configuration.

This issue could have been avoided by either (a) configuring Velero not to back up Pods directly so that the Statefulset would bring up the replicas one at a time OR (b) snapshot and migrate the persistent volumes so that the previous clustering information would also be migrated (this would present additional challenges of copying EBS volumes across AWS accounts and modifying the backup files to reference the new volume names/IDs).

Conclusion

Velero helped us streamline the migration of workloads between EKS clusters in separate AWS accounts. It is important to understand your specific backup configuration and how it might impact our system, also having a representative environment to test the environment can help identify issues before they reach your production environment.

In the end, the key to moving Kubernetes clusters successfully comes down to planning, knowing your workloads, and understanding your tools. This case study provides a glimpse into our journey and might help others who are about to do the same. We hope you can learn from our experience and avoid some of the challenges we faced.

container-init-process

2025-07-09T12:02:58.153Z

Container Init Process

TL;DR: When deploying applications in containers, understanding the role and implications of the init process is critical. In this article, we'll delve into how our application runs as Process ID #1 in a container, and what this means for our application's signal handling and zombie process management.

To init or not to init, that is the question...

Understanding the Init Process

When we boot up a Linux computer or virtual machine, the first process to start is known as the "init process". This process is assigned Process ID #1, given that PIDs are assigned sequentially. The init process carries several responsibilities including initializing the kernel, mounting filesystems, managing services and daemons, and supervising other processes.

This article is not meant to be an exhaustive discussion on init systems. We are only examining aspects of init systems that are relevant to running containers. If we want to learn more about Linux init systems, I recommend watching "Whats the Point of A Linux Init System" by Brodie Robertson.

The Implications of PID 1 in Containers

Now, what about Linux containers? Containers typically have their own isolated PID namespace. This implies that the process executed upon startup (the ENTRYPOINT or CMD within our Dockerfile) gets run as PID 1 inside the container.

In a standard Linux system, PID 1, also known as "init," is treated differently from other processes. As mentioned above it has a number of specific responsibilities, some of which may impact our container's operation. Two aspects of init processes are particularly relevant for containers:

Signal Handling: What happens when we send a SIGINT or SIGTERM signal to the container?
Zombie Process Handling: What happens when we spawn child processes, and the parent exits?

Let's examine these aspects in detail.

Signal Handling in Containers

Signal handling determines how our application responds to system signals, such as SIGINT (interrupt from keyboard) or SIGTERM (termination signal).

For instance, consider a Node.js program that loops indefinitely and doesn't handle system signals:

// javascript program that doesn't handle signals
while (true) {
  console.log("Looping forever...");
}

If this is running inside a container, when we press CTRL + C, which sends a SIGINT, or when we execute docker stop, which sends a SIGTERM, the application continues running. This behavior is due to the lack of appropriate signal handling in the application code.

So what can we do about this?

Option 1: Handle Signals in the Application Code

If we control the application source code, it is best to add handlers in our application code that respond appropriately to system signals such as SIGINT and SIGTERM. Moreover, it's best practice to include code that gracefully terminates any ongoing operations, such as closing open client connections before calling process.exit().

// Node.js program that handles SIGINT and SIGTERM signals
process.on('SIGINT', function() {
  console.log('Received SIGINT. Gracefully shutting down...');
  // Insert necessary commands to shut down gracefully here
process.exit();
});

process.on('SIGTERM', function() {
  console.log('Received SIGTERM. Gracefully shutting down...');
  // Insert necessary commands to shut down gracefully here
  process.exit();
});

while (true) {
  console.log("Looping forever...");
}

Option 2: Run a Separate Init Process

If we do not control the application source code, we can run a separate program as the init process which will in turn spawn the primary application. Tini is a "tiny buy valid init for containers" designed specifically for this purpose (https://github.com/krallin/tini).

Docker has the abiliy to do this at runtime by using the init flag (docker run --init <CONTAINER_IMAGE>). This approach works, but is specific to Docker. If we want a solution that will always run Tini without relying on a specific container runtime implementation, we can build it into the container image!

FROM node:18.16-bullseye

ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

COPY . .
CMD ["node", "index.js"]

It is important to note that this will NOT magically make our app terminate gracefully, but at least SIGINT and SIGTERM signals will stop the containerized application.

Zombie Process Handling in Containers

A "Zombie" process is a process that has finished its execution but still exists in the process table. Although these processes no longer consume CPU or memory, they should ideally be cleaned up. The init process typically handles this clean-up.

For example, consider a Node.js program that uses child_process.spawn in detached mode to run a shell script that initiates a sleep operation in the background before exiting. This sequence can create zombie processes.

// index.js
const { exec } = require('child_process');

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function main() {
  while (true) {
    console.log('Spawning child process!');
    exec('./child.sh');
    await sleep(1000);
  }
}

main();

Shell script child.sh:

#!usr/bin/env sh
# child.sh (called from Node.js program)
sleep 3 &
exit 0

In this scenario, each invocation of child.sh is a child process, and each sleep invocation within the child is a grandchild. When the child.sh processes exit, the grandchildren run to completion and never get cleaned up, turning into zombie processes.

To resolve this issue, we can update our code so that parent processes wait for their children to exit. In a shell script, we can achieve this by waiting for the PID of the grandchild. If we're spawning native Node.js processes, much of this will be handled automatically.

#!usr/bin/env sh
sleep 3 &

# By capturing the PID of the sleep command and waiting 
# for it to exit we no longer create zombies!
pid=$!
wait $pid

As with signal handling, if we don't control the source code, we can use a separate init process to handle zombie processes. The separate init process will reap zombie processes when they exit.

Summary

Running our application as PID 1 inside a container has implications for signal handling and zombie process reaping. The best practices are:

Properly handle system signals in our application code.
Manage child processes to prevent the creation of zombie processes.

If we cannot modify the application code, consider using a separate init process.

Hopefully, this article has helped we understand these considerations. To learn more about containers, check out my comprehensive course {{< link "https://courses.devopsdirective.com/docker-beginner-to-pro/lessons/00-introduction/01-main" "Docker: Beginner to Pro" >}}:

Additional Resources:

Here is a list of additional resources on the internet I found helpful when trying to understand this topic:

writing-a-nodejs-dockerfile

2025-07-09T02:27:47.040Z

TL;DR: Tutorials often show simplistic, naive Dockerfiles. We can do better.

Please don't do this! 🙏

Table of Contents:

Background

Node.js is a popular Javascript runtime environment that enables developers to write back-end applications. There are many tutorials online showing how to containerize Node.js applications, but many of them show a naive implementation that is functional, but lacking in so many ways!

Naive Implementation

This Dockerfile starts from the official node container image from DockerHub, copies in the entire build context, installs dependencies with npm, and sets a command to be run upon startup.

FROM node
COPY . .
RUN npm install
CMD [ "node", "index.js" ]

While this will technically work, there are SO many ways in which we can improve it. Let us incrementally improve this Dockerfile to make it more secure (🔒), build faster (🏎️), and more clear (👁️)!

Pin the Base Image (🔒+🏎️)

The first way we can improve the Dockerfile is by pinning the base image to a specific version. With no tag, Docker will use the "latest" tag which is the default tag applied to images. This would cause the base image to change with each new update to the upstream image, inevitably breaking your application.

We can choose a specific base image that is small and secure to meet the needs of our application. See this article for information about choosing a good base image: "TODO"

#-------------------------------------------
# Pin specific version
# Use slim for reduced image size
FROM node:19.6-bullseye-slim
#-------------------------------------------
COPY . .
RUN npm install
CMD [ "node", "index.js" ]

Pinning to the minor version should prevent known breaking changes while still allowing patch versions containing bugfixes to be utilized. If you want to truly lock the base image you can refer to a specific image hash such as:

FROM node:19.6-bullseye-slim@sha256:e684615bdfb71cb676b3d0dfcc538c416f7254697d8f9639bd87255062fd1681

Set a Working Directory (👁️)

By default, the working directory would be the root path (/) but you should set it to something else based on the conventions of your specific language + framework.

This will provide a dedicated place in the filesystem with your app.

FROM node:19.6-bullseye-slim
#-------------------------------------------
# Specify working directory other than /
WORKDIR /usr/src/app
#-------------------------------------------
COPY . .
RUN npm install
CMD [ "node", "index.js" ]

Copy package.json and package-lock.json Before Source Code (🏎️)

Each instruction within the Dockerfile creates a new layer within the image. Docker caches these layers to speed up subsequent builds. Previously, every change to the source code would invalidate the layer cache for COPY . . causing the build to reinstall all of the dependencies (which can be SLOW!).

By copying only the dependency configuration files before running npm install we can protect the layer cache and avoid reinstalling the dependencies with each source code change.

We can also use a .dockerignore file to specify files that should not be included in the container image (such as the node_modules directory).

FROM node:19.6-bullseye-slim
WORKDIR /usr/src/app
#-------------------------------------------
# Copy only files required to install
# dependencies (better layer caching)
COPY package*.json ./
RUN npm install
# Copy remaining source code AFTER installing dependencies. 
# Again, copy only the necessary files
COPY ./src/ .
#-------------------------------------------
CMD [ "node", "index.js" ]

Use a non-root USER (🔒)

If configured properly, containers provide some protection (via user namespaces) between a root user inside a container and the host system user, but setting to a non-root user provides another layer to our defense in depth security approach!

The node base image already has a user named node we can use for this purpose.

FROM node:19.6-bullseye-slim
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
#-------------------------------------------
# Use non-root user
# Use --chown on COPY commands to set file permissions
USER node
COPY --chown=node:node ./src/ .
#-------------------------------------------
CMD [ "node", "index.js" ]

Configure for the Production Environment (🔒 + 🏎️)

Many Node.js packages look for the NODE_ENV environment variable and behave differently if it is set to production (reduced logging, etc...). We can set this within the Dockerfile to ensure it will be set at runtime by default.

Also, rather than using npm install it is preferable to use npm ci or "clean install" which requires the use of a package-lock.json file and ensures the installed dependencies match the fully specified versions from that file. By using --only=production we can avoid installing unnecessary development dependencies reducing the attack surface area and further reducing the image size.

FROM node:19.6-bullseye-slim
#-------------------------------------------
# Set NODE_ENV
ENV NODE_ENV production
#-------------------------------------------
WORKDIR /usr/src/app
COPY package*.json ./
#-------------------------------------------
# Install only production dependencies
RUN npm ci --only=production
#-------------------------------------------
USER node
COPY --chown=node:node ./src/ .
CMD [ "node", "index.js" ]

Add Useful Metadata (👁️)

There are a few Dockerfile instructions that don't change the container runtime behavior, do provide useful metadata for users of the resulting container image.

We can add LABEL instructions with various annotations about the container image. For example we might want to include the Dockerfile author, version, licenses, etc... A set of suggested annotation keys from the Open Container Initiative can be found here: https://github.com/opencontainers/image-spec/blob/main/annotations.md.

The EXPOSE command tells end users the port number that the containerized application expects to listen on. The port will still need to be published at runtime, but it is useful to include this instruction to make it clear to end users which port should be opened.

FROM node:19.6-bullseye-slim
#-------------------------------------------
# Use LABELS to provide additional info
LABEL org.opencontainers.image.authors="sid@devopsdirective.com"
#-------------------------------------------
ENV NODE_ENV production
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci --only=production
USER node
COPY --chown=node:node ./src/ .
#-------------------------------------------
# Indicate expected port
EXPOSE 3000
#-------------------------------------------
CMD [ "node", "index.js" ]

Use a Cache Mount to Speed Up Dependency Installation (🏎️)

Buildkit provides many useful features, including the ability to specify a cache mount for specific RUN instructions within a Dockerifle. By specifying a cache in this way, changing a dependency won't require redownloading all dependencies from the internet because previously installed dependencies will be stored locally.

Note: If building the image in a remote continuous Integration system (e.g. GitHub Actions), you would need to configure that system to store and retrieve this cache across pipeline runs.

FROM node:19.6-bullseye-slim
LABEL org.opencontainers.image.authors="sid@devopsdirective.com"
ENV NODE_ENV production
WORKDIR /usr/src/app
COPY package*.json ./
#-------------------------------------------
# Use cache mount to speed up install of existing dependencies
RUN --mount=type=cache,target=/usr/src/app/.npm \
  npm set cache /usr/src/app/.npm && \
  npm ci --only=production
#-------------------------------------------
USER node
COPY --chown=node:node ./src/ .
EXPOSE 3000
CMD [ "node", "index.js" ]

Use a Multi-Stage Dockerfile (👁️)

By splitting out separate development and production image stages we can have an ergonomic dev environment with dev dependencies, hot reloading, etc... but retain the production improvements for deployment.

Shared steps can be built into a base stage and then customizations can be built on top of that base.

#-------------------------------------------
# Name the first stage to reference later
FROM node:19.6-bullseye-slim AS base
#-------------------------------------------
LABEL org.opencontainers.image.authors="sid@devopsdirective.com"
WORKDIR /usr/src/app
COPY package*.json ./
#-------------------------------------------
# Use the base stage to create dev image
FROM base AS dev
#-------------------------------------------
RUN --mount=type=cache,target=/usr/src/app/.npm \
  npm set cache /usr/src/app/.npm && \
  npm install
COPY . .
CMD ["npm", "run", "dev"]
#-------------------------------------------
# Use the base stage to create separate production image
FROM base AS production
#-------------------------------------------
ENV NODE_ENV production
RUN --mount=type=cache,target=/usr/src/app/.npm \
  npm set cache /usr/src/app/.npm && \
  npm ci --only=production
USER node
COPY --chown=node:node ./src/ .
EXPOSE 3000
CMD [ "node", "index.js" ]

References

This post from Liran Tal inspired many of the security-related improvements: https://snyk.io/blog/10-best-practices-to-containerize-nodejs-web-applications-with-docker/
My tweet thread on this topic went viral. See here for the dialog it generated: https://twitter.com/sidpalas/status/1634194026500096000

docker-cmd-vs-entrypoint

2025-07-09T02:25:56.031Z

TL;DR: The simplest possible example to demonstrate the difference between Docker's CMD and ENTRYPOINT instructions:

Dockerfile

The Dockerfile CMD and ENTRYPOINT instructions can be confusing. Let's create a mimimal docker image to explore how they behave and interact with eachother:

docker build -t entrypoint-cmd-example -<<EOF 
  FROM busybox 
  ENTRYPOINT [ "echo", "[ENTRYPOINT]" ]
  CMD [ "[CMD]" ]
EOF

No Arguments

If you run the image as is (no options/arguments), the ENTRYPOINT is used as the executable and the CMD is appended as arguments.

docker run entrypoint-cmd-example

The output will combine the ENTRYPOINT and CMD from the Dockerfile:

[ENTRYPOINT] [CMD]

With Arguments (override CMD)

If you specify arguments at runtime (e.g. docker run image-name FOO), those arguments override the CMD from the Dockerfile.

docker run entrypoint-cmd-example "[OVERRIDDEN CMD]"

The output will combine the ENTRYPOINT from the Dockerfile with the arguments provided at runtime:

[ENTRYPOINT] [OVERRIDDEN CMD]

Override Entrypoint

If you specify a new entrypoint at runtime (e.g. docker run --entrypoint=BAR image-name) that will override the ENTRYPOINT and IGNORE the CMD from the Dockerfile.

docker run --entrypoint echo entrypoint-cmd-example

Output is an empty string since echo is run with no arguments.

With Arguments (override CMD) AND Override Entrypoint

If you specify a new entrypoint AND arguments at runtime (e.g. docker run --entrypoint=BAZ image-name BING) those will be used instead of the values from the Dockerfile.

❯ docker run --entrypoint=echo entrypoint-cmd-example "[OVERRIDDEN CMD]"

The output will use the arguments supplied at runtime with the executable specified in the overriden entrypoint:

[OVERRIDDEN CMD]

When to Use Each

You should specify an ENTRYPOINT if you want the container to use the same executable (but potentially with different arguments) every time.

The CMD should contain the default arguments you want the container to run with.

The official docs have a nice summary of the various interactions as well: https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact

It is made slightly more complex by the fact that there is different behavior between:

ENTRYPOINT foo

AND

ENTRYPOINT ["foo"]

(With the latter being preferred)

cloudflare-workers-link-forwarder

2025-07-09T02:25:18.559Z

Cloudflare Workers Custom Link Forwarder

TL;DR: Setting up a custom link forwarder (e.g. links.devopsdirective.com/foo) takes about 3 minutes using Cloudflare workers!

Motivation

I have lots of content across many different platforms (YouTube, Twitter, LinkedIn, etc...) and wanted to create a was to easily set up custom links pointing to them. This way I can provide people links such as https://links.devopsdirective.com/terraform-gumroad rather than https://sidpalas.gumroad.com/l/hieekq.

This seemed like a perfect opportunity to try out cloudflare workers, a serverless compute offering from Cloudflare.

Create a Service

Each unit of compute within Cloudflare workers is called a "service". To create a service you can either use the CLI called "Wrangler" or navigate to the home page of your Cloudflare account at https://dash.cloudflare.com/<ACCOUNT_ID>/workers/overview. For this article I'll be showing the web UI.

You give your service a name and choose some simple starter code before clicking Create Service. This will deploy the service to https://<SERVICE_NAME>.<YOUR_CUSTOM_WORKERS_SUBDOMAIN>.workers.dev and within a few seconds it will be active.

Note: While Cloudflare workers do support other languages (either via WASM or cr languages that compile to JavaScript) JavaScript and TypeScript have the best native developer experience.

Updating the Code

After creating the service, it will then show up in your top-level workers dashboard where you can click it and choose Quick Edit.

This will bring up a simple code editor + HTTP request testing interface so you can make changes to your code and test them right in the browser. For more complex use cases, editing in your IDE of choice and using the Wrangler CLI to deploy makes sense, but here I am only going to have a few lines of code.

The Code

Your service will need to export a default object containing properties corresponding to the events the worker should handle (e.g. fetch events or scheduled events). A simple hello world service can be written as:

export default {
  async fetch(request, env) {
    return new Response("Hello world")
  }
}

For the link forwarding use case, we extract the pathname from the fetch event request.URL and then check if it matches one of our links. If so, redirect to the desired location, but if no links match, redirect to https://devopsdirective.com.

export default {
  async fetch(request, env) {
    try {
      const { pathname } = new URL(request.url);
      const statusCode=301;

      if (pathname.startsWith("/terraform-gumroad")) {
        const destinationURL="https://658236975647.gumroad.com/l/hieekq";
        return Response.redirect(destinationURL, statusCode);
      }

      // Insert additional links here!

      const defaultDestinationURL="https://devopsdirective.com";
      return Response.redirect(defaultDestinationURL, statusCode);
    } catch(e) {
      return new Response(err.stack, { status: 500 })
    }
  }
}

Custom Subdomain

At this point, the worker is doing what we want, but to make the links a bit nicer we can wire up custom trigger for a domain of our choosing.

You can choose a domain or subdomain of any domain managed by your CloudFlare account once the DNS settings propagate, your forwarding links will be active!

kubernetes-maintenance-page

2025-07-09T14:14:45.510Z

Kubernetes Maintenance Page

TL;DR: Need to set up a quick maintenance page for a service hosted in Kubernetes? You can create a custom page and host it with nginx without even needing a custom container image!

A gist containing the full code can be found here: {{< link "https://gist.github.com/sidpalas/e388f1a63bacc4c365d6cebf366f492d" "LINK" >}}

Why?

While it is usually best to attempt to avoid downtime for your production services, sometimes downtime is unavoidable (or at least not worth the effort to avoid). I recently had a Kubernetes-based project that called for a maintenance period, and this is the solution I came up with. The two key attributes I wanted to achieve were:

Easy + fast to toggle on/off (I wanted to avoid needing a DNS update because caching can cause them to be slow and unpredictable)
No additional container image required (Every time you build a custom container image, it is one more thing to maintain, host in a registry, etc...)

How?

To achieve these goals I used the default nginx image and took advantage of the fact that ConfigMaps can be mounted as volumes to inject my HTML, CSS, and nginx configuration into the pod at runtime!

This gives me an easy way to serve the maintenance page and then a minor change to my Ingress definition will route traffic to/away from it.

Nginx Configuration

I StackOverflow copy/pasted my way to the following default.conf nginx configuration which will take all requests (except for png|jpg|jpeg|css filetypes) and route them to the maintenance page located at /usr/share/nginx/html/maintenance/maintenance.html, returning the proper 503 Service Unavailable HTML code:

# default.conf
server { 
  listen 80 default_server;
  server_name  _ ;

  location / {
    if (-f /usr/share/nginx/html/maintenance/maintenance.html) {
      return 503;
    }
  }
  
  # for all routes, return maintenance page
  error_page 503 @maintenance;
  location @maintenance {
    root    /usr/share/nginx/html/maintenance/;
    rewrite ^(.*)$ /maintenance.html break;
  }
  
  # allow images and css to be retrieved
  location ~* \.(png|jpg|jpeg|css) {
    root /usr/share/nginx/html/maintenance/;
  }
}

Kubernetes resources

ConfigMap

I created a ConfigMap containing the site files (HTML + CSS) and nginx default.conf file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: maintenance-page
data:
  maintenance.html: |-
    <!--HTML GOES HERE-->
  maintenance.css: |-
    /* CSS GOES HERE */
  default.conf: |-
    # NGINX CONFIGURATION GOES HERE

Deployment

I then created a Deployment which mounts this ConfigMap to achieve the desired effect:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: maintenance-page
  labels:
    app: maintenance-page
spec:
  replicas: 1
  selector:
    matchLabels:
      app: maintenance-page
  template:
    metadata:
      labels:
        app: maintenance-page
    spec:
      containers:
      - name: nginx
        image: nginx:1.23
        ports:
        - containerPort: 80
        volumeMounts:
        # Because no subPath is specified, all keys in configmap will
        # be mounted as files at the specified mountPath
        - name: config-volume
          mountPath: /usr/share/nginx/html/maintenance/
        - name: config-volume
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: default.conf
      volumes:
        - name: config-volume
          configMap:
            name: maintenance-page

Service & Ingress

All that remained were the Service and Ingress objects to route traffic to this Deployment's pod (Note: the cluster is using the nginx ingress controller):

apiVersion: v1
kind: Service
metadata:
  name: maintenance-page
spec:
  selector:
    app: maintenance-page
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/rewrites: serviceName=maintenance-page rewrite=/;
  name: maintenance-page
spec:
  rules:
  - host: maintenance.devopsdirective.com
    http:
      paths:
      # Can also set this up only for specific paths
      - path: /
        pathType: Prefix
        backend:
          service:
            name: maintenance-page
            port:
              number: 8080
      # Can still have other paths defined

To turn on maintenance mode, I apply these resources to the cluster and to turn it off, I reapply the previous Ingress definition. The ConfigMap/Deployment/Service could be deleted, or the Deployment scaled to zero replicas if it will be used again in the future!

If you already have an nginx based deployment in the cluster, you could instead incorporate this configuration into it, but I like the fact that this is a standalone setup completely isolated from the standard config.

If you need to have a maintenance window for some services running in your cluster hopefully this helps make things easier for you!

why-k8s-exists

2025-07-09T02:23:14.315Z

TL;DR: Kubernetes is complex because it is solving complex problems. It builds on multiple decades of progress to provide a solid foundation for deploying web applications!

Pre-Cloud (2000s)

In the early days of the web, modern clouds didn't exist (AWS started in 2006), but deploying web applications still required access to servers!

You either racked them yourself or rented them from a colocation facility, but either way, they were slow to procure and painful to manage!

This doesn't look very fun...

The tooling to manage large fleets of servers and applications simply didn't exist, so you would often hire a team of system administrators, who would end up rolling their own tooling in the form of Bash or Batch/PowerShell scripts! Also, because the applications were running directly on the host system(s), dependency management was a nightmare!

These factors pushed teams to use monolithic architectures because it was simply too hard to manage anything else. Doing zero downtime deployments with this type of setup was incredibly difficult, so many companies leveraged scheduled maintenance windows to handle updates and roll out new versions.

Virtual Machine + Configuration Management (2010s)

Once cloud providers started maturing, the concept of creating and destroying Virtual Machines in minutes is a paradigm shift. Rather than maintain long-lived servers you could tear down the old ones and replace them!

Also, configuration management tools like Puppet and Chef matured, helping make it possible to manage fleets of servers more easily!

Virtualization technologies provide a level of isolation between applications that alleviate some of the dependency hell from before!

As web applications started tackling more complex problem domains, sometimes team size caused teams to start moving away from monolithic architectures. The improved tooling of the cloud made it possible to carve things up into smaller pieces.

However, because the size and complexity of these systems were now larger, we needed improved tooling to help automate deployments, monitor application health across all of those machines, aggregate logs, etc...

Workload Orchestrators (Late 2010s -> Today)

Moving towards the late 2010s and into the present day, solutions to those challenges have emerged in the form of what we now call "Workload Orchestrators".

There are a number of these types of systems:

Docker Swarm
Apache Mesos
Hashicorp Nomad
Kubernetes (today's industry darling)

These provide APIs to abstract away the individual machines and instead treat a cluster of instances as a single resource pool.

You provide information about the types of resources our workloads require and then let the system decide how to schedule and run them!

Kubernetes provides things like:

Automated health checks
Deployment strategies to enable zero downtime deployment (+ rollbacks)
Autoscaling based on metrics
Standard interfaces for networking, storage, and runtimes

Before these systems were available, large companies were building them internally in private.

Luckily for us, a few of those companies decided to release their systems. Google took many of the ideas from their internal system "Borg" and started the Kubernetes project!

Diagram from Google's public borg paper (https://research.google/pubs/pub43438/)

Now, rather than provisioning a bunch of instances/autoscaling groups for each app, you can provision a cluster and deploy many heterogeneous workloads into it!

You leverage the well-tested utilities of the system to handle what used to be done with your home-grown scripts.

So, yes Kubernetes IS quite complex, but that is because it is attempting to solve a complex problem!

Hopefully, this brief look over the past two decades has helped to give you an appreciation for why Kubernetes exists and operates the way it does.

Video Summary

blog-workflow-improvements

2025-07-09T02:21:02.955Z

Blog Workflow Improvements

TL;DR: I set up GitPod and improved the tooling for creating new posts. I am hoping that this reduces the activation energy for writing new posts significantly!

Why Don't I Post here Frequently?

If you look at the history of articles on this site, it is clear that I haven't done a great job staying consistent with posting. This is in contrast to my YouTube channel and Twitter.

When I thought about why that was, a big reason was that writing posts felt like a bit of a chore. There were several manual steps in the process and the just wasn't very fun. Here is what my old process looked like:

Open up VS Code
Navigate to the devops-directive-website project
Run hugo new <POST_NAME>
Manually change the posts from a single markdown file to a directory
Start up the Hugo development server
Write the post
Commit associated files to GitHub
GitHub Actions takes over from here

This workflow wasn't terrible... but there were some pain points:

It only works on my laptop with the environment preconfigured. What if I want to write a post from my iPad or a different computer?
Writing a blog post feels different than writing code and requires a different mindset. I would prefer if the workflow could be isolated such that I can go into "writing mode".
The manual manipulation of new post files.
I didn't have spelling/grammar checking set up which lead to lower quality writing with more errors.

What to do?

Content Management Systems

As I tried to determine how to best address the first two pain points, I considered all sorts of things. The obvious solution would be to use a web-based content management system (CMS). This could be a fully integrated blogging platform like WordPress or Ghost or a "headless" CMS that stores the content which is then displayed via a separate frontend.

These CMSs have a web interface for authoring content which would eliminate the need to have my environment properly set up and would provide a standalone editing experience. I even went so far as to set up both a WordPress and Ghost blog and export/import my content (via RSS feed) into them:

Looks pretty good!

This experimentation was reasonably successful and I was able to get my blog ported over and looking decent with some minor theme tweaking.

That being said, I like the fact that this blog uses static site generation (SSG) which is just a fancy way of saying that rather than storing content in a database and loading it when a user requests the page, the pages are all pre-generated ahead of time. This makes [hosting simple]({{< ref "/posts/2020/10/gcs-cloudflare-hosting/index.md" >}}) and makes handling large traffic spikes like 😅 rather than 🥵.

Containerized / Remote Dev Environment

I gave some more thought to how I could address the pain points without having to abandon my current setup entirely. How could I achieve that same browser-based, isolated writing environment while still using Hugo as the blogging engine? The answer was a remote development environment.

In the past few years, products like Gitpod and GitHub CodeSpaces have come along, offering development environments that can be spun up quickly with all of the necessary dependencies/configurations and torn down when not in use. The Gitpod free tier allows for 50hrs/month.

The only external dependency my dev environment really needs is Hugo, so I wrote a script to download it from GitHub and place it into a directory on the PATH:

#!/bin/bash
set -e -o pipefail

HUGO_VERSION=0.76.5
HUGO_TAR_FILE=hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz

wget https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/${HUGO_TAR_FILE}
tar -xf ${HUGO_TAR_FILE}
sudo mv hugo /usr/local/bin/
rm ${HUGO_TAR_FILE}

I then added this script to be run in the .gitpod.yml file:

tasks: 
  - before: ./gitpodinit.sh

One gotcha I ran into is that I initially ran the script from an init task rather than the before task. There is a limitation with init tasks that any changes made within them only persist within the /workspace directory (see here) so moving to /urs/local/bin didn't persist.

I also added some extensions to the .gitpod.yml file to ensure they are pre-installed in the workspace and that was all the config needed.

vscode:
  extensions:
    - znck.grammarly
    - yzhang.markdown-all-in-one

In fact, I'm editing this post from a Gitpod environment right now!

Whoa, meta...

Additional Tweaks

As you may have noticed in my .gitpod.yml code above, I added an extension that uses Grammarly to check for spelling and grammar issues, so that solves another one of my problems from before.

Finally, I actually read the Hugo documentation to set up a directory-based archetype and a new/improved Make target for creating new posts:

.PHONY: create-dir-post 
create-dir-post: check-post-name
	git checkout -b $(POST_NAME)
	hugo new -k=dir-post $(POST_PATH) 
	code content/$(POST_PATH)/index.md

As you can see, it checks out a new git branch, creates a post using the directory-based archetype I set up, and opens the index.md file so I can start editing right away. No more fiddling with moving/renaming files, etc...

Final Thoughts

I am hopeful that this new workflow will make it easier for me to write articles and therefore I will do so more frequently. Only time will tell! 🤞

stages-of-internal-development-systems

2025-07-09T02:19:33.546Z

The Six Stages of Internal Development Platforms

You either die supporting raw Kubernetes manifests, or live long enough to see yourself build an internal developer platform. ~Anonymous

What the Heck is an Internal Development Platform?

An Internal Development Platform (IDP) is the system that your organization uses to facilitate developing, deploying, and operating applications. The IDP defines how application developers create, modify, and test their applications, ideally in a self-service manner.

Instead of just handing application teams an AWS root account and saying "have fun", IDPs seek to provide abstractions that enable those teams to focus on building products, without needing to become DevOps/cloud experts!

As outlined by {{< link "https://internaldeveloperplatform.org/core-components/" "internaldeveloperplatform.org" >}}, the 5 core components of an Internal development platform are:

Application Configuration Management: Manage application configuration in a scalable and reliable way
Infrastructure Orchestration: Integrate with your existing and future infrastructure
Environment Management: Enable developers to create new environments whenever needed
Deployment Management: Implement a Continuous Delivery or even Continuous Deployment (CD) approach
Role-Based Access Control: Manage who can do what in a scalable way

Everywhere I turn these days I see another article/talk about companies building internal development platforms on top of Kubernetes:

2022-03: Applying Product Thinking to Slack’s Internal Compute Platform
2022-02: DevOps MUST Build Internal Developer Platform (IDP)
2022-01: Your Internal Developer Platform s**ks!
2020-05: More Power, Less Pain: Building an Internal Platform with CNCF Tools

This concept is not actually new! Kelsey Hightower, Joe Beda (and others) have been telling us that Kubernetes is a platform for building platforms since 2017!

It's pretty dope... but it's still not the endgame

"How can we turn Kubernetes into a platform for building platforms?"

Why am I hearing about this now?

At this point you might ask, what is fueling this renaissance of companies building (and publicly talking about) Internal Developer Platforms? In my opinion, it is driven by the rapid maturation of the cloud-native tools that represent the building blocks for these types of platforms. This makes it feasible for smaller teams to build and support platforms that actually work and add business value.

That being said, the {{< link "https://landscape.cncf.io/" "Cambrian explosion" >}} of cloud-native tools is both a blessing and a curse. While it is more likely than ever that a tool exists that somewhat addresses your particular need, choosing between them and ensuring interoperability is becoming increasingly difficult! (more on this in a future post... {{< link "https://roamresearch.com/#/app/devopsdirective/page/xi2CaPv5i" "but you can find some raw thoughts here" >}})

Designing the Right IDP

The size and complexity of the appropriate platform to suit your organization depends on three main factors:

The number and nature of the applications it needs to support
The size and skill set of the application development team(s)
The size and skill set of the platform development team

Building an internal development process is generally an iterative process and ideally, you will have some early adopter application developers willing to use it as soon as possible and provide feedback to ensure it is meeting their needs effectively.

Below is a rough spectrum (scale intentionally vague) of the different stages an internal developer platform often exhibits within an organization. This is based on my experience working with teams as they build out these types of application platforms and from discussions with others doing the same.

Evolution of an IDP

Stage 0: External Platform as a Service

For very small teams, you likely should avoid building your own platform entirely, and use a 3rd party Platform as a Service (PaaS) while focusing on product iteration and achieving product market fit. There are many options to choose from here including:

Eventually, though, it is likely that either:

Your workloads no longer fit the constraints imposed by the PaaS 📦
The PaaS pricing begins to hurt your wallet a bit too much 💸

at which point you may consider migrating off the PaaS, and in the past few years Kubernetes has become the de facto leader for doing so!

Stage 1: Vanilla Kubernetes

When you first move to Kubernetes and are still figuring out how to migrate things, you probably won't have built out additional abstractions. You will use the raw Kubernetes resource types directly and interact with the cluster via kubectl. This stage generally won't last long before moving on to the next stage.

Stage 2: Templating Tools

Very early on you will come across scenarios where you want to have variants of applications running in different environments. There are a variety of ways to accomplish this, ranging from custom bash scripts using envsubst to more robust tools such as helm. Your application developers will still need some familiarity with Kubernetes to be successful at this stage.

Stage 3: Initial Internal Developer Platform

Eventually, as the team and applications grow, creating bespoke definitions for each workload becomes unwieldy. To wrangle this complexity you will likely begin stitching together some collection of tools from the CNCF ecosystem to help.

Much of the work at this stage is selecting the tools that best suit your team's needs and configuring them to play nicely together. This is also the stage where you start to bake more control/opinions into the platform itself and you shape the application developer experience (for better or for worse).

There are several companies and projects such as backstage, humanitec, and shipa that are attempting to help make rolling an IDP easier. It's worth evaluating these options to see if they will suit your need and/or for inspiration in designing your own.

Stage 4: Extending Existing Platform Building Blocks

As you iterate on your platform, you will almost certainly start to push the boundaries of what existing tools can support or encounter use cases that aren't quite supported. In this case, you may start to extend those tools (either via a wrapper or a fork) to meet your own needs.

Stage 5 (Optional): Inventing New Platform Building Blocks

At a certain scale/complexity, there might not be an existing tool/project and your custom scripts and glue code are no longer enough. At that point, you might choose to build a new tool to fill the gap.

Stage 6: Full Internal Platform as a Service

At this point, you have come full circle! While the platform will never be complete, a mature internal developer platform starts to look and feel a lot like the 3rd party PaaS you started with, except it has all the specific features your organization needs.

Go Forth and Conquer!

Building an Internal Developer Platform is a significant undertaking, but if done properly can be a force multiplier for application developers and reduce the operational toil of building and operating applications on Kubernetes.

The sooner you start to think about what your ideal IDP might eventually look like, the easier it will be to make decisions that will help you set a course to reach that desired state.

Good luck! 🍀

cloud-control-spectrum

2025-07-09T02:15:01.068Z

Cloud Control Spectrum: How Should I Deploy My Web Application?

There are seemingly infinite ways to deploy web applications to the cloud:

Which option is right for you? Read on to find out!

🏢 On-Premise

If you want full control over everything, you can buy servers (computers), hook them up to power, set up networking, install any dependencies, and then deploy your software onto those systems.

Operating an on-premise deployment with high security and reliability comes with significant overhead and is generally only appropriate for large enterprises.

You are responsible for everything from physical security to climate control.

🤝 Colocation

If you want full control, but don't want to own and operate your own data center, colocation offloads that responsibility to a service provider. You still buy the servers but set them up in a building owned/operated by another company.

They will charge you rent for the space as well as a fee to cover utilities and other costs to operate the building.

This option can make sense if you have stable compute needs and are able to fully utilize the capacity you buy but don't want to deal with all of the operations.

🤘 Bare Metal Cloud

If you don't want to take on the capital expense of the servers, there are cloud providers that allow you to rent servers directly.

"Bare metal" refers to the fact that you interact with the server directly (instead of through a virtualization layer).
With bare metal servers, you are responsible for installing the OS, any dependencies, and your software.

If you mess something up, you may need to wipe the system and reinstall the OS from scratch (which can take a while). Bare metal servers have some performance benefits over virtual machines but are less dynamic.

This can be a good option if you have performance-critical workloads but your compute needs are variable and you don't want to buy your own servers!

📜 Dedicated Hosts/instances

Adding a virtualization layer, but renting a specific server(s) from a cloud provider makes it easier to create/destroy/configure servers while retaining hardware-level isolation from other customers (e.g. nobody else is running on the same servers).

You choose or create an "image" with an operating system (and potentially some dependencies) installed and the cloud provider creates a virtual server for you.
You then install additional dependencies and your software OR you might pre-build them into the image.

Dedicated hosts tend to cost more than standard virtual machines (next in this thread) but make sense in two cases:

Some software licenses charge you based on specific hardware usage so you need them
Some industries require data isolation guarantees

🤖 Virtual Machines

Similar to dedicated hosts, you choose or create an "image" with an OS (and potentially some dependencies) installed and the cloud provider creates a virtual server for you.

The main difference here is that standard VMs are cheaper and have no guarantees about which physical server your deployment goes on.

This is a great option to keep full control over the configuration but retain the flexibility to create and destroy servers quickly and easily!

⚙️ Managed Services

Many cloud providers offer specific products that further take over some of the management responsibility for portions of your application.

For example, AWS offers Relational Database Service (RDS) which provides a managed SQL database.

This allows you to avoid setting up and configuring your own database (with high availability, backups, etc).

You can leverage expertise from the provider, but you don't control the system so if something goes awry it can be an issue. It also costs more than operating yourself

✨ Platform as a Service

If you want to offload all of your deployment/operations, you can use a PaaS offering in which you generally hook up a code repo and let them handle the rest.
PaaS offerings can be a great choice for small teams who are focused on product iteration.

When using a PaaS, you are forced to adhere to the guidelines/requirements provided by the provider.

You give up control and pay a bit extra for the simplicity and streamlined developer experience!

Hopefully, this helps you understand the options and make a choice about what the right option is for you! 🎉

tiny-container-image

2025-07-09T02:11:27.801Z

Tiny Container Challenge: Building a 6kB Containerized HTTP Server!

TL;DR I set out to build the smallest container image that I could that was still able to do something useful. By taking advantage of multistage builds, the scratch base image, and a tiny assembly-based HTTP server, I was able to get it down to 6.32kB!

Table of Contents:

If you prefer video format, check out the corresponding YouTube video here!

Bloated Containers

Containers are often touted as a silver bullet to solve every challenge associated with operating software. While I love containers, I often encounter container images in the wild that have a variety of issues. One common issue is size, with images sometimes clocking in at multiple GB!

Because of this, I decided to challenge myself and others to build the smallest possible image.

Challenge

The rules are pretty simple:

The container should serve the contents of a file over http on the port of your choosing
No volume mounts are allowed (Also known as "The Marek Rule" 😜)

Naive Solution

To get a baseline image size, we can use node.js to create a simple server index.js:

const fs = require("fs");
const http = require('http');

const server = http.createServer((req, res) => {
  res.writeHead(200, { 'content-type': 'text/html' })
  fs.createReadStream('index.html').pipe(res)
})

server.listen(port, hostname, () => {
  console.log(`Server: http://0.0.0.0:8080/`);
});

and built it into an image starting the official node base image:

FROM node:14
COPY . .
CMD ["node", "index.js"]

This clocked in at 943MB! 😳

Smaller Base Image

One of the simplest and most obvious tactics for reducing image size is to use a smaller base image. The official node image has a slim variant (still debian based, but with fewer dependencies preinstalled) and an alpine variant based on Alpine Linux.

Using node:14-slim and node:14-alpine as the base image brought the image size down to 167MB and 116MB respectively.

Because docker images are additive, with each layer building on the next there isn't much else we can do to make the node.js solution smaller.

Compiled Languages

To take things to the next level we can move to a compiled language with many fewer runtime dependencies. There are a variety of options, but for building web services, golang is a popular choice.

I created a basic fileserver server.go:

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	fileServer := http.FileServer(http.Dir("./"))
	http.Handle("/", fileServer)
	fmt.Printf("Starting server at port 8080\n")
	if err := http.ListenAndServe(":8080", nil); err != nil {
			log.Fatal(err)
	}
}

and built it into a container image using the official golang base image:

FROM golang:1.14
COPY . .
RUN go build -o server .
CMD ["./server"]

Which clocked in at... 818MB. 😡

The issue here is that the golang base image has a lot of dependencies installed that are useful when building go software but aren't needed to run the software.

Multi-stage Builds

Docker has a feature called [multi-stage builds which makes it easy to build the code in an environment with all of the necessary dependencies, and then copy the resulting executable into a different image.

This is useful for a variety of reasons, but one of the most obvious is image size! By refactoring the Dockerfile as follows:

### build stage ###
FROM golang:1.14-alpine AS builder
COPY . .
RUN go build -o server .

### run stage ###
FROM alpine:3.12
COPY --from=builder /go/server ./server
COPY index.html index.html
CMD ["./server"]

The resulting image is just 13.2MB! 🙂

Static Compilation + Scratch image

13 MB isn't too bad, but there are still a couple of tricks we can play to make it even smaller.

There is a base image called scratch which is explicitly empty and has zero size. Because the scratch has nothing inside, any image built from it must bring all of its necessary dependencies.

To make this possible with our go-based server, we need to add a few flags to the compilation step to ensure the necessary libraries are statically linked into the executable:

### build stage ###
FROM golang:1.14 as builder
COPY . .
RUN go build \
  -ldflags "-linkmode external -extldflags -static" \
  -a server.go

### run stage ###
FROM scratch
COPY --from=builder /go/server ./server
COPY index.html index.html
CMD ["./server"]

Specifically, we set the link mode to external and pass the -static flag to the external linker.

These two changes bring the image size to 8.65MB 😀

ASM for the Win!

An image less than 10MB, written in a language like Go is plenty small for almost any circumstance... but we can go smaller! Github user nemasu fully functional HTTP server written in assembly on GitHub named assmttpd.

All that was required to containerize it was to install a few build dependencies into the ubuntu base image before running the provided make release recipe:

### build stage ###
FROM ubuntu:18.04 as builder
RUN apt update
RUN apt install -y make yasm as31 nasm binutils 
COPY . .
RUN make release

### run stage ###
FROM scratch
COPY --from=builder /asmttpd /asmttpd
COPY /web_root/index.html /web_root/index.html
CMD ["/asmttpd", "/web_root", "8080"]

The resulting asmttpd executable is then copied into the scratch image and invoked with the CMD. This resulted in an image size of just 6.34kB! 🥳

The progression of container image sizes!

Hopefully, you enjoyed this journey from our initial 943MB Node.js image all the way to this tiny 6.34kB Assembly image and learned some techniques you can apply to make your container images smaller in the future.

credential-management

2025-07-09T02:09:18.160Z

How to Manage Application Secrets (7 Levels of Credential Management)

TL;DR: If you develop web applications, inevitably you will have secrets (database credentials, 3rd party API keys, etc...) that you need to manage. I have seen a variety of approaches used here and wanted to walk through them, from least secure to most.

There are always trade-offs when writing software, and in this case, the tradeoff is between convenience and security. The ideal solution will establish convenient developer workflows while also protecting user data.

Table of Contents:

DISCLAIMER: Hopefully this is obvious from the tone of my writing, but PLEASE do not use levels -2, -1, or 0 in the real world. 🙏

If you prefer video format, check out the corresponding YouTube video here!

Level -2: No Authentication

For lazy people who like to live dangerously, you can simply turn off authentication for all services and cross your fingers 🤞.

This is the ultimate in convenience as you never have to deal with any secrets. It will be somewhat less convenient when inevitably your data is leaked on the dark web and you have to clean up the ensuing mess.

Level -1: All Passwords = "password"

If it is too hard to remember more than one password... how about setting them all to the same value? As long as you choose something easy to remember, you can then just share it with your teammates verbally and they will be good to go!

Level 0: Hardcode Everywhere

When I was learning to program, I received the advice to "define things close to where they are used". Well... you can't get any closer than hard coding it EXACTLY where it gets used!

const mongoose = require('mongoose');

const connectionString = 
  'mongodb://myUser:superSecretPassword@localhost:27017/databaseName';

mongoose.connect(connectionString, { useNewUrlParser: true }).catch((e) => {
  console.error('Connection error', e.message);
});
const db = mongoose.connection;

module.exports = db;

Okay, jokes aside -- there are many issues here. First, there is no way to avoid checking in the secret to your version control system. Second, it will be a nightmare to maintain because credentials will be sprinkled across many files. Third, there is no way to support multiple environments (separate dev/staging/production).

Level +1: Move Secrets into a Config File

Rather than hard code at the point of use, it is better to extract secrets into a separate configuration file and then load them in as environment variables. By doing this, you can treat the credential file as sensitive (including adding to .gitignore to avoid accidentally checking it into the codebase).

# secrets.env
DB_PASS=superSecretPassword

const dotenv = require('dotenv')
const mongoose = require('mongoose');

dotenv.config({ path: './secrets.env' })

const connectionString = 
  `mongodb://myUser:${process.env.DB_PASS}@localhost:27017/myDatabaseName`;

mongoose.connect(connectionString, { useNewUrlParser: true }).catch((e) => {
  console.error('Connection error', e.message);
});

const db = mongoose.connection;

module.exports = db;

For a side project where you are the only person working on it, this method might be sufficient, but it does have the downside of storing the credentials in plain text on your system. Also if you need to share credentials with teammates it is difficult to do so safely.

Level +2: Encrypt the Config File

One way to share the sensitive configuration file is to encrypt it. This can be done using a two-way encryption algorithm with a tool such as openssl:

export ENCRYPTION_KEY=Where-am-I-supposed-to-store-this?!

# encrypt
openssl aes-256-cbc -a -salt -in secrets.env -out secrets.env.enc -pass pass:$ENCRYPTION_KEY

# decrypt
openssl aes-256-cbc -d -a -salt -in secrets.env.enc -out secrets.env -pass pass:$ENCRYPTION_KEY

While certainly better than before, this does kick the can down the road a bit because now you have to figure out how to manage/share the encryption key.

One possible approach is to store it in a shared password manager such as 1Password or use a key management system such as Google Cloud KMS.

This approach doesn't provide any ability to monitor when individual developers are accessing the secrets. In certain industries where this type of audit log is required, this could be an issue.

Level +3: Use a Secret Manager

At this point we have taken the concept of a local configuration file about as far as we can. The next level is to move secrets into a dedicated secret manager. All of the major cloud providers offer a service like this, for example, AWS has the Secrets Manager.

The credentials can then be passed into your application as an environment variable at runtime. I often use the following Makefile snippet to retrieve credentials from GCP as needed:

PROJECT_ID:=<MY_GCP_PROJECT>
SECRET_NAME:=DB_PASS

define get-secret
$(shell gcloud secrets versions access latest --secret=$(1) --project=$(PROJECT_ID))
endef

run-app:
	@DB_PASS=$(call get-secret,$(DB_PASS))" npm start

Moving secrets into a system like this offer many benefits. First, these companies have entire teams of security experts building their products making them highly likely to be more secure than whatever system you roll yourself (analogous to "Don't roll your own crypto").

If you are already using the cloud provider to host your application, the tight integration with Identity and Access Management (IAM) functionality, and audit logging can be big wins from a security perspective.

The biggest shortfall with this approach is that rotating can still be a hassle, and often leads to a static set of long-lived credentials.

Level +4: Dynamic Ephemeral Credentials

To achieve fully enlightened credential management, we can move to a model of auto-generating credentials for each use case and only allow them to be active for a short period of time. This way if there is a leak, it greatly minimizes the potential blast radius.

The best implementation of this concept I have seen is HashiCorp's Vault. You can configure Vault so that whenever an application (or individual) needs access to a resource such as a database. It will create a new username/password and handle deleting those credentials after a pre-specified period of time.

There is a great talk on YouTube by Bench Accounting showcasing this approach in action (https://www.youtube.com/watch?v=Y0SdwZDy20Q), along with the corresponding Github Repo (https://github.com/BenchLabs/talk-vault-ephemeral-credentials).

To generate new credentials you can make a call to vault such as:

$ vault read database/creds/service-write

Which returns the new username/password along with some metadata:

Key                Value
---                -----
lease_id           database/creds/service-write/lVpzrysA5akqSvjZVtCgx1i9
lease_duration     240h
lease_renewable    true
password           A1a-9yW06ZdVk54I5KnX
username           v-token-service--1luYzAYl7SdMxvdpibYv-1555972312

Moving to this approach does require that you truly trust Vault (or whatever tool you use) because you have to grant it root-level permissions across many of your resources. You should protect those credentials as described in level +3.

Final Thoughts

Hopefully, this overview has helped you understand many of the available options for managing credentials for your web application. The right solution for your project will depend on its scale and the sensitivity of the resources being protected.

Before you start building your next application, think about where along this scale makes sense for you and your team so that you can keep your application secure.

gcs-cloudflare-hosting

2025-07-09T18:25:35.187Z

Static Site Hosting Using Google Cloud Storage and Cloudflare (with SSL!)

TL;DR: I used to host my site on a [virtual machine running Caddy webserver]({{< ref "/posts/2020/02/hugo-and-caddy-on-gcp/index.md" >}}), but recently migrated it to Google Cloud Storage with Cloudflare in front of it as a proxy/cache/SSL termination solution. It's pretty awesome! 😎

I also recorded a video about this setup on YouTube. ← Check out the video and subscribe if you are into this sort of thing 🙏

Does this count as serverless?

Table of Contents:

Overview

If you want to host a static website in 2020, the number of options can be a bit overwhelming. GitHub pages, Netlify, Vercel, oh my! This post describes my current favorite solution for hosting a static website. Here are the attributes of the setup described here:

Simple: No servers to maintain
Scalable: Can handle practically any load you throw at it
Affordable: Hosting this site costs me a few pennies per month
SSL: The site is served over HTTPS

Cloud Storage

Cloud object storage (S3, GCS, Azure Storage) is a pretty amazing technology. For a few pennies/GB/month you can store files and the cloud provider will serve them with high reliability to pretty much any level of traffic you can throw at it. Also, for nearly 10 years, AWS S3 has provided features to make hosting a website directly from a storage bucket simple.

That being said, those solutions don't support HTTPS by default. Google Cloud Platform does offer a guide for setting up a site with GCS with an HTTPS load balancer in front of it, but the load balancer costs $18/month at which point this solution becomes significantly less attractive relative to its competition.

Cloudflare

Cloudflare is a web-infrastructure company that provides a variety of network-related services, including DNS and content delivery caching. They also have a variety of SSL/TLS encryption options ranging from "Off" to "Full (strict)". The "Flexible" setting is the one we want here.

With this setting, SSL termination is done on the Cloudflare server, so the users' traffic is encrypted from their browser to Cloudflare and then requests between Cloudflare and the origin server (in our case the GCS server) will be un-encrypted.

If you were hosting a site with sensitive information, this would not be the right option, but for a public website (such as this one), this option gives us most of the benefits of HTTPS without having to set up an SSL certificate on our server!

Cloudflare flexible SSL

Cloudflare has the added benefit of automatically caching your content, speeding up delivery and reducing the data egress from GCS.

Setup

1) Purchase a Domain

The first step is to purchase a domain. Any domain provider will do, but purchasing with Google Domains makes the next step easier or unnecessary.

2) Verify Ownership

Before creating a GCS bucket associated with a particular domain, Google requires you to verify ownership of the domain.

There are a few methods to verify your ownership of the domain. If purchased through Google Domains, this can be done through the [Webmaster Central page, otherwise, you can use a DNS text record.

3) Create a GCS bucket

After verifying ownership, creating and configuring the bucket can be done using the gsutil tool.

	gsutil mb -p $PROJECT_ID -b on gs://$DOMAIN
	gsutil web set -m index.html -e 404.html gs://$DOMAIN
	gsutil iam ch allUsers:legacyObjectReader gs://$DOMAIN

The mb command creates the bucket, the web set command configures the main and error pages, and the iam ch command sets the permissions to enable public file access.

4) Configure Cloudflare

DNS Nameservers

When you add the domain to Cloudflare, it will prompt you to configure your domain provider to use a pair of Cloudflare nameservers. In my case this was:

elma.ns.cloudflare.com
james.ns.cloudflare.com

These will be use in place of the default nameservers provided by the domain provider.

CNAME Record

Because the website content will be served from a Google-managed server at c.storage.googleapis.com a CNAME record set that maps the custom domain to that:

Type   |  Name                 |  Content
------------------------------------------------------
CNAME  |  devopsdirective.com  |  c.storage.googleapis.com

SSL Setting

As described above, the site will use the "Flexible" SSL/TLS encryption mode, which can be set under the SSL/TLS tab on Cloudflare.

Page Rules

This step is optional, but without it, the www subdomain (e.g. www.devopsdirective.com won't work properly).

I prefer to use the root domain, but if a user adds a www. prefix I don't want them to encounter an error. This can be handled by creating the following forwarding rule.

The asterisk (*) will capture anything after the slash and gets substituted into the redirect URL at the $1. For example:

https://www.devopsdirective.com/about/ -> https://devopsdirective.com/about/

This page rule will only work properly if there is a DNS record for the www subdomain. This is accomplished by creating a dummy A record pointing to 192.0.2.1 (all addresses in the 192.0.2.0/24 are assigned to TEST-NET-1 for documentation and example code and don't actually correspond to a real server)

Type   |  Name                 |  Content
------------------------------------------------------
A      |  www                  |  192.0.2.1

5) Deploy Site

With everything configured, the only thing remaining is to upload the website files into the bucket. The easiest way to do this is using the gsutil rsync command:

gsutil -m rsync -d -r $LOCAL_SITE_DIR gs://$DOMAIN

The -m flag enables multi-threading which can help speed up the execution dramatically for large sets of files
The -d flag will delete files from the bucket not found in the $LOCAL_SITE_DIR. NOTE: If there is a chance the local dir could be empty, this would cause all the bucket contents to be deleted (so use at your own risk!).

Conclusions

I have been using this approach for a few months now and so far it has been rock solid. I love the simplicity of the setup, especially redeploying via a single rsync command.

Let me know how you end up hosting your next static site and why!

stupid-github-actions

2025-07-09T02:04:54.398Z

Doing Stupid Stuff with GitHub Actions

TL;DR: DevOps doesn't have to be all work and no play. I built 5 stupid (but fun!) GitHub actions... because why not?

The full code for these actions can all be found in this GitHub repo. I encourage you to fork and/or add issues/PRs with impractical actions of your own!

I also recorded a video about this project on YouTube. ← Check out the video and subscribe if you are into this sort of thing 🙏

Table of Contents:

What is GitHub Actions?

GitHub Actions is a CI/CD platform built into GitHub. It can be used to automate things such as building, testing, and deploying code and can be triggered by any GitHub event.

There is also a marketplace where developers can publish their actions for others to use.

While I have used many CI/CD systems including Jenkins, Google Cloud Build, and CircleCI, prior to this project I hadn't explored GitHub Actions, so I thought I would try it out and have some fun along the way.

Enough preamble, let's get to the stupid stuff!

The Actions

1 -- Holiday Reminder

Starting simple with this first action, I take advantage of the fact that actions can be triggered on a cron schedule to create the following 10 line action:

name: holiday-reminder-happy-new-year
on:
  schedule:
    - cron: '0 0 1 1 *'
jobs:
  happy-new-year:
    runs-on: ubuntu-latest
    steps:
    - name: throw error
      run: exit 1

The action will run at midnight on New Year's day and fail every time due to the non-zero exit code. This will cause GitHub to send me an email wishing me a Happy New Year 🎉🎉🎉.

2 -- Recursive Action

The next idea was proposed by a friend and former colleague (https://scotchka.github.io/). He suggested that I make an action that triggers itself, thus creating an infinite chain of actions.

Given that the GitHub documentation explicitly states "To minimize your GitHub Actions usage costs, ensure that you don't create recursive or unintended workflow runs," this seemed like a sufficiently stupid idea. To achieve this, I created an action triggered by commits that makes and commits a code change of its own.

There are two interesting parts to this action:

GitHub helps prevent users from accidentally doing this by not triggering actions based on events associated with the default GITHUB_TOKEN. In order to get around this, I created a personal access token. See the action yaml file for how this gets used.
In order to prevent the infinite action chain, I persist a count of the action executions within a file in the repo and increment it with each execution. This allows me to terminate the action chain when I reach a specified limit.

COUNTER_FILE=./recursive/counter.txt
MAX_COUNT=5

count=$(cat "$COUNTER_FILE") 
if (( $count > $MAX_COUNT ));
then 
    echo "Count too high... exiting";
else
    echo "Count okay... continuing";
    echo $(( $count + 1 )) > $COUNTER_FILE
    git config --global user.email "sid@devopsdirective.com"
    git config --global user.name "sid"
    git add $COUNTER_FILE
    git commit -m "Incremented counter file"
    git push 
fi;

3 -- Exponential Action

The previous action has the potential to run indefinitely, but only one instance executes at a time, so it could be stopped manually if necessary. What if instead, the action triggered itself multiple times? This way, if it got out of hand there would be no stopping the exponential growth 😰. That sounds dumb... let's do it!

The file counter trick from before no longer works because multiple actions would be executing in parallel, all trying to modify and commit simultaneously causing conflicts. Instead, I stored the state in Git tags!

A random UUID is used as the base for the tag and the current iteration is appended to prevent collisions. The current implementation only supports single digits, but if I make it past 9 iterations of exponential growth that means I messed up 💀.

function increment_tag_push {
    uuid=$(uuidgen)
    suffix=$(( $count + 1 ))
    tag=$1.$uuid.$suffix
    git tag -a $tag -m "New UUID tag"
    git push origin $tag
}

count="$(echo -n $GITHUB_REF| tail -c 1)"

echo $GITHUB_REF
echo $count

sleep 10 # In case something goes wrong (this saved me during development 😳)
MAX_COUNT=2

if (( $count > $MAX_COUNT ));
then 
    echo "Count too high... exiting";
else
    echo "Count okay... continuing";
    git config --global user.email "sid@devopsdirective.com"
    git config --global user.name "sid"
    for ((i = 1; i <= $1; i++ ));
    do
    increment_tag_push $i
    done
fi;

If you are looking for a quick way to burn through your 2000 free tier minutes... this is definitely the way to go!

4 -- Smart Lights

Thus far, these actions have been implemented in small bash scripts. To explore the capabilities of GitHub actions I decided the next action should utilize Docker.

I had a couple of WiFi smart plugs from VeSync that I received as a gift a few years ago and found a python client for their API. This made it simple to create an action that turns the lights on for a short period after each commit (what better way to incentivize code velocity?!💡)

Lights on... Lights off!

The code for this one isn't particularly interesting, but because the action is using more than just bash, it requires an action.yml file in which we can see how inputs get passed into the action:

name: 'Turn on Lights'
description: 'Turn on smart home lights for a few seconds'
inputs:
  VESYNC_PASS:  
    description: password for VESYNC_PASS
    required: true
runs:
  using: 'docker'
  image: 'Dockerfile'
  env:
    VESYNC_PASS: ${{ inputs.VESYNC_PASS }}

5 -- Tic-Tac-Toe

With the team starting to get a bit burnt out having to commit code constantly just to keep the lights on, I decided to implement a game of Tic-tac-toe to let them burn off some steam. The trick is that the computer player for this game executes within an action!

Command line interface + board state file

I wanted to try out the Javascript runtime, so I built the game with Node, persisting the board state in a text file. Since the point of this game was to learn about actions the computer's strategy is random, but this has the added benefit of letting the human win, boosting morale 🤔.

Closing Thoughts

Overall, GitHub Actions turned out to be fairly easy to work with and the option to Dockerize the action steps ensures that it should be able to support pretty much any CI/CD need. Also, GitHub has made sharing and reuse of actions a core part of the experience through the Marketplace.

While I won't be rushing off to port existing CI/CD workloads onto GitHub actions, I will certainly consider it for future greenfield projects!

password-protected-website

2025-07-08T20:32:49.696Z

password-protected-website

TL;DR: If you have a static website that needs to be password protected, using Google's Identity Aware Proxy along with App Engine is one of the simplest ways to accomplish this.

If you want to follow along with a site of your own, I have provided a working example in this GitHub repo.

Note: I have also seen S3 + Lambda used to accomplish this

~[[read-the-docs.png]]

Table of Contents:

The Need

A company I have been working with (Gauntlet Networks) needed to set up a website containing documentation for the simulation SDK they are building but wanted to restrict access to only their clients and employees.

Handling passwords

Historically, implementing authentication would require adding significant complexity to our website's architecture. We would need a database of some kind to store the (hashed) passwords and server-side processing to check if a login attempt is valid. We would also probably need to add some form of email confirmation/password reset functionality because people are terrible at remembering their passwords. All of the sudden the one-hour task of getting the documentation site set up just ballooned into something that could take weeks to accomplish!

Luckily, we don't need to take on all of this complexity ourselves. A standard called OAuth allows us to leverage the authentication system of another entity (for example, Google) to provide"secure delegated access" to our content.

Solution (IAP + App engine)

OAuth by itself would allow us to set up a sign-in flow using an external authentication provider, but for this use case, we want the process to be completely hands-off.

This is where Google's Identity Aware Proxy (IAP) comes in. Cloud IAP uses Google Sign-In and GCP's Identity and Access Management (IAM) to handle authentication and authorization to control access to GCP resources. Granting access to the site can then be managed using individual Google accounts and/or Google groups.

Given that Gauntlet was already using Google Cloud Platform, leveraging IAP in conjunction with Google App Engine to host the static site provided a simple, clean solution to satisfy the need.

Example Site

Because a documentation website was the use case that inspired this post, I am using the Sphinx Python documentation generator quick start as the basis for this demo. The static site files will be hosted using App Engine, with CircleCI for CI/CD.

Configuring App Engine

Google has documentation outlining how to use app engine to create a static site by adding an app.yaml file to the codebase that tells App Engine how to interpret URL request paths. Sphinx stores its output in the /docs/build/html directory by default, so I used the following app.yaml:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /
  static_files: docs/build/html/index.html
  upload: docs/build/html/index.html

- url: /(.*)
  static_files: docs/build/html/\1
  upload: docs/build/html/(.*)

With this in place, running the gcloud app deploy` command within the root of the project deploys the application.

Setting up IAP

With the website deployed, the next step was to follow Google's guide to enabling IAP for App Engine.

Just click the slider!

This involves configuring a few things for the Oauth Consent page (app name, support email, etc...), but the process is relatively painless.

The final step is to grant the IAP-secured Web App User access to all authorized individuals. In the screenshot below, I granted access to allAuthenticatedUsers which allows anyone with a Google account to access it. (Test it out here!)

Access can be granted to individuals or groups

Configuring CI/CD with CircleCI

At this point, the website is live and access controlled, but to minimize future work, it is useful to set up a system to handle continuous integration and deployment.

CircleCi offers simple integration with GitHub and a free tier that will easily handle a small project like this. The first step is to create a CircleCI account and then grant access to the GitHub repo containing the source files using their GitHub marketplace app.

Workflow

To define our workflow, we create a file at .circleci/config.yml containing two parts, a workflow definition and a job definition.

We define the workflow using version 2 of the CircleCI API. This workflow tells CircleCi to run the build_and_deploy job on any push to the master branch.

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build_and_deploy:
        filters:
          branches:
            only:
              - master

Next, we define the build_and_deploy job. Each CircleCI job runs within a Docker image, and because the goal is to deploy to Google App Engine, the google/cloud-sdk provides a good starting point.

version: 2.1
jobs:
  build_and_deploy:     
    docker:
        - image: google/cloud-sdk:slim
    steps:

The job is broken down into four steps:

checkout,
Sphinx installation,
Generating the website
Deploying to App Engine

1) checkout

      - checkout

As its name suggests, this step retrieves the source code within the repo and stores it in the working directory.

2) Install Sphinx

      - run:
          name: Install Sphinx
          command: |
            apt install -y python-pip && \
            python3 -m pip install -r requirements.txt

The google/cloud-sdk:slim container image we are using is based on Debian, so we can install Sphinx and its dependencies by first installing pip with apt and then using pip to install the requirements.txt file.

3) Make Docs

      - run:
          name: Make Docs
          command: |
            cd docs && make html

The Sphinx quickstart creates a Makefile within the /docs directory which can be used to generate the documentation site. This step changes directory into /docs and executes this make target passing html as the destination for Sphinx to store its output.

Note: html here needs to match the directory used in the App Engine app.yaml config (make html will build the site into docs/build/html where App Engine is configured to find them)

4) Deploy Docs

      - run: 
          name: Deploy Docs
          command: |
            echo ${GCLOUD_SERVICE_KEY} > /tmp/sa_key.json
            gcloud auth activate-service-account --key-file=/tmp/sa_key.json
            rm /tmp/sa_key.json
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
            gcloud --quiet app deploy

The final step in the job is to deploy the site. To do this, we first need to create a service account (https://console.cloud.google.com/iam-admin/serviceaccounts) and grant it the necessary permissions. Although it seems as though App Engine Deployer would be sufficient, it turns out there are actually a few additional permissions involved with deploying the site. The screenshot below shows the 5 roles I needed to add to achieve a successful deployment.

At this point I created a key for the service account (.json format) and stored its content as an environment variable in CircleCI. I also stored the Zone and Project ID as environment variables to ensure the deployment was targeting the correct app engine environment.

There was one final "gotcha" having to do with the service account key. My first attempt tried to pass it into the gcloud auth command by piping it in as follows:

echo ${GCLOUD_SERVICE_KEY} | gcloud auth activate-service-account --key-file=-

This caused gcloud to assume the key was a .p12 type and fail.

ERROR: (gcloud.auth.activate-service-account) Missing required argument [ACCOUNT]: An account is required when using .p12 keys

As shown in the full example at the beginning of the step, I was able to solve this by redirecting the echo output to a temporary file containing the JSON key and passing the file path to gcloud auth.

Success!

That's it! We now have a password-protected website, with CI/CD, that falls within the free usage tiers of GCP + CircleCI. If this site had heavy traffic or data egress it could exceed the free tier but for this internal documentation use case that is unlikely.

Next steps

There are a couple of additional steps that I will leave as an exercise to the reader.

Custom Domain

By default, App Engine assigns a domain such as https://devops-directive-project.uc.r.appspot.com/ to the website. It is possible to map a custom domain to the site by following this guide from Google.

Custom OAuth screen

The default OAuth consent screen is fairly boring but can be customized with a logo image, application homepage link, etc... from Google Cloud Console

kubernetes-the-hard-way-03

2025-07-10T13:11:51.358Z

Kubernetes the Hard Way Deep Dive -- 03 (Provisioning Compute Resources)

TL;DR: Setting up a Virtual Private Cloud (network), configuring firewall rules, and provisioning the Compute Engine virtual machine instances!

KTHW Lesson: 03-compute-resources.md

Table of Contents:

Networking

Default VPC

When the Compute Engine API is enabled within a GCP project, a default Virtual Private Cloud (named "default") is created.

Default VPC: 22 subnets (1 per GCP region) + 4 firewall rules are created by default!

Using the default network makes getting started with setting things up easy, but also enables any Compute Engine instances using that VPC to make network requests to one another using tcp, udp, or icmp protocols (based on the default-allow-internal firewall rule).

Custom VPC + Subnets

To isolate the Kubernetes cluster in this tutorial and minimize the attack surface area, we create a separate VPC with a single subnet and more restrictive firewall rules. Specifying the --subnet-mode custom when creating the VPC is what prevents GCP from auto-provisioning a subnet for each region. What about IP address range --range specified (10.240.0.0/24)?

The /24 is what is known as classless inter-domain routing (CIDR) notation and is a shorthand way of specifying the subnet mask. 24 corresponds to the number of active bits in the subnet mask, so in this case, the subnet mask is: 11111111.11111111.11111111.00000000 in binary (or 255.255.255.0 in decimal).

This subnet mask denotes the first 3 octets 10.240.0. will be the same across the entire subnet and the final octet can vary throughout its full range from 0 to 255. The highest address (10.240.0.255) is the "broadcast address", leaving 254 open for IP addresses (as Kelsey states).

For a more in-depth look at Network IDs and Subnet Masks, this YouTube video is a great resource:

Firewall Rules

We are also creating two firewall rules, one which will allow internal traffic and another for external traffic.

Internal

The internal firewall rule allows tcp, udp, and icmp traffic originating from the IP range specified for the above subnet (10.240.0.0/24) as well as 10.200.0.0/16 which is the CIDR range that will be passed into the kube-controller-manager to be used for pod IP address assignment.

External

The external firewall rule (it is external because the filter is 0.0.0.0/0 indicating that any originating IP address is acceptable) allows for tcp traffic on port 22 (used for SSH), tcp traffic on port 6443 (used to communicate with the Kubernetes API server), and icmp on any port (used by network devices to diagnose communication issues)

Compute Instances (Virtual Machines)

With the networking configuration set up, we can now provision the Compute Engine Instances (VMs) for the Kubernetes control plane as well as those that will serve as the Kubernetes workers.

Pricing

This tutorial sets up 3 control plane nodes and 3 worker nodes of type n1-standard-1 each with a 200 GB boot disk. looking at GCP's pricing calculator we can see within the us-central1 region this will cost $193.63/month

While the Compute Engine instances comprise the bulk of the costs, a load balancer will cost an additional $18.26/month bringing the total to $211.89/month.

NOTE: This is higher than the number cited in the tutorial because it accounts for the persistent storage charges.

VM configurations

Within the gcloud compute instances create command, there are a number of command line flags used to configure the virtual machines. Below I have detailed the noteworthy options:

--can-ip-forward: allows the instances to send and receive packets with non-matching destination or source IP addresses.
--private-network-ip 10.240.0.1${i} substituting in the iterator i we can see that our controller VMs will have all fall within the range of our subnet from earlier:

Controllers:
  10.240.0.10, 10.240.0.11, 10.240.0.12

Workers:
  10.240.0.20, 10.240.0.21, 10.240.0.22

--subnet kubernetes: references the subnet by name.
--metadata pod-cidr=10.200.${i}.0/24 (worker VMs only): This metadata is available to the guest operating system. It will be used in 09-bootstrapping-kubernetes-workers.md to generate a network configuration file that tells k8s which internal IP addresses to assign to pods.

Connecting to VMs

The use of the gcloud compute ssh command is straightforward and no additional commentary is needed.

At this point, we now have networking + compute resources provisioned and configured. Next, we will generate certificates and keys to enable secure communications amongst the machines.

See you in the next post!

kubernetes-the-hard-way-01-02

2025-07-10T13:09:05.967Z

Kubernetes the Hard Way Deep Dive -- 01 & 02 (System Set Up)

TL;DR: Setting up GCP account, and installing various software prerequisites (gcloud, tmux, cfssl and kubectl)

KTHW Lessons:

Is this the Matrix?

Table of Contents:

GCP + gcloud

I don't have anything to add for this step. The instructions are complete and will get your account/system set up to proceed.

A GCP project needs to be configured with the necessary APIs enabled, and gcloud will allow us to interact with GCP via the command line.

tmux

Embarrassingly, I had never used tmux prior to this. 😳😳😳

tmux, short for "Terminal Multiplexer" is a program that allows for splitting up your terminal into multiple panes and tabs and managing the associated shell sessions. At first glance, this may sound like the Split Pane feature of the macOS terminal or the Split Horizontally/Vertically feature of iterm2, but the difference is that shell sessions are managed independently from the terminal window.

There are a variety of benefits this enables, but the feature Kelsey is suggesting for this tutorial is the ability to synchronize inputs across multiple terminal panes. This will allow us to ssh into multiple virtual machines, synchronize those sessions, and then execute a series of commands in parallel.

To do this we can use the following workflow:

Install tmux ($ brew install tmux if using Homebrew on macOS)
Start a new tmux session with $ tmux
Split the current pane twice using Ctrl + b -> %
Navigate between panes using Ctrl + b -> ← OR →
SSH into the relevant virtual machines (gcloud ssh ...) in each of the panes
Once connected, synchronize the panes using Ctrl + b -> :setw synchronize-panes
Execute the tutorial commands
After finishing, shut down all running tmux sessions using $ tmux kill-server

This is just one very specific workflow using tmux. For more information about getting started with some of the other tmux functionality (and lots of other useful tools/skills adjacent to programming), check out The MIT Missing Semester Course.

CFSSL & cfssljson

CFSSL (of which cfssljson is a sub-project) is an open source project from Cloudflare that is "both a command line tool and an HTTP API server for signing, verifying, and bundling TLS certificates".

These will be used during 04-certificate-authority.md to provision a certificate authority, and then use it to generate TLS certificates and private keys for many k8s system components.

I will go into more detail about what that actually means in the post corresponding to that step in the tutorial. For now, simply installing the software and verifying the versions are correct is sufficient.

kubectl

Kubectl (It's okay... nobody else knows how to pronounce it either) is a command line tool for interacting with Kubernetes clusters. Once the cluster is up and running, kubectl will be the primary mechanism for getting information about the cluster (and things running within it), as well deploying and modifying software running within the cluster.

Kelsey's instructions for installation are sufficient, but if on MacOS, it can also be installed with Homebrew using:

$ brew install kubectl

At this point, your computer should have all the necessary prerequisite software and we can start setting up the cloud resources needed for the cluster.

See you in the next post!

kubernetes-the-hard-way-intro

2025-07-10T13:07:46.676Z

Kubernetes the Hard Way Deep Dive -- Intro

TL;DR: This is the introduction to a series of posts in which I will work through Kelsey Hightower's Kubernetes The Hard Way, explaining key concepts in detail along the way.

My Kubernetes Onboarding

My first experience with Kubernetes (k8s) was working with an existing cluster deployed using Google Kubernetes Engine (GKE). The yaml files all looked like gibberish, but I pattern-matched my way to getting a new service up and running.

Later, I needed to set up a new cluster to support a variety of data analysis workloads and going through the process of setting up + configuring the cluster helped solidify my understanding of the k8s concepts. That being said, I would still come across situations (such as a pod reaching the dreaded Unknown phase) that left me yearning for a deeper understanding of what was going on under the hood.

The Plan

To help illuminate the inner workings of k8s, I am going to take off the GKE training wheels and work my way through the process of setting up a Kubernetes cluster from scratch. Kelsey Hightower* published a fantastic guide to do exactly this, but it assumes a certain level of understanding of many of the foundational tools/technologies.

I plan to follow Kelsey's guide, and document the process as I try to dig one layer deeper to grok exactly what each step is doing and why it is necessary for setting up k8s.

This series will be most useful for individuals who have worked with k8s, but primarily in a managed configuration.

Let's dive in!

If you don't know who Kelsey Hightower is, do yourself a favor and go watch a few of his presentations on YouTube. You won't regret it!

hugo-google-analytics

2025-07-08T20:30:53.660Z

hugo-google-analytics

TL;DR: If using Google Analytics, most Hugo themes load the tracking script even when running locally. This causes a bunch of bogus page views when writing a new article. Creating separate config files for development and production is an easy way to fix this.

~[[bogus-google-analytics.png]]
Whoops... that's me visiting my site on localhost"

To solve this, take advantage of the fact that Hugo allows for specifying multiple configuration files. In this case, create a separate "production" config.toml file without the googleAnalytics code specified and rearrange the directory structure as follows:

├── config
│   ├── _default
│   │   └── config.toml
│   └── production
│       └── config.toml

When serving locally with hugo serve the config in _default will be used, but when generating the site with hugo, Hugo will default to the production configuration.

If you do want to test the production configuration locally, you can specify the environment via a command flag:

hugo serve --environment production

The full documentation for this feature can be found on the gohugo.io website.

I hope that helps anyone who encounters the same situation!

load-testing-f1-micro

2025-07-08T18:55:50.126Z

Load Testing Caddy Web Server on a GCP F1-Micro Instance Using K6 (k6.io)

TL;DR: I used the K6 load testing framework to benchmark the Compute Engine f1-micro and Caddy web server hosting this site. With Cloudflare caching turned off, the server was able to serve an onslaught of 800 virtual users continuously reloading the page (while maintaining a median request duration of <400ms), but started dropping requests when increasing the load further.

This is fine...

Background

DevOps Directive is a static website generated with Hugo and hosted using Caddy running on an f1-micro GCP Compute Engine instance with Cloudflare in front of it (see [The Making of This Site]({{< ref "/posts/2020/02/hugo-and-caddy-on-gcp/index.md" >}}) post for details). On a normal day, the site used to get between 1 (thanks Dad!) and 20 visitors, but recently, two articles made it to the front page of Hacker News ({{< link "https://news.ycombinator.com/item?id=22512149" "Link-1" >}} and {{< link "https://news.ycombinator.com/item?id=22661029" "Link-2" >}}) bringing outsized swells in traffic.

2178% increase from the previous peak!

In a single hour on March 7th, a total of 1307 people visited this site ({{< link "images/realtime-analytics.png" "from all over the world!" >}}). Thankfully, even with all of that traffic, CPU usage of the virtual machine never even reached 10% (and the short spikes correspond to redeploying the site with copy edits).

No flames yet...

At this point, the setup has proven itself capable of embracing a Hacker News hug without dying, but I wanted to get a sense of what kind of load it can handle.

Testing

Site Setup

In order to avoid causing any impact to my actual site, I spun up an identical replica on a separate virtual machine using this script and configured the https://test.devopsdirective.com/ subdomain (which will likely be inactive at the time you are reading this) to resolve to it.

Here is a summary of the configuration:

Compute Engine f1-micro Instance (0.2 vCPU burstable to 1 vCPU for short periods, 0.6GB Memory) running Container Optimized OS (COS)
Caddy (1.0.3) container image with the site content files built directly into the container
Cloudflare is configured to proxy traffic and set to the "standard" caching level (I performed tests with caching turned on and caching turned off)

NOTE: I didn't tune/configure the COS image running on the VM, nor specify resource requests in the docker run command.

K6 Performance Testing Framework

To perform the load test I used k6.io, an open source performance testing framework designed for building automated tests with a straightforward javascript config file. It uses the concept of "virtual users" (VUs) which in their words are "glorified, parallel while(true) loops" to load test a site.

*Want to learn more about K6? Watch this video!*

Replicating Current Peak

First, I created a script to approximate the load that the two HN posts brought. The peak hour had 1443 page views or 0.4 pageviews/second. To account for the load not being constant across the entire hour, I rounded this up to 1 pageview/second.

K6 is able to use a HAR file](https://en.wikipedia.org/wiki/HAR_(file_format)) to create a representative set of HTTP requests. I used 1 virtual user and adjusted the pause between iterations to achieve just over 1 pageload/second (with a "pageload" corresponding to the batch of HTTP requests). I excluded external requests for things like the Google Analytics script. The full K6 configuration script can be found as a GitHub gist and the resulting output can be seen below:

check_failure_rate.........: 0.00%   ✓ 0   ✗ 138
checks.....................: 100.00% ✓ 414 ✗ 0  
data_received..............: 66 MB   549 kB/s
data_sent..................: 219 kB  1.8 kB/s
group_duration.............: avg=873.64ms min=850.89ms med=864.69ms max=1.13s    p(90)=891.35ms p(95)=908.24ms
http_req_blocked...........: avg=81.77µs  min=156ns    med=446ns    max=133.52ms p(90)=861ns    p(95)=1.42µs  
http_req_connecting........: avg=7.06µs   min=0s       med=0s       max=11.69ms  p(90)=0s       p(95)=0s      
http_req_duration..........: avg=26.89ms  min=12.6ms   med=23.11ms  max=213.29ms p(90)=38.19ms  p(95)=47.38ms 
http_req_receiving.........: avg=726.31µs min=62.01µs  med=242.75µs max=29.96ms  p(90)=1.84ms   p(95)=2.41ms  
http_req_sending...........: avg=52.23µs  min=11.27µs  med=38.27µs  max=4.07ms   p(90)=106.59µs p(95)=115.18µs
http_req_tls_handshaking...: avg=72.85µs  min=0s       med=0s       max=120.64ms p(90)=0s       p(95)=0s      
http_req_waiting...........: avg=26.11ms  min=12.37ms  med=22.51ms  max=212.53ms p(90)=37.05ms  p(95)=45.32ms 
http_reqs..................: 1656    13.799982/s
iteration_duration.........: avg=873.71ms min=850.96ms med=864.75ms max=1.13s    p(90)=891.46ms p(95)=908.31ms
iterations.................: 137     1.141665/s
vus........................: 1       min=1 max=1
vus_max....................: 1       min=1 max=1

Key Takeaways

All requests returned successfully (Status: 200 OK)
The request duration ranged from 13ms to 213ms
The 95th percentile duration was 47ms

Ramping It Up!

With that test as a baseline, I proceeded to run a series of tests, each 60 seconds long, starting with 6 virtual users and increasing the number of VUs with each test. I also reduced the delay between virtual user iterations to 100ms. The most important metric is http_req_duration, which represents is the total request time (http_req_sending + http_req_waiting + http_req_receiving), which I have plotted below for the full set of tests.

Unsurprisingly... a CDN with caching makes a big difference

Zoomed to <200VU, <500ms

Up until around 50 VUs, the response time remains flat, with an uncached median of 68ms and a cached median of 31ms.

After 50 VUs, the response times begin to climb linearly. At 800 VUs the uncached median was 349ms and the cached median was 67ms. As would be expected at these higher loads, most (90+%) of the http_req_duration is spent in the http_req_waiting stage.

The uncached configuration finally gave out during the 1600 virtual user test, with only 414 successful responses, indicating that ~74% of the virtual users never received a response.

Virtual Users and Server Load

It is important to note that while the virtual users run in parallel with each other, they run in serial with themselves. Each individual VU waits until its current page load is complete before making a new set of requests. As the server slows down under load, this causes the total rate of requests to drop in the more demanding tests. The impact is clearly seen in the total amount of data received during the tests plotted below.

{{< img-link "images/data_received.png" "images/data_received.png" "Data received (and pageloads/s) peaks before the more demanding tests">}}

These were the two most informative plots, but all of the data and code to generate plots can be found in a notebook in this GitHub repo. You can load an interactive copy of the notebook using the following binder link:

https://mybinder.org/v2/gh/sidpalas/f1-micro-caddy-benchmark/master?filepath=plotting-load-test-results.ipynb

Snags Along the Way

I did run into some technical limitations when configuring and executing these tests. Here are the main issues and how I overcame them:

Bandwidth Limitations: My home internet was not sufficient to support the load test. Moving to a GCP virtual machine with sufficient bandwidth (Measured @ 900+ Mbps) as the test client running K6 solved this. For the later tests in the cached configuration, this actually still became a limiting factor.
Memory Limitations: After moving from my laptop to an n1-standard-1 instance as the testing client, the more demanding tests caused K6 to run out of memory (fatal error: runtime: out of memory). Moving to an n1-standard-8 (30GB memory) solved this.
Unix Resource Limits: Because each request group makes multiple HTTP requests, the final test with 1600 target virtual users surpasses the default maximum number of open files allowed by the OS for a single process to manage at once. Using multiple test client VMs in parallel solved this, but increasing the open file limit with ulimit -n <NEW_LARGER_LIMIT> is the approach I ended up using.

(Aside) Total Costs

The total cost to run this experiment was $2.82:

$1.60 for 40.1 GB of network egress
$0.50 for running the f1-micro server for a ~3 days
$0.72 for running the n1-standard-8 testing client for a ~2 hrs

Conclusions

I am continuously amazed at the level of load that even such a tiny virtual machine can handle when serving static content!

Utilizing a service like Cloudflare to help cache and serve content reduces the load on the server significantly. It cut the response times in half under light load and prevented the server from being overwhelmed under heavy load.

I would have liked to record real-time resource (CPU + Memory usage) on the server VM but the GCP cloud monitoring agent isn't compatible with Container Optimized OS, so I settled for the rough 1 min averaged view in the GCP console:

Now we're cooking with gas! (bursting above the 0.2 vCPU limit for a short period)

This test gives me confidence that my current server configuration should be able to handle quite a bit of growth before needing any major overhaul.

In the future, I hope to do similar benchmarking across other hosting options. If someone has a contact at @github or @netlify that could permit me to run a test against a Github Pages or Netlify Starter site let me know! Or maybe at @bluehost so I can benchmark some WordPress installs...

always-on-minikube

2025-07-08T18:35:37.770Z

Converting an Old MacBook Into an Always-On Personal Kubernetes Cluster

TL;DR: I set up Minikube on an old (2012) MacBook Air and configured it to be able to connect from outside my home network so I would have an always-on Kubernetes cluster at home to experiment with.

Photo Attribution: flickr

Table of Contents:

Why?

Over the past five years, Kubernetes (k8s) has risen to become the de facto standard for managing container orchestration at scale.

However, with many layers of abstraction and [non-trivial costs]({{< ref "/posts/2020/03/managed-kubernetes-comparison/index.md" >}}) to run even the smallest cloud-based cluster, the barriers to entry for an individual who wants to familiarize themselves with Kubernetes can be quite high.

There are a variety of options for running Kubernetes locally including Minikube, Docker Desktop, and Microk8s. These are all good options for aiding in the development process, but all suffer from the fact that they are tied to the power status of the system they are running on (i.e. if I shut down my laptop, the local cluster shuts down too). In order to run workloads that are either time-dependent (cronjobs) or event-driven (asynchronous data processing), it would be much better to have a cluster that is always on.

I happen to have a 2012 MacBook Air sitting around unused now that it is no longer my daily driver. Amazingly, this 8-year-old system has a CPU that supports the necessary software-based virtualization to run a Linux virtual machine where we can install Kubernetes.

Power/Cost Considerations

Because my goal is to achieve a low-cost solution, I wanted to do a quick calculation of how much power this laptop will use if I leave it running 24x7. Based on this environmental spec sheet from apple I estimate the system (with the display off and relatively light workloads) will draw approximately 10W on average.

Pricing Estimate:

  10 (W) 
x 1/1000 (W/kW) 
x 24 (hrs/day) 
x 30.4 (days/month) 
x 0.2 ($/kWh)
---
1.44 $/month -- 👍

Process

0) Prevent Server System from Sleeping

Because the MacBook Air is not designed to be used in this configuration, it is necessary to disable sleep using the power management settings (pmset) command line utility. The following commands will make the time to sleep infinite as well as prevent sleeping when there is no display connected:

sudo pmset -a sleep 0; 
sudo pmset -a disablesleep 1

1) Install Minikube on the Server System

I followed these instructions (Minikube Installation Instructions) using the VitualBox hypervisor + vm-driver to install Minikube on the MacBook air:

2) Install `kubectl` on the Client System

I already had kubectl installed on my primary development machine, but you could follow these instructions (Kubectl Installation Instructions) to install it on the system you plan to access the cluster from.

3) Add port forwarding rule from the Server System to the Minikube VM

In order for kubectl commands to make their way to the Minikube cluster, it is necessary to set up a port forwarding rule within the VM configuration to forward requests to port 8443. This can be accomplished via the VirtualBox GUI:

4) Update the `.kube/config` on the Client to Point to the Server

When executing the minikube start command on the server system, a kubectl context called 'minikube' is added to the kubectl configuration file @ ~/.kube/config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: <PATH-TO-CONFIG>/.minikube/ca.crt
    server: https://192.168.99.100:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: "minikube"
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: <PATH-TO-CONFIG>/.minikube/client.crt
    client-key: <PATH-TO-CONFIG>/.minikube/client.key

This configuration needs to be copied onto the client machine (along with the client.crt and client.key files).

I copied those files to <PATH-TO-CONFIG>/.minikube-macbook-air on the client system and updated the kubectl configuration as follows:

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://<INTERNAL-NETWORK-IP>:51928
  name: minikube-macbook-air
contexts:
- context:
    cluster: minikube-macbook-air
    user: minikube-macbook-air
  name: minikube-macbook-air
current-context: "minikube-macbook-air"
kind: Config
preferences: {}
users:
- name: minikube-macbook-air
  user:
    client-certificate: <PATH-TO-CONFIG>/.minikube-macbook-air/client.crt
    client-key: <PATH-TO-CONFIG>/.minikube-macbook-air/client.key

This can either be inserted as a context within the primary .kube/config file or stored as a separate configuration and used by setting the KUBECONFIG environment variable (documentation)

Note: The insecure-skip-tls-verify: true is used because the certificate authority set up by Minikube is not configured to accept incoming requests originating from another system in this way.

5) (Optional) Use DDNS + Router Port Forwarding to Access Outside Local Network

At this point, I was able to connect to the cluster, but only when the Client and Server systems were connected to the same network.

Two additional things needed to be configured to connect from outside of my home network:

I set up a free dynamic DNS using https://www.dynu.com/. After doing this I replaced <INTERNAL-NETWORK-IP> with <MY-DDNS-DOMAIN-NAME>
I configured my router to port forward incoming requests on port 51928 to the MacBook Air and reserve the IP address for the MAC Address of the machine to ensure it doesn't change.

At this point I was able to successfully connect to the cluster from an external network!

Conclusion

There we have it -- For about the price of a McDonald's coffee each month in electricity, I have an always-on Kubernetes cluster (with a single 2 vCPU + 2 GB Memory node) ready for experimentation!

If I didn't happen to have the MacBook Air sitting around, building a home cluster using Raspberry Pis is another popular option (and enables having more than just a single worker node).

managed-kubernetes-comparison

2025-07-08T18:56:11.849Z

TL;DR: Azure and Digital Ocean don't charge for the compute resources used for the control plane, making AKS and DO the cheapest for running many, smaller clusters. For running fewer, larger clusters GKE is the most affordable option. Also, running on spot/preemptible/low-priority nodes or long-term committed nodes makes a massive impact across all of the platforms.

Table of Contents:

Overview

With Google Cloud's recent announcement to start charging $0.10/hour for each cluster on GKE, it seemed like a good time to revisit pricing across the major managed Kubernetes offerings.

This upset some people on the internet...

This article will focus on:

Google Kubernetes Engine (GCP) -- pricing calculator
Elastic Kubernetes Service (AWS) -- pricing calculator
Azure Kubernetes Service (Azure) -- pricing calculator
Kubernetes on Digital Ocean -- pricing page

Cost Breakdown

The cost of running Kubernetes on each of these platforms is based on the following components:

Cluster Management Fee
Load Balancer (for Ingress)
Worker Node Compute Resources (vCPU & Memory)
Data Egress
Persistent Storage
Load Balancer Data Processing

Additionally, the cloud providers offer large discounts if you are willing/able to use preemptible/spot/low-priority nodes OR commit to using the same nodes for 1-3 years.

It is important to point out that while cost is a useful dimension to examine when evaluating providers, there are other factors that should also be considered including:

Uptime (Service Level Agreement)
Surrounding Cloud Ecosystem
k8s Version Availability
Documentation/Tooling Quality

Those factors are beyond the scope of this article/exploratory analysis. This blog post from StackRox (Feb. 2020) provides a detailed look at the non-price factors for EKS, AKS, and GKE.

Jupyter Notebook

To make it easier to explore the cost tradeoffs I created a Jupyter notebook using plotly + ipywidgets to facilitate rapid exploration of cluster sizes/tradeoffs across the different cloud providers.

You can interact with a live version of that notebook yourself using Binder:

https://mybinder.org/v2/gh/sidpalas/managed-kubernetes-pricing/master?filepath=%2Fmanaged-kubernetes-price-exploration.ipynb

If any of my calculations or pricing constants look incorrect, please let me know (via a Github Issue or Pull Request! -- Github repository)

Takeaways

There are too many variables to give much of a recommendation beyond my TL;DR up top, but here are some takeaways:

AKS and Digital Ocean do not charge for the control plane resources while GKE and EKS do. If your architecture incorporates many small clusters (e.g. 1 cluster per developer or per customer) AKS and DO can have an cost advantage.
Google's slightly cheaper compute resources result in lower costs as cluster sizes scale*.
Taking advantage of preemptible and/or committed use discounts can reduce costs by >50% (NOTE: Digital Ocean does not offer these types of discounts).
While Google's data egress fees are higher, the compute resources dominated the cost calculation (unless you are sending a significant amount of data out of the cluster).
Choosing machine types that match your workloads' CPU and Memory needs will avoid paying for wasted capacity.
Digital Ocean charges relatively less for vCPUs and more for memory which could be an important consideration depending on the nature of the compute workloads.

*NOTE: I used general-purpose compute node types for all 4 clouds (n1 GCP Compute Engine instances, m5 AWS ec2 instances, D2v3 Azure virtual machines, and dedicated CPU DO droplets). Further exploration could be done across the burstable and entry-level VM types. Also, it appeared that pricing for VMs scaled linearly with # of vCPU and GB of memory, but I am not sure that assumption holds as you move towards some of the less standard memory/cpu ratios.

The article "The Ultimate Kubernetes Cost Guide: AWS vs GCP vs Azure vs Digital Ocean" published in 2018 used a single reference cluster with 100 vCPU cores and 400 GB of memory. As a point of comparison, here is how much my calculation shows that cluster would cost (using on-demand prices) on each of these platforms.

AKS: $51,465/year
EKS: $43,138/year
GKE: $30,870/year
DO: $36,131/year

Hopefully, this article + notebook will help you in your journey to evaluate the major managed Kubernetes offerings and/or save $$$ on your cloud infrastructure by taking advantage of the available cost-saving opportunities.

hugo-cloud-build

2025-07-09T18:25:53.819Z

CI/CD for this site (Hugo + Cloud Build)

TL;DR: Configuring Cloud Build to automatically handle Continuous Integration and Continuous Deployment for this site based on Git triggers ended up being a bit trickier than I would have expected.

Table of Contents:

This is a continuation of the previous post ([The Making of This Site (Hugo, Caddy, + GCP)]({{< ref "/posts/2020/02/hugo-and-caddy-on-gcp/index.md" >}})) in which I walked through the set up of this site. In this post, I add automated builds/deploys to the site.

All of the commands for creating the site, as well as setting up this automation can be found in this GitHub Repo.

Picking a CI/CD Tool

Initially, I was going to use Circle CI to automate the process of building and deploying the site. Circle CI has direct integration with Github and posted to their blog in 2018 explaining how to Automate Your Static Site Deployment with CircleCI using Hugo as the example site generator.

That being said since everything in the site setup was GCP based, I decided to try out Cloud Build. Cloud Build also has a GitHub app and being within the same GCP project meant I wouldn't have to deal with shuffling additional service account credentials between platforms.

Also, just like with the server setup, Cloud Build is also included in the GCP free tier (up to 120 build minutes/day) so this shouldn't cost me anything.

Attempting to Use the Cloud Build GitHub App

Thinking this would be a 30-minute task, I eagerly installed the Cloud Build Github app and added a build trigger based on pushes to the master branch. When the build succeeded I was not greeted with the website, but instead with the Default Caddy home page. After manually navigating to the /articles endpoint I saw the following:

The content files were there, but I realized that It wasn't being rendered properly because the build failed to get the theme files. After doing more research I concluded that it has to do with the fact that the theme is not stored within the website Git repo, but is a Git submodule.

In an attempt to solve this, I added a step to the build pipeline to grab the submodule files using:

git submodule init
git submodule update

but Cloud Builds triggered from GitHub don't have access to the .git directory within the repo and these commands will fail. The best workaround I could find was to mirror the GitHub repo into a Cloud Source Repository. This is already getting more complicated than I had hoped, but the show must go on!

GCP Set Up

At this point, I had arrived what I thought was a viable plan that I just needed to execute.

NOTE: The commands that follow use $PROJECT_ID and other template variables that should reflect the relevant project and values.

export PROJECT_ID=my-awesome-project-1234

Video Walkthrough

To accompany this article, I also created a full video walkthrough of setting up the pipeline. Feel free to follow along or skip it depending on if you prefer the video or written format!

1) Mirroring GitHub Repo to Cloud Source

As noted above, I needed to mirror the GitHub repo for my website into a Cloud Source repo. I couldn't find a good way to script this, so I just followed the guidance in this Google post: Mirroring a GitHub repository

2) Enable the Cloud Build API

To use Cloud Build I had to enable the cloud build API using:

gcloud services enable cloudbuild.googleapis.com --project=$PROJECT_ID

3) Add IAM Roles for Cloud Build Service Account

When the Cloud Build API is enabled, a service account of the format projectNumber@cloudbuild.gserviceaccount.com is granted some IAM roles, but for this build pipeline the service account also needs the following two roles:

Compute Instance Admin (v1)
Service Account User

which I accomplished with the following commands:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
  --role roles/compute.instanceAdmin.v1
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
  --role roles/iam.serviceAccountUser

4) Creating Cloud Build Trigger

With all of the prerequisite configurations in place, it was then time to create the Cloud Build trigger:

export IMAGE_NAME=my-hugo-caddy-docker-image
export INSTANCE_NAME=my-f1-micro-instance
export ZONE=us-central1-a
gcloud beta builds triggers create cloud-source-repositories \
  --project=$PROJECT_ID \
  --repo=my-cloud-source-repo-mirroring-a-github-repo \
  --branch-pattern=master \
  --build-config=cloudbuild.yaml \
  --substitutions=_IMAGE_NAME=$IMAGE_NAME,_SSH_STRING=$USER@$INSTANCE_NAME,_ZONE=$ZONE,_HOME=/home/$USER

The --substitutions represent template variables that get used in the pipeline definition as will become apparent below.

Creating the Cloud Build Pipeline (cloudbuild.yaml)

At this point, Cloud Build was primed and ready to go... but I hadn't told it what to do yet. The actual build pipeline is defined within a cloudbuild.yaml file at the root of the repository.

Build Configuration Documentation

Each step in this pipeline takes in a container image as its "name" and can also take optional arguments which are executed inside that container.

Below I have broken down this pipeline into its 6 steps:

1) Initialize and Update the Submodules

As mentioned above, one of the initial challenges using Cloud Build was it failing to get the files associated with the Hugo theme because they are located in a git submodule. The following uses the git cloud-builders image to initialize and update the git submodules, resulting in the theme files being available for future steps:

steps:
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - -c
  - |
    git submodule init
    git submodule update

2) Build the Hugo Site

The whole reason for needing a build step is that only the content source files are version controlled (not the generated site files). This step runs the Hugo generator. I couldn't find a publicly available container image that was compatible with Cloud Build, so I created my own (based on this example) and posted it to DockerHub: https://hub.docker.com/r/sidpalas/cloud-builder-hugo.

# build hugo site
- name: 'sidpalas/cloud-builder-hugo:0.64.1'

3) Build + Push the Caddy Container Image

This step builds the website container image and pushes it to Google Container Registry. It uses the COMMIT_SHA (which is populated automatically by Cloud Build based on the triggering commit) to tag the image.

- name: 'gcr.io/cloud-builders/docker'
  # Overriding entrypoint to allow for running two docker commands
  entrypoint: 'bash'
  args: 
    - -c
    - |
      docker build -t gcr.io/$PROJECT_ID/$_IMAGE_NAME:$COMMIT_SHA . &&
      docker push gcr.io/$PROJECT_ID/$_IMAGE_NAME:$COMMIT_SHA

4) Stop Running Containers & Start New Container

With the new container available in GCR, the pipeline stops any running containers and then starts the new container using a gcloud container image to execute a gcloud ssh command on the VM.

- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - compute
  - ssh
  - $_SSH_STRING
  - --project=$PROJECT_ID
  - --zone=$_ZONE
  - --
  - docker container stop $(docker container ls -aq) && 
  - docker container rm $(docker container ls -aq) &&
  - docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v $_HOME/.caddy:/root/.caddy gcr.io/$PROJECT_ID/$_IMAGE_NAME:$COMMIT_SHA

NOTE: Pushing Images

Normally, the build configuration would have an images: section specifying which container images should be pushed to GCR. Because step #3 already tagged and pushed the container image, it is not necessary to include an images section:

images:
- 'gcr.io/$PROJECT_ID/$_IMAGE_NAME:$COMMIT_SHA'

Closing Thoughts

This process ended up being much more complex than I had initially hoped. The issues with Git submodules, and having to create my own Hugo builder image made it take much longer than I expected. That being said, it was quite satisfying when I got the configuration dialed in and the long stretch of red builds finally turned green.

While I am happy with the end result, I can't help but think using Circle CI might have been a smoother process. Perhaps sometime down the the road I'll attempt setting up an equivalent pipeline there and see how that goes!