Add automated tests

We need automated tests for this.

Many will need to be end-to-end, requiring an authentication technique
Each template will need to be tested

Claude suggestions:

Research: Automated Testing Approaches for coder-ddev and code.ddev.com

I've researched testing patterns across the Coder ecosystem and adjacent projects (coder/registry, DDEV add-ons, terraform test framework). Here's a comprehensive breakdown.

Current State

The repo has some testing infrastructure already but no CI pipelines:

make test — runs ddev --version, docker --version, node --version inside the built image. Validates binary presence only, not functional behavior.
drupal-core/scripts/test-issue-branches.sh — manual script that tests Drupal issue branch + composer resolution across D10/D11/D12. Runs locally, not in CI.
drupal-core/scripts/create-test-workspaces.sh — creates a matrix of test Coder workspaces (5 configurations covering D10/D11/D12 + issue forks). Has --create, --check, --delete modes. Also manual, not in CI.
No .github/workflows/ — no automated CI runs on push/PR.
bats-core, bats-support, bats-assert are already installed in the Docker image (via Homebrew) — unused.

Testing Layers

A comprehensive approach needs four layers. Each has different tooling and infrastructure requirements.

Layer 1: Static Validation (no live systems required)

terraform validate and terraform fmt --check should run on every PR. These catch HCL syntax errors and formatting issues.

# .github/workflows/validate.yml
- uses: hashicorp/setup-terraform@v3
- run: terraform init -backend=false
  working-directory: user-defined-web
- run: terraform validate
  working-directory: user-defined-web
- run: terraform fmt -check -recursive

Also useful: tflint for opinionated Terraform linting (deprecated resource arguments, required provider versions, etc.).

Layer 2: Terraform Unit Tests (no live Coder needed)

Terraform's native test framework (added in v1.6, mocking in v1.7) supports .tftest.hcl files that can run against mock providers. This is extremely useful for validating variable validation rules, required resource structure, and output correctness without requiring a live Coder deployment.

Example user-defined-web/tests/validate.tftest.hcl:

mock_provider "coder" {}
mock_provider "docker" {}

variables {
  workspace_image_registry = "index.docker.io/ddev/coder-ddev"
  image_version = "test"
}

run "workspace_creates_required_resources" {
  command = plan
  assert {
    condition     = length(docker_container.workspace) > 0
    error_message = "workspace container resource must be created"
  }
}

run "rejects_empty_image_version" {
  command = plan
  variables {
    image_version = ""
  }
  expect_failures = [var.image_version]
}

# Run all template tests
terraform test -verbose

This approach is used by the coder/registry modules and runs in ~seconds on any Linux/macOS machine.

Layer 3: Shell Script Unit Tests (Docker required, no Sysbox)

The startup script logic in template.tf (the inline startup_script) can be extracted into a testable shell script and tested with bats-core, which is already installed in the image. The DDEV project uses this pattern extensively for their add-on tests (ddev/github-action-add-on-test).

What can be tested with bats (mocking external binaries that aren't available in a normal CI runner):

DDEV global_config.yaml is correctly placed at ~/.ddev/global_config.yaml
The startup script sets expected environment variables (DDEV_NONINTERACTIVE, DDEV_NO_INSTRUMENTATION, etc.)
File copy logic from /home/coder-files/ to /home/coder/ works
WELCOME.txt is present after initialization
npm global bin path is in $PATH
Git SSH wrapper is configured correctly

Example tests/startup.bats:

#!/usr/bin/env bats
load 'test_helper/bats-support/load'
load 'test_helper/bats-assert/load'

setup() {
  export HOME="$(mktemp -d)"
  mkdir -p "$HOME/.ddev" /home/coder-files/.ddev
  cp image/scripts/.ddev/global_config.yaml /home/coder-files/.ddev/
}

@test "ddev global_config.yaml is placed correctly" {
  run bash -c "source startup_functions.sh && setup_ddev_config"
  assert_success
  assert [ -f "$HOME/.ddev/global_config.yaml" ]
}

@test "WELCOME.txt is present after init" {
  run bash -c "source startup_functions.sh && copy_home_files"
  assert_success
  assert [ -f "$HOME/WELCOME.txt" ]
}

This requires refactoring the inline startup script into sourced functions — which would also improve maintainability of the template itself.

Layer 4: Integration Tests (live Coder + Sysbox host required)

This is the most valuable but also most constrained layer. It requires:

A running Coder deployment (code.ddev.com or a dedicated test instance)
A machine with Sysbox installed as the Coder node (standard GitHub Actions runners don't support Sysbox)
A Coder machine token for CI authentication

The canonical Coder CI pattern uses the official coder/setup-action GitHub Action:

# .github/workflows/integration-test.yml
name: Integration Tests
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  test-user-defined-web:
    runs-on: [self-hosted, sysbox]  # Requires self-hosted runner with Sysbox
    env:
      TEMPLATE: user-defined-web
      WORKSPACE: ci-test-${{ github.run_id }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: coder/setup-action@v1
        with:
          access_url: ${{ vars.CODER_URL }}
          coder_session_token: ${{ secrets.CODER_SESSION_TOKEN }}

      # Push template as inactive (won't affect existing workspaces)
      - run: |
          cp VERSION $TEMPLATE/VERSION
          coder templates push $TEMPLATE --activate=false \
            --name ci-${{ github.sha }} --yes \
            --variable workspace_image_registry=index.docker.io/ddev/coder-ddev
          
      # Create a test workspace from this specific version
      - run: |
          coder create $WORKSPACE \
            --template $TEMPLATE \
            --template-version ci-${{ github.sha }} \
            --yes

      # Wait for startup and verify
      - run: coder ssh $WORKSPACE --wait=yes -- ddev --version
      - run: coder ssh $WORKSPACE --wait=yes -- docker ps
      - run: coder ssh $WORKSPACE --wait=yes -- bash -c "test -f ~/.ddev/global_config.yaml"
      - run: coder ssh $WORKSPACE --wait=yes -- bash -c "ddev config --project-type=php --docroot=web && ddev start"

      # Promote on success
      - run: |
          coder templates versions promote \
            --template $TEMPLATE \
            --template-version ci-${{ github.sha }}

      - if: always()
        run: coder delete $WORKSPACE --yes

The key insight from Coder's docs on template testing: push with --activate=false, test against that specific version, then promote only on success. This prevents broken templates from affecting existing workspaces.

Authentication setup: Create a dedicated CI user and machine token:

coder users create --email ci@code.ddev.com --username ci-bot --login-type none
coder tokens create --user ci-bot --lifetime 8760h
# Store the token as CODER_SESSION_TOKEN in GitHub Actions secrets

What Tests to Write, Per Template

`user-defined-web`

Test	Layer	Verifies
`terraform validate`	Static	HCL syntax is valid
Variable validation (mock provider)	Terraform unit	Required variables enforced
`make test` (binary versions)	Image	ddev, docker, node present
startup script copies home files	Shell (bats)	`/home/coder-files/` → `/home/coder/` works
`ddev --version` after startup	Integration	DDEV is functional in workspace
`docker ps` after startup	Integration	DinD daemon is running
`~/.ddev/global_config.yaml` exists	Integration	DDEV config is initialized
`ddev config && ddev start` (PHP)	Integration	DDEV can start a project

`drupal-core`

This template has the most complex logic (issue branch detection, seed DB import, Composer resolution). The existing test-issue-branches.sh and create-test-workspaces.sh scripts should be the basis for CI tests here.

Test	Layer	Verifies
`terraform validate`	Static	HCL syntax valid
Issue branch parameter validation	Terraform unit	Branch format validated
Drupal project cloned and configured	Integration	`~/drupal/web/index.php` exists
Seed DB imported successfully	Integration	`ddev drush status` shows DB connected
Issue fork branches resolvable	Integration	Composer resolves issue branch
Works across D10, D11, D12	Integration	Matrix across `drupal_version` values
DDEV site accessible via port-forward	Integration	HTTP 200 from `ddev describe` URL

For the Drupal issue branch tests specifically, a matrix strategy is ideal:

strategy:
  matrix:
    drupal_version: ["10", "11", "12"]
    issue_branch: ["0", "3001234-fix-something"]
  fail-fast: false

`freeform`

Lighter template. Primarily needs the Traefik/routing behavior verified.

Test	Layer	Verifies
`terraform validate`	Static	HCL valid
Container starts and agent connects	Integration	Workspace reaches `running` state
Traefik routing works	Integration	Coder port-forward returns HTTP response

The Sysbox Constraint

The hardest part of full integration testing is that Sysbox cannot run on standard GitHub Actions runners. Sysbox requires kernel-level changes (installation on the host via .deb package) and cannot be installed at CI runtime without a privileged runner, which GitHub doesn't offer.

Options:

Self-hosted GitHub Actions runner on code.ddev.com's Sysbox-capable node — registers as runs-on: [self-hosted, sysbox]. This is the cleanest path. The runner creates real workspaces on the production or staging Coder deployment.
Scheduled tests on a dedicated staging Coder instance — run integration tests on a schedule (nightly) rather than on every PR. Use workflow_dispatch + a cron trigger. Keeps production clean.
Separate "canary" workspace — keep a persistent test workspace up, run verification checks against it on a schedule via coder ssh.
GitHub Actions with alternative DinD — for tests that don't need the full Sysbox stack (e.g., testing that DDEV works with a regular Docker daemon), use ddev/github-action-setup-ddev in a standard ubuntu-latest runner.

Suggested Phased Implementation

Phase 1 (no infrastructure needed): Add .github/workflows/validate.yml with terraform validate and terraform fmt --check for all three templates. Add terraform test with mock providers for variable validation.

Phase 2 (Docker only): Extract startup script into testable functions. Write bats tests for file initialization, DDEV config placement, PATH setup. Run in standard ubuntu-latest with Docker available.

Phase 3 (self-hosted runner): Set up a self-hosted GitHub Actions runner on the Sysbox-capable host. Add integration test workflow that creates/verifies/deletes test workspaces. Start with user-defined-web, add drupal-core matrix tests second.

Phase 4 (drupal-core specific): Move test-issue-branches.sh logic into the integration test workflow. Run the existing workspace creation matrix automatically on drupal-core template changes.

References

Coder: Test Templates Through CI/CD
Coder: Change Management / GitOps
coder/setup-action
coder/registry CONTRIBUTING.md — TypeScript + Terraform test patterns
coder/terraform-provider-coderd — Terraform-native template management
Terraform Language Tests
bats-core
ddev/github-action-setup-ddev
ddev/github-action-add-on-test — DDEV bats testing patterns

Finalized Implementation Plan

This comment captures the agreed plan from design discussion. Decisions made:

All CI testing runs against staging-coder.ddev.com only — production (code.ddev.com) is never touched by CI
No startup script extraction into separate files — that refactor comes after test coverage exists
drupal-core seed cache is being removed in refactor: simplify drupal-core seed cache (unnecessary complexity) #99; plain-vanilla Drupal tests (::10, ::11, ::12) will be added to CI once that lands. Until then, Phase 5 focuses on issue-fork tests.

Infrastructure Prerequisites (one-time, on staging-coder.ddev.com)

Before any GitHub Actions CI runs against staging:

Register a self-hosted GitHub Actions runner on the staging host, tagged sysbox. Runs on the Sysbox-capable node, invokes coder CLI against the local Coder deployment.

Create a CI bot Coder user on staging:

coder users create --email ci@staging-coder.ddev.com --username ci-bot --login-type none
coder tokens create --user ci-bot --lifetime 8760h

Add GitHub Actions secrets to this repo:
- STAGING_CODER_URL = https://staging-coder.ddev.com
- STAGING_CODER_SESSION_TOKEN = token from step 2

Phase 1 — Static HCL Validation (no infrastructure required)

New files: .github/workflows/validate.yml
Makefile additions: make validate, make fmt-check

Runs terraform validate + terraform fmt -check -recursive for all three templates on every PR and push to main. Uses ubuntu-latest — no credentials, no Coder, no Docker needed.

Risk: Zero.

Phase 2 — Terraform Mock Unit Tests (no infrastructure required)

New files: user-defined-web/tests/validate.tftest.hcl, drupal-core/tests/validate.tftest.hcl, freeform/tests/validate.tftest.hcl

Uses Terraform 1.7+ mock providers (available via hashicorp/setup-terraform@v3). Validates that required variables reject empty/missing values and that mandatory resources (docker_container, coder_agent) appear in the plan. Added to the same workflow as Phase 1.

Trigger: Every push/PR.
Risk: Zero.

Phase 3 — Template Push Validation to Staging (`--activate=false`)

New files: .github/workflows/staging-validate.yml

Pushes all three templates to staging-coder.ddev.com with --activate=false. Catches Coder-provider-specific errors that terraform validate cannot (provider version compatibility, resource field validation against the live API). Cleans up the inactive test version after each run.

Trigger: Push to main, or workflow_dispatch. Runs on ubuntu-latest — only needs coder CLI + the staging credentials. No Sysbox.
Risk: Low — no workspaces created, no impact on existing workspaces.

Phase 4 — Integration Tests: `user-defined-web` and `freeform`

New files: .github/workflows/integration-test.yml

Runs on the self-hosted sysbox runner. Creates a workspace from the current template version, verifies it, then deletes it. Cleanup runs in if: always() to prevent orphaned workspaces.

Verifications for user-defined-web:

coder ssh $WS --wait -- ddev --version
coder ssh $WS --wait -- docker ps
coder ssh $WS --wait -- test -f ~/.ddev/global_config.yaml
coder ssh $WS --wait -- bash -c "mkdir /tmp/t && cd /tmp/t && ddev config --project-type=php --docroot=web && ddev start && ddev stop"

Verifications for freeform: Workspace reaches running state, agent connects, port-forward returns HTTP 200.

Trigger: Push to main, workflow_dispatch, or scheduled nightly (cron: '0 3 * * *').
Risk: Medium — creates/deletes real workspaces on staging. Always-cleanup step is critical.

Phase 5 — Integration Tests: `drupal-core`

Dependency: #99 (remove seed cache) should land first — it eliminates the cache_path required variable and all the cache-hit/miss branching, making this template straightforward to test.

New files: .github/workflows/drupal-integration-test.yml

Test matrix:

Test	`drupal_version`	`issue_fork`
Plain D10	10	(empty)
Plain D11	11	(empty)
Plain D12	12	(empty)
Issue fork D10	10	`3380334`
Issue fork D11	11	`3515218`

fail-fast: false so failures in one cell don't cancel others.

Verifications:

coder ssh $WS --wait -- ddev drush status
coder ssh $WS --wait -- ddev describe

Plain installs take 5–10 minutes (no cache, full composer create-project). Total matrix runtime ~30–45 min.

Trigger: Scheduled nightly only (too slow for per-PR), plus workflow_dispatch. Changes to drupal-core/** also trigger.
Risk: Medium — longer running, multiple concurrent workspaces. Nightly schedule with always-cleanup is the right tradeoff.

Order of Work

Set up staging self-hosted runner + CI bot user + GitHub secrets
Phase 1: HCL validate + fmt workflow (immediate value, zero risk)
Phase 2: Terraform mock tests (same workflow, zero risk)
Phase 3: Staging push validation (needs secrets only)
Phase 4: Integration tests for user-defined-web + freeform
Land refactor: simplify drupal-core seed cache (unnecessary complexity) #99 (remove drupal-core seed cache)
Phase 5: drupal-core nightly matrix tests

ci: add HCL validate/fmt workflow and Makefile targets for #71 (Phase…

ci: add Terraform mock unit tests for all templates for #71 (Phase 2) (…

ci: add staging push validation workflow for #71 (Phase 3)

ci: add staging push validation workflow for #71 (Phase 3) (#103)

ci: add make archive-inactive-versions and pull_request trigger for #71

ci: add drupal-core matrix, unzip doc, and vscode_extensions param for …

ci: cancel stale runs and document parallel runner setup for #71 (Pha…

ci: fix --wait flag syntax in coder ssh for #71 (Phase 4)

ci: fix tilde expansion in remote coder ssh command for #71 (Phase 4)

ci: fix drupal-core params, freeform global_config race, and unzip do…

ci: fix startup race for global_config and drupal-core params for #71 …

fix: create ddev global_config.yaml early via ddev config global for #71

ci: use system coder binary instead of coder/setup-action for #71 (Ph…

ci: use bash heredoc with ddev start stdin fix for #71 (Phase 4)

ci: fix coder ssh heredoc+PTY hang — use scp to push script file for #71

ci: add integration test workflow for user-defined-web, freeform, dru…

ci: run drupal integration tests on pull requests, for #71

ci: fix template version name collision in drupal-plain matrix, for #71

ci: fix web root path, add branch/URL verification, for #71

Status update — remaining work

Done (this branch, PR #113)

Phase 1–3: HCL validate, Terraform mock tests, staging push validation
Phase 4: Integration tests for user-defined-web and freeform
Phase 5: drupal-core integration tests — plain D11/D12 matrix plus issue fork job
Issue fork job derives branch and Drupal version automatically from drupal.org/GitLab APIs; only DRUPAL_TEST_ISSUE_FORK repo variable needed
External URL verification for Drupal (via drupal-site--... coder_app proxy) and user-defined-web (via ddev-web--... proxy)
Issue branch verification in the fork job
All jobs now run on PRs

Remaining

freeform integration test improvements — deferred to "freeform" template struggles with port 80, switch to 8080 #87
drupal_version parameter validation — enforced by the Coder API (coder_parameter options), not Terraform; cannot be tested at the terraform test layer. No action needed.
Runner count — TODO comment left in integration-test.yml; revisit if more matrix cells or templates are added

Open question: do we need self-hosted runners at all?

It's possible none of the integration tests actually require a self-hosted runner. We do need a target Coder server (staging-coder.ddev.com) with Sysbox-capable provisioners — but the GitHub Actions jobs themselves only run coder CLI commands (coder templates push, coder create, coder ssh, curl). All of that could run on a standard ubuntu-latest GitHub-hosted runner as long as it has credentials to reach staging-coder.ddev.com.

The self-hosted runner was assumed necessary because of the Sysbox requirement, but Sysbox runs on the Coder provisioner host (the server where workspaces are created), not on the runner. The runner just talks to the Coder API.

If that's right, we could drop the runs-on: [self-hosted, sysbox] requirement entirely and use ubuntu-latest, which would simplify setup significantly (no runner registration, no runner maintenance).

Worth verifying before investing more time in runner infrastructure.

ci: fix tilde expansion in coder ssh commands, for #71

ci: fix DDEV test step gating and cleanup command, for #71

ci: fix branch carriage return and curl redirect for Drupal URL check…

fix: pass Coder session token for drupal-site URL check, add diagnost…

ci: add drupal integration test workflow with issue fork support, for #…

This is mostly where it should be. Keeping it open for suggestions about future basics.

In great shape and very useful, closing

Navigation Menu

Repository navigation

Add automated tests #71

Research: Automated Testing Approaches for coder-ddev and code.ddev.com

Current State

Testing Layers

Layer 1: Static Validation (no live systems required)

Layer 2: Terraform Unit Tests (no live Coder needed)

Layer 3: Shell Script Unit Tests (Docker required, no Sysbox)

Layer 4: Integration Tests (live Coder + Sysbox host required)

What Tests to Write, Per Template

`user-defined-web`

`drupal-core`

`freeform`

The Sysbox Constraint

Suggested Phased Implementation

References

Finalized Implementation Plan

Infrastructure Prerequisites (one-time, on staging-coder.ddev.com)

Phase 1 — Static HCL Validation (no infrastructure required)

Phase 2 — Terraform Mock Unit Tests (no infrastructure required)

Phase 3 — Template Push Validation to Staging (`--activate=false`)

Phase 4 — Integration Tests: `user-defined-web` and `freeform`

Phase 5 — Integration Tests: `drupal-core`

Order of Work

18 remaining items

Status update — remaining work

Done (this branch, PR #113)

Remaining

Open question: do we need self-hosted runners at all?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add automated tests #71

Description

Activity

rfay commented on Mar 23, 2026

Research: Automated Testing Approaches for coder-ddev and code.ddev.com

Current State

Testing Layers

Layer 1: Static Validation (no live systems required)

Layer 2: Terraform Unit Tests (no live Coder needed)

Layer 3: Shell Script Unit Tests (Docker required, no Sysbox)

Layer 4: Integration Tests (live Coder + Sysbox host required)

What Tests to Write, Per Template

user-defined-web

drupal-core

freeform

The Sysbox Constraint

Suggested Phased Implementation

References

rfay commented on Apr 30, 2026

Finalized Implementation Plan

Infrastructure Prerequisites (one-time, on staging-coder.ddev.com)

Phase 1 — Static HCL Validation (no infrastructure required)

Phase 2 — Terraform Mock Unit Tests (no infrastructure required)

Phase 3 — Template Push Validation to Staging (--activate=false)

Phase 4 — Integration Tests: user-defined-web and freeform

Phase 5 — Integration Tests: drupal-core

Order of Work

18 remaining items

rfay commented on May 1, 2026

Status update — remaining work

Done (this branch, PR #113)

Remaining

Open question: do we need self-hosted runners at all?

rfay commented on May 6, 2026

rfay commented on May 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`user-defined-web`

`drupal-core`

`freeform`

Phase 3 — Template Push Validation to Staging (`--activate=false`)

Phase 4 — Integration Tests: `user-defined-web` and `freeform`

Phase 5 — Integration Tests: `drupal-core`