# Buildkite Test Agent Setup

We are using [Buildkite](<https://buildkite.com/ddev>) for Windows and macOS testing. The build machines and `buildkite-agent` must be set up before use.

## Git Mirrors (all platforms)

Set `git-mirrors-path` in `buildkite-agent.cfg` on every agent. Buildkite keeps a local bare mirror of the repository there, so a wiped or first-time build directory does an incremental fetch from the mirror instead of a fresh clone against GitHub — this avoids the exposure to transient network failures that a full clone has. See [Buildkite’s Git mirrors docs](<https://buildkite.com/docs/agent/self-hosted/configure/git-mirrors>).

`buildkite-agent.cfg` doesn’t expand `$HOME`-style variables, only a leading `~`. Point `git-mirrors-path` at a sibling of that agent’s own `build-path`, matching whichever form (`~` or absolute) `build-path` already uses. To check or fix an existing agent, run the matching one-liner below (idempotent — safe to run again) and restart the agent afterward:

- macOS/Pi, `build-path="~/tmp/buildkite-agent/builds"`:

  ```
  CFG=/usr/local/etc/buildkite-agent/buildkite-agent.cfg; [ -f "$CFG" ] || CFG=/opt/homebrew/etc/buildkite-agent/buildkite-agent.cfg
  grep -q '^git-mirrors-path=' "$CFG" || echo 'git-mirrors-path="~/tmp/buildkite-agent/git-mirrors"' | sudo tee -a "$CFG"
  brew services restart buildkite-agent@4
  ```
- Linux/WSL2 package default, `build-path="/var/lib/buildkite-agent/builds"`:

  ```
  grep -q '^git-mirrors-path=' /etc/buildkite-agent/buildkite-agent.cfg || echo 'git-mirrors-path="/var/lib/buildkite-agent/git-mirrors"' | sudo tee -a /etc/buildkite-agent/buildkite-agent.cfg
  sudo systemctl restart buildkite-agent
  ```
- Windows, `build-path=C:\Users\testbot\tmp\buildkite` (Git Bash):

  ```
  grep -q '^git-mirrors-path=' /c/buildkite-agent/buildkite-agent.cfg || echo 'git-mirrors-path="C:\Users\testbot\tmp\buildkite-git-mirrors"' >> /c/buildkite-agent/buildkite-agent.cfg
  nssm.exe restart buildkite-agent
  ```

Windows agents provisioned with the package default `build-path=C:\buildkite-agent\builds` instead use `git-mirrors-path="C:\buildkite-agent\git-mirrors"` — check `build-path` in the file first.

## Windows Test Agent Setup

1. Create the user “testbot” on the machine. Use the password for `ddevtestbot@gmail.com`, available in 1Password.
2. In admin PowerShell, `wsl --install`.
3. (WSL2/Docker Desktop and traditional Windows only): Install either [Docker Desktop for Windows](<https://docs.docker.com/desktop/release-notes/>) (from the release notes page, which is best maintained) or [Rancher Desktop](<https://rancherdesktop.io/>). With Rancher Desktop, turn off Kubernetes. (Rancher Desktop on Windows appears to be too unreliable to use at this point in its history.)
4. In admin PowerShell, `Set-ExecutionPolicy -Scope "CurrentUser" -ExecutionPolicy "Unrestricted"`.
5. In admin PowerShell, download and run [windows\_buildkite\_start.ps1](<https://ddev.github.io/ddev/en/stable/developers/scripts/windows_buildkite_start.ps1>) with `curl <url> -O windows_buildkite_start.ps1`.
6. Install items as needed with `winget` as the testbot user. Everything but `mkcert` is only required for a traditional Windows test machine, which also needs `nc.exe` (netcat) on the `PATH`; it has no `winget` package. Don’t use Chocolatey, which needs an administrator for upgrades that the unprivileged test jobs can’t do. Don’t install DDEV either: the test jobs build and use the DDEV under test.

   ```
   winget install -e --accept-package-agreements --accept-source-agreements Git.Git jqlang.jq Oracle.MySQL GoLang.Go ezwinports.make FiloSottile.mkcert GnuWin32.Zip OpenJS.NodeJS.LTS Ngrok.Ngrok
   ```
7. After restart, in **administrative** Git Bash window, `Rename-Computer <testbot-win10(home|pro)-<description>-1` and then `export BUILDKITE_AGENT_TOKEN=<token>`.
8. (Traditional Windows test runner only): Download and run [windows\_buildkite\_setup.sh](<https://ddev.github.io/ddev/en/stable/developers/scripts/windows_buildkite_setup.sh>).
9. If using Rancher Desktop, adjust the /c/buildkite-agent/buildkite-agent.cfg file to set `rancher-desktop=true` in the tags instead of `docker-desktop`. If using Docker Desktop, set `docker-desktop=true`. While editing that file, also add `git-mirrors-path` (see [Git Mirrors](<#git-mirrors-all-platforms>)): `echo 'git-mirrors-path="C:\buildkite-agent\git-mirrors"' >> /c/buildkite-agent/buildkite-agent.cfg`.
10. (Traditional Windows test runner only): Download and run [windows\_postinstall.sh](<https://ddev.github.io/ddev/en/stable/developers/scripts/windows_postinstall.sh>).
11. (Traditional Windows or Docker Desktop WSL2 Only) Launch Docker. It may require you to take further actions.

    - Check “Start Docker Desktop when you sign in” or the equivalent with Rancher Desktop.
    - Check “Add the \*.docker.internal names to the host’s /etc/hosts file”
    - Uncheck “SBOM Indexing”
    - Under “Resources” uncheck “Resource Saver” — **important**: Docker Desktop may re-enable this after a full restart. Verify it is unchecked after any Docker Desktop upgrade or manual restart. When Resource Saver is active, Docker Desktop pauses the Linux VM when no containers are running; `docker ps` and WSL2 integration fail until it is un-paused, causing spurious test failures.
12. After starting Docker Desktop or Rancher Desktop, set the correct Docker context in the Git Bash window with `docker context use desktop-linux` (Docker Desktop) or `docker context use default` (Rancher Desktop).
13. Log into Chrome with the user `ddevtestbot@gmail.com` and enable Chrome Remote Desktop.
14. (Traditional Windows test runner only): Enable `gd`, `fileinfo`, and `curl` extensions in `/c/tools/php*/php.ini`.
15. Set the “Sleep after time” setting in settings to never.
16. Install [winaero tweaker](<https://winaero.com/request.php?1796>) and “Enable user autologin checkbox”. Set up the machine to [automatically log in on boot](<https://www.cnet.com/how-to/automatically-log-in-to-your-windows-10-pc/>). Then run netplwiz, provide the password for the main user, uncheck “require a password to log in”.
17. (Traditional Windows test runner only): Set the `buildkite-agent` service to run as the testbot user and use delayed start: Choose “Automatic, delayed start” and on the “Log On” tab in the services widget it must be set up to log in as the testbot user, so it inherits environment variables and home directory (and has testbot Git config, etc).
18. (Traditional Windows test runner only): `git config --global --add safe.directory '*'`.
19. (Traditional Windows test runner only): Manually run `testbot_maintenance.sh`, `curl -sL -O https://raw.githubusercontent.com/ddev/ddev/main/.buildkite/testbot_maintenance.sh && bash testbot_maintenance.sh`.
20. (Traditional Windows test runner only): Run `.buildkite/sanetestbot.sh` to check your work.
21. (Traditional Windows test runner only): Reboot the machine and do a test run. (On Windows, the machine name only takes effect on reboot.)
22. (Traditional Windows test runner only): Verify that `go`, `git-bash`, and `%USERPROFILE%\go\bin` are in the path.
23. In “Advanced Windows Update Settings” enable “Receive updates for other Microsoft products” to make sure you get WSL2 kernel upgrades. Make sure to run Windows Update to get the latest kernel.
24. Turn off the settings that cause the “windows experience” prompts after new upgrades: ![disable\_windows\_experience](<https://ddev.github.io/ddev/en/stable/images/disable_windows_experience.png>)
25. In PowerShell: `wsl.exe --update`. Watch for the elevation to complete, it may require elevation.

## Both Docker Desktop/WSL2 and Docker-ce/WSL2

1. The Ubuntu distro should be set up with the user `buildkite-agent` and the password used for `ddevtestbot@gmail.com`.
2. Stop Docker Desktop if it is running.
3. Log into Chrome with the user `ddevtestbot@gmail.com` and enable Chrome Remote Desktop.
4. Windows Terminal should be installed. Set “Ubuntu” (or this distro) as the default and have it start on Windows startup. Enable “copy on select” in behaviors.
5. `nc.exe -L -p 9003` on Windows to trigger and allow Windows Defender.
6. For a docker-ce/WSL2 runner (`BUILDKITE_DOCKER_TYPE=wsl2` below), Mirrored Mode is normal: edit `~/.wslconfig` on Windows to add the WSL2 settings below. A Docker Desktop runner (`BUILDKITE_DOCKER_TYPE=dockerforwindows`) keeps the default NAT networking instead, so skip this step for it.

   ```
   [wsl2]
   networkingMode=Mirrored
   [experimental]
   hostAddressLoopback=true
   ```
7. **Before running the setup script**, configure mkcert CAROOT on Windows (PowerShell as the testbot user). This must happen first so that `mkcert -install` in WSL2 installs the Windows CA (not a separate Linux-local one) into the Linux trust store:

   ```
   winget install -e FiloSottile.mkcert   # if not already installed
   mkcert -install
   setx CAROOT (mkcert -CAROOT)
   setx WSLENV "CAROOT/up"
   wsl --terminate Ubuntu    # restart distro so WSLENV takes effect
   ```
8. In the Ubuntu distro:

   1. `export BUILDKITE_AGENT_TOKEN=<token>` with the token from 1Password `BUILDKITE_AGENT_TOKEN`.
   2. `export BUILDKITE_DOCKER_TYPE=dockerforwindows` or `export BUILDKITE_DOCKER_TYPE=wsl2` — this also picks the `os=` agent tag the setup script writes: `wsl2-mirrored` for `wsl2` (docker-ce), plain `wsl2` for `dockerforwindows`, matching step 6 above.
   3. Optionally `export NGROK_TOKEN=<token>` with the `NGROK_TOKEN` from 1Password ngrok.com `nopaid` account.
   4. Run the script [wsl2-test-runner-setup.sh](<https://ddev.github.io/ddev/en/stable/developers/scripts/wsl2-test-runner-setup.sh>) in the Ubuntu distro. This script reads `CAROOT` from the Windows registry via `powershell.exe`, exports it before calling `mkcert -install`, and then creates `/etc/buildkite-agent/hooks/environment` to repeat this for every buildkite-agent job (since systemd does not propagate `WSLENV`). It also sets `git-mirrors-path="/var/lib/buildkite-agent/git-mirrors"` in `/etc/buildkite-agent/buildkite-agent.cfg` (see [Git Mirrors](<#git-mirrors-all-platforms>)).
9. Restart the distro with `wsl.exe -t Ubuntu` and then restart it by opening the Ubuntu window.
10. If using Docker Desktop, start Docker Desktop.
11. In `~/workspace/ddev/.buildkite`, run `./testbot_maintenance.sh`.
12. In `~/workspace/ddev/.buildkite`, run `./sanetestbot.sh` to check your work.

## Windows Installer Test Distros

The Windows GUI installer tests run in their own dedicated Buildkite pipeline (`windows-installer`, driven by `.buildkite/windows-installer.yml` + `.buildkite/installer-test.sh`), separate from the traditional Windows suite. Each matrix case is a parallel job that installs DDEV into a dedicated WSL2 distro using a specific Docker provider.

These tests need a **Windows-native** (`os=windows`) `buildkite-agent` — the same kind of agent as the traditional Windows runner — because they drive `wsl.exe`, `reg.exe`, and the installer `.exe`. The matrix spreads its jobs across all available `os=windows` agents, so adding more such machines increases parallelism.

> [!WARNING]
>
> **One `os=windows` agent per machine**
>
> An installer run mutates global Windows state (registry `CAROOT`/`WSLENV` and the machine-wide `mkcert` CA root). Two installer jobs must never run on the same machine at the same time. A Buildkite agent runs one job at a time, so running exactly **one** `os=windows` agent per physical machine guarantees this; the jobs then only overlap across separate machines.

The test does **not** create distros — it resets and reuses them. Provision the six named instances **once per runner** (the normal install handles user creation; complete first-run with the `testbot` user, which is the default user the test harness expects):

```
wsl --install Ubuntu       --name ddev-test-ubuntu-ce
wsl --install Ubuntu       --name ddev-test-ubuntu-desktop
wsl --install Ubuntu-24.04 --name ddev-test-ubuntu2404-ce
wsl --install Ubuntu-24.04 --name ddev-test-ubuntu2404-desktop
wsl --install Debian       --name ddev-test-debian-ce
wsl --install Debian       --name ddev-test-debian-desktop
```

For the `*-desktop` instances only, enable Docker Desktop’s WSL integration for that instance (Docker Desktop → Settings → Resources → WSL integration). Otherwise the `docker-desktop` cases skip gracefully. The base distro names above (`Ubuntu`, `Ubuntu-24.04`, `Debian`) must be available in the WSL catalog; verify with `wsl -l -o`.

Remove `unattended-upgrades` from all test instances. Ubuntu 24.04 in particular runs it on boot and holds the dpkg lock while the installer is trying to run `apt-get`, causing intermittent failures. It serves no purpose in these short-lived test distros:

```
foreach ($d in @("ddev-test-ubuntu-ce","ddev-test-ubuntu-desktop","ddev-test-ubuntu2404-ce","ddev-test-ubuntu2404-desktop","ddev-test-debian-ce","ddev-test-debian-desktop")) {
    wsl -d $d -u root apt-get remove -y unattended-upgrades 2>$null
}
```

Install [wsl-fix-interop](<https://github.com/rfay/wsl-fix-interop>) in all test instances and configure it to run on every distro boot. WSL interop (binfmt\_misc) loses its `WSLInterop` entry whenever `docker-ce` is removed — the package’s post-remove scripts clear binfmt\_misc entries. Without it, any Windows `.exe` called from within WSL fails with “Exec format error”, causing `mkcert.exe` failures and TLS test failures.

First install the script and sudoers entry:

```
foreach ($d in @("ddev-test-ubuntu-ce","ddev-test-ubuntu-desktop","ddev-test-ubuntu2404-ce","ddev-test-ubuntu2404-desktop","ddev-test-debian-ce","ddev-test-debian-desktop")) {
    wsl -d $d -u root bash -c "cd /tmp && curl -fsSL https://raw.githubusercontent.com/rfay/wsl-fix-interop/main/wsl-fix-interop -o wsl-fix-interop && curl -fsSL https://raw.githubusercontent.com/rfay/wsl-fix-interop/main/install.sh -o install.sh && bash install.sh testbot"
}
```

Then wire it into `/etc/wsl.conf` in each distro so it re-registers automatically on every distro boot (including after `docker-ce` removal clears binfmt\_misc):

```
foreach ($d in @("ddev-test-ubuntu-ce","ddev-test-ubuntu-desktop","ddev-test-ubuntu2404-ce","ddev-test-ubuntu2404-desktop","ddev-test-debian-ce","ddev-test-debian-desktop")) {
    wsl -d $d -u root bash -c "grep -q 'wsl-fix-interop' /etc/wsl.conf 2>/dev/null || printf '\n[boot]\ncommand = /usr/local/sbin/wsl-fix-interop\n' >> /etc/wsl.conf"
}
```

To run a single case manually (e.g. for debugging), select it by instance name, which is also the Go subtest name:

```
make testwininstaller TESTARGS="-run TestWindowsInstallerWSL2/ddev-test-debian-ce"
```

### WSL2 install-script tests

The `ddev-windows-installer` pipeline also tests the manual WSL2 install PowerShell scripts (`scripts/install_ddev_wsl2_docker_inside.ps1` and `scripts/install_ddev_wsl2_docker_desktop.ps1`) against current Ubuntu, as the `ps1-docker-inside` and `ps1-docker-desktop` matrix cases. These run only when a ps1 script (or its test/plumbing) changes, plus on `main` and manual builds.

They **reuse the two Ubuntu instances** above — no extra provisioning. The scripts now accept a `-Distro` parameter so the test passes the distro name directly without changing the system default:

- `ps1-docker-inside` → `ddev-test-ubuntu-ce`, which must have Docker Desktop WSL integration **disabled** (the script aborts if `/mnt/wsl/docker-desktop` exists). This is the same state the docker-ce installer case wants.
- `ps1-docker-desktop` → `ddev-test-ubuntu-desktop`, which must have Docker Desktop running with WSL integration **enabled**.
- `mkcert.exe` must be on the Windows PATH (already installed with `winget` above).

Run one manually with:

```
make testwsl2scripts TESTARGS="-run TestWSL2InstallScripts/docker-inside"
```

## Icinga2 monitoring setup for WSL2 instances

1. Icinga Director web UI, configure the host on `monitor.ddev.com`, normally making a copy of an existing identical item.
2. Deploy the new host using Icinga Director.
3. `sudo icinga2 node wizard` to configure the agent, see [docs](<https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#agentsatellite-setup-on-linux>).

   ```
   buildkite-agent@tb-wsldd-16:~$ sudo icinga2 node wizard
   Welcome to the Icinga 2 Setup Wizard!

   We will guide you through all required configuration details.

   Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]:

   Starting the Agent/Satellite setup routine...

   Please specify the common name (CN) [tb-wsldd-16.localdomain]: tb-wsldd-16

   Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
   Master/Satellite Common Name (CN from your master/satellite node): monitor.ddev.com

   Do you want to establish a connection to the parent node from this node? [Y/n]: y
   Please specify the master/satellite connection information:
   Master/Satellite endpoint host (IP address or FQDN): monitor.ddev.com
   Master/Satellite endpoint port [5665]:

   Add more master/satellite endpoints? [y/N]:
   Parent certificate information:

   Version:             3
   Subject:             CN = monitor.ddev.com
   Issuer:              CN = Icinga CA
   Valid From:          Feb 22 22:34:09 2026 GMT
   Valid Until:         Mar 26 22:34:09 2027 GMT
   Serial:              02:0e:20:7e:a1:a9:f0:b4:8d:07:63:49:46:fb:d7:90:29:fd:c4:bc

   Signature Algorithm: sha256WithRSAEncryption
   Subject Alt Names:   monitor.ddev.com
   Fingerprint:         4A 99 90 F6 F4 F7 F9 20 1B DD 9D 51 EE 86 50 C7 BB 38 09 9B B0 40 4A E0 52 C9 52 2D A1 B8 72 D5

   Is this information correct? [y/N]: y

   Please specify the request ticket generated on your Icinga 2 master (optional).
   (Hint: # icinga2 pki ticket --cn 'tb-wsldd-16'): c5b9fa649792abcde3941862a664be0fe8d126
   Please specify the API bind host/port (optional):
   Bind Host []:
   Bind Port []:

   Accept config from parent node? [y/N]: y
   Accept commands from parent node? [y/N]: y

   Reconfiguring Icinga...

   Local zone name [tb-wsldd-16]:
   Parent zone name [master]:

   Default global zones: global-templates director-global
   Do you want to specify additional global zones? [y/N]:

   Do you want to disable the inclusion of the conf.d directory [Y/n]:
   Disabling the inclusion of the conf.d directory...

   Done.

   Now restart your Icinga 2 daemon to finish the installation!
   buildkite-agent@tb-wsldd-16:~$ sudo systemctl restart icinga2
   ```
4. Restart `sudo systemctl restart icinga2`
5. On `monitor.ddev.com` edit `/usr/local/bin/check_buildkite_agents.sh` to include the hostname of the new instance.

## macOS Docker Desktop Test Agent Setup (Apple Silicon)

1. Create the user “testbot” on the machine. Use the password for `ddevtestbot@gmail.com`, available in 1Password.
2. Change the name of the machine to something in keeping with current style, perhaps `testbot-macos-arm64-8`. This is done in **Settings** → **General** → **About** → **Name** and in **Sharing** → **Computer Name** and in **Sharing** → **Local Hostname**.
3. Download and install Chrome and log the browser into the account used for test runners. It will pick up the Chrome Remote Desktop setup as a result. Configure Chrome Remote Desktop to serve. When this is done, the machine will be available for remote access and most other tasks can be done using Chrome Remote Desktop.
4. The machine should be on the correct network and have a static IP handed out by DHCP. IP addresses are listed in /etc/hosts on `pi.ddev.site`, so this one should be added.
5. Power should be set up as in ![macos power settings](<https://ddev.github.io/ddev/en/stable/images/macos_power_settings.png>).
6. Auto login should be set up as in ![macos users and groups](<https://ddev.github.io/ddev/en/stable/images/macos_users_and_groups.png>), see [automatically log in on boot](<https://support.apple.com/en-us/HT201476>).
7. Remote login should be enabled as in ![macos remote login](<https://ddev.github.io/ddev/en/stable/images/macos_remote_login.png>).
8. Automatic updates should be set to mostly security only as in ![macos automatic\_updatees](<https://ddev.github.io/ddev/en/stable/images/macos_automatic_updates.png>).
9. Set the time zone to US MT (nearest city: Denver, Colorado).
10. `sudo mkdir -p /usr/local/bin && sudo chown -R testbot /usr/local/bin`
11. Install [Homebrew](<https://brew.sh/>) `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"`
12. After installing Homebrew follow the instructions it gives you at the end to add brew to your PATH.
13. Install everything you’ll need with `brew install buildkite/buildkite/buildkite-agent@4 bats-core composer git golang jq mysql-client@8.0 mkcert netcat p7zip  && brew install --cask docker iterm2 ngrok && brew link --force mysql-client`.
14. Run `ngrok authtoken <token>` with token for free account from 1Password.
15. Run `mkcert -install`.
16. If Docker Desktop will be deployed, run Docker manually and go through its configuration routine.
17. If OrbStack will be deployed, install it from [orbstack.dev](<https://orbstack.dev>).

    - Install with Docker only.
    - Click “Sign in” in the lower left to sign in with OrbStack credentials (normal test runner gmail address; it will receive an email with a login code).
    - Configure it to automatically start and download updates, see ![OrbStack configuration](<https://ddev.github.io/ddev/en/stable/images/orbstack_configuration.png>).
18. If Rancher Desktop will be deployed, install it.

    - Turn off kubernetes.
    - Set virtualization to VZ
    - Set mount type to VirtioFS
19. Run iTerm. You may need to allow full disk access permissions.
20. Run `mkdir ~/workspace && cd ~/workspace && git clone https://github.com/ddev/ddev`.
21. `git config --global --add safe.directory '*'`.
22. Edit `/usr/local/etc/buildkite-agent/buildkite-agent.cfg` or `/opt/homebrew/etc/buildkite-agent/buildkite-agent.cfg` to add

    - the agent `token` (from [agents tab](<https://buildkite.com/organizations/ddev/agents>), “Reveal Agent Token”).
    - the agent `name` (the name of the machine).
    - `tags`, like `"os=macos,architecture=arm64,osvariant=sonoma,dockertype=dockerformac,rancher-desktop=true,orbstack=true,docker-desktop=true"`
    - `build-path="~/tmp/buildkite-agent/builds"`
    - `git-mirrors-path="~/tmp/buildkite-agent/git-mirrors"` (see [Git Mirrors](<#git-mirrors-all-platforms>))
23. Run `brew services start buildkite-agent@4`.
24. Run `bash ~/workspace/ddev/.buildkite/testbot_maintenance.sh`.
25. Run `bash ~/workspace/ddev/.buildkite/sanetestbot.sh` to check your work.
26. The `testbot` user’s SSH account is used for monitoring, so `ssh-keygen` and then add the public key `id_testbot` from 1Password to `~/.ssh/authorized_keys` and `chmod 600 ~/.ssh/authorized_keys`.
27. Add the new machine to Icinga by copying an existing Icinga service to the new one. This is done in **Icinga Director** → **Services** → **Single Services** → **Select a Service** → **Clone** → **Deploy**. The new service has to have `by-ssh-address` set to the name of the test runner, and that address needs to be added to `pi.ddev.site`’s `/etc/hosts` file.
28. If `zsh` is the shell configured, add `/etc/zshenv` so that `/usr/local/bin/docker` will be picked up:

    ```
    PATH=$PATH:/usr/local/bin:/opt/homebrew/bin
    ```
29. In macOS Settings visit “full disk access” and grant access to `buildkite-agent`, `docker`, `iterm`, `orbstack`. This may prevent startup modal dialogs that prevent `buildkite-agent` or `docker` from continuing properly.

## Additional Colima macOS setup

1. `brew install colima`
2. `colima start vz --cpu 4 --memory 6 --disk 60 --vm-type=vz --mount-type=virtiofs --dns=1.1.1.1`
3. `colima stop vz`

Then the Buildkite agent must be configured with tags `colima_vz=true`.

## Additional Lima macOS setup

1. `limactl create --name=lima-vz --vm-type=vz --mount-type=virtiofs --mount-writable --mount="~/:w" --memory=6 --cpus=4 --disk=60 template:docker`
2. `limactl start lima-vz`
3. `docker context use lima-lima-vz`

Then the Buildkite agent must be configured with tags `lima=true`.

## Additional Podman Rootless macOS Setup

Podman rootless on macOS runs containers in a lightweight VM managed by `podman machine`.

1. Follow the instructions for [Podman on macOS](<https://ddev.github.io/ddev/en/stable/users/install/docker-installation/#macos-podman-rootless>) to install Podman and set up a rootless Podman machine with 6GB of memory. On a testbot whose machine was created with the 2GB default, resize it with `podman machine stop && podman machine set --memory 6144`.
2. Verify connectivity: `docker context use podman-rootless && docker ps` should return an empty container list.
3. Stop the machine when done with initial setup: `podman machine stop`

> [!NOTE]
>
> Rootless containers cannot bind to ports below 1024. `test.sh` handles this automatically by setting `DDEV_XDG_CONFIG_HOME` to an isolated config directory and configuring `router-http-port=8080` and `router-https-port=8443` via `ddev config global`.

Then configure the Buildkite agent with tag `podman-rootless=true` in addition to the standard macOS tags:

```
tags="os=macos,architecture=arm64,osvariant=sonoma,...,podman-rootless=true"
```

The `test.sh` script uses `podman machine start` and `docker context use podman-rootless` to activate the provider, then stops the machine in the cleanup trap on exit.

## Performance Benchmark Harness Prerequisites

The nightly performance benchmark harness (`perf/`, see `perf/README.md`) needs Node.js on every testbot, for the Puppeteer-driven Drupal install metric (`perf/metrics/03-drupal-install/`). On Linux, Puppeteer’s downloaded Chrome build also needs a handful of OS shared libraries a bare testbot image doesn’t ship with.

### WSL2 / Linux testbots (docker-ce or Docker Desktop’s WSL2 backend)

1. Install Node.js 22, matching the version `perf-linux.yml` uses:

   ```
   curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
   sudo apt-get install -y nodejs
   ```
2. Install Chrome’s runtime shared library dependencies. A bare Ubuntu 24.04 image is missing all of these; without them Puppeteer fails the first time it tries to launch Chrome, with an error like `error while loading shared    libraries: libatk-1.0.so.0` (or any of the others below, one at a time as each is fixed):

   ```
   sudo apt-get install -y \
     libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 libasound2t64 \
     libxkbcommon0 libgbm1 libx11-6 libxext6 libxcb1 libcairo2 \
     libpango-1.0-0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
     libatspi2.0-0
   ```

   The `t64` suffix on some of these is Ubuntu 24.04-specific: it renamed several packages during the 64-bit `time_t` ABI transition (`libatk1.0-0` → `libatk1.0-0t64`, etc.). Older or non-Ubuntu distros use the un-suffixed names. This list was derived by running `ldd` against the actual downloaded Chrome binary (`~/.cache/puppeteer/chrome/linux-*/chrome-linux64/chrome`); if a future Chrome version needs more, `ldd <path-to-chrome> | grep "not found"` shows what’s still missing.

### macOS testbots

Add `node` to the `brew install` list in step 13 of the macOS setup above.

### Traditional (native) Windows testbots

Install Node.js with `winget install -e OpenJS.NodeJS.LTS` (see step 6 of Windows Test Agent Setup above). Confirmed working, including headless Chrome launching from the NSSM-installed `buildkite-agent` service context.

## Running Targeted Builds on Specific Pipelines

To test a branch against only selected pipelines (e.g. WSL2 only) or to run a subset of tests without waiting for the full matrix:

1. Push your branch to upstream.
2. In the [Buildkite dashboard](<https://buildkite.com/ddev>), open the pipeline you want (e.g. “wsl2-mirrored”).
3. Click **New Build** and set the branch to your branch name.
4. Expand **Environment Variables** and add:

   ```
   TESTARGS=-run TestCheck -failfast
   ```

   Adjust the `-run` pattern to match the tests you want. Use a common prefix where possible. If you need `|` for alternation, wrap the regular expression in single quotes so the shell does not interpret the pipe as a pipeline operator:

   ```
   TESTARGS=-run 'TestCheckLiveConnectivity|TestCheckMkcertInstallation' -failfast
   ```

   `TESTARGS` defaults to `-failfast` when not set, so include `-failfast` explicitly if you want it.
5. Click **Create Build**.

Use `[skip ci]` in your commit message to prevent all pipelines from triggering automatically when you push, then manually trigger only the pipelines you need.