Skip to content

Release Management & Docker Images

Release process and tools

  • GoReleaser Pro is used to do the actual releasing using .goreleaser.yml. GoReleaser Pro is a licensed product that requires separate installation and a license key, which is in the GitHub Workflow configuration and is available in 1Password to DDEV maintainers who need it.
  • The Main Build/Release GitHub Action does the actual running of the GoReleaser actions and provides the needed secrets.

GitHub Actions Required Secrets

How to add new people to these accounts

  • AUR is Arch Linux User Repository. ddev-bin is at https://aur.archlinux.org/packages/ddev-bin. The current maintainer of this is @rfay, who can add co-maintainers.
  • The chocolatey package. Additional maintainers can be added at (login required) https://community.chocolatey.org/packages/ddev/ManagePackageOwners; they could then create tokens to push it.
  • GitHub requires write access to this repository, either via permissions on the repository or on the org.
  • Apple signing and notarization requires access to the DDEV Foundation group on https://developer.apple.com. It’s easy enough to add additional people.
  • Windows signing is an awkward process that requires a dongle. When the current signing certificate expires we definitely want the simpler approach.
  • Discord
  • Docker

Environment variables required

These are normally configured in the repository environment variables.

  • AUR_EDGE_GIT_URL: The Git URL for AUR edge (normally ddev-edge-bin), for example ssh://aur@aur.archlinux.org/ddev-edge-bin.git.
  • AUR_PACKAGE_NAME: The base name of the AUR package. Normally ddev for production, but ddev-test for testing repository.
  • AUR_STABLE_GIT_URL: The Git URL for AUR stable (normally ddev-bin), for example ssh://aur@aur.archlinux.org/ddev-bin.git.
  • CLOUDSMITH_ORG: The organization on Cloudsmith that receives apt/yum package pushes.
  • CLOUDSMITH_REPO: The repository within CLOUDSMITH_ORG that receives package pushes, so ddev-test/ddev can push to a separate test repository from ddev/ddev.
  • DDEV_IGNORE_EXPIRING_KEYS: If the value is "false" then daily tests will fail if signing keys are expiring soon, default is "false" on ddev/ddev
  • DDEV_MAX_DAYS_BEFORE_CERT_EXPIRATION: Number of days before expiration to warn about expiring signing keys, default is 90 on ddev/ddev.
  • DDEV_WINDOWS_SIGN: If the value is "true" then make will attempt to sign the Windows executables, which requires building on our self-hosted Windows runner.
  • DISABLE_NOTARIZATION: If the value is "true" then notarization of macOS apps will be skipped, otherwise notarization is attempted.
  • DOCKERHUB_USERNAME: Username for pushing to hub.docker.com or updating image descriptions. Usually ddevmachinepush.
  • DOCKER_ORG: The organization on hub.docker.org to push to. Currently ddev on ddev/ddev and ddevhq on ddev-test/ddev.
  • FURY_ACCOUNT: Gemfury account that receives package pushes. drud on ddev/ddev for historical reasons, and rfay on ddev-test/ddev because that’s a spare account there.
  • HOMEBREW_EDGE_REPOSITORY: Like ddev/homebrew-ddev-edge but might be another repository like be ddev-test/homebrew-ddev-edge.
  • HOMEBREW_STABLE_REPOSITORY: Like ddev/homebrew-ddev but might be another repository like ddev-test/homebrew-ddev.

GitHub Actions Secrets Required

  • AMPLITUDE_API_KEY: Key that enables Amplitude reporting. Environment variable for Make is AmplitudeAPIKey. Unfortunately, the 1password/load-secrets-action does not work with Windows (see issue).
  • AMPLITUDE_API_KEY_DEV: Key that enables Amplitude reporting for development versions e.g. a PR build. Environment variable for Make is AmplitudeAPIKey.

1Password secrets required

The following “Repository secret” environment variables must be configured in 1Password:

  • AUR_SSH_PRIVATE_KEY: Private SSH key for the ddev-releaser user. This must be processed into a single line, for example, perl -p -e 's/\n/<SPLIT>/' ~/.ssh/id_rsa_ddev_releaser| pbcopy.
  • CHOCOLATEY_API_KEY: API key for Chocolatey.
  • DDEV_CLOUDSMITH_API_TOKEN: API token for the CLOUDSMITH_ORG/CLOUDSMITH_REPO above.
  • DDEV_GITHUB_TOKEN: GitHub personal token (repo scope, classic PAT) that gives access to create releases and push to the Homebrew repositories.
  • DDEV_MACOS_APP_PASSWORD: Password used for notarization, see signing_tools.
  • DDEV_MACOS_SIGNING_PASSWORD: Password for the macOS signing key, see signing_tools.
  • DDEV_WINDOWS_SIGNING_PASSWORD: Windows signing password.
  • DOCKERHUB_TOKEN: Token for pushing to hub.docker.com. or updating image descriptions.
  • FURY_TOKEN: Push token assigned to the above Gemfury account.
  • GORELEASER_KEY: License key for GoReleaser Pro.

Creating a Release

This is completely automated now, so nothing needs to be done unless something goes wrong.

Prerelease Tasks

  • Create and execute a test plan.
  • Run make release-prep TAG=vX.Y.Z and open a pull request with the result, so every image is rebuilt, pushed and tested under the release. See Preparing the image tags below.
  • Merge that pull request into main before creating the release. pkg/versionconstants/versionconstants.go’s image tags are plain source constants, not -ldflags, so a release tagged before the merge builds a binary that still names the prerelease image tags, even though the registry already has vX.Y.Z images under the new ones.

Preparing the image tags: make release-prep

make release-prep TAG=v1.25.4 — or bare make release-prep, which asks for the tag — does the source-file half of a release:

  • Stamps # ddev-release-marker: v1.25.4 into every image’s Dockerfile, replacing any marker left by a previous release. A Dockerfile comment is stripped by the parser, so this moves the image’s content hash without changing a single layer of the image it produces, which is what makes CI rebuild all of them.
  • Rewrites each tag in pkg/versionconstants/versionconstants.go to the resulting hash, and each <TagVar>Branch to v1.25.4 so ddev version names the release.

It builds nothing and commits nothing. Commit the result and open a pull request: the Image build workflow builds every image, publishes it under its content hash and under v1.25.4 and latest, and the test suites then run against those exact images, before the release is cut. Nothing has to be dispatched by hand.

Open that pull request from a branch in this repository, not from a fork. create-manifests, the job that adds the release names, doesn’t run for fork pull requests, so a release branch pushed to a fork would build every image and quietly leave vX.Y.Z and latest off. detect fails the run when it finds a release marker on a fork pull request, before anything is built.

Because latest moves when that pull request builds rather than when the GitHub release is created, don’t run release-prep for an edge/prerelease unless you want latest to follow it.

The trailing // <branch>-<hash> comment on each tag keeps naming the branch rather than the release, because that is the alias the push actually publishes; containers/validate-image-tag.sh rejects an alias whose prefix is release-shaped, so a v1.25.4-<hash> tag never exists.

Actual Release Creation

  1. Confirm the release-prep pull request above is merged to main — the release tag has to be cut from a commit that already has the new tags in versionconstants.go, not from a main that still points at the previous release’s images.
  2. Create a release for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
  3. Make sure you’re about to create the right release tag.
  4. Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.

Automatic Image Build and Push

Any pull request that changes containers/ — including from a fork — is built and pushed automatically by the Image build / Image push workflow pair. See Automatic Image Build and Push in the contributing guide for how the flow works and why it’s safe to run on fork-authored Dockerfiles.

A release no longer needs any of this. When detect sees a release marker in an image’s Dockerfile, create-manifests publishes vX.Y.Z and latest as further names on the manifest it is already creating — the same build the release pull request’s tests then run against, rather than a separate rebuild. It skips both if vX.Y.Z already exists in the registry, so the marker left behind after the release merges can’t move a released tag onto a later build.

The two workflows below (manual workflow_dispatch) remain for re-pushing a specific tag by hand. Their tag input is optional: leave it empty and the workflow uses the tag the checkout actually needs, which is safer than retyping a content hash.

When the tag is a content hash, these workflows also publish the <branch>-<hash> alias next to it — the same second name the automatic flow creates — so a manual push doesn’t leave a bare hash with nothing readable beside it in the registry. A vX.Y.Z tag is already readable and gets no alias.

A containers/ddev-dbserver change builds and pushes all 20 database variants (36 jobs), because they all share a single BaseDBTag. That variant matrix lives in containers/ddev-dbserver/variants.txt and is read by variants.sh, which also generates that directory’s make targets, the automatic flow’s image list, and push-tagged-dbimage.yml’s matrix — add a database version there and every consumer picks it up.

One-time setup: the image-push GitHub Environment

Fork PRs from a contributor without push access go through a single approval, gated by the image-push GitHub Environment (Settings → Environments), before anything is built — as soon as detect finds an image that needs building:

  1. Create the environment image-push.
  2. Add required reviewers (the maintainers/dev team).
  3. Leave PUSH_SERVICE_ACCOUNT_TOKEN as a repository secret. The environment gates the approval, not the secret: one job holds the approval and the build/push jobs run after it, so a run costs one approval rather than one per image, and those jobs read the repository secret — the same one the non-fork path has always used.

This approval only applies to fork PRs from a contributor without push access — checked via author_association. A fork PR from a maintainer’s own fork (OWNER/MEMBER/COLLABORATOR), a push to main, or a same-repo PR all build and push without any approval at all, using the repository-level PUSH_SERVICE_ACCOUNT_TOKEN secret directly (those paths never gate on environment:, so this environment’s protection rules don’t apply to them).

When testing this on ddev-test/ddev, do the same steps there first, and confirm vars.DOCKER_ORG on that repository points at the DockerHub org used for testing.

That org is not the one versionconstants.go names, so a binary built from ddev-test/ddev asks for ddev/ddev-webserver:vX.Y.Z while its images were published to ddevhq. Set DDEV_DOCKER_ORG to pull them from where they actually landed:

DDEV_DOCKER_ORG=ddevhq ddev start

It replaces the org on the web, db, router, ssh-agent, and xhgui images, leaving upstream images like postgres alone, and ddev version shows what it resolved to. Unset, it changes nothing.

Since a job referencing an environment that doesn’t exist yet gets auto-created with no protection rules (silently not gating), verify the environment actually has a required_reviewers rule before relying on it, e.g. gh api repos/<owner>/<repo>/environments/image-push.

Pushing Docker Images with the GitHub Actions Workflow

The easiest way to push Docker images is to use the GitHub Actions workflow, especially if the code for the image is already in the ddev/ddev repository. For a normal container change on a pull request, you shouldn’t need this — see Automatic Image Build and Push above.

Actual release creation

  1. Create a release for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
  2. Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.
  3. Verify that Homebrew (Linux and macOS) and Chocolatey and AUR are working correctly with the right versions.

You can push all images besides ddev-dbserver at https://github.com/ddev/ddev/actions/workflows/push-tagged-image.yml

You can push ddev-dbserver images at https://github.com/ddev/ddev/actions/workflows/push-tagged-dbimage.yml

A forked PR that changes a container image no longer needs any of this — see Automatic Image Build and Push above. The fork-your-own-secrets workaround described in earlier versions of this doc is superseded by that flow.

  • Visit https://github.com/ddev/ddev/actions/workflows/push-tagged-image.yml.
  • Click the “Push tagged image” workflow on the left side of the page.
  • Click the “Run workflow” button in the blue section above the workflow runs.
  • Choose the branch to build from (usually main).
  • Enter the image (ddev-webserver, etc.).
  • Enter the tag that will be used in pkg/versionconstants/versionconstants.go. For a normal (non-release) dev change, running make at the repository root already builds the image locally and computes/writes this tag automatically — run make print-image-tags to read the exact value to enter here.

Pushing Docker Images Manually

While it’s more error-prone, images can be pushed from the command line:

  1. docker login with a user that has push privileges.
  2. docker buildx use multi-arch-builder || docker buildx create --name multi-arch-builder --use.
  3. cd containers/<image>.
  4. make push VERSION=<release_version> DOCKER_ARGS=--no-cache for most of the images. For ddev-dbserver it’s make PUSH=true VERSION=<release_version> DOCKER_ARGS=--no-cache. There’s a push-all.sh script to update all of them, but it takes forever.
  5. ddev-dbserver images can be pushed with make PUSH=true VERSION=<release_version> DOCKER_ARGS=--no-cache from the containers/ddev-dbserver directory.

Maintaining ddev-dbserver MySQL 5.7 ARM64 Images

We don’t currently have a way to get xtrabackup for ARM64 Docker images for MySQL 5.7, so we have our own process to maintain ddev/mysql-arm64-images, which uses Ubuntu 18.04 Docker images, where xtrabackup was available.

  • ddev/mysql:5.7 uses Ubuntu 18.04 as the base image, and Ubuntu 18.04 ARM64 has mysql-server 5.7 in it, so we can install.
  • To build ddev/mysql (5.7) ARM64 images, follow the instructions on ddev/mysql-arm64-images. After the files, you can push a new release and the proper images will be pushed. Since MySQL 5.7 (and Ubuntu 18.04) are EOL, it’s unlikely that there will be any new minor releases.

Actual Release Docker Image Updates

We may not build every image for every point release. If there have been no changes to ddev-traefik-router or ddev-ssh-agent, for example, we may not push those and update pkg/version/version.go on major releases.

But here are the steps for building:

  1. If you’re bumping ddev-dbserver 8.0 minor release, follow the upstream Maintaining ddev-dbserver MySQL 5.7 instructions.
  2. Update pkg/version/version.go with the correct versions for the new images, and run all the tests.

Manually Updating Homebrew Formulas

Homebrew formulas normally update with the release process, so nothing needs to be done.

If you have to temporarily update the Homebrew formulas, you can do that with a commit to https://github.com/ddev/homebrew-ddev and https://github.com/ddev/homebrew-ddev-edge. The bottles and checksums for macOS (High Sierra) and x86_64_linux are built and pushed to the release page automatically by the release build process (see bump_homebrew.sh). Test brew upgrade ddev both on macOS and Linux and make sure DDEV is the right version and behaves well.

Manually Updating Chocolatey

Normally the release process does okay with pushing to Chocolatey, but at times a failure can happen and it’s not worth doing the whole release process again.

Note that if an existing approved release is being updated you have to have a new version. So for example, if v1.21.3 failed, you’ll need to work with v1.21.3.1, so make chocolatey VERSION=v1.21.3.1 below.

  • Open up GitHub Codespaces and
cd /workspace/ddev
git checkout <tag>
sudo apt-get update && sudo apt-get install -y nsis
sudo .ci-scripts/nsis_setup.sh /usr/share/nsis
  • Edit the checksum in tools/chocolateyinstall.ps1 to match the released checksum of the ddev-windows-installer in checksums.txt of the release that is being repaired, for example, for v1.21.3 this would be the checksum for ddev_windows_installer.v1.21.3.exe in v1.21.3 checksums.txt.
  • Edit url64 in tools/chocolateyinstall.ps1 to be the intended actual DDEV download version - edit the version where it appears and edit the GitHub org. For example, if the actual version of DDEV to be downloaded is v1.21.3 then put that there.
make chocolatey VERSION=<tag>
export CHOCOLATEY_API_KEY=key33333
cd .gotmp/bin/windows_amd64/chocolatey
docker run --rm -v $PWD:/tmp/chocolatey -w /tmp/chocolatey linuturk/mono-choco push -s https://push.chocolatey.org/ --api-key "${CHOCOLATEY_API_KEY}"

Manually Updating AUR Repository

The AUR repository normally updates with the release process, so nothing needs to be done.

However, you can manually publish the release to the DDEV AUR repository. The README.md in the AUR Git repository (https://aur.archlinux.org/ddev-bin.git) has instructions on how to update, including how to do it with a Docker container, so it doesn’t have to be done on an ArchLinux or Manjaro VM.

Manually Signing the Windows Installer

This is done by the release process, but the manual process is documented here.

This is done automatically by the release build on a dedicated Windows test runner (GitHub Actions runner) named testbot-asus-win10pro. You would need to do this process manually on that build machine or install the fob on another machine.

After rebooting this machine, sometimes an automated reboot, the password for the security fob has to be re-entered or Windows signing will fail. We do this by opening up tb-win11-06 using Chrome Remote Desktop (or manually physically opening it), opening Git Bash, and cd ~/tmp && signtool sign gsudo.exe. There happens to be a gsudo.exe there but it doesn’t matter what you sign—the idea is to pop up the GUI where you enter the password (which is in 1Password).

Basic Instructions

  1. Install the suggested Windows SDK. Only the signing component is required.
  2. Add the path of the kit binaries to the Windows system PATH, C:/Program Files (x86)/Windows Kits/10/bin/10.0.22621.0/x64/.
  3. The keyfob and Safenet Authentication Client must be installed. The best documentation for the Safenet software is at https://support.globalsign.com/digital-certificates/manage-safeNet-eToken/safenet-drivers. You must configure the advanced client settings to “Enable single logon” or it will require the password on each run.
  4. After make windows_amd64_install the ddev_windows_amd64_installer.exe will be in .ddev/bin/windows_amd64/ddev_windows_amd64_installer.exe and you can sign it with signtool sign ddev_windows_amd64_installer.exe.
  5. If you need to install the GitHub self-hosted Windows runner, do it with the instructions in project settings → Actions → Runners.
  6. Currently the actions/cache runner does not work out of the box on Windows, so you have to install tar and zstd as described in this issue.

We shouldn’t use this high-security keyfob approach to signing on the next go-around with the certs.

It’s way too difficult to manage, and the Safenet software is atrocious.

APT and YUM/RPM Package Management

  • The nfpms section of .goreleaser.yml builds the actual .deb/.rpm packages; furies and cloudsmiths each push those same built packages to their respective repository.
  • The primary location for APT/YUM packages is Cloudsmith, pushed via the cloudsmiths section of .goreleaser.yml, using the CLOUDSMITH_ORG/CLOUDSMITH_REPO variables and DDEV_CLOUDSMITH_API_TOKEN secret.
  • We provided and uploaded a custom GPG signing key to the Cloudsmith repository (backed up as DDEV_CLOUDSMITH_SIGNING_KEY in 1Password). This is a one-time repository setting on Cloudsmith’s side, not part of the release process — Cloudsmith signs every pushed package with it automatically, and neither .goreleaser.yml nor the GitHub Actions workflow references it.
  • Cloudsmith’s custom domain packages.ddev.com is live and serves the same content as dl.cloudsmith.io/public/ddev/ddev/... (drop the ddev/ddev organization/repository segment, keep public, e.g. packages.ddev.com/public/deb/ubuntu, packages.ddev.com/public/rpm/any-distro/any-version, packages.ddev.com/public/gpg.key).
  • Linux apt and yum/rpm packages are also pushed in parallel to the historical Gemfury repository using the furies sections of the .goreleaser.yml file. This will eventually be phased out.
  • The Gemfury packages are served by gemfury.com.
  • The name of the organization in GemFury is drud, managed at https://manage.fury.io/dashboard/drud.
  • Randy Fay, Matt Stein, and Simon Gillis are authorized as owners on this dashboard.
  • The pkg.ddev.com domain name has been used as a custom alias for the Gemfury package repositories; see https://manage.fury.io/manage/drud/domains. (Users do not see drud anywhere. Although we could have moved to a new organization for this, the existing repositories contain all the historical versions so it made sense to be less disruptive.)
  • The packages.ddev.com and pkg.ddev.com CNAME records are managed in Cloudflare because ddev.com is managed there.
  • The fury.io tokens are in DDEV’s shared 1Password account.

Testing Release Creation

When significant changes are made to the .goreleaser.yml or related configuration, it’s important to be able to test without actually deploying to ddev/ddev/releases of course. We have two ways to test the configuration; we can run goreleaser manually for simpler tests, or run a full release on ddev-test/ddev where needed.

Running goreleaser manually to create test packages and releases

This approach is great for seeing what artifacts get created, without deploying them.

Prerequisites:

You can test the GoReleaser configuration and package building locally without publishing:

First, build all artifacts, as GoReleaser uses them as prebuilt.

make linux_amd64 linux_arm64 darwin_amd64 darwin_arm64 windows_amd64 windows_arm64 wsl_amd64 wsl_arm64

Then, you can use goreleaser to check the configuration and build packages. You must have GoReleaser Pro installed, as DDEV uses it for configuration. If you don’t have it installed, see the GoReleaser installation instructions.

# Check configuration syntax
REPOSITORY_OWNER=ddev goreleaser check

# Build packages in snapshot mode (no publishing)
git tag <tagname> # Try to include context like PR number, for example v1.24.7-PR5824
REPOSITORY_OWNER=ddev goreleaser release --snapshot --clean

Built packages will appear in the dist/ directory. You can examine package contents:

# List created packages
ls -la dist/*.{deb,rpm}

# Examine DEB package contents
dpkg-deb -c dist/ddev_*_linux_amd64.deb
dpkg-deb -c dist/ddev-wsl2_*_linux_amd64.deb  # WSL2 package

# Examine RPM package contents
rpm -qlp dist/ddev_*_linux_amd64.rpm

Creating a test release on ddev-test/ddev

ddev-test/ddev is now set up for actual release testing. It has all or most of the environment variables set up already. It also acts against ddev-test/homebrew-ddev and ddev-test/homebrew-ddev-edge so you can test Homebrew publishing.

  1. Create a branch on ddev-test/ddev.
  2. Using the web UI, create a release using that branch as base. The release tag must start with v1.. Where possible, please use a release tag that includes context about the PR you are working against, like v1.28.8-PR2022FixStuff, and include in the release notes a link to the issue. The tag must be a valid Semantic Version tag, so don’t use underscores, etc.
  3. Test out the resulting artifacts that get published or deployed.