# Release Management & Docker Images
## Release process and tools
- [GoReleaser Pro]() is used to do the actual releasing using [.goreleaser.yml](). GoReleaser Pro is a licensed product that requires separate installation and a license key, which is in the GitHub Workflow configuration and is available in 1Password to DDEV maintainers who need it.
- The [Main Build/Release GitHub Action]() does the actual running of the GoReleaser actions and provides the needed secrets.
## GitHub Actions Required Secrets
### How to add new people to these accounts
- AUR is Arch Linux User Repository. `ddev-bin` is at `https://aur.archlinux.org/packages/ddev-bin`. The current maintainer of this is [@rfay]( "GitHub User: rfay"), who can add co-maintainers.
- The [chocolatey]() package. Additional maintainers can be added at (login required) `https://community.chocolatey.org/packages/ddev/ManagePackageOwners`; they could then create tokens to push it.
- GitHub requires write access to this repository, either via permissions on the repository or on the org.
- Apple signing and notarization requires access to the DDEV Foundation group on `https://developer.apple.com`. It’s easy enough to add additional people.
- Windows signing is an awkward process that requires a dongle. When the current signing certificate expires we definitely want the simpler approach.
- Discord
- Docker
### Environment variables required
These are normally configured in the repository environment variables.
- `AUR_EDGE_GIT_URL`: The Git URL for AUR edge (normally `ddev-edge-bin`), for example `ssh://aur@aur.archlinux.org/ddev-edge-bin.git`.
- `AUR_PACKAGE_NAME`: The base name of the AUR package. Normally `ddev` for production, but `ddev-test` for testing repository.
- `AUR_STABLE_GIT_URL`: The Git URL for AUR stable (normally `ddev-bin`), for example `ssh://aur@aur.archlinux.org/ddev-bin.git`.
- `CLOUDSMITH_ORG`: The organization on [Cloudsmith]() that receives `apt`/`yum` package pushes.
- `CLOUDSMITH_REPO`: The repository within `CLOUDSMITH_ORG` that receives package pushes, so `ddev-test/ddev` can push to a separate test repository from `ddev/ddev`.
- `DDEV_IGNORE_EXPIRING_KEYS`: If the value is `"false"` then daily tests will fail if signing keys are expiring soon, default is `"false"` on `ddev/ddev`
- `DDEV_MAX_DAYS_BEFORE_CERT_EXPIRATION`: Number of days before expiration to warn about expiring signing keys, default is `90` on `ddev/ddev`.
- `DDEV_WINDOWS_SIGN`: If the value is `"true"` then `make` will attempt to sign the Windows executables, which requires building on our self-hosted Windows runner.
- `DISABLE_NOTARIZATION`: If the value is `"true"` then notarization of macOS apps will be skipped, otherwise notarization is attempted.
- `DOCKERHUB_USERNAME`: Username for pushing to `hub.docker.com` or updating image descriptions. Usually `ddevmachinepush`.
- `DOCKER_ORG`: The organization on `hub.docker.org` to push to. Currently `ddev` on `ddev/ddev` and `ddevhq` on `ddev-test/ddev`.
- `FURY_ACCOUNT`: [Gemfury]() account that receives package pushes. `drud` on `ddev/ddev` for historical reasons, and `rfay` on `ddev-test/ddev` because that’s a spare account there.
- `HOMEBREW_EDGE_REPOSITORY`: Like `ddev/homebrew-ddev-edge` but might be another repository like be `ddev-test/homebrew-ddev-edge`.
- `HOMEBREW_STABLE_REPOSITORY`: Like `ddev/homebrew-ddev` but might be another repository like `ddev-test/homebrew-ddev`.
### GitHub Actions Secrets Required
- `AMPLITUDE_API_KEY`: Key that enables Amplitude reporting. Environment variable for Make is `AmplitudeAPIKey`. Unfortunately, the `1password/load-secrets-action` does not work with Windows (see [issue]()).
- `AMPLITUDE_API_KEY_DEV`: Key that enables Amplitude reporting for development versions e.g. a PR build. Environment variable for Make is `AmplitudeAPIKey`.
### 1Password secrets required
The following “Repository secret” environment variables must be configured in 1Password:
- `AUR_SSH_PRIVATE_KEY`: Private SSH key for the `ddev-releaser` user. This must be processed into a single line, for example, `perl -p -e 's/\n//' ~/.ssh/id_rsa_ddev_releaser| pbcopy`.
- `CHOCOLATEY_API_KEY`: API key for Chocolatey.
- `DDEV_CLOUDSMITH_API_TOKEN`: API token for the `CLOUDSMITH_ORG`/`CLOUDSMITH_REPO` above.
- `DDEV_GITHUB_TOKEN`: GitHub personal token (`repo` scope, classic PAT) that gives access to create releases and push to the Homebrew repositories.
- `DDEV_MACOS_APP_PASSWORD`: Password used for notarization, see [signing\_tools]().
- `DDEV_WINDOWS_SIGNING_PASSWORD`: Windows signing password.
- `DOCKERHUB_TOKEN`: Token for pushing to `hub.docker.com`. or updating image descriptions.
- `FURY_TOKEN`: Push token assigned to the above Gemfury account.
- `GORELEASER_KEY`: License key for GoReleaser Pro.
The macOS Developer ID certificate and its password are stored in the `ddev-signing` vault, not `push-secrets`. Configure the `DDEV_SIGNING_SERVICE_ACCOUNT_TOKEN` GitHub Actions secret with a dedicated, read-only service account that has access only to that vault. The macOS release job downloads the certificate to a temporary file and deletes it when the job exits.
## Creating a Release
> [!TIP]
>
> **This is completely automated now, so nothing needs to be done unless something goes wrong.**
### Prerelease Tasks
- Create and execute a test plan.
- Run `make release-prep TAG=vX.Y.Z` and open a pull request with the result, so every image is rebuilt, pushed and tested under the release. See [Preparing the image tags](<#preparing-the-image-tags-make-release-prep>) below.
- Merge that pull request into `main` before creating the release. `pkg/versionconstants/versionconstants.go`’s image tags are plain source constants, not `-ldflags`, so a release tagged before the merge builds a binary that still names the prerelease image tags, even though the registry already has `vX.Y.Z` images under the new ones.
### Preparing the image tags: `make release-prep`
`make release-prep TAG=v1.25.4` — or bare `make release-prep`, which asks for the tag — does the source-file half of a release:
- Stamps `# ddev-release-marker: v1.25.4` into every image’s `Dockerfile`, replacing any marker left by a previous release. A Dockerfile comment is stripped by the parser, so this moves the image’s content hash without changing a single layer of the image it produces, which is what makes CI rebuild all of them.
- Rewrites each tag in [`pkg/versionconstants/versionconstants.go`]() to the resulting hash, and each `Branch` to `v1.25.4` so `ddev version` names the release.
It builds nothing and commits nothing. Commit the result and open a pull request: the [Image build]() workflow builds every image, publishes it under its content hash *and* under `v1.25.4` and `latest`, and the test suites then run against those exact images, before the release is cut. Nothing has to be dispatched by hand.
Open that pull request from a branch in this repository, not from a fork. `create-manifests`, the job that adds the release names, doesn’t run for fork pull requests, so a release branch pushed to a fork would build every image and quietly leave `vX.Y.Z` and `latest` off. `detect` fails the run when it finds a release marker on a fork pull request, before anything is built.
`release-prep` also takes a prerelease tag (`v1.25.4-alpha1`, `-beta2`, `-rc1`), which follows exactly the same path except that `latest` is left pointing at the last stable release. No other suffix is accepted, so `git describe` output can’t be stamped as a release by accident.
For a stable release, `latest` moves when that pull request builds rather than when the GitHub release is created.
The trailing `// -` comment on each tag keeps naming the branch rather than the release, because that is the alias the push actually publishes; `containers/validate-image-tag.sh` rejects an alias whose prefix is release-shaped, so a `v1.25.4-` tag never exists.
### Actual Release Creation
1. Confirm the `release-prep` pull request above is merged to `main` — the release tag has to be cut from a commit that already has the new tags in `versionconstants.go`, not from a `main` that still points at the previous release’s images.
2. Create a [release]() for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
3. Make sure you’re about to create the right release tag.
4. Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.
## Automatic Image Build and Push
Any pull request that changes `containers/` — including from a fork — is built and pushed automatically by the [Image build]() / [Image push]() workflow pair. See [Automatic Image Build and Push]() in the contributing guide for how the flow works and why it’s safe to run on fork-authored Dockerfiles.
A release no longer needs any of this. When `detect` sees a release marker in an image’s `Dockerfile`, `create-manifests` publishes `vX.Y.Z` and `latest` as further names on the manifest it is already creating — the same build the release pull request’s tests then run against, rather than a separate rebuild. It skips both if `vX.Y.Z` already exists in the registry, so the marker left behind after the release merges can’t move a released tag onto a later build.
The two workflows below (manual `workflow_dispatch`) remain for re-pushing a specific tag by hand. Their `tag` input is optional: leave it empty and the workflow uses the tag the checkout actually needs, which is safer than retyping a content hash.
When the tag is a content hash, these workflows also publish the `-` alias next to it — the same second name the automatic flow creates — so a manual push doesn’t leave a bare hash with nothing readable beside it in the registry. A `vX.Y.Z` tag is already readable and gets no alias.
A `containers/ddev-dbserver` change builds and pushes all 20 database variants (36 jobs), because they all share a single `BaseDBTag`. That variant matrix lives in `containers/ddev-dbserver/variants.txt` and is read by `variants.sh`, which also generates that directory’s make targets, the automatic flow’s image list, and `push-tagged-dbimage.yml`’s matrix — add a database version there and every consumer picks it up.
### One-time setup: the `image-push` GitHub Environment
Fork PRs from a contributor without push access go through a single approval, gated by the `image-push` GitHub Environment (Settings → Environments), *before* anything is built — as soon as `detect` finds an image that needs building:
1. Create the environment `image-push`.
2. Add required reviewers (the maintainers/dev team).
3. Leave `PUSH_SERVICE_ACCOUNT_TOKEN` as a repository secret. The environment gates the approval, not the secret: one job holds the approval and the build/push jobs run after it, so a run costs one approval rather than one per image, and those jobs read the repository secret — the same one the non-fork path has always used.
This approval only applies to fork PRs from a contributor without push access — checked via `author_association`. A fork PR from a maintainer’s own fork (`OWNER`/`MEMBER`/`COLLABORATOR`), a push to `main`, or a same-repo PR all build and push without any approval at all, using the repository-level `PUSH_SERVICE_ACCOUNT_TOKEN` secret directly (those paths never gate on `environment:`, so this environment’s protection rules don’t apply to them).
When testing this on `ddev-test/ddev`, do the same steps there first, and confirm `vars.DOCKER_ORG` on that repository points at the DockerHub org used for testing.
That org is not the one `versionconstants.go` names, so a binary built from `ddev-test/ddev` asks for `ddev/ddev-webserver:vX.Y.Z` while its images were published to `ddevhq`. Set `DDEV_DOCKER_ORG` to pull them from where they actually landed:
```
DDEV_DOCKER_ORG=ddevhq ddev start
```
It replaces the org on the web, db, router, ssh-agent, and xhgui images, leaving upstream images like `postgres` alone, and `ddev version` shows what it resolved to. Unset, it changes nothing.
Since a job referencing an environment that doesn’t exist yet gets auto-created with no protection rules (silently *not* gating), verify the environment actually has a `required_reviewers` rule before relying on it, e.g. `gh api repos///environments/image-push`.
## Pushing Docker Images with the GitHub Actions Workflow
The easiest way to push Docker images is to use the GitHub Actions workflow, especially if the code for the image is already in the [ddev/ddev]() repository. For a normal container change on a pull request, you shouldn’t need this — see [Automatic Image Build and Push](<#automatic-image-build-and-push>) above.
### Actual release creation
1. Create a [release]() for the new version using the GitHub UI. It should be “prerelease” if it’s an edge release.
2. Use the “Auto-generate release notes” option to get the commit list, then edit to add all the other necessary info.
3. Verify that Homebrew (Linux and macOS) and Chocolatey and AUR are working correctly with the right versions.
You can push all images besides `ddev-dbserver` at [https://github.com/ddev/ddev/actions/workflows/push-tagged-image.yml]()
You can push `ddev-dbserver` images at [https://github.com/ddev/ddev/actions/workflows/push-tagged-dbimage.yml]()
A forked PR that changes a container image no longer needs any of this — see [Automatic Image Build and Push](<#automatic-image-build-and-push>) above. The fork-your-own-secrets workaround described in earlier versions of this doc is superseded by that flow.
- Visit `https://github.com/ddev/ddev/actions/workflows/push-tagged-image.yml`.
- Click the “Push tagged image” workflow on the left side of the page.
- Click the “Run workflow” button in the blue section above the workflow runs.
- Choose the branch to build from (usually `main`).
- Enter the image (`ddev-webserver`, etc.).
- Enter the tag that will be used in `pkg/versionconstants/versionconstants.go`. For a normal (non-release) dev change, running `make` at the repository root already builds the image locally and computes/writes this tag automatically — run `make print-image-tags` to read the exact value to enter here.
## Pushing Docker Images Manually
While it’s more error-prone, images can be pushed from the command line:
1. `docker login` with a user that has push privileges.
2. `docker buildx use multi-arch-builder || docker buildx create --name multi-arch-builder --use`.
3. `cd containers/`.
4. `make push VERSION= DOCKER_ARGS=--no-cache` for most of the images. For `ddev-dbserver` it’s `make PUSH=true VERSION= DOCKER_ARGS=--no-cache`. There’s a [push-all.sh]() script to update all of them, but it takes forever.
5. `ddev-dbserver` images can be pushed with `make PUSH=true VERSION= DOCKER_ARGS=--no-cache` from the `containers/ddev-dbserver` directory.
## Maintaining `ddev-dbserver` MySQL 5.7 ARM64 Images
We don’t currently have a way to get `xtrabackup` for ARM64 Docker images for MySQL 5.7, so we have our own process to maintain [ddev/mysql-arm64-images](), which uses Ubuntu 18.04 Docker images, where `xtrabackup` was available.
- `ddev/mysql:5.7` uses Ubuntu 18.04 as the base image, and Ubuntu 18.04 ARM64 has `mysql-server` 5.7 in it, so we can install.
- To build `ddev/mysql` (5.7) ARM64 images, follow the instructions on [ddev/mysql-arm64-images](). After the files, you can push a new release and the proper images will be pushed. Since MySQL 5.7 (and Ubuntu 18.04) are EOL, it’s unlikely that there will be any new minor releases.
## Actual Release Docker Image Updates
We may not build every image for every point release. If there have been no changes to `ddev-traefik-router` or `ddev-ssh-agent`, for example, we may not push those and update `pkg/version/version.go` on major releases.
But here are the steps for building:
1. If you’re bumping `ddev-dbserver` 8.0 minor release, follow the upstream [Maintaining ddev-dbserver MySQL 5.7](<#maintaining-ddev-dbserver-mysql-57-arm64-images>) instructions.
2. Update `pkg/version/version.go` with the correct versions for the new images, and run all the tests.
## Manually Updating Homebrew Formulas
Homebrew formulas normally update with the release process, so nothing needs to be done.
If you have to temporarily update the Homebrew formulas, you can do that with a commit to [https://github.com/ddev/homebrew-ddev]() and [https://github.com/ddev/homebrew-ddev-edge](). The bottles and checksums for macOS (High Sierra) and x86\_64\_linux are built and pushed to the release page automatically by the release build process (see [bump\_homebrew.sh]()). Test `brew upgrade ddev` both on macOS and Linux and make sure DDEV is the right version and behaves well.
## Manually Updating Chocolatey
Normally the release process does okay with pushing to Chocolatey, but at times a failure can happen and it’s not worth doing the whole release process again.
Note that if an existing approved release is being updated you have to have a new version. So for example, if `v1.21.3` failed, you’ll need to work with `v1.21.3.1`, so `make chocolatey VERSION=v1.21.3.1` below.
- Open up GitHub Codespaces and
```
cd /workspace/ddev
git checkout
sudo apt-get update && sudo apt-get install -y nsis
sudo .ci-scripts/nsis_setup.sh /usr/share/nsis
```
- Edit the checksum in `tools/chocolateyinstall.ps1` to match the released checksum of the `ddev-windows-installer` in `checksums.txt` of the release that is being repaired, for example, for `v1.21.3` this would be the checksum for `ddev_windows_installer.v1.21.3.exe` in [v1.21.3 checksums.txt]().
- Edit `url64` in `tools/chocolateyinstall.ps1` to be the intended actual DDEV download version - edit the version where it appears and edit the GitHub org. For example, if the actual version of DDEV to be downloaded is `v1.21.3` then put that there.
```
make chocolatey VERSION=
export CHOCOLATEY_API_KEY=key33333
cd .gotmp/bin/windows_amd64/chocolatey
docker run --rm -v $PWD:/tmp/chocolatey -w /tmp/chocolatey linuturk/mono-choco push -s https://push.chocolatey.org/ --api-key "${CHOCOLATEY_API_KEY}"
```
## Manually Updating AUR Repository
The AUR repository normally updates with the release process, so nothing needs to be done.
However, you can manually publish the release to [the DDEV AUR repository](). The README.md in the AUR Git repository (`https://aur.archlinux.org/ddev-bin.git`) has instructions on how to update, including how to do it with a Docker container, so it doesn’t have to be done on an ArchLinux or Manjaro VM.
## Manually Signing the Windows Installer
> [!TIP]
>
> **This is done by the release process, but the manual process is documented here.**
This is done automatically by the release build on a dedicated Windows test runner (GitHub Actions runner) named `testbot-asus-win10pro`. You would need to do this process manually on that build machine or install the fob on another machine.
**After rebooting this machine, sometimes an automated reboot, the password for the security fob has to be re-entered or Windows signing will fail. We do this by opening up `tb-win11-06` using Chrome Remote Desktop (or manually physically opening it), opening Git Bash, and `cd ~/tmp && signtool sign gsudo.exe`. There happens to be a `gsudo.exe` there but it doesn’t matter what you sign—the idea is to pop up the GUI where you enter the password (which is in 1Password).**
### Basic Instructions
1. Install the suggested [Windows SDK](). Only the signing component is required.
2. Add the path of the kit binaries to the Windows system PATH, `C:/Program Files (x86)/Windows Kits/10/bin/10.0.22621.0/x64/`.
3. The keyfob and Safenet Authentication Client must be installed. The best documentation for the Safenet software is at [https://support.globalsign.com/digital-certificates/manage-safeNet-eToken/safenet-drivers](). You must configure the advanced client settings to “Enable single logon” or it will require the password on each run.
4. After `make windows_amd64_install` the `ddev_windows_amd64_installer.exe` will be in `.ddev/bin/windows_amd64/ddev_windows_amd64_installer.exe` and you can sign it with `signtool sign ddev_windows_amd64_installer.exe`.
5. If you need to install the GitHub self-hosted Windows runner, do it with the instructions in project settings → Actions → Runners.
6. Currently the `actions/cache` runner does not work out of the box on Windows, so you have to install tar and zstd as described in [this issue]().
> [!TIP]
>
> **We shouldn’t use this high-security keyfob approach to signing on the next go-around with the certs.**
>
> It’s way too difficult to manage, and the Safenet software is atrocious.
## APT and YUM/RPM Package Management
- The `nfpms` section of [.goreleaser.yml]() builds the actual `.deb`/`.rpm` packages; `furies` and `cloudsmiths` each push those same built packages to their respective repository.
- The primary location for APT/YUM packages is [Cloudsmith](), pushed via the `cloudsmiths` section of [.goreleaser.yml](), using the `CLOUDSMITH_ORG`/`CLOUDSMITH_REPO` variables and `DDEV_CLOUDSMITH_API_TOKEN` secret.
- We provided and uploaded a custom GPG signing key to the Cloudsmith repository (backed up as `DDEV_CLOUDSMITH_SIGNING_KEY` in 1Password). This is a one-time repository setting on Cloudsmith’s side, not part of the release process — Cloudsmith signs every pushed package with it automatically, and neither `.goreleaser.yml` nor the GitHub Actions workflow references it.
- Cloudsmith’s [custom domain]() `packages.ddev.com` is live and serves the same content as `dl.cloudsmith.io/public/ddev/ddev/...` (drop the `ddev/ddev` organization/repository segment, keep `public`, e.g. `packages.ddev.com/public/deb/ubuntu`, `packages.ddev.com/public/rpm/any-distro/any-version`, `packages.ddev.com/public/gpg.key`).
- Linux `apt` and `yum`/`rpm` packages are also pushed in parallel to the historical Gemfury repository using the `furies` sections of the [.goreleaser.yml]() file. This will eventually be phased out.
- The Gemfury packages are served by [gemfury.com]().
- The name of the organization in GemFury is `drud`, managed at `https://manage.fury.io/dashboard/drud`.
- [Randy Fay](), [Matt Stein](), and [Simon Gillis]() are authorized as owners on this dashboard.
- The `pkg.ddev.com` domain name has been used as a custom alias for the Gemfury package repositories; see `https://manage.fury.io/manage/drud/domains`. (Users do not see `drud` anywhere. Although we could have moved to a new organization for this, the existing repositories contain all the historical versions so it made sense to be less disruptive.)
- The `packages.ddev.com` and `pkg.ddev.com` `CNAME` records are managed in Cloudflare because `ddev.com` is managed there.
- The fury.io tokens are in DDEV’s shared 1Password account.
## Testing Release Creation
When significant changes are made to the `.goreleaser.yml` or related configuration, it’s important to be able to test without actually deploying to `ddev/ddev/releases` of course. We have two ways to test the configuration; we can run `goreleaser` manually for simpler tests, or run a full release on `ddev-test/ddev` where needed.
### Running `goreleaser` manually to create test packages and releases
This approach is great for seeing what artifacts get created, without deploying them.
Prerequisites:
- GoReleaser Pro must be installed, see [GoReleaser installation instructions]().
- `export GORELEASER_KEY=` (not needed for simple snapshot)
You can test the GoReleaser configuration and package building locally without publishing:
First, build all artifacts, as GoReleaser uses them as `prebuilt`.
```
make linux_amd64 linux_arm64 darwin_amd64 darwin_arm64 windows_amd64 windows_arm64 wsl_amd64 wsl_arm64
```
Then, you can use `goreleaser` to check the configuration and build packages. You must have [GoReleaser Pro]() installed, as DDEV uses it for configuration. If you don’t have it installed, see the [GoReleaser installation instructions]().
```
# Check configuration syntax
REPOSITORY_OWNER=ddev goreleaser check
# Build packages in snapshot mode (no publishing)
git tag # Try to include context like PR number, for example v1.24.7-PR5824
REPOSITORY_OWNER=ddev goreleaser release --snapshot --clean
```
Built packages will appear in the `dist/` directory. You can examine package contents:
```
# List created packages
ls -la dist/*.{deb,rpm}
# Examine DEB package contents
dpkg-deb -c dist/ddev_*_linux_amd64.deb
dpkg-deb -c dist/ddev-wsl2_*_linux_amd64.deb # WSL2 package
# Examine RPM package contents
rpm -qlp dist/ddev_*_linux_amd64.rpm
```
### Creating a test release on `ddev-test/ddev`
[ddev-test/ddev]() is now set up for actual release testing. It has all or most of the environment variables set up already. It also acts against `ddev-test/homebrew-ddev` and `ddev-test/homebrew-ddev-edge` so you can test Homebrew publishing.
1. Create a branch on `ddev-test/ddev`.
2. Using the web UI, create a release using that branch as base. The release tag must start with `v1.`. Where possible, please use a release tag that includes context about the PR you are working against, like `v1.28.8-PR2022FixStuff`, and include in the release notes a link to the issue. The tag must be a valid Semantic Version tag, so don’t use underscores, etc.
3. Test out the resulting artifacts that get published or deployed.