# Router Customization and Debugging (Traefik)

DDEV uses the well-known open-source [Traefik](<https://traefik.io/traefik/>) for its router container.

DDEV’s router plays an important role in its [container architecture](<https://ddev.github.io/ddev/en/stable/users/usage/architecture/#container-architecture>), receiving HTTP and HTTPS traffic for requests like `*.ddev.site` and delivering them to the appropriate project’s web container.

## Traefik Configuration

Before continuing, it’s important to note that **very few users ever experiment with custom Traefik configuration**. This is an advanced topic, and the vast majority of users never need to know anything about it, as DDEV generates all the necessary configuration. In general you don’t need to understand Traefik configuration.

However, you can fully customize the router’s [Traefik configuration](<https://doc.traefik.io/traefik/getting-started/configuration-overview/>). (DDEV uses the Traefik `v3` rule syntax.)

All Traefik configuration uses the *file* provider, not the *Docker* provider. Even though the Traefik daemon itself is running inside the `ddev-router` container, it uses mounted files for configuration, rather than listening to the Docker socket.

> [!TIP]
>
> Like other DDEV configuration, any file with `#ddev-generated` will be overwritten unless you choose to “take over” it yourself. You can do this by removing the `#ddev-generated` line. DDEV will stop making changes to that file, and you’ll be responsible for updating it.

### Traefik Static Configuration

Static configuration is automatically generated in the `$HOME/.ddev/traefik` directory (see [global configuration directory](<https://ddev.github.io/ddev/en/stable/users/usage/architecture/#global-files>)). “Static” configuration means Traefik configuration which is only read when the router is started.

- `.static_config.yaml` (a hidden file) is the configuration that gets used. It is not to be edited; it is generated from DDEV’s base configuration while merging any files named `static_config.*.yaml`. It is read on router startup, and does not change until the router starts again (normally after `ddev poweroff`).
- Additional static configuration may be added by adding `static_config.*.yaml` files, which will be merged into the generated `.static_config.yaml`. For example, a `static_config.loglevel.yaml` might override logging configuration, a `static_config.plugin.yaml` might contain external Traefik plugins, or a `static_config.dnschallenge.yaml` might provide configuration for additional `certificatesResolvers`. Merging is done with an **override** strategy, meaning that the final file in alphanumeric sort to touch a particular element of the YAML structure wins. Some examples of `static_config.*.yaml` files are:

  - `static_config.loglevel.yaml`:

    ```
    # Enable extensive error and access logging
    log:
      level: DEBUG
    accessLog:
      filters:
        statusCodes: {}
    ```
  - `static_config.cloudflare.yaml`:

    ```
    certificatesResolvers:
      acme-dnsChallenge:
        acme:
          email: admin@example.com
          dnsChallenge:
            provider: cloudflare
    ```
  - `static_config.fail2ban.yaml`

    ```
    experimental:
      plugins:
        fail2ban:
          moduleName: "github.com/tomMoulard/fail2ban"
          version: "v0.8.1"
    ```
- `certs/default_cert.*` files are the default DDEV-generated certificates, normally created by `mkcert`.
- `config/default_config.yaml` contains global *dynamic* configuration, including pointers to the default certificates. It’s unusual to change its content, but if you wanted to change the content you would override with a file in `custom-global-config`.
- `custom-global-config` is a directory where you can add custom Traefik dynamic configuration files (such as middleware, routers, or services) that will be automatically included for all projects. Files in this directory are copied to the router’s config volume on each `ddev start`. For example, a `custom-global-config/global_middlewares.yaml` file might provide middleware implementations that could be used in all projects:

  ```
  http:
    middlewares:
      global-ratelimit:
        rateLimit:
          average: 100
          burst: 50
  ```

### Project Traefik Configuration

Project-specific configuration is automatically generated in the project’s `.ddev/traefik/config` directory. For example, a project named `example` will have a `.ddev/traefik/config/example.yaml` which describes the routers, middlewares, and services generated by default for that project. These are based on the base hostname, `additional_hostnames`, and `additional_fqdns` defined for the project. They also include support for add-ons and services that use `HTTP_EXPOSE`, `HTTPS_EXPOSE`, and `VIRTUAL_HOST` configurations (see [Conventions for Defining Additional Services](<https://ddev.github.io/ddev/en/stable/users/extend/custom-compose-files/#conventions-for-defining-additional-services>) for more details).

- The `certs` directory contains the `<projectname>.crt` and `<projectname>.key` certificate generated for the project.
- The `config/<projectname>.yaml` file contains the base configuration for the project, including information about routers, services, and certificates.
- **Adding Custom Configuration**: You can add custom Traefik configuration by creating additional `*.yaml` files in the `.ddev/traefik/config` directory. All YAML files in this directory will be automatically merged together, with `<projectname>.yaml` serving as the base. The merged result is then copied to the global Traefik configuration as `<projectname>_merged.yaml`. This allows you to add custom routers, middlewares, or services without modifying the main configuration file.

  For example, to add custom middleware to a project named `example`, you could create `.ddev/traefik/config/custom-middleware.yaml`:

  ```
  http:
    middlewares:
      example-custom-headers:
        headers:
          customRequestHeaders:
            X-Custom-Header: "my-value"
  ```
- **Taking Over Configuration**: If you want to completely customize the configuration and prevent DDEV from regenerating it, remove the `#ddev-generated` line from the top of the `config/<projectname>.yaml` file. DDEV will then stop regenerating it, and you’ll be responsible for maintaining it. If you do this and then change the name of the project, you have to either remove the file and have DDEV regenerate it, or you have to update it yourself.

## Router `docker-compose` Customization

The default Docker Compose configuration for the router container is found in `$HOME/.ddev/.router-compose.yaml` (see [global configuration directory](<https://ddev.github.io/ddev/en/stable/users/usage/architecture/#global-files>)). It is quite unusual to override this configuration, but it can be overridden in the same way project configuration can be overridden (project `.ddev/docker-compose.*.yaml`). These ultimately get merged into `$HOME/.ddev/.router-compose-full.yaml`

You can create a `$HOME/.ddev/router-compose.*.yaml`. For example, as `$HOME/.ddev/router-compose.cloudflare.yaml` might contain environment variables like this:

```
# $HOME/.ddev/router-compose.cloudflare.yaml
services:
  ddev-router:
    environment:
      - CLOUDFLARE_EMAIL=you@example.com
      - CLOUDFLARE_API_KEY=some-key
```

### Custom Maintenance Page

When a request reaches the router for a hostname that no running project serves, for example while a project is stopped or restarting, the router answers with a DDEV page explaining why. If you [host with DDEV](<https://ddev.github.io/ddev/en/stable/users/topics/hosting/index.md>), you can replace it with your own maintenance page using `configs` in a `$HOME/.ddev/router-compose.maintenance.yaml`:

```
# $HOME/.ddev/router-compose.maintenance.yaml
services:
  ddev-router:
    configs:
      - source: unmatched-route.html
        target: /var/www/ddev-router-fallback/unmatched-route.html
        mode: "0444"
      - source: ddev-router-fallback-responder.sh
        target: /usr/local/bin/ddev-router-fallback-responder.sh
        mode: "0755"

configs:
  unmatched-route.html:
    content: |
      <h1>Under maintenance</h1>
      <p>We'll be back shortly.</p>
  ddev-router-fallback-responder.sh:
    content: |
      #!/bin/sh
      printf 'HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/html; charset=utf-8\r\nRetry-After: 60\r\nConnection: close\r\n\r\n'
      cat /var/www/ddev-router-fallback/unmatched-route.html
```

Run `ddev poweroff` and `ddev start` to recreate the router. The page is served with a `503` status instead of DDEV’s `404`, and without DDEV’s `X-Ddev-404-Source` header. To change only the page, keep just the `unmatched-route.html` config. In `content`, write each `$` as `$$` so that Docker Compose doesn’t treat it as a variable.

## Troubleshooting Traefik Routing

Traefik provides a dynamic description of its configuration you can visit at `http://localhost:10999`. When things seem to be going wrong, run [`ddev poweroff`](<https://ddev.github.io/ddev/en/stable/users/usage/commands/#poweroff>) and then start your project again by running [`ddev start`](<https://ddev.github.io/ddev/en/stable/users/usage/commands/#start>). Examine the router’s logs to see what the Traefik daemon is doing (or failing at) by running `docker logs ddev-router` or `docker logs -f ddev-router`. The Traefik logs are set to a minimal set by default, but you can enable much more extensive logging and access logs with a `static_config.loglevel.yaml` as [described above](<#traefik-static-configuration>).

### Warning: There are router configuration problems

If you see a warning on `ddev start` about router configuration problems, it is most likely a result of custom configuration problems, which could be an invalid `docker-compose.*.yaml` or leftover project `.ddev/traefik/config/*.yaml` files. Only Traefik **error**-level (ERR) messages are shown there; **warning**-level (WRN) and debug output from Traefik appear only in `docker logs ddev-router`.

On Linux systems, file watcher errors related to the `*file.Provider` are a common cause. See the section below for solutions.

### Linux file watcher errors

If you encounter Traefik file watcher errors on Linux systems, these are common solutions:

#### Error adding file watcher: no space left on device

This error appears during `ddev start` or in the router logs, `docker logs ddev-router`, as:

> ERR Cannot start the provider \*file.Provider error=”error adding file watcher: no space left on device”

This indicates that the system has reached the limit for inotify watches on your Linux computer. First, check the current value:

```
sysctl fs.inotify.max_user_watches
```

Then create a configuration file to increase the limit (for example, if the current value is `65536`, you can increase it to `524288`):

```
echo 'fs.inotify.max_user_watches=524288' | sudo tee -a /etc/sysctl.d/60-inotify.conf
```

Apply the changes without restarting the system:

```
sudo sysctl -p /etc/sysctl.d/60-inotify.conf
```

#### Error creating file watcher: too many open files

This error appears during `ddev start` or in the router logs, `docker logs ddev-router`, as:

> ERR Cannot start the provider \*file.Provider error=”error creating file watcher: too many open files”

This indicates that too many inotify instances are open. First, check the current value:

```
sysctl fs.inotify.max_user_instances
```

Then create a configuration file to increase the limit (for example, if the current value is `128`, you can increase it to `8192`):

```
echo 'fs.inotify.max_user_instances=8192' | sudo tee -a /etc/sysctl.d/60-inotify.conf
```

Apply the changes without restarting the system:

```
sudo sysctl -p /etc/sysctl.d/60-inotify.conf
```