[BUG] Using env_file in compose includes has inconsistent behavior with .env

[psanchezeverhealth]

Description

I'm currently using the env_file feature in compose files to try to automate which dotenv files are automatically used for variable interpolation. The expectation is that if there is a list of env files, the files later in the list override values earlier in the list.

This typically works as expected unless one of the files is .env. Then no matter where it is in that list, those files don't get overridden by the other env files.

Steps To Reproduce

With the following setup:

compose.yml

include:
  - path:
      - ./compose.include.base.yml
    env_file:
      - ./.env
      - ./.env.local
      - ./.env.dev

compose.include.base.yml

services:
  env_file_test:
    image: env_file_test
    build:
      args:
        TEST_VAL: ${TEST_VAL}

Dockerfile (not technically needed)

FROM debian
ARG TEST_VAL
ENV TEST_VAL=$TEST_VAL

.env

TEST_VAL=".env"

.env.local

TEST_VAL=".env.local"

.env.dev

TEST_VAL=".env.dev"

If you run docker compose config, you'll roughly get the following:

name: compose_test
services:
  env_file_test:
    build:
      context: /Users/me/compose_test
      dockerfile: Dockerfile
      args:
        TEST_VAL: .env
    image: env_file_test
    networks:
      default: null
networks:
  default:
    name: compose_test_default

Or with docker buildx bake --print:

{
  "group": {
    "default": {
      "targets": [
        "env_file_test"
      ]
    }
  },
  "target": {
    "env_file_test": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "args": {
        "TEST_VAL": ".env"
      },
      "tags": [
        "env_file_test"
      ]
    }
  }
}

The expectation is that you would get:

name: compose_test
services:
  env_file_test:
    build:
      context: /Users/me/compose_test
      dockerfile: Dockerfile
      args:
        TEST_VAL: .env.dev
    image: env_file_test
    networks:
      default: null
networks:
  default:
    name: compose_test_default

If you remove the line in the .env file and rerun docker compose config, you get the expected output where the .env.dev file overrides the value from .env.local.

Doing it the manual way with docker compose --env-file .env --env-file .env.local --env-file .env.dev config does produce the expected result.

Compose Version

Docker Compose version v5.3.0

Docker Environment

Client:
 Version:    29.6.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  agent: Docker AI Agent Runner (Docker Inc.)
    Version:  v1.98.0
    Path:     /Users/me/.docker/cli-plugins/docker-agent
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.27.0
    Path:     /Users/me/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.35.0-desktop.2
    Path:     /Users/me/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.3.0
    Path:     /Users/me/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.47
    Path:     /Users/me/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.4.1
    Path:     /Users/me/.docker/cli-plugins/docker-desktop
  dhi: CLI for managing Docker Hardened Images (Docker Inc.)
    Version:  v0.0.5
    Path:     /Users/me/.docker/cli-plugins/docker-dhi
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     /Users/me/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/me/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.43.1
    Path:     /Users/me/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.2.5
    Path:     /Users/me/.docker/cli-plugins/docker-model
  offload: Docker Offload (Docker Inc.)
    Version:  v0.6.7
    Path:     /Users/me/.docker/cli-plugins/docker-offload
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.2.0
    Path:     /Users/me/.docker/cli-plugins/docker-pass
  sandbox: "docker sandbox" is deprecated, use Docker Sandboxes instead (Docker Inc.)
    Version:  v0.13.0
    Path:     /Users/me/.docker/cli-plugins/docker-sandbox
  scout: Docker Scout (Docker Inc.)
    Version:  v1.23.1
    Path:     /Users/me/.docker/cli-plugins/docker-scout

Server:
 Containers: 12
  Running: 12
  Paused: 0
  Stopped: 0
 Images: 46
 Server Version: 29.6.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
 runc version: v1.3.6-0-g491b69ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 16
 Total Memory: 46.96GiB
 Name: docker-desktop
 ID: 491a6c9f-6cac-4c22-b16f-8c0db75dc061
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/me/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Anything else?

My guess is that this is due to compose using environment files for variable interpolation (which defaults to just the .env file) happens in its own phase before the inclusion process.

Since there's no easy way to just configure which env files a compose file should use without specifying everything in command line, this leaves no easy alternative on using dotenv hierarchy files and this edge case breaks a pretty typical setup of dotenv files on developer machines.

[ndeloof]

This is known limitation: env_file using while parsing includes is used to process the yaml stream, but not preserved for later use, and can't apply to build definition - this is pretty challenging in terms of compose-go parser architecture.