Skip to content

frontend/dockerui: share context load across bake targets#6729

Open
vanackere wants to merge 1 commit intomoby:masterfrom
vanackere:fix/dockerui-shared-context-followpaths
Open

frontend/dockerui: share context load across bake targets#6729
vanackere wants to merge 1 commit intomoby:masterfrom
vanackere:fix/dockerui-shared-context-followpaths

Conversation

@vanackere
Copy link
Copy Markdown

When buildx runs a multi-target bake with the same local context, it can pass that context as a shared session with local-sessionid:context. The dockerfile frontend still computes FollowPaths per target from the context files reachable by that target and passes the filter to MainContext.

Small reproducer:

# Dockerfile
FROM alpine AS app
RUN --mount=type=bind,source=app/cfg,target=/cfg cat /cfg

FROM alpine AS tools
RUN --mount=type=bind,source=tools/cfg,target=/cfg cat /cfg

# docker-bake.hcl
target "app"   { context = "."; target = "app" }
target "tools" { context = "."; target = "tools" }
group "default" { targets = ["app", "tools"] }

With app/cfg and tools/cfg present, a bake of both targets can show two separate context-load vertices before this change:

=> [app internal] load build context
=> [tools internal] load build context

The targets compute different FollowPaths values, so the marshaled llb.Local("context") ops differ even though they use the same shared local session.

For shared-session main contexts, clear FollowPaths after applying the caller-provided local options. Non-shared builds keep the existing per-target FollowPaths optimization.

Add a dockerui unit test covering the option behavior: non-shared sessions preserve FollowPaths, while shared sessions clear it and produce identical context source ops for divergent caller filters.

On my real-world - and quite complex - multi-stage Dockerfile this patch significantly improves the build time (times with cold/empty build cache):

Before:

  • build time: 1261s
  • context transfers: 4

After:

  • build time: 870s
  • context transfers: 3

When buildx runs a multi-target bake with the same local context, it can pass that context as a shared session with local-sessionid:context. The dockerfile frontend still computes FollowPaths per target from the context files reachable by that target and passes the filter to MainContext.

Small reproducer:

    # Dockerfile
    FROM alpine AS app
    RUN --mount=type=bind,source=app/cfg,target=/cfg cat /cfg

    FROM alpine AS tools
    RUN --mount=type=bind,source=tools/cfg,target=/cfg cat /cfg

    # docker-bake.hcl
    target "app"   { context = "."; target = "app" }
    target "tools" { context = "."; target = "tools" }
    group "default" { targets = ["app", "tools"] }

With app/cfg and tools/cfg present, a bake of both targets can show two separate context-load vertices before this change:

    => [app internal] load build context
    => [tools internal] load build context

The targets compute different FollowPaths values, so the marshaled llb.Local("context") ops differ even though they use the same shared local session.

For shared-session main contexts, clear FollowPaths after applying the caller-provided local options. Non-shared builds keep the existing per-target FollowPaths optimization.

Add a dockerui unit test covering the option behavior: non-shared sessions preserve FollowPaths, while shared sessions clear it and produce identical context source ops for divergent caller filters.
@vanackere
Copy link
Copy Markdown
Author

@tonistiigi this commit replaces #6725 (this one was definitely wrong, sorry for that I messed-up while trying to make a minimal case to reproduce my issue)

I made a small reproducer case here if it helps: https://github.com/vanackere/misc/tree/main/docker-debug/context-load

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant