Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/command-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,13 @@ docket export --resource 'dokku_config[app=api]' --output -
The correctness contract is idempotency: applying an exported pair back to the same server reports
no drift (`plan` shows every task `[ok]`).

Export only reads a task type when the server has the plugins it needs. A task that needs a
third-party plugin - dokku-acl, dokku-http-auth, dokku-maintenance, dokku-letsencrypt, or
dokku-global-cert for the global certificate - is skipped on a server that does not have that plugin
installed: export reads `plugin:list` once, runs none of that plugin's commands, prints no warning
for it, and `--verbose` prints no line for it. A plugin that is installed but fails is still warned
about.

The two halves are written with different modes, because they hold different things. The recipe
carries interpolations rather than values, so it lands at `0644` like every other file docket
writes. The vars-file holds the values themselves in the clear - every `config` value, the
Expand Down
7 changes: 7 additions & 0 deletions docs/embedding.md
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,13 @@ the version could not be read or parsed, and is nil when the server is simply ol
set `ExportOptions.IgnoreVersion` to read it anyway. That option wins over a gate already on the
context.

`sdk.ExportRecipe` also skips an exporter whose task needs a plugin the server does not have
installed - `dokku_acl_app` without dokku-acl, the global `dokku_certs` without dokku-global-cert -
whether or not the gate is on. It reads `plugin:list` once per run, sharing the gate's read when there
is one, and a skipped exporter adds no warning and no progress event. When the plugin list cannot be
read, every exporter runs. A task's own `ExportApp` or `ExportGlobal` makes no such check and runs
the plugin's command regardless.

### Listing task types

`sdk.TaskTypes()` returns every registered task type, sorted. `sdk.Lookup` reports whether a type is
Expand Down
2 changes: 2 additions & 0 deletions docs/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ which servers an app holds a credential for but nothing about the credential its
to supply at apply time. Each task's [reference page](tasks/README.md) has an Export support
section noting its limits. Those warnings mask the secrets the export read, so the log of a
migration run is safe to paste into a ticket even though the pair of files it wrote is not.
A task whose plugin the old server does not have installed - dokku-acl, for example - is not read at
all, so it adds nothing to the recipe and no warning to the log.

`tasks.vars.yml` is written `0600` on the box that produced it, which says nothing about where you
put it next: move it the way you would move a private key, and delete it once the migration is done.
Expand Down
18 changes: 6 additions & 12 deletions internal/tasks/acl_service_task.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package tasks

import (
"context"
"errors"
"fmt"

"github.com/dokku/docket/internal/subprocess"
Expand Down Expand Up @@ -70,26 +69,21 @@ func (t AclServiceTask) MinimumVersions() VersionSupport {
// service on the server. Discovery is via listServices; the members are read
// from `acl:list-service <type> <name>` (reusing getAclServiceUsers). Note the
// field inversion versus the other service tasks: Service holds the instance
// name and Type the datastore type. Services with no ACL entries - and every
// service when the dokku-acl plugin is absent - are skipped. state:set
// replaces the whole list for an exact match.
// name and Type the datastore type. Services with no ACL entries are skipped,
// and a failed read is returned. state:set replaces the whole list for an
// exact match.
func (t AclServiceTask) ExportGlobal(ctx context.Context) ([]interface{}, error) {
services, err := listServices(ctx)
if err != nil {
return nil, err
}
var out []interface{}
for _, s := range services {
// The export engine does not run this exporter on a server without
// dokku-acl (#672), so a failure here is the installed plugin's own.
users, err := getAclServiceUsers(ctx, s.Type, s.Name)
if err != nil {
var sshErr *subprocess.SSHError
if errors.As(err, &sshErr) {
return nil, err
}
// acl:list-service fails when the dokku-acl plugin is not installed;
// treat that (and any other dokku-level failure) as "no ACL" so
// export stays quiet on servers without dokku-acl.
continue
return nil, err
}
if len(users) == 0 {
continue
Expand Down
10 changes: 6 additions & 4 deletions internal/tasks/certs_task.go
Original file line number Diff line number Diff line change
Expand Up @@ -578,14 +578,16 @@ func (t CertsTask) ExportGlobal(ctx context.Context) ([]interface{}, error) {

// exportCert probes whether a certificate is installed for the given scope (app
// or global) and, if so, reads its PEM back via certs:show / global-cert:show.
// A transport failure aborts the export; any other probe error (for example the
// dokku-global-cert plugin not being installed) is swallowed so the scope is
// skipped silently.
// A transport failure aborts the export, as does any global probe error: the
// export engine does not run the global read on a server without
// dokku-global-cert (#672), so a failure there is the installed plugin's own.
// Any other app probe error is swallowed so the app's scope is skipped
// silently.
func exportCert(ctx context.Context, t CertsTask) ([]interface{}, error) {
enabled, err := certsEnabled(ctx, t)
if err != nil {
var sshErr *subprocess.SSHError
if errors.As(err, &sshErr) {
if t.Global || errors.As(err, &sshErr) {
return nil, err
}
return nil, nil
Expand Down
95 changes: 92 additions & 3 deletions internal/tasks/export.go
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,88 @@ type ExportResult struct {
// filter narrows emission to the --resource addresses. nil for an
// unrestricted export, where every check it performs answers "keep".
filter *resourceFilter

// plugins is the server's installed plugin list, read the first time an
// exporter that needs a plugin is reached (see missingPlugin). pluginsRead
// says that read has happened, and pluginsKnown that it succeeded.
plugins map[string]string
pluginsRead bool
pluginsKnown bool
}

// missingPlugin reports whether the exporter of proto needs a plugin the
// server does not have installed, in which case the export does not run it
// (#672): its read would only fail, or come back empty, and say nothing the
// user needs. global says whether the exporter reads the global play, where a
// requirement gated on the task's Global field (dokku_certs and global-cert)
// applies.
//
// Only a plugin a task names outright is checked. A requirement named by a
// field (the datastore tasks' service type) has no name on a prototype, and
// those exporters only find services through service-list, which lists the
// installed datastore plugins' services alone.
//
// The plugin list is read once per run. When it cannot be read the exporter
// runs as it would have without the check, and reports whatever its own read
// returns.
func (res *ExportResult) missingPlugin(ctx context.Context, proto Task, global bool) bool {
support, ok := TaskMinimumVersions(proto)
if !ok || len(support.Plugins) == 0 {
return false
}
scoped := exportScopePrototype(proto, global)
var needed []string
for _, plugin := range support.Plugins {
if !pluginApplies(scoped, plugin) {
continue
}
if name, ok := pluginListName(scoped, plugin); ok {
needed = append(needed, name)
}
}
if len(needed) == 0 {
return false
}
if !res.pluginsRead {
res.pluginsRead = true
plugins, err := installedPlugins(ctx)
res.plugins, res.pluginsKnown = plugins, err == nil
}
if !res.pluginsKnown {
return false
}
for _, name := range needed {
if _, installed := res.plugins[name]; !installed {
return true
}
}
return false
}

// exportScopePrototype returns a copy of proto for the scope an exporter
// reads: with its Global field set for the global play, and left zero for an
// app's. proto is the pointer Lookup returns; a task without a Global field
// is returned as-is.
func exportScopePrototype(proto Task, global bool) Task {
if !global {
return proto
}
value := reflect.ValueOf(proto)
if value.Kind() != reflect.Pointer || value.IsNil() {
return proto
}
copied := reflect.New(value.Elem().Type())
copied.Elem().Set(value.Elem())
field, ok := walkStructField(copied.Elem(), "Global")
if !ok || field.Kind() != reflect.Bool || !field.CanSet() {
return proto
}
field.SetBool(true)
task, ok := copied.Interface().(Task)
if !ok {
return proto
}
return task
}

// noteSensitive records values this export has identified as credential
Expand Down Expand Up @@ -459,6 +541,12 @@ func (res *ExportResult) SensitiveValues() []string {
// with a warning and listed in Report.VersionErrors. Set
// ExportOptions.IgnoreVersion to read it anyway.
//
// An exporter whose task needs a plugin the server does not have installed -
// dokku_acl_app without dokku-acl, the global dokku_certs without
// dokku-global-cert - is not run at all, so it adds no warning and no
// progress event. Calling a task's ExportApp or ExportGlobal directly does
// not check; it runs the plugin's command regardless.
//
//sdk:export ExportRecipe
func GatedExportRecipe(ctx context.Context, opts ExportOptions) (*ExportResult, error) {
if !opts.IgnoreVersion {
Expand All @@ -469,7 +557,8 @@ func GatedExportRecipe(ctx context.Context, opts ExportOptions) (*ExportResult,

// ExportRecipe reads the live Dokku server (via the current subprocess host)
// and assembles a recipe describing it. It enumerates apps, runs every
// registered AppExporter for each, lifts sensitive values into a vars map
// registered AppExporter whose plugins are installed (see missingPlugin) for
// each, lifts sensitive values into a vars map
// (unless opts.Inline), and returns the result for the caller to marshal.
// A --resource run whose addresses all pin an app or are global does not
// enumerate: it reads only the pinned apps (#567).
Expand Down Expand Up @@ -610,7 +699,7 @@ func (res *ExportResult) exportGlobalPlay(ctx context.Context, scope ExportProgr
continue
}
exporter, ok := proto.(GlobalExporter)
if !ok {
if !ok || res.missingPlugin(ctx, proto, true) {
continue
}
opts.progress(scope.withType(typeKey))
Expand Down Expand Up @@ -752,7 +841,7 @@ func (res *ExportResult) exportAppPlay(ctx context.Context, scope ExportProgress
continue
}
exporter, ok := proto.(AppExporter)
if !ok {
if !ok || res.missingPlugin(ctx, proto, false) {
continue
}
opts.progress(scope.withType(typeKey))
Expand Down
26 changes: 19 additions & 7 deletions internal/tasks/export_dependencies_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -159,11 +159,7 @@ func TestExportAppSkipDependencies(t *testing.T) {
if got := globalAddresses(t, res); got != nil {
t.Errorf("SkipDependencies exported a global play: %v", got)
}
for _, call := range calls {
if strings.Contains(call, "network:list") || strings.Contains(call, "plugin:list --format json") {
t.Errorf("SkipDependencies still read %q", call)
}
}
assertNoGlobalReads(t, calls, "SkipDependencies still read")
}

func TestExportAppWithoutDependenciesReadsNoGlobals(t *testing.T) {
Expand All @@ -178,10 +174,26 @@ func TestExportAppWithoutDependenciesReadsNoGlobals(t *testing.T) {
if got := globalAddresses(t, res); got != nil {
t.Errorf("an app with no dependencies exported a global play: %v", got)
}
assertNoGlobalReads(t, calls, "an app with no dependencies read")
}

// assertNoGlobalReads fails when calls show a global exporter ran. network:list
// is dokku_network's read. plugin:list --format json is read once by the
// installed-plugin check the app play makes before an exporter that needs a
// plugin (#672), so a second read is dokku_plugin's exporter.
func assertNoGlobalReads(t *testing.T, calls []string, prefix string) {
t.Helper()
pluginLists := 0
for _, call := range calls {
if strings.Contains(call, "network:list") || strings.Contains(call, "plugin:list --format json") {
t.Errorf("an app with no dependencies read %q", call)
if strings.Contains(call, "network:list") {
t.Errorf("%s %q", prefix, call)
}
if strings.Contains(call, "plugin:list --format json") {
pluginLists++
}
}
if pluginLists > 1 {
t.Errorf("%s plugin:list --format json %d times, want at most once for the installed-plugin check", prefix, pluginLists)
}
}

Expand Down
4 changes: 2 additions & 2 deletions internal/tasks/export_masking_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -131,10 +131,10 @@ func TestExportRegistersHttpAuthHashes(t *testing.T) {
// masks for the same task, and over-masking is the safe direction.
func TestExportRegistersEveryLetsencryptPropertyValue(t *testing.T) {
t.Parallel()
ctx := subprocess.ContextWithRunner(testCtx(), fakeDokku(map[string]string{
ctx := subprocess.ContextWithRunner(testCtx(), fakeDokku(withInstalledPlugins(map[string]string{
"--quiet apps:list": "web",
"--quiet letsencrypt:report web --format json": `{"email":"admin@example.com","dns-provider-CLOUDFLARE_API_TOKEN":"tok3n"}`,
}))
}, "letsencrypt")))

res, err := ExportRecipe(ctx, ExportOptions{})
if err != nil {
Expand Down
Loading
Loading