Add experimental filtered IoT Agent installer - #448
Draft
thieman wants to merge 12 commits into
Draft
Conversation
Generate install_script_agent7_iot.sh from the common Agent template with Agent 7 metadata and a target-specific IoT mode flag. Existing common-template targets explicitly disable the mode so every generated script resolves the placeholder. Wire the ignored artifact into Make clean/all and tagged manual deployment, document its future filtered normal-Agent purpose, and teach local installation checks its package, major version, and install-info identity. Package filtering remains intentionally deferred to the follow-up tasks.
Run a lightweight contract check immediately after script generation so unresolved placeholders, incorrect IoT modes, metadata, or executable bits fail in CI. Keep the IoT output available as a generated artifact while removing it from the public deploy matrix until package filtering is implemented.
Add extracted helpers that validate the generated IoT mode, render deterministic dpkg filters, and derive bounded RPM exclusion prefixes from the package manifest. The helpers retain only the required Agent, data-plane, rtloader, support-view, and IoT check assets without changing any live package-manager branch.\n\nValidate filtered layouts, query installed package versions by family, and atomically write the durable install profile. Add shunit coverage for compatible and rejected options, filter ordering and retention, RPM quoting and bounds, aggregated layout failures, marker replacement failures, and package queries.
Validate the full retained IoT payload and every representative pruned class while tolerating empty dpkg-preserved directories and dangling links. Use targeted DEB and RPM exclusions so Agent SSL and support assets remain available, with explicit DEB parent re-includes. Reject RPM exclusion-prefix collisions before producing output, and cover missing retained content, pruned payloads, parent rules, support retention, and collision cases with shunit tests.
Write the dpkg filter and install profile through same-directory temporary files, replace exact destinations atomically, and reject unsafe post-replacement types. Broaden filtered layout validation across retained file types, disallowed payload classes, and live links while keeping RPM exclusion deduplication compatible with Bash 3. Extend shunit coverage for each failure mode.
Apply the same Python-script, requirements, and shared MS ODBC pruning contract to DEB filters, RPM exclusion fixtures, and post-install validation. Capture every validator find traversal in a checked NUL-delimited temporary listing so inspection errors are aggregated and validation fails closed without relying on process-substitution status.
Wire the generated Agent 7 IoT variant into fresh-install-only option validation and force the normal datadog-agent package with IoT infrastructure mode. Install the dpkg path filter atomically before package unpack, retain the package installer only for post-install configuration, then persist the final filter, validate the retained layout, and capture the installed package version. Add orchestration, generated-script, local container, checksum, and dedicated Ubuntu/Debian CI coverage while keeping every existing generated installer byte-identical and excluding the draft from deploy.
Pass the installer-facing DD_AGENT_MINOR_VERSION variable so the filtered Debian job actually selects and verifies Agent 7.82. Extend the generated-script contract to catch an unconnected pinned-version variable.
Exercise the persistent Debian filter through the localtest decision helper so retained Agent, ADP, rtloader, views, and IoT check paths stay included while pruned payload classes stay excluded. Also reject the transient installer override and any retained installer binary, with a generated-script contract requiring the real-install check.
Force IoT infrastructure mode into the Debian package environment and normalize the fresh Agent configuration after postinst, while leaving generated ordinary installers byte-for-byte unchanged. Treat every queryable dpkg state as pre-existing and add a rollback transaction around the transient dpkg path filter. The transaction snapshots prior content, ownership, and mode in private storage, restores or removes the filter on APT and EXIT failures, and is disarmed only after the persistent policy and filtered layout validate. Strengthen filtered checksum evidence with a required dpkg md5 manifest and accurate retained-file reporting. Run orchestration regressions in GitLab, keep the pinned 7.82 job push-only, and document fresh-install and upgrade constraints plus the experimental same-version disk result. Tests cover unset infrastructure input, all dpkg states, empty checksum evidence, silent debsums success, and APT rollback with and without a prior filter.
Clarify that the experimental filtered installer currently supports Debian and Ubuntu only. Record the RPM post-install installer dependency that prevents retaining a verified RPM path in this draft, and add the feature to the unreleased changelog.
Publish the reviewed filtered IoT profile only after Debian package, layout, configuration, and service handling complete. Stage the profile outside the root context, install it atomically as root:root mode 0644, and verify its exact content against the captured package version and installer identity before success telemetry.\n\nUpdate generated-script, shunit, local container, and documentation coverage for publication ordering, controlled pre-marker failure, prior-marker preservation, and the Agent companion-guard contract while keeping ordinary generated installers byte-identical.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an experimental
install_script_agent7_iot.shvariant that installs the normal Agent 7 DEB with a reduced IoT filesystem profile instead of installing the separately packageddatadog-iot-agentartifact.The generated script:
datadog-agentpackage;infrastructure_mode: iot;dpkgpath-filter policy before package unpack;/etc/datadog-agent/install_profilemarker only after package, configuration, layout, and service handling succeed;Existing generated installers remain byte-for-byte unchanged. The new script is generated as a CI artifact but is intentionally absent from the public deploy matrix while this remains a prototype.
Disk measurement
Same-version Agent 7.82.2 comparison on Ubuntu 22.04:
A prior same-version allocated-block comparison measured a 77.81% reduction. These are experimental measurements rather than CI thresholds and vary with package version and filesystem.
Validation
Prototype limitations
%postrequires the embedded installer before the final filtered state; reconciling that with RPM's recorded file state needs a separate design.install_profile. A companion Agent change should requireinfrastructure_mode: iotand strict capability guards wheneverprofile: iot-filteredis present.