[Enh]: Telemetry in CLI

[JerryNixon]

What?

Add usage telemetry to the CLI.

Why?

Use telemetry to answer product questions, not surveillance.

Opt out

Support environment variable DAB_TELEMETRY_CLI_OPT_OUT=1 or true.

Disclosure

We will detect the first time a DAB subcommand is used after installation and emit:

Telemetry
---------
The DAB tools collect usage data to help us improve your experience. The data is collected by Microsoft. You can opt out of telemetry by setting the DAB_TELEMETRY_CLI_OPT_OUT environment variable to '1' or 'true' using your favorite shell.

Read more about DAB CLI Tools telemetry: https://aka.ms/dab/docs/cli

To disable this message and the DAB welcome message, set the DAB_NOLOGO environment variable to 1 or true. This variable has no effect on telemetry opt out. After the first time a DAB subcommand is used, the DAB CLI will somehow set this variable to true.

Note

This will almost certainly require a sentinel file, but that is an ENG decision.

Important

We need a plan for upgrade, reinstall, and uninstall behavior.

Allow-listed data only

Collect only these fields:

Good	Bad
command name, subcommand, known switches used, DAB version, OS, install mode, CI yes/no, duration bucket, success/failure, exception type, application_name	connection strings, file paths, entity names, table names, view names, stored procedure names, descriptions, database policies, hostnames, tenant IDs, Key Vault references, environment variable values

Exception data

If the DAB CLI crashes, it collects the exception name and the stack trace of the CLI code. The DAB CLI collects this information to assess problems and improve quality.

The DAB CLI collects information for CLI exceptions only, not exceptions in your application. The collected data contains the exception name and the stack trace. This stack trace is from CLI code.

We will follow this approach: https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry?tabs=dotnet10#crash-exception-telemetry

Important

We need to be sure CLI stack traces cannot accidentally include user paths, config values, or other sensitive strings in exception messages.

Instrumentation point

Instrument once at the command dispatcher.

Add CLI middleware around every command to emit:

• CommandStarted
• CommandCompleted
• CommandFailed

Transport

Keep transport simple and non-blocking.

Use Application Insights. Embed the default connection string in the CLI.

Tip

Microsoft recommends the Azure Monitor OpenTelemetry Distro instead of the classic Application Insights SDK. We will use this only if it makes sense to us.

Support a hidden override for testing:

DAB_TELEMETRY_AI_CONN_STRING=XYZ

When set, this overrides the internal Application Insights connection string.

Commands to instrument

Instrument every DAB CLI subcommand:

• init
• add
• update
• configure
• auto-config
• start
• validate
• add-telemetry
• export

Flag capture rules

Capture every known flag used, but never its value.

Normalize flags to their canonical name.

Examples:

• --connection-string → capture --connection-string
• --config dab-config.json → capture --config
• --permissions "anonymous:*" → capture --permissions
• -c → capture --config
• --runtime.mcp.enabled → capture --runtime.mcp.enabled

Important

Make sure this means allow-listed switches only, resolved after alias normalization, with no fallback that logs unknown arguments.

Drop unknown flags.

Capture repeated flags once only.

Existing context

Include the current DAB-specific telemetry application_name: dab_oss_1.2.3+xxx|123|123|123+

Note

There will be no context group in the connection string. It is shown here as xxx.

Additional context

Detect additional context values for the payload.

Context	Value
Multi-config	N = quantity
CI Environment	1 = yes, 0 = no
LLM Environment	1 = yes, 0 = no

We might need to extend this as we continue.

Example payload

{
  "dab_version": "1.2.3",
  "command": "add",
  "flags": ["--source", "--permissions", "--description"],
  "app_name": "dab_oss_1.2.3+xxx|123|123|123+",
  "multi_config": 1,
  "ci_env": 1,
  "llm_env": 1,
  "os": "windows",
  "duration_ms_bucket": "100-500",
  "success": true
}

Duration buckets:

• <10
• 10-100
• 100-500
• 500-1000
• >1000

os values:

• windows
• linux
• macos

os values rules:

1. Lowercase only
2. Map from RuntimeInformation.OSDescription or OSPlatform
3. Do not include version, distro, or build numbers
4. Always emit one value