title	Experiment Catalog
description	Catalog, compare, and analyze experiment runs with a .NET API, Svelte UI, and deterministic development harness.
ms.date	2026-08-28
ms.topic	overview

Experiment Catalog

A tool for cataloging, comparing, and analyzing experiment results. The Experiment Catalog enables teams to track evaluation runs across projects, compare metrics against baselines, and identify performance regressions or improvements in AI/ML experimentation workflows.

Overview

The Experiment Catalog is designed for teams running iterative experiments. It is particularly useful for AI evaluation pipelines where you need to:

• Track results across multiple evaluation runs
• Compare experiment metrics against established baselines
• Analyze performance trends and identify regressions
• Filter and drill down into specific ground-truth results
• Annotate experiments with links to commits, configurations, or documentation

Watch these walkthroughs:

• Installation (6:08)
• Usage (30:56)
• Configuration (16:36)

Architecture

The application consists of several main components:

Component	Description
catalog	ASP.NET Core API that stores experiment data in Azure Blob Storage
MCP interface	MCP tools hosted by the catalog API for project, experiment, and analysis actions
ui	Svelte frontend for visualizing and comparing experiments
catalog.tests	xUnit tests for the catalog API

Key Concepts

• A project is a collection of experiments sharing the same baseline, grounding data, and evaluation configuration. It typically aligns to a sprint. See the experimentation process for details.
• An experiment is a hypothesis-driven collection of evaluation runs within a project.
• A set is a group of results from a single evaluation run, also commonly called a permutation (for example, 3 iterations × 12 ground truths).
• A ref identifies a specific ground-truth entity being evaluated, allowing aggregation across iterations.
• A baseline is a reference point for comparison at the project or experiment level.

Features

Experiment Management

• Create projects and experiments with hypotheses
• Set project-level and experiment-level baselines
• Record arbitrary metrics without pre-definition
• Annotate sets with commit hashes, configuration links, or notes

Comparison & Analysis

• Compare experiment results against baselines
• View aggregate statistics across sets
• Drill down into individual ground-truth results
• Compare metrics across multiple evaluation runs

Filtering Capabilities

• Use the metrics filter to show or hide specific metrics in comparison views
• Use the tags filter to select ground truths by tags extracted from source data
• Use the free filter to write custom expressions that find specific results

Free Filter Examples

# Find poor performers
[generation_correctness] < 0.8

# Find regressions compared to baseline
[generation_correctness] < [baseline.generation_correctness]

# Find significant improvements (>20% better)
[generation_correctness] > [baseline.generation_correctness] * 1.2

# Find absolute metric differences
[generation_correctness] - [baseline.generation_correctness] > 0.05

# Find noisy aggregate metrics by standard deviation
result.metrics["generation_correctness"].std_dev > 0.10

# Find unstable aggregate metrics by coefficient of variation
result.metrics["generation_correctness"].coefficient_of_variation > 0.20

# Complex analysis - retrieval got worse but generation improved
[retrieval_recall] < [baseline.retrieval_recall] AND [generation_correctness] > [baseline.generation_correctness]

# Find specific ground truths
ref == "TQ10" OR ref == "TQ25"

You can find out more about the Free Filter syntax and use cases in the UI README.

Getting Started

Prerequisites

• .NET 10 SDK
• Node.js 20+
• Docker (for containerized deployment)
• Azure Storage Account
• Azure CLI (when using INCLUDE_CREDENTIAL_TYPES=azcli)

Running Locally

Backend API

1. Navigate to the catalog directory:

   cd catalog

2. Create a .env file with required configuration:

   # if using az-cli for login
   INCLUDE_CREDENTIAL_TYPES=azcli
   AZURE_STORAGE_ACCOUNT_NAME=<your-storage-account>
   
   # or if using a connection string
   AZURE_STORAGE_ACCOUNT_CONNSTRING=<your-connection-string>

   Full configuration for the API can be found in the Catalog README.

3. Run the API:

   dotnet run

The API will be available at http://localhost:6010 with API documentation at /scalar/v1.

Frontend UI

1. Navigate to the UI directory:

   cd ui

2. Install dependencies:

   npm install

3. Start the development server:

   npm run dev

The UI will be available at http://localhost:6020.

Docker Deployment

Build the complete application (UI + API) as a Docker container:

docker build --rm -t exp-catalog:latest -f catalog.Dockerfile .

Run the container:

docker run -p 6010:6010 -e AZURE_STORAGE_ACCOUNT_CONNSTRING="<your-connection-string>" exp-catalog:latest

You can instead provide AZURE_STORAGE_ACCOUNT_NAME when the container has access to a supported Azure credential, such as a service principal or managed identity.

ISE OSS Usage Attribution Disclosure

Experiment Catalog deployment automation may include Microsoft ISE OSS usage attribution for Azure Resource Manager (ARM) control-plane operations. This is not Experiment Catalog application telemetry. When enabled, deployment tools append Asset ID acce1e78-0cec-4c66-9e3b-900c69b1c199 to the ARM User-Agent so Microsoft ISE can measure adoption of this open source tool. The catalog application runtime does not add this ID to normal Blob Storage data-plane operations such as creating project containers or writing experiment results.

For ARM control-plane requests that include this Asset ID, Microsoft collects aggregated request telemetry such as tenantId, subscriptionId, userAgent, action, and statusCode. This reporting is intended for aggregate OSS usage measurement and does not collect catalog project names, experiment names, results, prompts, customer data, or Blob Storage contents.

Users are free to opt out. Opting out does not affect Experiment Catalog functionality. Users can opt out by removing the Asset ID from generated deployment artifacts or disabling the attribution flag exposed by those artifacts. For command-based deployment, opt out by not setting AZURE_HTTP_USER_AGENT to include this Asset ID. Underlying tools also have their own telemetry controls:

• Azure CLI telemetry opt-out: https://learn.microsoft.com/cli/azure/azure-cli-configuration#cli-configuration-values
• Terraform AzureRM provider telemetry opt-out: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#disable_terraform_partner_id-1

Microsoft open source telemetry guidance is available at https://docs.opensource.microsoft.com/releasing/general-guidance/telemetry/.

Development Harness

A make-based harness provides deterministic commands for local development and CI. Run all commands from the repository root:

Command	Description
make setup	Install dependencies and prepare the dev environment
make smoke	Build the .NET solution and UI
make lint	Run the configured repository linters
make typecheck	Run type checking across all projects
make check	Run both lint and typecheck
make test	Run the full test suite
make security	Run the configured repository security checks
make ci	CI-equivalent local run (smoke + check + test)

Start with make setup after cloning, then use make ci before pushing changes to verify everything passes locally.

Governance and Branch Policy

This repository uses branch protection and CI checks as quality gates for main.

Required merge policy:

• Pull requests are required for all changes to main.
• At least 2 approvals are required before merge.
• Code owner review is required for protected areas.
• Stale approvals are dismissed when new commits are pushed.
• The latest push must be approved by someone other than its author.
• All review threads must be resolved before merge.
• Required checks must pass before merge: Harness CI, CodeQL / Analyze (csharp), CodeQL / Analyze (javascript), CodeQL / Analyze (python), PR Title Validation / validate-title, and Scorecard analysis.
• Force pushes and branch deletion are blocked on main.

These checks are defined in repository workflows and should be set as required status checks in GitHub branch rules.

Community and Roadmap

Current focus areas:

• Expand API, MCP, and UI test coverage.
• Improve analytics and baseline-comparison usability in the UI.
• Harden operational readiness with clearer SLO reporting.

Contribution and triage cadence:

• New issues are triaged weekly.
• Pull requests are reviewed based on priority and reviewer availability.
• Security reports follow the process in SECURITY.md.

API Usage

All examples for using the API can be found in catalog.http.

Synthetic and Sample Data Provenance

This repository includes sample data intended only for local demos, testing, and validation workflows.

Data provenance policy:

• Sample files are non-production artifacts and must not contain customer or regulated data.
• Any generated or synthetic examples should be clearly labeled in-file and in documentation.
• Contributors must document data origin, generation method, and intended usage when adding new sample datasets.

The catalog service sample files are documented in catalog/README.md.

Community Triage and Ownership

• Issue intake and triage workflow: docs/TRIAGE.md
• Maintainer role model and escalation paths: MAINTAINERS.md