Architecture

A Docker-out-of-Docker development environment for building Splunk applications — custom search commands (Python), React UI apps with Dashboard Studio components, and any combination of backend + frontend Splunk apps.

What This Template Supports

Custom search commands — Python chunked-protocol commands, modular inputs, REST handlers, alert actions
React UI apps — built with @splunk/create, using @splunk/dashboard-core and Dashboard Studio exported components
Multi-app development — multiple Splunk apps developed and tested simultaneously
Simultaneous dev + staging — dev on port 8000 (bind-mounted, live reload) and staging on port 18000 (packaged apps mounted from splunk/stage/)
Python debugging — VSCode attaches to debugpy inside the Splunk container via SA-VSCode add-on
React debugging — Chrome DevTools via source maps in Splunk Web (webpack watch updates stage/ live)

System Overview

┌──────────────────────┐     ┌──────────────────────────────────────┐
│  Dev Container       │     │  Docker Compose                      │
│  (tools only)        │     │                                      │
│  Node 22 / Python 3.12    │  splunk-dev     (port 8000, 8089)    │
│  Docker CLI / Go Task│────▶│    bind-mounted apps, skip-provision │
│  AppInspect / ruff   │     │                                      │
│  debugpy             │     │  splunk-staging (port 18000, 18089)  │
└──────────────────────┘     │    mounted apps from splunk/stage/   │
        │                    └──────────────────────────────────────┘
        │                                    ▲
        ├── splunk/config/apps/ ─────────────┘  (bind-mounted + symlinked)
        └── react/             → stage/ symlinked → /opt/splunk/etc/apps/<app>/

Directory Structure

.devcontainer/
  devcontainer.json              # Tools-only container (Node, Python, Docker CLI, Task)
  docker-compose.yml             # Dev Splunk (target: dev, bind mounts, port 8000)
  docker-compose.staging.yml     # Staging Splunk (reuses dev image, mounts splunk/stage/, port 18000)
  post-create.sh                 # Tool install + Splunk image build
splunk/
  Dockerfile                     # Multi-stage: base → dev
  entrypoint-wrapper.sh          # Skip-provision + auto-discover /tmp/apps/*.tgz for SPLUNK_APPS_URL
  config/
    apps/                        # Splunk app source directories (symlinked into Splunk)
      splunk-config-dev/         # Dev-mode web.conf (js_no_cache, enableWebDebug, etc.)
      <your_app>/                  # Created via `task app:create APP_NAME=<your_app>`
        bin/                     # Python scripts (custom commands, modular inputs)
          lib/                   # Shared Python libraries
        default/                 # commands.conf, searchbnf.conf, restmap.conf, app.conf
        metadata/
    deps.yml                     # Splunkbase dependencies (MLTK, SA-VSCode, etc.)
  stage/                         # Built tarballs (.tgz) — gitignored, created on demand
react/                           # React/JS source (monorepo via @splunk/create) — created on demand
  packages/
    main/                        # Shared component library (@splunk/main)
    <app>/                       # Splunk app package (scaffolded by @splunk/create)
      src/main/
        webapp/pages/            # React page entry points (one bundle per page)
        resources/splunk/        # Splunk app config (app.conf, views, nav, templates)
      stage/                     # Webpack output — a COMPLETE Splunk app dir (gitignored)
      webpack.config.js          # Outputs to stage/appserver/static/pages/
Taskfile.yml                     # All automation
.env                             # Secrets/config — gitignored
splunk.env.example               # Template for .env
AGENTS.md                        # Concise repo summary + windloop hint
ARCHITECTURE.md                  # This file — full architecture docs
.vscode/launch.json              # Debug configs (Python attach, Chrome, React dev server)
tests/e2e/                       # E2E test scripts (devcontainer + Splunk lifecycle)
.ref/                            # Reference repos (docker-splunk, kveditor) — gitignored

Environment Setup

Copy splunk.env.example to .env and fill in your values. .env is gitignored.

cp splunk.env.example .env

Devcontainer (recommended)

LOCAL_WORKSPACE_FOLDER is injected automatically via runArgs in devcontainer.json — no manual setup needed. It resolves to the Mac host path of this repo, which is required so docker compose bind mounts point to the correct location on the Docker host.

Native Mac (without devcontainer)

docker compose resolves relative paths in docker-compose.yml relative to the compose file location on the Mac host — so bind mounts work correctly without any extra configuration. Just run task commands directly from the repo root.

Example: App created via `task app:create`

splunk/config/apps/
  splunk-config-dev/                     # Dev settings (always present, ships with template)
  my_custom_app/                         # Created via: task app:create APP_NAME=my_custom_app
    bin/                                 # Python scripts (custom commands, modular inputs)
    default/
      app.conf                           # [package] id = my_custom_app, [ui] label = my custom app
      commands.conf                      # Custom search command definitions
      data/ui/views/                     # Dashboard XML views
      data/ui/nav/default.xml
    metadata/default.meta

App Naming Model

APP_NAME is the single identifier that ties everything together:

Concept	Value	Location
Folder name	`APP_NAME`	`splunk/config/apps/<APP_NAME>/`
Package ID (`app.conf [package] id`)	`APP_NAME`	Splunkbase, REST API
UI Label (`app.conf [ui] label`)	`APP_NAME` with underscores → spaces	Splunk Web
React source (if applicable)	`APP_NAME`	`react/packages/<APP_NAME>/`

Set APP_NAME in .env to identify the primary app you're developing. All app:*, react:*, and CI/CD commands use it as the default when no explicit APP_NAME= argument is passed.

For React-based apps, @splunk/create outputs a complete Splunk app to react/packages/<APP_NAME>/stage/. In dev, react:link symlinks stage/ directly into Splunk's etc/apps/ — no separate app skeleton needed. react:package builds and packages stage/ as a tgz for staging image bake-in.

Helper apps (like splunk-config-dev) are always present and don't need APP_NAME.

Commands

Dev Splunk Lifecycle

task dev:build-image        # build dev Splunk Docker image (first time / Dockerfile change)
task dev:up                 # start + sync app symlinks (no rebuild)
task dev:refresh            # reload configs + static assets via REST API (~2s)
task dev:restartd           # restart splunkd process (~10s)
task dev:restart            # restart container (skip-provision ~30s)
task dev:reprovision        # force full Ansible re-provisioning
task dev:down               # stop container
task dev:clean              # stop + remove volumes (full reset)
task dev:logs               # follow container logs
task dev:status             # check container status

Staging Splunk (port 18000) — runs alongside dev

task stage:deploy           # package + start + install (full pipeline)
task stage:package          # package all apps to splunk/stage/
task stage:up               # start staging container + health wait
task stage:install          # install tgz into running staging
task stage:down             # stop staging container
task stage:clean            # stop staging + remove volumes
task stage:logs             # follow staging container logs

App Development

task app:create APP_NAME=x  # scaffold + symlink + refresh Splunk (no recreate)
task app:package APP_NAME=x # package to splunk/stage/x.tgz (for staging)

Which one do I need?

Situation	Command
New app just created, want it visible in Splunk	`app:create` (calls sync-links automatically)
Symlinks got out of sync (e.g. after `dev:clean`)	`dev:ensure-links`
Preparing a release build for staging	`stage:deploy` (packages + starts + installs)

dev:ensure-links — fastest (~1s). Makes the app visible via symlink. Enough for .conf edits, dashboards, and Python scripts (no packaging needed). Called automatically by dev:up and app:create.
package — produces a .tgz in splunk/stage/. Used as input for staging (stage:install installs from there). Not needed for day-to-day dev.

React UI

task react:create           # scaffold + initial build, syncs APP_NAME in .env (interactive)
task react:add-page         # add a page/component via @splunk/create (interactive)
task react:link             # symlink stage/ into dev Splunk (auto-builds if stage/ missing)
task react:start            # webpack watch — stage/ updates live via symlink
task react:build            # production webpack build (always rebuilds stage/)
task react:package          # build + package stage/ as splunk/stage/<APP_NAME>.tgz for staging

Dependencies, Python & Testing

task deps:install           # install Splunkbase deps from deps.yml (idempotent)
task python:lint            # ruff check
task python:format          # ruff format
task python:test            # pytest
task test:lifecycle          # Splunk lifecycle tests — 7 suites (any host)
task test:devcontainer      # build devcontainer + static checks + lifecycle
task test:all               # lint + devcontainer

Developer Workflow

First-Time Setup

Open repo in VS Code → "Reopen in Container"
post-create.sh installs tools and builds the Splunk image
task dev:up → starts Splunk, syncs app symlinks (full Ansible provisioning on first boot, ~50s)
task deps:install → install Splunkbase dependencies (SA-VSCode, MLTK, etc.)

Creating a New App

task app:create APP_NAME=my_app → scaffolds app, creates symlink, refreshes Splunk (~2-10s, no container recreation)
App is immediately visible in Splunk
Edit .conf files → task dev:refresh to reload (~2s, no restart)

Creating a React App

task react:create → scaffolds via @splunk/create, detects app name, updates .env, runs initial build
task react:link → symlinks stage/ into Splunk etc/apps/ (run once, or after dev:clean)
task react:start → webpack watch; stage/ updates live through the symlink
task react:add-page → add more pages interactively via @splunk/create
task react:package → build + package stage/ as tgz for staging

Daily Dev Loop (fastest → slowest)

Change	Command	Time
Edit `.conf` / dashboard	`task dev:refresh`	~2s
Edit Python code	`task dev:restartd`	~10s
New app	`task app:create` (symlink + refresh)	~2-10s
Dockerfile change	`task dev:build-image` then `task dev:up`	varies
Full reset	`task dev:clean` then `task dev:up`	~90s

Staging Verification

task stage:deploy → packages all apps, starts staging, installs via CLI
- Or manually: task stage:package → task stage:up → task stage:install
For React apps: task react:package first (puts tgz in splunk/stage/), then task stage:deploy

How It Works

Multi-Stage Dockerfile

The splunk/Dockerfile has two stages:

base — installs acl + entrypoint wrapper on the official Splunk image
dev — entrypoint-wrapper for skip-provision; used by both dev and staging compose files

Staging reuses the dev image. Packaged apps are bind-mounted from splunk/stage/ → /tmp/apps/; the entrypoint auto-discovers and installs them on first start.

Dev + Staging Side by Side

Dev and staging run as separate compose projects with different ports:

	Dev	Staging
Compose file	`docker-compose.yml`	`docker-compose.staging.yml`
Container	`splunk-dev`	`splunk-staging`
Web	`:8000`	`:18000`
REST API	`:8089`	`:18089`
HEC	`:8088`	`:18088`
Apps	Bind-mounted (live edit)	Mounted from splunk/stage/ (auto-installed on first start)
Entrypoint	entrypoint-wrapper.sh	entrypoint-wrapper.sh (same)

Skip-Provision Entrypoint

First start: No marker → full Ansible → writes /opt/splunk/var/.provisioned
Subsequent starts: Marker exists → starts splunkd directly (~30s vs ~90s)
Force reprovision: task dev:reprovision removes marker and restarts
Clean reset: task dev:clean removes volumes (including marker)

The wrapper also writes the Docker healthcheck state file so checkstate.sh passes in skip-provision mode.

Build / Up Separation

The Splunk image is built once during post-create.sh (or explicitly via task dev:build-image). task dev:up only starts the container — it does not rebuild. This avoids unnecessary rebuilds during daily development.

App Symlink Mounts

The entire splunk/config/apps/ directory is bind-mounted to /opt/splunk/dev-apps/ inside the container. Symlinks in /opt/splunk/etc/apps/ point to each app under /opt/splunk/dev-apps/. This means:

No container recreation when adding a new app — just docker exec ln -s + refresh
Live editing — file changes on the host are immediately visible through the symlink
task dev:ensure-links manages symlinks (creates missing, removes stale)
This pattern is validated by Splunk's own @splunk/create tooling (yarn run link:app)

Change type	Action needed
`.conf` files, dashboards	`task dev:refresh` (~2s, no restart)
Python code in `bin/`	Re-run the search command (no restart)
Python code needing restart	`task dev:restartd` (~10s)
React source (`src/`)	`task react:start` (webpack watch → `stage/` live via symlink)
React production build	`task react:package` (build + package tgz for staging)
Dashboard Studio JSON	Edit `definition.json` → HMR picks it up
New app added	`task app:create` (symlink + refresh ~2-10s)

Persistence

splunk-var volume — indexed data, KV store, search artifacts, provisioning marker
splunk-etc volume — Splunk config, installed apps, and app symlinks; persists across container recreation
/opt/splunk/dev-apps/ — bind mount of splunk/config/apps/ (live source)
splunk/stage/ — built tarballs mounted to /tmp/apps in the container

task dev:clean removes all volumes for a full reset.

React + Dashboard Studio Dev Loop

Scaffold: task react:create → runs @splunk/create interactively, creates monorepo under react/packages/, syncs APP_NAME in .env, runs initial build
Link: task react:link → symlinks react/packages/<app>/stage/ into Splunk etc/apps/<app>/
Add dashboard page: task react:add-page → @splunk/create offers "Add a Dashboard Page"
Install dashboard packages: yarn add @splunk/dashboard-core @splunk/dashboard-presets @splunk/dashboard-context (from react/)
Dev: task react:start → webpack watch; edit src/ → stage/ updates live through symlink
Production: task react:package → yarn build → package stage/ as tgz for staging

Dependency Management

splunk/config/deps.yml declares Splunkbase dependencies. task deps:install downloads and installs them idempotently. Default deps include:

Python for Scientific Computing — NumPy/SciPy for custom commands
Splunk ML Toolkit — ML framework
SA-VSCode — enables Python debugging from VSCode

Debugging

Python (custom search commands, modular inputs, REST handlers)

Uses the SA-VSCode add-on + debugpy for remote debugging:

Install SA-VSCode: task deps:install (included in deps.yml)

Add debug hook to your Python code:

import sys, os
sys.path.append(os.path.join(os.environ['SPLUNK_HOME'], 'etc', 'apps', 'SA-VSCode', 'bin'))
import splunk_debug as dbg
dbg.enable_debugging(timeout=25)

Trigger the code (run a search, enable an input, etc.)
In VSCode: Run → "Python: Attach to Splunk (debugpy)"
Debugger connects to localhost:5678 (forwarded from Splunk container)

Path mapping: splunk/config/apps/<app>/bin/ ↔ /opt/splunk/etc/apps/<app>/bin/

React / JavaScript

Webpack watch mode (recommended for development):

task react:link → symlink stage/ into Splunk (run once)
task react:start → webpack watch; edits to src/ rebuild stage/ live
Refresh Splunk Web to see changes (source maps available for debugging)
VSCode: Run → "Chrome: Splunk Web App" for breakpoints

Production build (for staging verification):

task react:package → builds and packages stage/ as tgz
task stage:deploy
VSCode: Run → "Chrome: Staging Splunk Web"

Staging mode:

task stage:deploy → staging on :18000
VSCode: Run → "Chrome: Staging Splunk Web"

splunk-config-dev

The splunk-config-dev app sets web.conf options essential for debugging:

js_no_cache = True — disables JS caching
minify_js = False — preserves readable JS
enableWebDebug = True — enables Splunk Web debug mode
minify_css = False — preserves readable CSS

This app is always bind-mounted in dev. Do not include it in staging builds.

Environment Variables

Required (in .env):

SPLUNK_PASSWORD — Splunk admin password

Optional:

APP_NAME — default app for tasks
SPLUNKBASE_USERNAME / SPLUNKBASE_PASSWORD — for Splunkbase downloads
SPLUNK_APPS_URL — comma-separated URLs for first-run app install
SPLUNK_VERSION — Splunk image version (default: 9.4.0)

Ports

Port	Service
8000	Splunk Web (dev)
8089	splunkd REST API (dev)
8088	HTTP Event Collector (dev)
18000	Splunk Web (staging)
18089	splunkd REST API (staging)
18088	HTTP Event Collector (staging)
5678	Python debugpy (SA-VSCode)
3000	React dev server (webpack HMR)

Testing

Native Mac (fastest — no devcontainer needed)

Requires: Docker Desktop running, .env configured, task installed on Mac.

task test:lifecycle        # Splunk lifecycle tests — 7 suites (guards, boot, app, deps, react, staging, skip-provision)
task python:lint          # ruff lint

test:lifecycle runs tests/e2e/run-lifecycle.sh directly — starts Splunk, runs all 7 lifecycle suites, cleans up. Works on any host with Docker + task.

Inside devcontainer (full E2E including devcontainer build)

task test:devcontainer    # devcontainer build + static checks + lifecycle tests
task test:all             # python:lint + test:devcontainer

test:devcontainer uses @devcontainers/cli to build and start the devcontainer, runs static checks (tool availability, LOCAL_WORKSPACE_FOLDER, compose config), then calls task test:lifecycle inside it — same test code path as running on the host.

Test suites (in `tests/e2e/`)

Suite	Script	What it tests
guards	`test-guards.sh`	Guard tasks fail fast with clear errors (no container, no image, no APP_NAME)
boot	`test-boot.sh`	`dev:up`, health, `SPLUNK_PASSWORD` auth, `splunk-config-dev` symlink
app-lifecycle	`test-app-lifecycle.sh`	`app:create`, symlinks, `app:package`, REST verify
deps-install	`test-deps-install.sh`	`deps:install` idempotency
react-build	`test-react-build.sh`	`react:build`, `react:package`, tgz validation, idempotency
staging	`test-staging.sh`	`stage:deploy` (package + start + install)
skip-provision	`test-skip-provision.sh`	container restart skips Ansible

Troubleshooting

Splunk won't start: task dev:logs → check for errors
App changes not visible: task dev:refresh (reloads configs without restart); if that doesn't work, task dev:restartd
New app not visible: task dev:ensure-links creates missing symlinks; task dev:refresh reloads configs
Python debugger won't connect: ensure SA-VSCode is installed (task deps:install), code has dbg.enable_debugging(), and port 5678 is exposed
React HMR not working: check webpack proxy config points to splunk-dev:8089
Corrupt state after restart: task dev:reprovision (forces full Ansible)
Full reset: task dev:clean && task dev:up
Port conflicts: edit ports in .devcontainer/docker-compose.yml or docker-compose.staging.yml

FilesExpand file tree

ARCHITECTURE.md

Latest commit

History