Skip to content

perf(finding): drop four dojo_finding indexes that no query uses - #15659

Open
devGregA wants to merge 1 commit into
DefectDojo:devfrom
devGregA:perf/drop-unused-finding-indexes
Open

perf(finding): drop four dojo_finding indexes that no query uses#15659
devGregA wants to merge 1 commit into
DefectDojo:devfrom
devGregA:perf/drop-unused-finding-indexes

Conversation

@devGregA

Copy link
Copy Markdown
Contributor

What

dojo_finding is the busiest table in the schema and carries close to fifty indexes, every one maintained on each row written. Import is where that lands: a scan creating thousands of findings pays for all of them, per row.

This drops four that no query uses.

Index Column(s)
dojo_findin_epss_pe_567499_idx (epss_percentile) — the largest of the four
dojo_findin_line_fea329_idx (line)
dojo_findin_known_e_8c584e_idx (known_exploited)
idx_finding_sev_open_unver partial (severity, -numerical_severity) WHERE active AND NOT verified

Evidence

pg_stat_user_indexes.idx_scan = 0 on four long-running production instances, three of them carrying roughly seventeen months of accumulated statistics, ranging from a few thousand to ~184k findings.

The EPSS pair is the informative one: (epss_score) is used, (epss_percentile) is not — anywhere. Findings get sorted by score; the percentile is stored and displayed but is never an access path.

What was deliberately left alone

Several other indexes were unused on some instances and busy on others — (test_id, false_p), (test_id, unique_id_from_tool, duplicate), (test_id, out_of_scope), (unique_id_from_tool), and the three review-workflow FK indexes.

Those track optional features. A single-instance reading would have "proved" them dead and broken every install that uses them, so only indexes unused across every instance sampled are dropped here.

The limit of that evidence, stated plainly: four instances is a sample, and they share a deployment style. If any of these four serves a use case that sampling wouldn't see, I'd rather hear it than merge it — happy to drop any of the four from this PR.

Migration safety

DROP INDEX CONCURRENTLY in a non-atomic migration, following the pattern in 0280_vulnerability_id_upper_index, so it takes no exclusive lock on a large dojo_finding. IF EXISTS makes it idempotent, and the reverse rebuilds concurrently, so a downgrade doesn't lock the table either.

Verification

  • makemigrations dojo --check --dry-run"No changes detected" — state operations match the model, so a later makemigrations stays a no-op
  • Applied against a migrated database: 49 indexes → 45
  • Reversed: back to 49
  • ruff (repo config) clean

dojo_finding is the busiest table in the schema and carries close to fifty indexes, every one
of which is maintained on each row written. Import is where that is felt: a scan creating
thousands of findings pays for all of them, per row.

These four measured as completely unused (pg_stat_user_indexes.idx_scan = 0) on four
long-running production instances, three with roughly seventeen months of accumulated
statistics, ranging from a few thousand to ~184k findings:

    (epss_percentile)          the largest of the four
    (line)
    (known_exploited)
    idx_finding_sev_open_unver partial: (severity, -numerical_severity)
                               WHERE active AND NOT verified

The EPSS pair is the informative one. (epss_score) IS used, while (epss_percentile) is unused
everywhere: findings are sorted by score, and the percentile is stored and displayed but is
never an access path.

WHAT WAS DELIBERATELY LEFT ALONE. Several other indexes were unused on SOME instances and busy
on others -- (test_id, false_p), (test_id, unique_id_from_tool, duplicate),
(test_id, out_of_scope), (unique_id_from_tool), and the three review-workflow FK indexes. They
track optional features, so a single-instance reading would have "proved" them dead and broken
the installs that use them. Only indexes unused across every instance sampled are dropped here.

Built with DROP INDEX CONCURRENTLY in a non-atomic migration, following 0280, so it takes no
exclusive lock on a large dojo_finding. IF EXISTS makes it idempotent; the reverse rebuilds
concurrently, so a downgrade does not lock the table either.

Verified:
- makemigrations dojo --check --dry-run reports "No changes detected", so the state operations
  match the model and a later makemigrations stays a no-op.
- Applied against a migrated database: dojo_finding goes 49 indexes -> 45.
- Reversed: back to 49.
- ruff (repo config) clean.
@devGregA devGregA added this to the 3.3.0 milestone Aug 13, 2026
@github-actions github-actions Bot added the New Migration Adding a new migration file. Take care when merging. label Aug 13, 2026
@dryrunsecurity

Copy link
Copy Markdown

DryRun Security

This pull request contains critical security findings because two sensitive files were modified by an author not on the allowed list. Specifically, changes to dojo/db_migrations/0295_drop_unused_finding_indexes.py and dojo/finding/models.py violate configured codepath policies.

🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/db_migrations/0295_drop_unused_finding_indexes.py (drs_a7380a0c)
Vulnerability Configured Sensitive Codepath Modified by Non-Allowed Author
Description File 'dojo/db_migrations/0295_drop_unused_finding_indexes.py' matches configured sensitive codepath pattern 'dojo/db_migrations/*.py' and was modified by '' (commit aba7dc6) who is not in the allowed authors list.
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/models.py (drs_96652083)
Vulnerability Configured Sensitive Codepath Modified by Non-Allowed Author
Description File 'dojo/finding/models.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by '' (commit aba7dc6) who is not in the allowed authors list.

We've notified @mtesauro.


Comment to provide feedback on these findings.

Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]

Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing

All finding details can be found in the DryRun Security Dashboard.

@valentijnscholten valentijnscholten left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with a focus on migration mechanics and whether any real query path still wants these indexes.

Per-index check against actual query paths

  • epss_percentile — one precision note on "never an access path": the column header sort is client-side DataTables only (no dojo_sort link, findings_list_snippet.html:330) and the API ordering allowlist excludes it, but the UI filter panel's Ordering dropdown does offer "EPSS Percentile" (dojo/finding/ui/filters.py:227, :860), producing a genuine server-side ORDER BY epss_percentile. That said, those queries are always permission/product-scoped, so the planner would essentially never satisfy the sort via a single-column index scan — consistent with idx_scan = 0 even on instances where users do use that ordering. Drop still looks safe; the claim just deserves the softer phrasing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

New Migration Adding a new migration file. Take care when merging.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants