refactor(importers): track reimport finding buckets by id, not instance - #15600
Draft
valentijnscholten wants to merge 4 commits into
Draft
refactor(importers): track reimport finding buckets by id, not instance#15600valentijnscholten wants to merge 4 commits into
valentijnscholten wants to merge 4 commits into
Conversation
valentijnscholten
force-pushed
the
feat/persist-new-findings-seam
branch
from
August 10, 2026 18:57
65d33c4 to
1c5a996
Compare
valentijnscholten
force-pushed
the
feat/track-finding-ids-not-instances
branch
from
August 10, 2026 18:58
180aac2 to
e5e0e94
Compare
valentijnscholten
force-pushed
the
feat/persist-new-findings-seam
branch
from
August 12, 2026 06:57
1c5a996 to
5565aa2
Compare
valentijnscholten
force-pushed
the
feat/track-finding-ids-not-instances
branch
from
August 12, 2026 06:57
e5e0e94 to
97fcd4d
Compare
valentijnscholten
added
the
affects_pro
DefaultReImporter kept every original/new/reactivated/unchanged Finding instance alive for the whole run just so notify_scan_added() and update_import_history() could read a few scalar fields at the end -- on a large reimport that pins the full result set in memory. new_items, reactivated_items, unchanged_items and original_items now hold ids; the two consumers that need real rows (close_old_findings, the JIRA finding-group push) requery deliberately and in bounded chunks at the point they need them, sharing self.test instead of copying the parent chain per row. notify_scan_added also gains NOTIFICATION_SCAN_ADDED_MAX_FINDINGS (default 100) so a reimport touching thousands of findings no longer templates all of them into a single notification body.
process_matched_active_finding()'s two inline-closing branches (report re-marks a matched, previously-active finding as mitigated, or as risk- accepted/false-p/out-of-scope) never tracked the finding into any bucket, and never left an audit note -- both were only papered over by a latent double-processing bug: since the finding was never removed from to_mitigate's arithmetic, close_old_findings() picked it up too and (redundantly) re-mitigated it, which is where the "closed" accounting and the "Mitigated by ... re-upload." note actually came from. M1's fresh, honest re-hydrate in close_old_findings() (rather than reusing a stale original_items instance) surfaces this: it correctly recognizes the finding is already mitigated and skips it, so the redundant re-save no longer happens -- and with it, the accounting and the note disappear too. The stale-instance re-save also happened to stomp the finding's `verified` field back to its pre-reimport value, masking that the reimport itself had legitimately just applied an explicit verified=False. Track these ids in a new self.actively_closed_matches list (disjoint from close_old_findings()'s own output by construction) and fold them into closed_finding_ids at the call site, and leave the same audit note inline that close_old_findings() would have left. Updates test_import_veracode_reimport_veracode_active_verified_mitigated's verified assertions to the corrected value, and re-pins the query counts in test_importers_performance.py that this fix and the seam changes shifted (net effect on the empty-report reimport step: 3 fewer queries, from no longer redundantly re-processing already-closed findings).
_finalize_pending_new_finding() read finding.id immediately after persist_new_findings() returned, on the assumption that the write had already happened. That assumption only holds for the default per-finding save; a downstream edition is explicitly allowed to override persist_new_finding() and defer the write to its own batch boundary (see that method's docstring), and for such an edition the finding is still unsaved at this point. new_items ended up holding None for every deferred new finding instead of its real id. Move the read to _flush_post_processing_batch(), after any such buffer has been flushed and every finding in new_findings_in_batch is guaranteed to have a primary key -- the same point batch_findings_to_dispatch already reads .id from, a few lines down in the same method.
valentijnscholten
force-pushed
the
feat/persist-new-findings-seam
branch
from
August 12, 2026 20:41
5565aa2 to
2b81778
Compare
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
valentijnscholten
force-pushed
the
feat/track-finding-ids-not-instances
branch
from
August 12, 2026 20:41
b885147 to
148a0ca
Compare
Contributor
|
Conflicts have been resolved. A maintainer will review the pull request shortly. |
…issing id column
process_groups_for_all_findings() re-derives which finding groups to push to
JIRA from the reactivated/unchanged id buckets with a query. It selected
values_list("finding_group_id"), but Finding has no finding_group_id column --
Finding.finding_group is a cached_property over the reverse M2M
finding_group_set -- so the query raised FieldError and 500'd the reimport
whenever finding groups were enabled and JIRA push/sync was on. Select the
finding_group relation (the reverse query name, already used in the filter)
instead. Add a regression test covering that groups+JIRA path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #15599.
DefaultReImporter._process_findings_internalkept everyoriginal/new/reactivated/unchanged
Findinginstance alive in memory for thewhole run, purely so
notify_scan_added()andupdate_import_history()couldread a few scalar fields off them at the very end. On a large reimport that
pins the full result set in memory for no reason.
original_items,new_items,reactivated_items,unchanged_itemsnowhold ids, not instances.
close_old_findings()and the JIRA finding-group push are the onlyconsumers that need real rows; both now requery deliberately, in bounded
chunks, at the point they need them -- and share
self.testrather thaneach row carrying its own copy of the
test -> engagement -> productchain, the same pattern reimport matching already uses.
notify_scan_added()is rewritten to requery a capped, ordered slice(
NOTIFICATION_SCAN_ADDED_MAX_FINDINGS, default 100) instead of templatingevery touched finding into a notification body.
dojo.finding.helper.filter_findings_by_existence(instance-based, onecaller) is replaced by
filter_finding_ids_by_existence(id-based).A pre-existing bug this surfaced (not introduced by this PR)
close_old_findings()used to receiveself.to_mitigateas the original,never-refreshed
Findinginstances fetched at the very start of the run.process_matched_active_finding()'s two inline-closing branches (reportre-marks a matched, previously-active finding as mitigated, or as
risk-accepted/false-p/out-of-scope) never removed that finding from
to_mitigate's bucket arithmetic, soclose_old_findings()picked it upagain and redundantly re-mitigated it using the stale, pre-reimport
instance -- silently overwriting fields like
verifiedback to theirpre-reimport value, and only that redundant pass is what produced the
"closed" accounting entry and the "Mitigated by ... re-upload." note; the
inline branches themselves never wrote either.
This PR's honest re-hydrate (a fresh DB read instead of reusing the stale
instance) makes
close_old_findings()correctly recognize the finding asalready mitigated and skip it -- which surfaced that skipping it also lost
the accounting entry, the note, and let the stale re-save's field-stomping
stop (a real, if obscure, improvement: a reimport's explicit
verified=override now actually sticks instead of being silently reverted).
Added
self.actively_closed_matchestracking and an inline audit note soboth branches now do directly what the accidental double-processing used to
paper over. See the added commit for the full explanation, and
test_import_veracode_reimport_veracode_active_verified_mitigated's updatedassertions for what changed observably.
Test plan
unittests.test_reimport_batch_flush-- updated assertions for the newid-based buckets, passes.
unittests.test_importers_importer-- no new failures vs baseline(pre-existing 301-redirect failures in
FlexibleImportTestAPI/FlexibleReimportTestAPIreproduce identically with this branchstashed out, confirmed unrelated).
unittests.test_import_reimport-- full file green (256 assertionsacross
test_import_reimport.py+ neighboring importer test filescombined), including the veracode double-processing fix above, found
by running this file for the first time against the complete stack
rather than assuming "zero behavior change" held everywhere.
ProReImporter) -- 2391 passed, 119 skipped, 0 failed after fixing alocal dev-stack env misconfiguration (
DD_V3_FEATURE_LOCATIONS) thathad produced 46 false failures unrelated to this change.
test_importers_performance.pyassertNumQueriesbaselinesupdated; the refactor is a net reduction in query count on every step
except one that initially regressed by 21 queries (an N+1 from
freshly-hydrated
close_old_findingscandidates not sharingself.test) -- fixed at the root rather than absorbed into the pinnedcount, which now shows a net decrease vs the pre-refactor baseline.
unittests.test_importers_performancere-pinned for thequery-count shift from the veracode double-processing fix above (net
-3 on the empty-report reimport step, from no longer redundantly
re-processing an already-closed finding; +1-3 elsewhere from the new
inline audit note).
unittests/test_tag_inheritance_perf.py's reimport-with-new-findingsbaselines were hand-recomputed while rebasing onto a
devcommit thatindependently changed the same constants (two additive deltas from a
common ancestor, combined arithmetically:
EXPECTED_ZAP_REIMPORT_WITH_NEW_V3193 -> 194). Could not run locally -- this dev stack's Pro integration
permanently disables
watson, which this test class's fixture needs.Flagging for CI to confirm.