UP Deduplication Design Documentation

[JFisk42]

This PR is for the UP Deduplication design. There are some outstanding research items which could affect minor pieces of the described design.

Changes

• Added UP Deduplication Design Documentation

Linked Issues

• Fixes Deduplication in the UP MVP #14103

Outstanding TODOs

Should be resolved before merge:

• Database processing efficiency investigation.
• Is sender name collision a concern?
  • A: No, sender names are unique to an organization.
• Product Q: Are Azure Event logs needed for instances of duplicate detection?
  • A: From a platform perspective this is not needed
• Engagement Q: Are Azure Event logs needed for instances of duplicate detection?
  • A: Engagement would like an Azure Event logged with parent_report_id, child_report_id, and sender org/name.
• Update UP SRD with dedupe requirements/convert step changes as necessary.
  • A: This will be completed as part of the implementation.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files

[github-actions]

Test Results

1 279 tests  ±0   1 275 ✅ ±0   7m 35s ⏱️ +7s
  168 suites ±0       4 💤 ±0 
  168 files   ±0       0 ❌ ±0 

Results for commit 186698b. ± Comparison against base commit a99306f.

♻️ This comment has been updated with latest results.

[github-actions]

Integration Test Results

 60 files   60 suites   42m 45s ⏱️
428 tests 418 ✅ 10 💤 0 ❌
431 runs  421 ✅ 10 💤 0 ❌

Results for commit 186698b.

♻️ This comment has been updated with latest results.

[arnejduranovic]

Looks great! Please address TODOs before merging.

[JFisk42]

Key changes since last reviews:

• New DB recommendations based on discovery that only 5% of item_lineage.item_hash values are necessary for deduplication. (See "Other Updates")
• Added proposal for 2-step rollout. Talked with Jessica and having some time to gather item_hash data to analyze potential false positives would be a good safety check.
• Added UP SRD update proposals
• Documented existing sender setting for enabling duplicate detection
• Addition of Azure Events for engagement team
• Mock class design for containing key field business logic

Update on TODOs:

• While investigating memory requirements, item_lineage queries continually timed out. An index was pushed for this field to be able to help this.
• UP SRD will be updated once changes are approved

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud