feat(test): unified fixture framework with e2e DB test runner#1972
Merged
feat(test): unified fixture framework with e2e DB test runner#1972
Conversation
Split extract testdata into unit/ and e2e/ subdirectories. Unit fixtures use MockResolver (no DB), e2e fixtures run the full scraper pipeline (Run -> SaveResults -> CEL assertions). Add 4 e2e fixtures testing metadata-only config changes with variations of default vs explicit config_type and cross-config change targeting.
Contributor
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
WalkthroughThe pull request adds comprehensive test coverage for metadata-only extraction scenarios, including new JSON fixture files and YAML test specifications for both unit and end-to-end tests. It introduces a new end-to-end test framework in Go, refactors test fixture discovery, and modifies the default-detection logic in the extraction code. A .gitignore entry is also added. Changes
Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
.gitignore (1)
52-52: Remove the duplicate ignore rule.Line 46 already ignores
.claudeas both a file and a directory, so.claude/is redundant here. Keeping just one entry makes the ignore list easier to maintain.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.gitignore at line 52, The .gitignore contains a duplicate entry for the same ignore target: remove the redundant ".claude/" line so only the existing ".claude" entry remains; edit the .gitignore and delete the ".claude/" entry (the unique symbol to look for is the ".claude" ignore rule) to avoid duplicate rules and keep the file concise.scrapers/extract_e2e_test.go (1)
144-151: Consider handling JSON marshal/unmarshal errors for better debuggability.Silently ignoring errors on lines 145 and 147 could mask issues during test debugging. While the nil check on line 148 provides a fallback, unexpected serialization failures would be harder to diagnose.
♻️ Optional: Add explicit error handling
// Marshal changes to map form for CEL - changesRaw, _ := json.Marshal(allChanges) + changesRaw, err := json.Marshal(allChanges) + if err != nil { + // Fall back to empty slice if marshal fails + return map[string]any{"config": nil, "changes": []any{}} + } var changesSlice []any - _ = json.Unmarshal(changesRaw, &changesSlice) + if err := json.Unmarshal(changesRaw, &changesSlice); err != nil { + changesSlice = []any{} + } if changesSlice == nil { changesSlice = []any{} }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scrapers/extract_e2e_test.go` around lines 144 - 151, The code silently ignores errors from json.Marshal(allChanges) and json.Unmarshal into changesSlice; update the block around changesRaw, changesSlice and env["changes"] to capture and handle both errors (from json.Marshal and json.Unmarshal) — e.g., check the returned error values and call t.Fatalf or t.Errorf with descriptive messages including the error and relevant data (allChanges or changesRaw) so the test fails loudly and is debuggable, and keep the nil fallback for changesSlice if unmarshalling yields nil.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scrapers/extract_e2e_test.go`:
- Around line 58-63: The defer calling os.Remove(tmpFile.Name()) has its error
return unchecked; change it to explicitly ignore the error (e.g. defer func() {
_ = os.Remove(tmpFile.Name()) }() or defer func() { if err :=
os.Remove(tmpFile.Name()); err != nil && !os.IsNotExist(err) { /* optional log
*/ } }()) so the linter is satisfied; update the defer that references tmpFile
and os.Remove accordingly in extract_e2e_test.go near the tmpFile
creation/cleanup block.
- Around line 88-95: The test cleanup currently deletes ConfigChange,
ConfigItem, and ConfigScraper but misses ConfigLocation rows created by
SaveResults; update the deferred cleanup to also delete orphaned ConfigLocation
records for each id in createdItems by issuing DefaultContext.DB().Where("id =
?", id).Delete(&dutymodels.ConfigLocation{}) alongside the existing deletes so
ConfigLocation entries related to createdItems are removed.
---
Nitpick comments:
In @.gitignore:
- Line 52: The .gitignore contains a duplicate entry for the same ignore target:
remove the redundant ".claude/" line so only the existing ".claude" entry
remains; edit the .gitignore and delete the ".claude/" entry (the unique symbol
to look for is the ".claude" ignore rule) to avoid duplicate rules and keep the
file concise.
In `@scrapers/extract_e2e_test.go`:
- Around line 144-151: The code silently ignores errors from
json.Marshal(allChanges) and json.Unmarshal into changesSlice; update the block
around changesRaw, changesSlice and env["changes"] to capture and handle both
errors (from json.Marshal and json.Unmarshal) — e.g., check the returned error
values and call t.Fatalf or t.Errorf with descriptive messages including the
error and relevant data (allChanges or changesRaw) so the test fails loudly and
is debuggable, and keep the nil fallback for changesSlice if unmarshalling
yields nil.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 550277f0-fce3-4f4a-8d6c-c8502056be06
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (52)
.gitignorefixtures/data/metadata_only_changes.jsonfixtures/data/metadata_only_cross_config.jsonfixtures/data/metadata_only_no_config_type.jsonfixtures/data/metadata_only_with_config_type.jsonscrapers/extract/extract.goscrapers/extract/extract_test.goscrapers/extract/testdata/e2e/metadata_only_cross_config.yamlscrapers/extract/testdata/e2e/metadata_only_default_config_type.yamlscrapers/extract/testdata/e2e/metadata_only_e2e.yamlscrapers/extract/testdata/e2e/metadata_only_explicit_config_type.yamlscrapers/extract/testdata/unit/access_invalid_ref.yamlscrapers/extract/testdata/unit/access_logs_basic.yamlscrapers/extract/testdata/unit/access_logs_missing_user.yamlscrapers/extract/testdata/unit/access_logs_resolve_user_alias.yamlscrapers/extract/testdata/unit/access_logs_shorthand_external.yamlscrapers/extract/testdata/unit/access_logs_shorthand_uuid.yamlscrapers/extract/testdata/unit/access_logs_user_shorthand.yamlscrapers/extract/testdata/unit/access_skipped_config_not_found.yamlscrapers/extract/testdata/unit/analysis_basic.yamlscrapers/extract/testdata/unit/analysis_default_from_parent.yamlscrapers/extract/testdata/unit/analysis_invalid_ref.yamlscrapers/extract/testdata/unit/analysis_no_parent.yamlscrapers/extract/testdata/unit/changes_and_access.yamlscrapers/extract/testdata/unit/changes_basic.yamlscrapers/extract/testdata/unit/changes_default_from_parent.yamlscrapers/extract/testdata/unit/changes_invalid_ref.yamlscrapers/extract/testdata/unit/changes_missing_change_id.yamlscrapers/extract/testdata/unit/changes_missing_change_type.yamlscrapers/extract/testdata/unit/changes_no_parent.yamlscrapers/extract/testdata/unit/changes_with_details.yamlscrapers/extract/testdata/unit/combined_aliases.yamlscrapers/extract/testdata/unit/config_ref_defaults.yamlscrapers/extract/testdata/unit/config_ref_shorthand.yamlscrapers/extract/testdata/unit/config_ref_uuid.yamlscrapers/extract/testdata/unit/drop_nil_user_groups.yamlscrapers/extract/testdata/unit/empty_config_access.yamlscrapers/extract/testdata/unit/external_config_id.yamlscrapers/extract/testdata/unit/group_aliases.yamlscrapers/extract/testdata/unit/json_roundtrip.yamlscrapers/extract/testdata/unit/metadata_only_changes.yamlscrapers/extract/testdata/unit/metadata_only_external_id_only.yamlscrapers/extract/testdata/unit/missing_config_access.yamlscrapers/extract/testdata/unit/remap_user_groups.yamlscrapers/extract/testdata/unit/resolve_after_sync.yamlscrapers/extract/testdata/unit/resolve_aliases.yamlscrapers/extract/testdata/unit/reuse_existing_ids.yamlscrapers/extract/testdata/unit/role_aliases.yamlscrapers/extract/testdata/unit/stub_entity_creation.yamlscrapers/extract/testdata/unit/unmarshal_all_aliases.yamlscrapers/extract/testdata/unit/user_aliases.yamlscrapers/extract_e2e_test.go
Comment on lines
+88
to
+95
| // Cleanup after test | ||
| defer func() { | ||
| for _, id := range createdItems { | ||
| DefaultContext.DB().Where("config_id = ?", id).Delete(&models.ConfigChange{}) | ||
| DefaultContext.DB().Delete(&models.ConfigItem{}, "id = ?", id) | ||
| } | ||
| DefaultContext.DB().Where("id = ?", scraperModel.ID).Delete(&dutymodels.ConfigScraper{}) | ||
| }() |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check what tables SaveResults writes to
rg -n '\.Create|\.Save|\.Clauses.*Create' db/update.go -A2 | head -50Repository: flanksource/config-db
Length of output: 1797
🏁 Script executed:
#!/bin/bash
# Find SaveResults function and what it does
rg -n 'func.*SaveResults|func (.*) SaveResults' db/update.go -A 100 | head -150Repository: flanksource/config-db
Length of output: 3233
🏁 Script executed:
#!/bin/bash
# Search for config_locations table writes across the codebase
rg -n 'config_locations|ConfigLocation' --type go -i | head -30Repository: flanksource/config-db
Length of output: 234
🏁 Script executed:
#!/bin/bash
# Check what the test actually creates and verifies
cat -n scrapers/extract_e2e_test.go | sed -n '70,100p'Repository: flanksource/config-db
Length of output: 1401
🏁 Script executed:
#!/bin/bash
# Find the actual saveResults implementation - should show what it does with locations
rg -n 'func saveResults|func \(.*\) saveResults' db/update.go -A 200 | grep -E '(func|locations|CreateInBatches|Create.*Location)' | head -40Repository: flanksource/config-db
Length of output: 505
🏁 Script executed:
#!/bin/bash
# More targeted search for where locations are persisted
rg -n 'locations.*CreateInBatches|CreateInBatches.*locations|\.CreateInBatches.*Location' db/update.go -B 3 -A 3Repository: flanksource/config-db
Length of output: 47
🏁 Script executed:
#!/bin/bash
# Check if the test actually invokes SaveResults or just creates test data
rg -n 'SaveResults|saveResults' scrapers/extract_e2e_test.go -B 2 -A 2Repository: flanksource/config-db
Length of output: 215
🏁 Script executed:
#!/bin/bash
# Check ConfigLocation model definition and foreign keys
rg -n 'type ConfigLocation|ConfigLocation struct' --type go -A 10 | head -30Repository: flanksource/config-db
Length of output: 47
🏁 Script executed:
#!/bin/bash
# Check if tests use transactions or DB isolation
rg -n 'Transact|tx\.|Transaction|BeginTx' scrapers/extract_e2e_test.go -B 2 -A 2Repository: flanksource/config-db
Length of output: 236
🏁 Script executed:
#!/bin/bash
# Search more broadly for ConfigLocation definition
rg -n 'ConfigLocation' --type go | head -30Repository: flanksource/config-db
Length of output: 234
🏁 Script executed:
#!/bin/bash
# Check if test uses any DB isolation or transaction mechanism in the setup
cat -n scrapers/extract_e2e_test.go | sed -n '1,70p'Repository: flanksource/config-db
Length of output: 2645
🏁 Script executed:
#!/bin/bash
# Search for ConfigLocation model definition in any go files
find . -name "*.go" -type f | xargs grep -l "type ConfigLocation struct" 2>/dev/nullRepository: flanksource/config-db
Length of output: 49
🏁 Script executed:
#!/bin/bash
# Check the duty models package import and look for where it might be defined
rg -n 'dutyModels.ConfigLocation' --type go -B 2 -A 2 | head -30Repository: flanksource/config-db
Length of output: 685
🏁 Script executed:
#!/bin/bash
# Check if there's a go.mod to understand if duty is external
cat go.mod | grep -i dutyRepository: flanksource/config-db
Length of output: 153
Add ConfigLocation to test cleanup.
The test calls SaveResults at line 105, which creates ConfigLocation records (see db/update.go:631-635). The cleanup only removes ConfigChange, ConfigItem, and ConfigScraper records. Add cleanup for orphaned ConfigLocation records related to the test's ConfigItem IDs:
for _, id := range createdItems {
DefaultContext.DB().Where("id = ?", id).Delete(&dutymodels.ConfigLocation{})
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@scrapers/extract_e2e_test.go` around lines 88 - 95, The test cleanup
currently deletes ConfigChange, ConfigItem, and ConfigScraper but misses
ConfigLocation rows created by SaveResults; update the deferred cleanup to also
delete orphaned ConfigLocation records for each id in createdItems by issuing
DefaultContext.DB().Where("id = ?", id).Delete(&dutymodels.ConfigLocation{})
alongside the existing deletes so ConfigLocation entries related to createdItems
are removed.
This was referenced Mar 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unit/ande2e/subdirectories for clear separationscrapers/extract_e2e_test.go) reads fixtures with aspecfield and runs the full scraper pipeline (Run -> SaveResults -> CEL assertions)config_typeand cross-config change targetingTest plan
make test-fastpasses (40 unit fixtures, 4 e2e fixtures)Summary by CodeRabbit