Skip to content

git scraper #1925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
moshloop wants to merge 4 commits into
base: main
Choose a base branch
from
Open

git scraper #1925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5784b99
feat(git): filter tags by commit date using depth-based since window
moshloop Mar 1, 2026
405391f
fix(git): use tagger date as fallback when tag target is not a commit
moshloop Mar 1, 2026
587670e
fix(git): improve integration tests and filetree author format
moshloop Mar 1, 2026
cee32dd
chore: apply go fmt formatting fixes
moshloop Mar 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions api/v1/git.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package v1

import "github.com/flanksource/duty/connection"

type GitFileStrategy string

const (
GitFileStrategyIgnore GitFileStrategy = "ignore"
GitFileStrategyTrack GitFileStrategy = "track"
GitFileStrategyDiff GitFileStrategy = "diff"
)

type GitFileRule struct {
Pattern string `json:"pattern" yaml:"pattern"`
Strategy GitFileStrategy `json:"strategy" yaml:"strategy"`
}
Comment on lines +5 to +16
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Constrain GitFileStrategy values at the API layer.

GitFileRule.Strategy currently accepts arbitrary strings. Invalid values can silently bypass expected file handling logic.

🛡️ Proposed fix 
+// +kubebuilder:validation:Enum=ignore;track;diff
 type GitFileStrategy string
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
type GitFileStrategy string
const (
GitFileStrategyIgnore GitFileStrategy = "ignore"
GitFileStrategyTrack GitFileStrategy = "track"
GitFileStrategyDiff GitFileStrategy = "diff"
)
type GitFileRule struct {
Pattern string `json:"pattern" yaml:"pattern"`
Strategy GitFileStrategy `json:"strategy" yaml:"strategy"`
}
// +kubebuilder:validation:Enum=ignore;track;diff
type GitFileStrategy string
const (
GitFileStrategyIgnore GitFileStrategy = "ignore"
GitFileStrategyTrack GitFileStrategy = "track"
GitFileStrategyDiff GitFileStrategy = "diff"
)
type GitFileRule struct {
Pattern string `json:"pattern" yaml:"pattern"`
Strategy GitFileStrategy `json:"strategy" yaml:"strategy"`
}
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@api/v1/git.go` around lines 5 - 16, The API currently allows any string for
GitFileRule.Strategy; add explicit validation to reject unknown values by
implementing a strict unmarshaler/validator for GitFileStrategy that checks
incoming JSON/YAML against the allowed constants (GitFileStrategyIgnore,
GitFileStrategyTrack, GitFileStrategyDiff) and returns an error for anything
else, and ensure GitFileRule uses that by either adding a Validate() method on
GitFileRule or relying on GitFileStrategy.UnmarshalJSON/UnmarshalYAML so parsing
fails on invalid inputs; update callers/tests to expect and handle the
validation error.


// +kubebuilder:object:generate=true
type Git struct {
BaseScraper `json:",inline" yaml:",inline"`
connection.GitConnection `json:",inline" yaml:",inline"`

// Branches to track. Supports glob patterns (e.g. "release/*").
// Defaults to the default branch only.
Branches []string `json:"branches,omitempty" yaml:"branches,omitempty"`

// Files configures per-glob file handling strategy.
// Default strategy for unmatched files is "track".
Files []GitFileRule `json:"files,omitempty" yaml:"files,omitempty"`
}
3 changes: 3 additions & 0 deletions api/v1/scrapeconfig_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ func (t *ScrapeConfig) Type() string {
if len(t.Spec.File) != 0 {
return "file"
}
if len(t.Spec.Git) != 0 {
return "git"
}
if len(t.Spec.Kubernetes) != 0 {
return "kubernetes"
}
Expand Down
6 changes: 6 additions & 0 deletions api/v1/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ var AllScraperConfigs = map[string]any{
"exec": Exec{},
"file": File{},
"gcp": GCP{},
"git": Git{},
"github": GitHub{},
"githubactions": GitHubActions{},
"http": HTTP{},
Expand Down Expand Up @@ -66,6 +67,7 @@ type ScraperSpec struct {
GCP []GCP `json:"gcp,omitempty" yaml:"gcp,omitempty"`
AWS []AWS `json:"aws,omitempty" yaml:"aws,omitempty"`
File []File `json:"file,omitempty" yaml:"file,omitempty"`
Git []Git `json:"git,omitempty" yaml:"git,omitempty"`
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

ScraperSpec.IsEmpty() is now inconsistent with the new Git field.

Git-only scraper specs can be misclassified as empty because IsEmpty() still ignores Git.

🔧 Proposed fix 
 func (c ScraperSpec) IsEmpty() bool {
-	return len(c.AWS) == 0 && len(c.File) == 0
+	return len(c.AWS) == 0 && len(c.File) == 0 && len(c.Git) == 0
 }
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@api/v1/types.go` at line 70, ScraperSpec.IsEmpty() currently ignores the new
Git field so a spec that only sets Git can be reported empty; update
ScraperSpec.IsEmpty() to treat Git as non-empty when s.Git is non-nil and has
elements (e.g., check len(s.Git) > 0) or, if there is a Git.IsEmpty() helper on
the element type, iterate and treat the spec as non-empty if any Git element is
non-empty; modify the IsEmpty logic in ScraperSpec.IsEmpty() to include this Git
presence check alongside the existing field checks.

Kubernetes []Kubernetes `json:"kubernetes,omitempty" yaml:"kubernetes,omitempty"`
KubernetesFile []KubernetesFile `json:"kubernetesFile,omitempty" yaml:"kubernetesFile,omitempty"`
AzureDevops []AzureDevops `json:"azureDevops,omitempty" yaml:"azureDevops,omitempty"`
Expand Down Expand Up @@ -115,6 +117,10 @@ func (c ScraperSpec) ApplyPlugin(plugins []ScrapePluginSpec) ScraperSpec {
spec.File[i].BaseScraper = spec.File[i].BaseScraper.ApplyPlugins(plugins...)
}

for i := range spec.Git {
spec.Git[i].BaseScraper = spec.Git[i].BaseScraper.ApplyPlugins(plugins...)
}

for i := range spec.Kubernetes {
spec.Kubernetes[i].BaseScraper = spec.Kubernetes[i].BaseScraper.ApplyPlugins(plugins...)
}
Expand Down
119 changes: 119 additions & 0 deletions api/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading