referee, prow, plugin: add plugin that holds PRs on too many retest comments (kubevirt#3251)

* referee, prow, plugin: initial prototype

This commit provides a simple tool that counts the number of retest
comments after the latest push to the given PR.

Also it provides the server implementation for the prow external plugin.

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: add bazel build files

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: add files for publishing image

Add Makefile, Containerfile and job definition for publishing the plugin
image.

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: add service and deployment for k8s

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: enable in prow plugin config

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: test event handling

Extract interfaces to be able to mock out graphql client and github
client dependencies. Then create a couple of test cases to make sure
that the botuser doesn't handle it's own comments.

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: ping team on hold comment

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: leave PR alone if hold placed

In case a PR already has received a hold comment we don't handle it again.
The intent is that the author and a team has already been pinged on the
PR, and we leave it up to them to handle this now.

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: fix review issues

Signed-off-by: Daniel Hiller <[email protected]>

* referee, prow, plugin: add PR labels to evaluation

We now also consider whether a previous hold was already issued before
by the bot user, and if so, we do not place another hold again.
The hold is not placed even if it is not present any more.
This covers the case where the hold was placed once by the botuser and
then was taken back by someone else.

Note: updated robots/cmd/retests-to-merge to fetch PRs on itself.

Signed-off-by: Daniel Hiller <[email protected]>

* prow, plugin, referee: add unit tests for labels

Signed-off-by: Daniel Hiller <[email protected]>

* robots, retests-to-merge: add README, refine output

Refines the output of the cli tool to sort the output starting with
highest number of retest PRs first. Adds README and updates main README
to contain entry for tool.

Signed-off-by: Daniel Hiller <[email protected]>

* prow, plugin, referee: update wording in comment

Add references to k/kubevirt CONTRIBUTING and
getting-started#testing.

Signed-off-by: Daniel Hiller <[email protected]>

---------

Signed-off-by: Daniel Hiller <[email protected]>

Author: dhiller

Diff for: README.md

@@ -55,8 +55,11 @@ depending on billing alerts. See [README](limiter/README.md)
 
   * `robots/cmd/labels-checker`: Checks whether a PR has certain labels
 
+  * `robots/cmd/retests-to-merge`: Tool to check recent approved PRs for retest comments. See
+  [README](robots/cmd/retests-to-merge/README.md)
+
   * `robots/cmd/uploader`: Tool to mirror bazel dependencies on GCS. See
-  [README](plugins/cmd/uploader/README.md)
+  [README](robots/cmd/uploader/README.md)
 
 ## Contributing
 

Diff for: external-plugins/referee/BUILD.bazel

@@ -0,0 +1,26 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["main.go"],
+    importpath = "kubevirt.io/project-infra/external-plugins/referee",
+    visibility = ["//visibility:private"],
+    deps = [
+        "//external-plugins/referee/ghgraphql:go_default_library",
+        "//external-plugins/referee/server:go_default_library",
+        "@com_github_shurcool_githubv4//:go_default_library",
+        "@com_github_sirupsen_logrus//:go_default_library",
+        "@io_k8s_test_infra//pkg/flagutil:go_default_library",
+        "@io_k8s_test_infra//prow/config/secret:go_default_library",
+        "@io_k8s_test_infra//prow/flagutil:go_default_library",
+        "@io_k8s_test_infra//prow/interrupts:go_default_library",
+        "@io_k8s_test_infra//prow/pluginhelp/externalplugins:go_default_library",
+        "@org_golang_x_oauth2//:go_default_library",
+    ],
+)
+
+go_binary(
+    name = "referee",
+    embed = [":go_default_library"],
+    visibility = ["//visibility:public"],
+)

Diff for: external-plugins/referee/Makefile

@@ -0,0 +1,16 @@
+
+.PHONY: all build test format push
+all: format push
+bazelbin := bazelisk
+
+build:
+	$(bazelbin) build //external-plugins/referee/...
+
+test:
+	$(bazelbin) test //external-plugins/referee/...
+
+format:
+	gofmt -w .
+
+push:
+	podman build -f ../../images/referee/Containerfile -t quay.io/kubevirtci/referee:latest && podman push quay.io/kubevirtci/referee:latest

Diff for: external-plugins/referee/ghgraphql/BUILD.bazel

@@ -0,0 +1,31 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = [
+        "client.go",
+        "labels.go",
+        "pullrequests.go",
+        "timeline.go",
+        "types.go",
+    ],
+    importpath = "kubevirt.io/project-infra/external-plugins/referee/ghgraphql",
+    visibility = ["//visibility:public"],
+    deps = [
+        "@com_github_shurcool_githubv4//:go_default_library",
+        "@com_github_sirupsen_logrus//:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = [
+        "ghgraphql_suite_test.go",
+        "labels_test.go",
+    ],
+    deps = [
+        ":go_default_library",
+        "@com_github_onsi_ginkgo_v2//:go_default_library",
+        "@com_github_onsi_gomega//:go_default_library",
+    ],
+)

Diff for: external-plugins/referee/ghgraphql/client.go

@@ -0,0 +1,40 @@
+/*
+ * This file is part of the KubeVirt project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ * Copyright the KubeVirt Authors.
+ *
+ */
+
+package ghgraphql
+
+import "github.com/shurcooL/githubv4"
+
+type GitHubGraphQLClient interface {
+	// FetchPRTimeLineForLastCommit returns specific events a PR has received
+	// after the last commit or force push.
+	FetchPRTimeLineForLastCommit(org string, repo string, prNumber int) (PRTimelineForLastCommit, error)
+
+	FetchPRLabels(org string, repo string, prNumber int) (PRLabels, error)
+
+	FetchOpenApprovedAndLGTMedPRs(org string, repo string) (PullRequests, error)
+}
+
+type gitHubGraphQLClient struct {
+	gitHubClient *githubv4.Client
+}
+
+func NewClient(gitHubClient *githubv4.Client) GitHubGraphQLClient {
+	return gitHubGraphQLClient{gitHubClient: gitHubClient}
+}

Diff for: external-plugins/referee/ghgraphql/ghgraphql_suite_test.go

@@ -0,0 +1,32 @@
+/*
+ * This file is part of the KubeVirt project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ * Copyright the KubeVirt Authors.
+ *
+ */
+
+package ghgraphql_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestGHGraphQL(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "GHGraphQL Suite")
+}

Diff for: external-plugins/referee/ghgraphql/labels.go

@@ -0,0 +1,67 @@
+/*
+ * This file is part of the KubeVirt project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ * Copyright the KubeVirt Authors.
+ *
+ */
+
+package ghgraphql
+
+import (
+	"context"
+	"fmt"
+	"github.com/shurcooL/githubv4"
+)
+
+func (g gitHubGraphQLClient) FetchPRLabels(org string, repo string, prNumber int) (PRLabels, error) {
+	labels, err := g.fetchLabelsForPR(org, repo, prNumber)
+	if err != nil {
+		return PRLabels{}, err
+	}
+	return NewPRLabels(labels), nil
+}
+
+func (g gitHubGraphQLClient) fetchLabelsForPR(org string, repo string, prNumber int) ([]Label, error) {
+	var query struct {
+		Repository struct {
+			PullRequest struct {
+				Labels Labels `graphql:"labels (first: 100)"`
+			} `graphql:"pullRequest(number: $prNumber)"`
+		} `graphql:"repository(owner: $org, name: $repo)"`
+	}
+	variables := map[string]interface{}{
+		"prNumber": githubv4.Int(prNumber),
+		"org":      githubv4.String(org),
+		"repo":     githubv4.String(repo),
+	}
+
+	err := g.gitHubClient.Query(context.Background(), &query, variables)
+	if err != nil {
+		return []Label{}, fmt.Errorf("failed to use github query %+v with variables %v: %w", query, variables, err)
+	}
+	return query.Repository.PullRequest.Labels.Nodes, nil
+}
+
+func NewPRLabels(labels []Label) PRLabels {
+	prLabels := PRLabels{
+		Labels: labels,
+	}
+	for _, label := range labels {
+		if label.Name == "do-not-merge/hold" {
+			prLabels.IsHoldPresent = true
+		}
+	}
+	return prLabels
+}

Diff for: external-plugins/referee/ghgraphql/labels_test.go

@@ -0,0 +1,56 @@
+/*
+ * This file is part of the KubeVirt project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ * Copyright the KubeVirt Authors.
+ *
+ */
+
+package ghgraphql_test
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"kubevirt.io/project-infra/external-plugins/referee/ghgraphql"
+)
+
+var _ = Describe("Labels", func() {
+	DescribeTable("NewLabels", func(labels []ghgraphql.Label, expected ghgraphql.PRLabels) {
+		Expect(ghgraphql.NewPRLabels(labels)).To(BeEquivalentTo(expected))
+	},
+		Entry("no labels don't set IsHoldPresent", nil, ghgraphql.PRLabels{
+			IsHoldPresent: false,
+		}),
+		Entry("some labels don't set IsHoldPresent", []ghgraphql.Label{
+			{Name: "test"},
+		}, ghgraphql.PRLabels{
+			IsHoldPresent: false,
+			Labels: []ghgraphql.Label{
+				{Name: "test"},
+			},
+		}),
+		Entry("exactly one label sets IsHoldPresent", []ghgraphql.Label{
+			{Name: "ok-to-test"},
+			{Name: "do-not-merge/hold"},
+			{Name: "needs-rebase"},
+		}, ghgraphql.PRLabels{
+			IsHoldPresent: true,
+			Labels: []ghgraphql.Label{
+				{Name: "ok-to-test"},
+				{Name: "do-not-merge/hold"},
+				{Name: "needs-rebase"},
+			},
+		}),
+	)
+})

Diff for: external-plugins/referee/ghgraphql/pullrequests.go

@@ -0,0 +1,57 @@
+/*
+ * This file is part of the KubeVirt project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ * Copyright the KubeVirt Authors.
+ *
+ */
+
+package ghgraphql
+
+import (
+	"context"
+	"fmt"
+	"github.com/shurcooL/githubv4"
+)
+
+func (g gitHubGraphQLClient) FetchOpenApprovedAndLGTMedPRs(org string, repo string) (PullRequests, error) {
+	labels, err := g.fetchPRs(org, repo)
+	if err != nil {
+		return PullRequests{}, err
+	}
+	pullRequests := PullRequests{
+		PRs: labels,
+	}
+	return pullRequests, nil
+}
+
+func (g gitHubGraphQLClient) fetchPRs(org string, repo string) ([]PullRequest, error) {
+	var query struct {
+		Repository struct {
+			PullRequests struct {
+				Nodes []PullRequest
+			} `graphql:"pullRequests(first: 100, baseRefName: \"main\", labels: [\"lgtm\", \"approved\"], states: [OPEN])"`
+		} `graphql:"repository(owner: $org, name: $repo)"`
+	}
+	variables := map[string]interface{}{
+		"org":  githubv4.String(org),
+		"repo": githubv4.String(repo),
+	}
+
+	err := g.gitHubClient.Query(context.Background(), &query, variables)
+	if err != nil {
+		return []PullRequest{}, fmt.Errorf("failed to use github query %+v with variables %v: %w", query, variables, err)
+	}
+	return query.Repository.PullRequests.Nodes, nil
+}