Mintmaker: create a cronjob for generating the OSV database #5379
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| apiVersion: batch/v1 | ||
| kind: CronJob | ||
| metadata: | ||
| name: create-database-generator | ||
| namespace: mintmaker | ||
| spec: | ||
| schedule: "0 */4 * * *" | ||
| concurrencyPolicy: Forbid | ||
| jobTemplate: | ||
| spec: | ||
| template: | ||
| spec: | ||
| restartPolicy: Never | ||
| containers: | ||
| - name: database-generator | ||
| image: quay.io/konflux-ci/mintmaker:next | ||
| imagePullPolicy: IfNotPresent | ||
| command: | ||
| - /bin/sh | ||
| - -c | ||
| - /osv-generator -destination-dir /mnt/database -docker-filename docker.nedb -rpm-filename rpm.nedb -days 1 | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| memory: 500Mi | ||
| limits: | ||
| cpu: 100m | ||
| memory: 500Mi | ||
| volumeMounts: | ||
| - name: osv-db | ||
| mountPath: /mnt/database | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| readOnlyRootFilesystem: true | ||
| volumes: | ||
| - name: osv-db | ||
| persistentVolumeClaim: | ||
| claimName: osv-offline-db | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: database-generator | ||
| namespace: mintmaker | ||
| annotations: | ||
| argocd.argoproj.io/hook: Sync | ||
| argocd.argoproj.io/hook-delete-policy: HookSucceeded | ||
| spec: | ||
| template: | ||
| spec: | ||
| restartPolicy: Never | ||
| containers: | ||
| - name: database-generator | ||
| image: quay.io/konflux-ci/mintmaker:next | ||
| imagePullPolicy: IfNotPresent | ||
| command: | ||
| - /bin/sh | ||
| - -c | ||
| - /osv-generator -destination-dir /mnt/database -docker-filename docker.nedb -rpm-filename rpm.nedb -days 1 | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| memory: 500Mi | ||
| limits: | ||
| cpu: 100m | ||
| memory: 500Mi | ||
| volumeMounts: | ||
| - name: osv-db | ||
| mountPath: /mnt/database | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| readOnlyRootFilesystem: true | ||
| volumes: | ||
| - name: osv-db | ||
| persistentVolumeClaim: | ||
| claimName: osv-offline-db |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| apiVersion: v1 | ||
|
There was a problem hiding this comment. The default storage class provides EBS volumes. The pvc is going to be bounded to a single node, it means that all the jobs that will use it would need to be scheduled on the same node, this can cause a bottleneck. There was a problem hiding this comment. Interesting. Checking the storage classes, it seems all of them use EBS volumes. Is there a way to overcome this limitation? |
||
| kind: PersistentVolumeClaim | ||
| metadata: | ||
| name: osv-offline-db | ||
| namespace: mintmaker | ||
| spec: | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 10Gi | ||
| volumeMode: Filesystem | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| resources: | ||
| - create-database-pvc.yaml | ||
| - create-database-generator-cronjob.yaml | ||
| - create-database-generator-job.yaml | ||
| namespace: mintmaker |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to check how many data will be generated with days=1 (which can be done after Feb 10 when released.csv is available), but I would consider to increase this to 3 possibly, so we can still fix vulnerabilities in case of an outage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a placeholder value, I expect that we will gather more than just one day's worth of data. But for testing purposes, changes.csv has a lot of entries even for 1 day, so it will generate quite sizable databases.