Skip to content

Update sync_vulnerablecode pipeline to import vulnerablecode git repo #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Update sync_vulnerablecode pipeline to import vulnerablecode git repo #80

wants to merge 6 commits into from

Conversation

ziadhany
Copy link
Collaborator

No description provided.

@ziadhany
Update sync_vulnerablecode pipeline to import vulnerablecode Git repo…
46953eb 
…sitories with new schema format

Signed-off-by: ziad hany <[email protected]>
@ziadhany
Implement soft delete for Note and Package models
dd680ef 
Update sync_vulnerablecode pipeline to handle deletions and updates.

Signed-off-by: ziad hany <[email protected]>
@ziadhany
Add docs and enable bulk update for VulnerableCode pipeline
92f7130 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Add bulk update support to note_handler
d8ef695 
Signed-off-by: ziad hany <[email protected]>
@ziadhany
Copy link
Collaborator Author

sync_vulnerablecode pipeline logs

/home/ziad-hany/PycharmProjects/federatedcode/venv/bin/python /home/ziad-hany/PycharmProjects/federatedcode/manage.py sync sync_vulnerablecode 
Syncing data using sync_vulnerablecode
INFO 2025-08-22 22:01:15.921 Pipeline [SyncVulnerableCode] starting
INFO 2025-08-22 22:01:15.921 Step [get_git_repos] starting
INFO 2025-08-22 22:01:15.921 Step [get_git_repos] completed in 0 seconds
INFO 2025-08-22 22:01:15.921 Step [sync_vulnerablecode_repositories] starting
INFO 2025-08-22 22:01:15.933 Syncing vulnerability from 1 repositories
INFO 2025-08-22 22:02:09.464 Syncing 123,954 vulnerability scan from https://github.com/aboutcode-data/vulnerablecode-data/
INFO 2025-08-22 22:25:55.273 Progress: 10% (12396/123954) ETA: 12832 seconds (3.6 hours)
INFO 2025-08-22 22:51:35.978 Processed 10000 purls.yml files, flushing bulk changes...
INFO 2025-08-22 22:56:09.916 Progress: 20% (24791/123954) ETA: 12962 seconds (3.6 hours)
INFO 2025-08-22 23:12:38.901 Progress: 30% (37187/123954) ETA: 9869 seconds (2.7 hours)
INFO 2025-08-22 23:15:28.285 Processed 20000 purls.yml files, flushing bulk changes...
INFO 2025-08-22 23:23:06.593 Progress: 40% (49582/123954) ETA: 7286 seconds (2.0 hours)
INFO 2025-08-22 23:25:46.176 Progress: 50% (61977/123954) ETA: 5017 seconds (1.4 hours)
INFO 2025-08-22 23:25:47.094 Processed 10000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:26:33.825 Processed 20000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:26:44.647 Progress: 60% (74373/123954) ETA: 3383 seconds (56.4 minutes)
INFO 2025-08-22 23:27:19.765 Processed 30000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:27:41.000 Progress: 70% (86768/123954) ETA: 2199 seconds (36.6 minutes)
INFO 2025-08-22 23:28:05.058 Processed 40000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:28:36.993 Progress: 80% (99164/123954) ETA: 1297 seconds (21.6 minutes)
INFO 2025-08-22 23:28:50.458 Processed 50000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:29:32.493 Progress: 90% (111559/123954) ETA: 583 seconds (9.7 minutes)
INFO 2025-08-22 23:29:35.135 Processed 60000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:30:18.894 Processed 70000 VCID files, flushing bulk changes...
INFO 2025-08-22 23:30:27.569 Progress: 100% (123954/123954)
INFO 2025-08-22 23:30:31.257 The Importer run successfully
INFO 2025-08-22 23:30:31.330 Step [sync_vulnerablecode_repositories] completed in 5355 seconds (1.5 hours)
INFO 2025-08-22 23:30:31.331 Pipeline completed in 5355 seconds (1.5 hours)

Process finished with exit code 0

@ziadhany
Remove all password recommendations.
23a5e54 
Add a function to retrieve the vulnerabilities file path using vulnerability ID.

Signed-off-by: ziad hany <[email protected]>
@ziadhany
Add a test for sync_vulnerablecode pipeline
ff9c6d1 
Signed-off-by: ziad hany <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ziadhany