Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add: standalone plugin for evaluating dependencies with a graph #774

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

NiklasHargarter
Copy link
Contributor

@NiklasHargarter NiklasHargarter commented Dec 11, 2024

What

Adds a standalone plugin for evaluating script dependencies with a directed networkx graph.

checks for:

  • duplicate dependencies (a script declaring dependency on other script multiple times)
  • checking for cyclic dependencies
  • checking for missing dependencies
  • checking for cross feed dependencies (community script dependence on enterprise script). It is differentiated between dependencies that are behind a enterprise feed gate and those that are not.
  • category order
  • dependency on deprecated script

included functionality of normal plugins

  • dependencies (not included the subdirectory placement warning)
  • dependency_category_order (missing ACT_SCANNER error)
  • deprecated_dependency (i use the helper pattern regex not the one from the plugin)

Output

python logging levels for system information (error, warning, info)
normal additive verbosity up to -vv for result output.

Feed options

  • 21.04 (21.04 + common)
  • 22.04 (22.04 + common)
  • common
  • full (21.0 + 22.04 + common)

example call:
poetry run troubadix-dependency-graph ~/gb/vulnerability-tests/nasl --feed full --log info -vv

Execution Time

locally ~13 seconds

Why

When checking dependencies, it makes sense to analyse the whole feed, rather than just working on changed scripts. And working on the whole feed is easier with a standalone plugin that doesn't have to adhere to the Troubadix structure.

References

Checklist

  • Tests

Copy link

github-actions bot commented Dec 11, 2024

Conventional Commits Report

Type Number
Changed 15
Added 5

🚀 Conventional commits found.

Copy link

codecov bot commented Dec 11, 2024

Codecov Report

Attention: Patch coverage is 91.84549% with 19 lines in your changes missing coverage. Please review.

Project coverage is 80.88%. Comparing base (fede29e) to head (5457bd2).

Files with missing lines Patch % Lines
...alone_plugins/dependency_graph/dependency_graph.py 86.72% 9 Missing and 6 partials ⚠️
...adix/standalone_plugins/dependency_graph/checks.py 95.65% 1 Missing and 1 partial ⚠️
...oubadix/standalone_plugins/dependency_graph/cli.py 91.30% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #774      +/-   ##
==========================================
+ Coverage   79.81%   80.88%   +1.07%     
==========================================
  Files          87       91       +4     
  Lines        3022     3207     +185     
  Branches      589      606      +17     
==========================================
+ Hits         2412     2594     +182     
- Misses        462      463       +1     
- Partials      148      150       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Dec 12, 2024

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 3 package(s) with unknown licenses.
See the Details below.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 5457bd2.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

License Issues

poetry.lock

PackageVersionLicenseIssue Type
lxml5.3.1NullUnknown License
networkx3.4.2NullUnknown License

pyproject.toml

PackageVersionLicenseIssue Type
networkx^ 3.4.2NullUnknown License
Allowed Licenses: 0BSD, AGPL-3.0-or-later, Apache-2.0, BlueOak-1.0.0, BSD-2-Clause, BSD-3-Clause-Clear, BSD-3-Clause, BSL-1.0, CAL-1.0, CC-BY-3.0, CC-BY-4.0, CC-BY-SA-4.0, CC0-1.0, EPL-2.0, GPL-2.0-only, GPL-2.0-or-later, GPL-2.0, GPL-3.0-or-later, ISC, LGPL-2.0-only, LGPL-2.0-or-later, LGPL-2.1-only, LGPL-2.1-or-later, LGPL-2.1, LGPL-3.0-only, LGPL-3.0, LGPL-3.0-or-later, MIT, MIT-CMU, MPL-1.1, MPL-2.0, OFL-1.1, PSF-2.0, Python-2.0, Python-2.0.1, Unicode-DFS-2016, Unlicense, Zlib, ZPL-2.1

OpenSSF Scorecard

PackageVersionScoreDetails
pip/lxml 5.3.1 🟢 6.6
Details
CheckScoreReason
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Code-Review⚠️ 0Found 2/29 approved changesets -- score normalized to 0
Packaging⚠️ -1packaging workflow not detected
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Maintained🟢 1030 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Security-Policy🟢 10security policy file detected
Binary-Artifacts🟢 10no binaries found in the repo
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Pinned-Dependencies⚠️ 1dependency not pinned by hash detected -- score normalized to 1
License🟢 9license file detected
Fuzzing🟢 10project is fuzzed
Vulnerabilities🟢 100 existing vulnerabilities detected
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
Signed-Releases⚠️ 0Project has not signed or included provenance with any releases.
pip/networkx 3.4.2 🟢 4.9
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 17 issue activity found in the last 90 days -- score normalized to 10
Code-Review🟢 9Found 27/28 approved changesets -- score normalized to 9
Dangerous-Workflow⚠️ 0dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Binary-Artifacts🟢 10no binaries found in the repo
License🟢 9license file detected
Vulnerabilities🟢 100 existing vulnerabilities detected
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Signed-Releases⚠️ -1no releases found
Packaging🟢 10packaging workflow detected
Fuzzing🟢 10project is fuzzed
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy⚠️ 0security policy file not detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
pip/ruff 0.9.6 UnknownUnknown
pip/networkx ^ 3.4.2 🟢 4.9
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 17 issue activity found in the last 90 days -- score normalized to 10
Code-Review🟢 9Found 27/28 approved changesets -- score normalized to 9
Dangerous-Workflow⚠️ 0dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Binary-Artifacts🟢 10no binaries found in the repo
License🟢 9license file detected
Vulnerabilities🟢 100 existing vulnerabilities detected
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Signed-Releases⚠️ -1no releases found
Packaging🟢 10packaging workflow detected
Fuzzing🟢 10project is fuzzed
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy⚠️ 0security policy file not detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0

Scanned Files

  • poetry.lock
  • pyproject.toml

@NiklasHargarter NiklasHargarter force-pushed the dependency_graph branch 2 times, most recently from f43bc1f to 63c0f64 Compare January 20, 2025 08:45
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
checks for a script depending on a script multiple times
"""
warnings = []
for script in scripts:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to come back to this, reminder to myself



def cross_feed_dependencies(graph, gated_status: bool) -> list[tuple[str, str]]:
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to come back to this

Copy link
Contributor

@amy-gb amy-gb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it locally and it works great! Left a couple of small comments but I will check back when Max has finished his review

troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
troubadix/standalone_plugins/dependency_graph.py Outdated Show resolved Hide resolved
@mbrinkhoff mbrinkhoff mentioned this pull request Jan 29, 2025
1 task
@NiklasHargarter NiklasHargarter marked this pull request as ready for review February 11, 2025 11:16
@NiklasHargarter NiklasHargarter requested a review from a team as a code owner February 11, 2025 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants