Skip to content

Add GraphsSharedArraysExt and remove Distributed dependency #430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

topolarity
Copy link

This PR follows up on #429 to enable parallel=:threads by default and move Distributed-based parallelism to an extension, for faster loading times and reduced dependencies.

On Julia <1.9, the dependency is still there so this shouldn't break backwards compatibility (1.6 tests are passing for me locally)

These implementations are extremely basic, but they try to follow the
patterns in the other parts of the Parallel module. My real motivation
is to be able to move the Distributed implementations into an extension,
so that Graphs.jl does not depend on Distributed.
@topolarity topolarity force-pushed the ct/distributed-ext branch from b44e9a0 to 3476c5f Compare May 15, 2025 02:10
Copy link

codecov bot commented May 15, 2025

Codecov Report

Attention: Patch coverage is 86.42857% with 19 lines in your changes missing coverage. Please review.

Project coverage is 96.63%. Comparing base (6130332) to head (3476c5f).
Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
src/Parallel/distance.jl 81.25% 3 Missing ⚠️
src/Parallel/shortestpaths/dijkstra.jl 75.00% 3 Missing ⚠️
src/Parallel/traversals/greedy_color.jl 78.57% 3 Missing ⚠️
src/Parallel/centrality/betweenness.jl 0.00% 2 Missing ⚠️
src/Parallel/centrality/closeness.jl 0.00% 2 Missing ⚠️
src/Parallel/centrality/radiality.jl 33.33% 2 Missing ⚠️
src/Parallel/centrality/stress.jl 33.33% 2 Missing ⚠️
src/Parallel/utils.jl 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #430      +/-   ##
==========================================
- Coverage   97.31%   96.63%   -0.68%     
==========================================
  Files         117      118       +1     
  Lines        6956     7011      +55     
==========================================
+ Hits         6769     6775       +6     
- Misses        187      236      +49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@topolarity
Copy link
Author

Expression: result ⊜ true
Evaluated: ⟪result: 😭 FAILED: /Users/runner/work/Graphs.jl/Graphs.jl/Project.toml
The file /Users/runner/work/Graphs.jl/Graphs.jl/Project.toml is not in canonical format.

Looks like 1.6 / 1.9 have a different notion of "canonical" for [weakdeps] entries in the Project.toml

I probably need to disable the test on Julia <1.9

@simonschoelly
Copy link
Member

I don't want to be too negative here - but I am not sure if this is something we want right now. My reasons for this:

  • It will make maintaining this package more difficult. It will also make testing it more difficult.
  • It will make using it slightly more difficult as one has to explicitly import the relevant packages - unfortunately Julia's package manager does not have the abilities to declare features like Rust does for example.
  • It is unclear how much this will increase loading time.
  • Having no or very few dependencies was never the the goal of this package. In the past we removed some dependencies because some packages (like the linear optimization ones) were breaking this package during the transition from Julia v0.7 to v1.0.

If there is really the wish to have weak dependencies I am open to it - but first we should reach a consensus on that - so we should discus it in a Github issue.

@simonschoelly
Copy link
Member

Expression: result ⊜ true
Evaluated: ⟪result: 😭 FAILED: /Users/runner/work/Graphs.jl/Graphs.jl/Project.toml
The file /Users/runner/work/Graphs.jl/Graphs.jl/Project.toml is not in canonical format.

Looks like 1.6 / 1.9 have a different notion of "canonical" for [weakdeps] entries in the Project.toml

I probably need to disable the test on Julia <1.9

Disabling tests for the lower bound version cannot be an option. We could raise the lower bounds to the latest long-term support release though - that would be v1.10. We can do that in a separate PR though.

@topolarity
Copy link
Author

topolarity commented May 21, 2025

Disabling tests for the lower bound version cannot be an option. We could raise the lower bounds to the latest long-term support release though - that would be v1.10. We can do that in a separate PR though.

It's just a formatting check though, which is implemented differently on v1.6 vs. 1.9 / 1.10. The Project.toml would still be checked for canonicity on 1.10, so I don't think there's any loss of coverage by disabling the check on 1.6

@simonschoelly
Copy link
Member

I think we are due for bumping the min version of this package anyway so I created a PR: #432 - let's see if someone objects.

@Krastanov
Copy link
Member

  • It is unclear how much this will increase loading time.

Wouldn't this strictly decrease loading time? And decrease compilation time as well? It is a small change for this specific package, but for packages that depend on Graphs.jl (and other packages) this can start adding up. It is why DiffEq.jl split out all of its solvers in separate packages.

e.g. running julia> @time @eval using Graphs on 1.11.5

master: 0.540166 seconds (411.04 k allocations: 26.379 MiB, 5.20% gc time, 15.12% compilation time)

this pr: 0.502729 seconds (380.79 k allocations: 24.282 MiB, 4.67% gc time, 8.19% compilation time)

  • Having no or very few dependencies was never the the goal of this package.

But happening to have few dependencies makes the life of downstream package maintainers betters. If Graphs unconditionally depends on Distributed, then other packages that have extensions for Distributed will run precompilation for the extensions. Changes like this are overall beneficial for the user because way less unnecessary precompilation will be happening.

@thchr
Copy link
Contributor

thchr commented May 22, 2025

  • It will make maintaining this package more difficult. It will also make testing it more difficult.

How will it become more difficult?

  • Having no or very few dependencies was never the the goal of this package. In the past we removed some dependencies because some packages (like the linear optimization ones) were breaking this package during the transition from Julia v0.7 to v1.0.

There's no downside to having fewer dependencies? The fewer dependencies Graphs.jl has, the lower the barrier is to some other package deciding to take a dependency on Graphs.jl.

Comment on lines +17 to +18
[weakdeps]
SharedArrays = "1a1011a3-84de-559e-8e89-a11a2f7dc383"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't SharedArrays then be removed from the [deps] section above as well?

@rafaqz
Copy link

rafaqz commented May 22, 2025

As the maintainer of a package that uses Graphs.jl but never Distributed or SharedArrays, I support this change.

We should also move Inflate.jl to an extension, it pulls in binary dependencies that most users wont use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants