Skip to content

Commit 7c7c63a

Browse files
committed
Remove ElasticManager field printing_kwargs
The name of the field is confusing. It's much cleaner for the users to call `get_connect_cmd` directly. Workers may be added in different ways to the same ElasticManager and require a different startup. Also, `get_connect_cmd` was not using printing_kwargs for it's kwargs by default, which was inconsistent.
1 parent fc6f1f8 commit 7c7c63a

File tree

3 files changed

+34
-13
lines changed

3 files changed

+34
-13
lines changed

README.md

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ port and publish their own host/port information for other workers to connect to
1919
On the master, you need to instantiate an instance of `ElasticManager`. The constructors defined are:
2020

2121
```julia
22-
ElasticManager(;addr=IPv4("127.0.0.1"), port=9009, cookie=nothing, topology=:all_to_all, printing_kwargs=())
22+
ElasticManager(;addr=IPv4("127.0.0.1"), port=9009, cookie=nothing, topology=:all_to_all)
2323
ElasticManager(port) = ElasticManager(;port=port)
2424
ElasticManager(addr, port) = ElasticManager(;addr=addr, port=port)
2525
ElasticManager(addr, port, cookie) = ElasticManager(;addr=addr, port=port, cookie=cookie)
@@ -37,6 +37,17 @@ ElasticManager:
3737
/home/user/bin/julia --project=/home/user/myproject/Project.toml -e 'using ElasticClusterManager; ElasticClusterManager.elastic_worker("4cOSyaYpgSl6BC0C","127.0.1.1",36275)'
3838
```
3939

40-
By default, the printed command uses the absolute path to the current Julia executable and activates the same project as the current session. You can change either of these defaults by passing `printing_kwargs=(absolute_exename=false, same_project=false))` to the first form of the `ElasticManager` constructor.
40+
Use `ElasticClusterManager.get_connect_cmd(em; kwargs...)` to generate a suitable system command to start up a
41+
worker process, e.g.:
4142

42-
Once workers are connected, you can print the `em` object again to see them added to the list of active workers.
43+
```julia
44+
print(ElasticClusterManager.get_connect_cmd(em; absolute_exename=false, same_project=false))
45+
```
46+
47+
```
48+
/home/user/bin/julia --project=/home/user/myproject/Project.toml -e 'using ElasticClusterManager; ElasticClusterManager.elastic_worker("4cOSyaYpgSl6BC0C","127.0.1.1",36275)'
49+
```
50+
51+
By default, the printed command uses the absolute path to the current Julia executable and activates the same project as the current session.
52+
53+
Once workers have connected, `show(em)` should show them added to the list of active workers.

src/elastic.jl

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,18 @@ Constructor:
2121
```julia
2222
function ElasticManager(;
2323
addr=IPv4("127.0.0.1"), port=9009, cookie=nothing,
24-
topology=:all_to_all, manage_callback=elastic_no_op_callback, printing_kwargs=()
24+
topology=:all_to_all, manage_callback=elastic_no_op_callback
2525
)
2626
```
2727
2828
The manager will listen for worker connections on `addr:port`. Workers have
2929
to authenticate themselves with `cookie` (randomly generated by default).
30+
`topology` sets the Julia cluster topology and may be `:all_to_all` or
31+
`:master_worker` (see documentation of `Distributed.addprocs` for details.
32+
33+
Workers are not added via `Distributed.addprocs`, instead started via
34+
[`ElasticClusterManager.elastic_worker`](@ref) from external Julia
35+
processes, this is up to the user.
3036
3137
If `manage_callback` is set to a user-provided `my_manage_callback`, it will
3238
be called on the primary Julia process when `Distributed.manage` adds or
@@ -47,11 +53,10 @@ struct ElasticManager <: Distributed.ClusterManager
4753
topology::Symbol
4854
sockname
4955
manage_callback
50-
printing_kwargs
5156

5257
function ElasticManager(;
53-
addr=IPv4("127.0.0.1"), port=9009, cookie=nothing,
54-
topology=:all_to_all, manage_callback=elastic_no_op_callback, printing_kwargs=()
58+
addr=Sockets.IPv4("127.0.0.1"), port=9009, cookie=nothing,
59+
topology=:all_to_all, manage_callback=elastic_no_op_callback
5560
)
5661
Distributed.init_multi()
5762
cookie !== nothing && Distributed.cluster_cookie(cookie)
@@ -67,7 +72,7 @@ struct ElasticManager <: Distributed.ClusterManager
6772

6873
l_sock = Distributed.listen(addr, port)
6974

70-
lman = new(Dict{Int, Distributed.WorkerConfig}(), Channel{Sockets.TCPSocket}(typemax(Int)), Set{Int}(), topology, Sockets.getsockname(l_sock), manage_callback, printing_kwargs)
75+
lman = new(Dict{Int, Distributed.WorkerConfig}(), Channel{Sockets.TCPSocket}(typemax(Int)), Set{Int}(), topology, Sockets.getsockname(l_sock), manage_callback)
7176

7277
t1 = @async begin
7378
while true
@@ -171,9 +176,6 @@ function Base.show(io::IO, mgr::ElasticManager)
171176
seek(iob, position(iob)-1)
172177
println(iob, "]")
173178

174-
println(iob, " Worker connect command : ")
175-
print(iob, " ", get_connect_cmd(mgr; mgr.printing_kwargs...))
176-
177179
print(io, String(take!(iob)))
178180
end
179181

@@ -207,6 +209,16 @@ function elastic_worker(
207209
Distributed.start_worker(c, cookie)
208210
end
209211

212+
"""
213+
ElasticManager.get_connect_cmd(em::ElasticManager; absolute_exename=true, same_project=true, exeflags::Tuple=())
214+
215+
Return a suitable system command that starts a Julia process with an elastic
216+
worker that will try to connect the the primary Julia process running the
217+
[`ElasticManager`](@ref) instance `em`.
218+
219+
The output of `get_connect_cmd` should be taken as a baseline. Complex use
220+
cases (on some batch systems, etc.) may require a more involved command.
221+
"""
210222
function get_connect_cmd(em::ElasticManager; absolute_exename=true, same_project=true, exeflags::Tuple=())
211223
ip = string(em.sockname[1])
212224
port = convert(Int,em.sockname[2])
@@ -220,5 +232,4 @@ function get_connect_cmd(em::ElasticManager; absolute_exename=true, same_project
220232
project...,
221233
"-e 'import ElasticClusterManager; ElasticClusterManager.elastic_worker(\"$cookie\",\"$ip\",$port)'"
222234
]," ")
223-
224235
end

test/elastic.jl

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@
2525
@test lines[2] == "Active workers : []"
2626
@test lines[3] == "Number of workers to be added : 0"
2727
@test lines[4] == "Terminated workers : [ 2]"
28-
@test lines[5] == "Worker connect command :"
2928
end
3029

3130
@testset "Other constructors for ElasticManager()" begin

0 commit comments

Comments
 (0)