Giant refactor to move all state into a Kernel struct

[JamesWrigley]

Pretty much what the title says 😛 This is not ready yet but I'm opening it for transparency. There are two reasons for the refactor:

1. Easier testing, so that we can more confidently make changes like Make message receive and handling async #1140 and Jupyter messaging updates/correctness #1138.
2. Allow for creating a temporary kernel that can be used in a precompilation workload for Add precompile statements to reduce startup latency #1074.

Overview:

• Almost all global state has been moved into the new Kernel struct, and all relevant internal methods now require a kernel be passed to them. That's the bulk of the diff. I've tried to minimize other changes as much as possible, so there should 🤞 be no change in behaviour from master.

• There are certain fields like ans and n which are documented as globals and must stay global, so setproperty!(::Kernel,...) is implemented such that it will copy changes to those fields to their corresponding global variable. Similarly there are certain public methods like IJulia.history() that must still work without a kernel argument, so a new _default_kernel global is defined for normal usage and that's used by default for all public methods.

  An edge-case are the dicts In and Out which are by default assigned to the corresponding fields of _default_kernel and will not contain changes from temporary test kernels. I don't think that's a particularly important discrepancy so I'm inclined to ignore it.

• I implemented a basic test in waitloop.jl that spawns the IJulia waitloop and then makes a request to it using jupyter_client. This isn't working very well right now because blocking calls to the client causes hangs, I suspect there's a deadlock somewhere.

This will need a lot more work before it can be merged, but feel free to review the design already (cc @stevengj, @fredrikekre).

@MasonProtter, you should be able to do... something... with this for a precompilation workload. I would suggest creating a Kernel, connecting to some of the sockets with ZMQ.jl, and then sending some mock requests following the wire protocol: https://jupyter-client.readthedocs.io/en/latest/messaging.html#the-wire-protocol

[codecov]

Codecov Report

Attention: Patch coverage is 88.25215% with 41 lines in your changes missing coverage. Please review.

Project coverage is 66.56%. Comparing base (b340817) to head (72e3590).

Files with missing lines	Patch %	Lines
src/handlers.jl	66.66%	8 Missing ⚠️
src/IJulia.jl	93.10%	6 Missing ⚠️
src/init.jl	88.67%	6 Missing ⚠️
src/inline.jl	57.14%	6 Missing ⚠️
src/stdio.jl	85.00%	6 Missing ⚠️
src/eventloop.jl	86.36%	3 Missing ⚠️
src/kernel.jl	0.00%	3 Missing ⚠️
src/comm_manager.jl	90.00%	2 Missing ⚠️
src/execute_request.jl	96.29%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #1145       +/-   ##
===========================================
+ Coverage    9.90%   66.56%   +56.65%     
===========================================
  Files          15       16        +1     
  Lines         848      963      +115     
===========================================
+ Hits           84      641      +557     
+ Misses        764      322      -442     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JamesWrigley]

Figured out the deadlock, we can now start writing full tests with jupyter_client :)

[stevengj]

Probably using rather than import since we only use these, not extend them? Also invokelatest is exported automatically these days.

(This code dates back to the days before using Foo: bar)

[JamesWrigley]

Agreed, but I think I'd prefer to do those sorts of changes in a later PR. This one is already quite large 😅

[stevengj]

should be called by a finalizer?

[JamesWrigley]

Unfortunately that wouldn't be easy because it wait's on the stdio tasks so it might yield. I think it's ok to rely on the do-constructor closing it in the tests.

[JamesWrigley]

Progress update:

• Moved the waitloop task into Kernel to make testing easier, and added a wait(::Kernel) method to wait for it to finish.
• Added/expanded tests for autocompletion, MIME display, and clear_output() using jupyter_kernel_test. I think that covers everything we can use jupyter_kernel_test for.
• Moved the calls to watch_stdio() and pushdisplay() into init(), now close(::Kernel) will ensure they're cleaned up properly.
• Added a bunch of tests for some of the public API: clear_history(), load(), the hook system, and history().
• Tweaked history() to print all the entries on new lines in 20dc591. From the docstring, I think that's what it should be doing?

To-do:

• Address remaining review comments.
• Add a test for kernel.jl to ensure that the kernel works outside of the test suite.
• Figure out why the load() test is failing on Windows.
• Go through the coverage reports to find more low-hanging fruit to test.

[JamesWrigley]

I believe this is ready to be reviewed now. One thorny issue is segfaults on windows that appear to be coming from ZMQ, using @threadcall for the heartbeat seemed to fix it: d491800
But I'm not convinced I understand the problem fully, will try to debug it a bit more before merging.

For reviewers, I've tried to keep the commits atomic so I would recommend reviewing each commit one-by-one.

[JamesWrigley]

Explicitly waiting for the heartbeat thread also seemed to fix it, which seems preferable to @threadcall in the interest of minimizing changes. But now mac is hanging...

[JamesWrigley]

Back in the green with 0b6e7be 😅 I think this is in pretty good shape now.

[JamesWrigley]

I went ahead and added a simple precompilation workload in 9bb03a1. On my system it reduces the running time of getting the kernel info and executing 42 from 4s to 0.5s.

Not sure why the tests are failing, it seems to be related to CondaPkg 🤔 Will investigate later, but I believe this is ready for review anyway.

[JamesWrigley]

Back in the green, had to explicitly specify MicroMamba after the latest CondaPkg update.

[MasonProtter]

Testing locally and installing a kernel, the kernel seems to die as soon as I attempt to connect

[JamesWrigley]

Is the path to kernel.jl in the kernel.json file correct? If precompilation succeeds but Jupyter can't connect to the kernel I think that points to a problem in kernel.json (I ran into that while testing locally).

[MasonProtter]

Figured out the problem I was having, it was crashing at startup for me because comm_open, comm_close, and comm_msg had the wrong signature relative to the invokelatest call in the event loop.

Github isn't letting me give a suggestion for the comm_close and comm_msg functions, but they need to all have the same signature.

[JamesWrigley]

Ouch, that's quite a serious oversight 😬 I've added tests and hopefully fixed the comms code in bb7cef4, with the one change of making the :notarget Comm sent in the comm_open error case to not be a primary comm, so it won't send a comm_open message to the client:

IJulia.jl/src/comm_manager.jl

Line 114 in 4390ef4

send_ipython(IJulia.publish[],

One other thing I noticed is that close_comm() (closing a comm from the kernel side) doesn't remove the comm from the comms dictionary:

IJulia.jl/src/comm_manager.jl

Lines 85 to 90 in 4390ef4

	function close_comm(comm::Comm, data::Dict = Dict(),
	metadata::Dict = Dict(); kwargs...)
	msg = msg_comm(comm, IJulia.execute_msg, "comm_close", data,
	metadata; kwargs...)
	send_ipython(IJulia.publish[], msg)
	end

@stevengj, should we be removing it like the comm_close() handler does?

[stevengj]

@shashi, can you common on the close_comm question?

[JamesWrigley]

Rebased to fix merge conflicts. About close_comm(), in the absence of clarification I'd suggest we go ahead with the existing behaviour for now since it's been working so far.

[JamesWrigley]

(bump)

[JamesWrigley]

(rebased to fix merge conflicts from #1149)

[JamesWrigley]

(bump)

[MasonProtter]

Is it known why Windows is failing to precompile here? It seems to build and run fine on my machine.

[JamesWrigley]

Nope, I still need to debug that (bit swamped this week). But glad it works on your machine now 🎉 Do you see a TTFX improvement from the precompilation?

[MasonProtter]

Yes, I am seeing a slight noticeable reduction in TTFX!

[MasonProtter]

I should mention though that my main usecase (https://github.com/emacs-jupyter/jupyter) remains super slow the first time and seems to be unaffected by this. But baby steps, and I don't think that should hold back this effort.