feat: jbang line magic

[maxandersen]

This is a PR to discuss how/if support for jbang styled code would be of your interest in jjava.

This PR adds two jbang related magics.

%jbang <scriptRef>

Line magic that takes an argument and will build and add its resulting jar + classpath.

The argument can be any meaningfull jbang reference, i.e. local or remote (https://..) file (.java,.kt,.groovy), maven artifact (g:a:v:), gist or git repo url etc.

Examples:

%jbang myfile.java
%jbang https://acme.org/my.jar
%jbang a.b:artifact:1.0

One caveat due to jshell - scripts must define a package as the default/empty package are not accessible to jshell (annoying :)

But otherwise it lets you use almost every feature of jbang.

In future could add support for command line arguments beyond just scriptRef (ie. --java to use different java versions for building)

%%jbang cell magic

Basically same setup as %%loadfrompom, where we take the content of the cell and pass to jbang and the resulting classpath added to the running shell.

Example:

%%jbang
//DEPS org.dflib:dflib-jupyter:1.0.0
...

Ultimately I would have preferred I could also eval the code via the kernel and show the result but afaics magics cannot affect the display. There is DisplayMagics but does not seem to be enabled afaics.

Proper JBang integration

This pr made me realize that a more sensible form of jbang integration would be to let jbang hook into eval and thus not require additional magic lines. I can open PR for that too but wanted to start with more "traditional" integration.

Also, this PR currently require jbang on the PATH - a proper one would go look for it and possibly even fetch it if not installed (jbang-maven and jbang-gradle does similar)

Question

1. would you be interested in these jbang based magics?

2. would you be interested in a PR that would make it so if a cell has jbang directives (so only activated if such are present - without requiring use of %magics. ?

[maxandersen]

note: the build failure is due to jbang not being available in the environment. I'll handle that aif need be.

[andrus]

Hey Max, this sounds interesting. Need to understand better the full functionality scope from a user perspective.

So //DEPS (and a short-hand notation of %jbang a.b:artifact:1.0) will be an alternative to Maven for dependency resolution. This part is clear. What other features of jbang would make sense for a Java developer in a context of a Jupyter notebook?

[maxandersen]

@andrus %jbang yourapp.java would be able to build arbitrary full java apps, not be limited to jshell - but without having to make any additional setup/config.

That includes //FILES and //SOURCES that lets you add resources and additional sources from the filesystem.

Having JBang syntax/approach available inside jupyter notebook in same way as is possible outside means there is one way to explain and learn - making it easier to transition from jupyter to "real" java and of course also the other way.

Thats my main interest.

Something I'm considering is if default eval would support using jbang to extract/manage using //DEPS etc. it could make sense to make %jbang line magic expose all of jbang commands - i.e. %jbang run scriptref would actually run and having %jbang build scriptref be what to use to get a classpath update ...this would let one use jupyter notebook to orchestrate and run any java based artifact - possibly even running upcoming jbang test and have the results somehow rendered back to the jupyter notebook?

I think it could open up for lots of interesting interactions and visualizations.

[maxandersen]

just to be clear: So //DEPS (and a short-hand notation of %jbang a.b:artifact:1.0) will be an alternative to Maven for dependency resolution. is not really true. %jbang <scriptref> is a build and the output of that is placed on the classpath - so //DEPS and %jbang <scirptref> are not equivalent. //DEPSand%classpath` are similar.

Ultimately I wish there was a way to reset the classpath in jjava but i think atm only way of doing that is forcing a restart.

[maxandersen]

any update/comments on this idea?

I'm considering doing a dedicate jbang-style kernel to have less duplicate behaviors for the user but to do that a lot more changes needed as lots of hardwired deps to JavaKernel/Kernel.getInstance() in the current codebase.

[andrus]

Sorry, been (am) swamped at work (and still digesting the idea). I think I am starting to get the big picture though 🙂

Still struggling conceptually whether this should be its own kernel or some kind of “kernel extension” for the existing one (essentially what you mentioned about overriding eval)

So in the current PR Jshell is the main environment, with kernel calling external jbang, and essentially using a „pipe” to scrape its text output and load to JShell. If you are able to override eval, will jbang operate in the same JVM as the kernel with a more direct interaction with JShell ?

[maxandersen]

So in the current PR Jshell is the main environment, with kernel calling external jbang, and essentially using a „pipe” to scrape its text output and load to JShell. If you are able to override eval, will jbang operate in the same JVM as the kernel with a more direct interaction with JShell ?

for now I'm using/preferring the external call as it means it will be able to use any jbang version - not just the one bundled with the kernel.

But we could also make jbang-cli.jar a dependency of kernel but I'm not sure it gives us much win. I would at least not put it through jbang unless there is a //DEPS or other jbang directive so overhead will be minimal.

[maxandersen]

Still struggling conceptually whether this should be its own kernel or some kind of “kernel extension” for the existing one (essentially what you mentioned about overriding eval)

reasons for own kernel is to not also expose the 'raw' classpaths option but its also not a biggie. I tried make separate but the current kernel is too hard coupled.

[andrus]

reasons for own kernel is to not also expose the 'raw' classpaths option but its also not a biggie. I tried make separate but the current kernel is too hard coupled.

Yeah, from a user perspective, there'd be a "jjava way" (Maven, really) to setup classpath and a "jbang way", so a separate kernel would make more sense... jjbang 🙂

We inherited from iJava a split between "basekernel" with Jupyter protocol knowledge and the Java kernel. I wouldn't mind further modularizing the Java kernel. It is a JShell-based code executor + Maven dependency resolver. The Maven part can be replaced with a pluggable resolver API, so that bespoke kernels could be built with other alternatives (and their own set of magics). Do you think this will get you unstuck?

[maxandersen]

@andrus it could - if jjava would have way for me to have jupyter-jbang repo that extends jjava and the code in jjava gets its global statics decoupled things would be good.

[andrus]

Sounds like a plan.

Would it be possible for you to sign our individual CLA at https://www.objectstyle.com/f/cla/icla.pdf (modeled after Apache and Spring), and, if your employer is involved, also a corporate CLA at https://www.objectstyle.com/f/cla/ccla.pdf ) and email a scan to the address mentioned in the CLA. Then I will create a repo.

And separately, I'll do my best to jump on the jjava refactoring mentioned above (will probably start on Friday, and work on it over the weekend).

[maxandersen]

for jupyter-jbang i was thinking about just putting it in github.com/jbangdev and use dflib as dependency.

[andrus]

Great. Let me work on the refactoring then

[andrus]

I just did the first refactoring pass - properly modeling magics and streamlining their evaluation flow per #80 . This is already on main.

The next step is modularization of jjava itself and externalizing the kernel bootstrap out of its constructor. This should hopefully make it fully reusable for building alternative kernels on top of JShell

[maxandersen]

Cool.

Will try update jbang kernel to utilize it.

Probably will still hit the global kernel instance limit but sounds you are targeting solving that too!

[andrus]

I just finished the refactoring scope that I had in mind:

• Magics as interfaces #80 - Magics as interfaces instead of annotations
• Refactoring: JavaKernel must manage JShell #81, Refactoring: Move JavaKernel init logic from constructor to Builder #82 - refactoring BaseKernel and JavaKernel with proper builders for maximum reusability
• Refactoring: reorg Maven modules for reusability #84 - refactoring modules to decouple dependencies (again, for maximum reusability)

When implementing own kernel based on JJava, you might follow this approach:

1. Add jjava-kernel as a dependency
2. (Optional) Add jjava-maven as a dependency, but only if you care for Maven magics in your kernel.
3. (Optional) Inherit your own kernel from JavaKernel, and your builder - from JavaKernelBuilder. This is only needed if you can't implement certain customizations as either magics or JShell / Java startup scripts (e.g., if you want to override eval(..)). Otherwise, you can directly use our kernel.
4. To create and package a working kernel, you will need to do some copy/pasting. jjava-kernel only provides the builder, and is intentionally environment-agnostic (doesn't even reference JJAVA_ vars). So take a look at and copy to your own project parts of jjava-distro as appropriate. The interesting parts are:
   1. JJava.main(..) - builds the kernel, adds all magics, etc.
   2. pom.xml - has all the assembly stuff. It is a bit loaded. We package 2 runnable jars in the distro. One of them is a "launcher" that takes some env config and passes it to the kernel JVM. You may or may not care about this.

Let me know how it goes and whether you have all the extension points now.

[andrus]

(Optional) Inherit your own kernel from JavaKernel, and your builder - from JavaKernelBuilder. This is only needed if you can't implement certain customizations as either magics or JShell / Java startup scripts (e.g., if you want to override eval(..)).

In fact, if you can implement everything via magics or notebook scripts, I can easily change the magic definition process to be allowed after kernel startup, and then you'd just use JJava, adding an extra jar to classpath. Let me know

[maxandersen]

Will give this as a try and let you know how it goes!

[maxandersen]

okey - got https://github.com/jbangdev/jbang-jupyter up and running.

still early days but at least works on my machine :)

[maxandersen]

about magics - It would be nice to enable %jbang line magic and possibly also cell magic on jjava using just a jar dependency. how would that work?

for clean //DEPS usage i still need to override eval.

should we consider allowing something like %enablejbang or similar which will allow participating in eval?

because then in jjava you could do

%maven dev.jbang:jupyter-jbang
%enablejbang

or similar?

[maxandersen]

fyi, I realized most of the default magics (display, print etc.) wasn't available - had to copy the NotebookMagics.java that configure the static imports to make them available.

I was wondering if those "magics" shuold all be in jjava and then only jjava-distro have the extensions declared? because as is now two gets loaded no matter what, but the static imports part does not. seems non-symmetrical.

[andrus]

about magics - It would be nice to enable %jbang line magic and possibly also cell magic on jjava using just a jar dependency. how would that work?

Right. This aligns with what I wanted to do with dynamic magic registration from a custom org.dflib.jjava.jupyter.Extension. Then you'd be able to drop a jar with JBang support on the classpath of JJava and may not even need a custom kernel. For this we'll need to address a couple of issues:

• Extension loading from classpath #71
• Run extensions in JShell ClassLoader #85
• Add back the API to load magics in the running kernel

Will deal with these next

[andrus]

fyi, I realized most of the default magics (display, print etc.) wasn't available - had to copy the NotebookMagics.java that configure the static imports to make them available.

I was wondering if those "magics" shuold all be in jjava and then only jjava-distro have the extensions declared? because as is now two gets loaded no matter what, but the static imports part does not. seems non-symmetrical.

This was intentional. The idea is that core module extensions are installed in every kernel. All they do is store a reference to the kernel in a static var and nothing else. But then the distro module decides what gets exposed via static imports (i.e. which methods are relevant in a given kernel).

In fact we need the BaseNotebookStatics extension from jjava-jupyer for MagicTranspiler to operate regardless of whether its static import is exposed in the notebook.