port Julia parser to JavaScript

[StefanKarpinski]

The Julia parser is written in Scheme and evaluated using Jeff's femtolisp implementation. It would be handy for many things, including the web repl to be able to parse Julia code in JavaScript. A couple of possible approaches exist:

1. Find a JavaScript Scheme implementation and port the existing flisp code to it.
2. Use emscripten to compile flisp to JavaScript.

Not sure which approach is better, but this would be an interesting and largely self-contained project. See discussion here: https://groups.google.com/d/topic/julia-dev/SWkJWxrS5xo/discussion.

[JeffBezanson]

Hey, it's not April 1 yet!

[StefanKarpinski]

We can make that the due date for the issue.

[JeffBezanson]

We will also have to go through the code and make sure it is standard scheme. Unfortunately flisp has useful functions like string.find that scheme would never dirty its hands with. The use of ungetc will probably also be painful.

[StefanKarpinski]

The alternative would be to try compiling flisp to JavaScript using LLVM, which should certainly be possible.

[JeffBezanson]

That would probably work but feels really heavyweight to me. Biwa scheme seems to be alive and even moving to R6RS.

[StefanKarpinski]

Cool. Well, either way, it's a good "up for grabs" project.

[despeset]

I've written some basic parsers in Javascript before, I'd love to work on this -- I don't think I could take the lead but I'd definitely be up for some hard work.

[StefanKarpinski]

@danielmendel: there's no parser writing in JavaScript involved here. The parser is already written in FemptoLisp, which is a dialect of Scheme (mostly R5RS compliant, I believe). So it's a matter of either getting FemtoLisp running in JavaScript via emscripten (option 2), or porting the parser Scheme code from the FemtoLisp Scheme dialect, which runs on x86, to e.g. the Biwa Scheme dialect, which runs on JavaScript (option 2). The port should involve relatively minor changes to the Scheme code, which @JeffBezanson will know better than anyone, but would involve getting into and understanding both Scheme implementations and the the parser framework to some extent.

[despeset]

@StefanKarpinski Right okay, I understand -- that makes more sense. I am definitely up to working on this but am even more sure that I'm not going to dive in without someone else who knows better how to proceed. I'll agitate some of the HS batch to see if I can round up some folks to look at it with me -- there was some talk of implementing a simple Scheme dialect in Javascript by folks a couple days ago, they might be excited about this as an alternative project with more practical applications.

[StefanKarpinski]

Cool. Looking into how Biwa Scheme works would certainly be a good alternative project (although implementing your own Scheme is always fun), and there's nothing like trying to get stuff working in a system like that to get a feel for how it actually works. The comparison across two rather different Scheme implementations would also be interesting.

[nathanwbrei]

@danielmendel l I can't do this by myself either, but I definitely want to contribute.

[ncollins]

I don't have any experience of writing parsers but I was hoping to write one at HS and was considering Scheme/Racket. I'll speak to you on Monday :)

[nathanwbrei]

Sounds good!

[ncollins]

Hi Jeff, Stefan

Daniel and I have been looking over the differences between femtolisp and Biwa. It looks like they are reasonably compatible. We will need to include some of the functions from the SRFI1 list library (but the reference implementation is in Scheme anyway) that are used in the parser, and (as mentioned above) implement the string.find function.

The Biwa implementation on node.js only includes a little more than the browser version (IO and process), so we think it would be easier to test on this to start with. Do you have any idea of what kind of testing environment we would need to setup? Are there existing tests for the parser?

[ViralBShah]

I don't think we have specific tests for the parser only.

[bradengroom]

Is this being actively worked on by anyone? I am interested.

[StefanKarpinski]

It is not – go for it!

[bradengroom]

Great! Is there a formal grammar for Julia defined anywhere?

[kmsquire]

No, only in julia-parser.scm

[tshort]

Stefan mentioned Emscripten in his original post. While the parser wasn't a goal of mine, I have played around with compiling Julia with Emscripten. I got Emscripten to compile the flisp directory to libflisp.a. I didn't try anything further. It was over a month ago, but I don't think I touched anything in the flisp directory. I still have the files around, so let me know if that would help.

[bradengroom]

I believe that I could write a parser in Biwa fairly easily if I had a grammar. I will just have to study julia-parser.scm a bit I guess.

[StefanKarpinski]

The grammar is the scheme code ;-) Just get it to run under that scheme and you're done.

[TheSeamau5]

Hey there, I'm interested in giving a hand to port the Julia parser to Javascript.
I was wondering if I could help.

I really like Julia and I think that it's a wonderful language, especially since I have previously programmed a lot in Matlab.

I'm mostly a Javascript programmer but for the I've been exploring different approaches to get the parser to work in Javascript:

1. Writing the parser from scratch in Javascript using peg.js
2. Using Emscripten to compile the flisp directory to Javascript

I'm kinda new to Lisp, Scheme and Emscripten and am a bit overwhelmed by the number of errors Emscripten is throwing at me so I'm still going to try hacking my way through FemtoLisp and Emscripten.

[JeffBezanson]

Cool.

My guess is that the easiest approach would be to port julia-parser.scm to a js-based scheme such as Biwa. That way you only have to deal with ~1 file, ~1800 lines, and just a small amount of scheme knowledge to translate the non-standard functions.

[TheSeamau5]

Sounds awesome. I'll get to improving my Scheme then. Yay parentheses!

I do have a worry though about this approach. My worry is that Emscripten tends to spit out horrendously horrible javascript code. Just to try out I wrote a simple hello world in C (4 lines) and Emscripten a 6000 line Javascript file. Sure, it's probably super fast and optimized (although hello world is just 1 line long in Javascript) but I feel that it'll make it hard to operate with other Javascript libraries or even the DOM for that matter.

In any case, I'll go work on my Scheme.

[JeffBezanson]

Running the scheme code in Biwa (or similar) would not use Emscripten.

[pao]

Just to try out I wrote a simple hello world in C (4 lines) and Emscripten a 6000 line Javascript file.

Most of that overhead will amortize out with larger programs, and a line of asm.js is roughly like a line of assembly--it doesn't do much. But as @JeffBezanson said, avoiding that altogether is probably a saner approach.

[mictadlo]

To reduce the JavaScript code you can use Google's closure https://developers.google.com/closure/.

[jahewson]

I had a look at getting the parser running with Biwa but it doesn't support the non-standard bound? built-in which seems to be used by table.

[ihnorton]

Closing in favor of #9430.

[EricForgy]

Can we reopen this? I still think the Biwa option (or something more recent in the past year) is a viable alternative to Emscripten, which - in my limited experience - doesn't seem to be so friendly.

If we were to try Biwa, is that still the best alternative to Emscripten? Does the bound? thing make this a non starter? For something else I'm working on, I was thinking about writing a mini parser just to handle interpolation only. Then, I thought to use JuliaParser, but that seems to be broken.

I'm doing a lot of work getting Julia and JS to talk to each other so something like this would be great.

[tkelman]

Would be more broadly useful to try to fix JuliaParser.

[sc0ttwad3]

How about we open a new issue and broaden/update it for at least a discussion around a toolchain for going from Julia -> to -> WebAssembly? With all the work being done around WebAssembly: LLVM targeting WASM, things like LLVM JIT VM, the infrastructure and toolchain library for WebAssembly binaryen, and the WebAssembly Binary Toolkit wabt, many more, and support in nearly all the browsers, it seems like a great time to get a toolchain going, or at least a discussion around how to get there?

[StefanKarpinski]

We can certainly have a discussion, but until 1.0 is out, this is definitely not a priority. Refactoring bootstrapping and tooling is an excellent 1.x project since it shouldn't affect anything user-facing.

[sc0ttwad3]

Watched your Julia Roadmap talk from JuliaCon 2017 at Berkeley and figured out it wouldn't be part of current coding efforts in the deprecation cycle, I guess I was asking where the best place for keeping Julia pkg/tools development and any toolchain efforts to WebAssembly aware of each other. Probably something to handle over in Julia discourse for now.

Thanks for all your work, BTW!