Separate out "trivia" from "notation"

[Keno]

The usual definition of "trivia" are "things that can be removed from the input without affecting the semantics of the output of the parse" and includes whitespace, comments (and depending on who you ask, docstrings). However, when JuliaSyntax talks about "trivia" this includes everything that does not end up in the final AST and includes fixed keywords, notation, etc. We should separate these categories so that downstream users can make use of this information. In some circumstances, it might make sense to print code without trivia (e.g. when space constrained), or to change the highlighting of trivial, but keeping notation intact.

[Keno]

I've changed my mind on this a little bit - I think it's fine to have a single flag on whether this is included in the AST or not that is used for both trivia and notation (although we should find a different word for it) - we can distinguish trivia and notation by kind as long as we have a separate (non-terminal) kind for semantic whitespace, so that people who want a notation-full tree can keep it.

[c42f]

I think it could be nice to do something here ... but I'm not sure what it should be. For example, even comments can act as semantic whitespace:

julia> :(function#==#foo()
       end)
:(function foo()
      #= REPL[9]:1 =#
      #= REPL[9]:1 =#
  end)

In JuilaSyntaxFormatter I go in the opposite direction: I'm building a model of semantic whitespace in the formatter, rather than in the parser. (This is required if we want to print valid ASTs to text without the source code existing - a case that the parser can never cover.)

So maybe a good option is to have a standard API which users can call in order to discover the required semantic whitespace. "Required" vs "suggested" is quite subtle and I think it can even depend on the content of token strings which the parser shouldn't be inspecting.

[c42f]

The general approach of JuliaSyntaxFormatter is to emit a stream of tokens including the semantic whitespace as "flexible space". Then later run a heuristic formatting pass which decides what each piece of whitespace should be. (The thought is to plug in a simple learned model for this, perhaps)