Refactor NbBlock

[pietroppeter]

this is a major change, implements #168 (and also #117). Very WIP in a sandbox folder (see sandbox/notes.md)

[pietroppeter]

milestone reached, now in sandbox/minib3.nim both a minimal html and json backend work. (your json work was great @HugoGranstrom). Still lots of details to work on, but I am feeling more confident now!

A big realization was that in the end we might not need after all nim mustache anymore (we might still provide it for legacy reason and we should be able to provide a somewhat backward compatible NbLegacyDoc object that implements the default theme as before with partials), in a sense nimib itself is some code based (and block based) templating system with superpowers...

[pietroppeter]

(moved this note in sandbox/notes.md)

[HugoGranstrom]

Wow! Great work! 🥳 Looking good, it definitely seems way easier to define custom blocks now that the mustache backend is gone 😄 But I'm wondering how we will solve the problem of customizing an existing block type without it 🤔 But maybe that is what this idea is about?:

• can I also use a super method or something to wrap the rendered block in a NbBlock?
  • for example it could be used to add a class name inside a div
    • this could be important for example to customize code block appearance
  • or to add an optional id to a block
  • (this could anyway be added later)
  • and it could be added as something that by default a block does during rendering
  • it might have a different signature (takes rendering of content and outputs new rendering)

Mmh, no to customize an existing block the idea is to replace the function that renders it in the NbRenderobject here:

nimib/sandbox/minib3/minib.nim

Line 26 in 5e6727e

var nbToHtml: NbRender # since we need it for json, let's make it also for html

This sounds magical! Being able to modify the rendered code would be so cool and useful. The custom block code could stay very simple while we add all sort of features behind the scenes. Would we be able to do this with raw HTML though? Parse the HTML to and AST and modify the AST and then rewrite it as HTML?

The part above is still an idea and will need more fleshing out. I would definitely start with bare html and would allow customization only in the sense of: give me the rendered output and I can put things before or after. Any kind of refinement to that idea would need a use case that we think makes it worth it. And I would not probably push this during this refactor, maybe for now we do not even do what is sketched above (just to keep the scope manageable).

there is a tension between what you want to be able to serialize (content and specificities of a page) and what you want to be controlled (later) by the SSG (like the theme) and should not be serialized (also because it is redundant). Tension also with a third element which is the fact that rendering does need a theme

Is there really a tension? Couldn't we theoretically serialize the theme as well and just ignore/overwrite it in the SSG?

Yes, that is an option (serializing also the theme), but I guess it is kind of wasteful and also not "clean". I would like if possible to have a somewhat readable json. Especially if it is a low cost thing and a simple rule such as: everything that is in theme field of NbBlock is not serialized, the rest will.

Also, while we are refactoring the code, one part that has always confused me is nb.blk. I don't really see the point of it. Can't we just use nb.blocks[^1] instead (behind a template for easy access)? What value does adding it as its own variable (that you need to remember to set) bring?

Yeah, that is a good observation and it might not be needed after all. In principle you should not remember to set it since it should be in the NbBlock generating sugar, but if we can avoid needing it I could probably remove it. It will get trickier once I introduce real custom container blocks. My idea there at the moment is that NbDoc would have a add or similar field that represent what is used to add the block (the container could have multiple blocks fields, or maybe add, adds first to field first then to field second and that is it...). Then it could still be useful to have a reference to last block processed (which is needed for post processing, for the moment we have only the nbClearOutput in nimib, but I guess there could be other reasonable use cases).

Thanks for the feedback on the work-in-progress, it is useful :) nothing here is yet set in stone but I think a cleaner implementation is finally coming out.

[pietroppeter]

note, I just pushed a possible implementation of a NbContainer object (and NbDoc would inherit from this):

  NbContainer = ref object of NbBlock
    blocks: seq[NbBlock]
    parent: NbContainer

while the html backend works fine with it the json backend fails to serialize because of the circular reference. This is an issue with jsony that does not support this out of the box. It does not seem easy to fix. I tried to use skipHook (with a dev version of jsony), but since ref objects delegate to the non ref version, it does not work (I do not know if there is a way to get the type of the non ref version). I also tried with a custom dumpHook (which would not be nice to automate) but it also does not work. This issues should be probably better documented (maybe in a jsony issue?).

My current plan is to actually skip the parent field in NbContainer and have a containers: seq[NbContainer] field in Nb object (and add(nb: var Nb, blk: NbBlock) as a generic api that might support a different implementation.

[pietroppeter]

ok the new NbContainer implementation is indeed simpler, easier to implement and... it actually works! :)
I have also "hidden" the nb.blk api which is something that we might get rid of in the future.

Next steps:

• implement nbCode in minib
• implement nbCodeFromJs in minib
• think if there are essentially other types of blocks we need to test with the new refactoring
• if not go ahead and implement sugar to implement custom blocks (and refactor)
• clean up minib implementation
• reimplement all of nimib starting with the new minib poc

[HugoGranstrom]

Nice work! 🤩

I really like the withContainer and nb.add. Now that nb.blk is hidden, I don't have that much against it anymore tbh.

[pietroppeter]

notes from another discussion. in this PR we should also:

• reduce the number of global variables created (see the gotchas section of nimislides)
• have a non-global api where many blocks could be called without creating any globals (e.g. nb.text("hi"))

[HugoGranstrom]

So, I've started working on implementing nbJsFromCode. The part that I'm not sure about is where we should do the Nim->JS compilation. Should it happen before or after the transformation to JSON? It feels like it should happen before so that the JSON->HTML can happen without a Nim-compiler present + that steps become faster (important for SSG?).

Another thing that I'm wondering about is whether we should do the compilation the same way we are doing it now. Now we do it in nbSave: code
The thing is that we compile all of the nbJsFromCode blocks in the same file at the end, so we can't do the compilation before we have created all of them. I think the way we do it should be fine but I'm open to other ideas.

[HugoGranstrom]

I've started thinking about the sugar for defining a block now and we have for example:

type
  NbImage = ref object of NbBlock
    url: string
template nbImage*(turl: string) =
  let blk = NbImage(url: turl, kind: "NbImage")
  nb.add blk
func nbImageToHtml*(blk: NbBlock, nb: Nb): string =
  let blk = blk.NbImage
  "<img src= '" & blk.url & "'>"
nbToHtml.funcs["NbImage"] = nbImageToHtml
addNbBlockToJson(NbImage)

Which should be shortened to:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

I'm not sure how the template could be automated with this sugar. It feels like it should be created manually unless we can come up with a nice way of abstracting it. So currently I think this is feasible:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

template nbImage*(turl: string) =
  let blk = NbImage(url: turl, kind: "NbImage")
  nb.add blk

Something else to consider is inheritance. By default the block inherits from NbBlock but you could also write:

newNbBlock(nbImage of NbContainer):
  ...

to signify that it should inherit another block?

[HugoGranstrom]

Something we could do though to prevent the error of writing the wrong kind (wrong casing probably) in the template when creating the block, is to generate a newNbImage proc which has all the inputs except kind which is already prefilled. So the example would become:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

template nbImage*(turl: string) =
  let blk = newNbImage(url=turl)
  nb.add blk

[pietroppeter]

Why the template cannot be created from the sugar macro? Is there some macro limitation I am not aware of (I know I am not aware of many things regarding macros... :))

[pietroppeter]

Btw I forgot to add them but I think it might worth adding also a new nb. api that is not template based, e.g.:

proc image*(nb: var NbContext, url: string) =
  nb.blk.add NbImage(url: url)

template nbImage*(url: string) =
  nb.image(url)

The new procedural api would also help to reduce the number of templates that nimib needs and incidentally should reduce the number of globals (of course for nbCode and similar we still would have a template in order to have untyped arguments)

[pietroppeter]

Also agree on the additional sugar for inheriting from other blocks

[HugoGranstrom]

Because we don't know the inputs and what to do with them 😄 Say we have two fields on the object and we only want the user to provide one of them. So the problem is that we don't know what the input to the template should be from just the information given right now. We could of course add the template body as a new section in the sugar syntax with the inputs as well. But then we have basically written the template already. So I think it is more flexible if we write the template manually. For example when you have a block that you want to create a few different templates with different defaults for.

[HugoGranstrom]

Yes it's a good idea to write the functional api as well 😄 I feel like I have regained some coding motivation now so I'll just need to find some time the coming weeks 😁

[pietroppeter]

Because we don't know the inputs and what to do with them

Ah right. I think we could provide maybe a default with all fields or indeed think of additional sugar but I guess it might be wiser for the moment to limit ourselves to leave it out of the sugar for now and see later if there is some good pattern that emerges

[pietroppeter]

And yeah probably we should just provide the functional api as default (with all the fields) and leave all the templates as a customization (indeed nbCode has a lot of variants, the skip one, the block one, ... and they all refer to the same type)

[pietroppeter]

In that case I would suggest the templates to use the functional api in their implementation (and this could be general advice).

[HugoGranstrom]

Because we don't know the inputs and what to do with them

Ah right. I think we could provide maybe a default with all fields or indeed think of additional sugar but I guess it might be wiser for the moment to limit ourselves to leave it out of the sugar for now and see later if there is some good pattern that emerges

The problem with creating a default template is that it can cause conflict with a manually created one.

[HugoGranstrom]

And yeah probably we should just provide the functional api as default (with all the fields) and leave all the templates as a customization (indeed nbCode has a lot of variants, the skip one, the block one, ... and they all refer to the same type)

So we automatically generate the functional api? How would we do that for blocks that require untyped arguments? Or do you mean we just create a constructor automatically?

[HugoGranstrom]

In that case I would suggest the templates to use the functional api in their implementation (and this could be general advice).

Agreed, when possible, the templates should be sugar on top of the functional api 👍

[HugoGranstrom]

Looking at this again, I'll try to summarize the last conversation we had:

• The sugar should generate a functional initializer with the kind prefilled
• The sugar should not generate the template (as we don't know what the inputs to it as they could be untyped)
• The sugar should not generate the functional API for the same reason, at least not by default. This is because it would create problems if we want to create a custom template/proc.

This means that for now, when defining a new block. We would have to create a functional api proc/template for each block type. And then create a "old-style" template using that (e.g. nbCode using nb.code)

This could be included in the sugar, but then we are basically defining the proc/template in there instead. Which could be a good thing as stuff would be nicely contained.

[HugoGranstrom]

Sooo, now the sugar is finally finished. So instead of this:

type
  NbImage = ref object of NbBlock
    url: string
template nbImage*(turl: string) =
  let blk = NbImage(url: turl, kind: "NbImage")
  nb.add blk
func nbImageToHtml*(blk: NbBlock, nb: Nb): string =
  let blk = blk.NbImage
  "<img src= '" & blk.url & "'>"
nbToHtml.funcs["NbImage"] = nbImageToHtml
addNbBlockToJson(NbImage)

We now do this:

newNbBlock(nbImage):
  url: string
  toHtml:
    "<img src= '" & blk.url & "'>"

func image*(nb: var Nb, url: string) =
  let blk = newNbImage(url=url)
  nb.add blk

template nbImage*(url: string) =
  nb.image(url)

Note that nbImage must be a template as it needs to access nb which isn't defined until we run nbInit

[HugoGranstrom]

I'm trying to think about what's left and it seems like I haven't converted nbJs to the sugar style yet. Will give it a try during the weekend. Other than that, is there anything else holding us back from actually starting to implement this in Nimib?

[pietroppeter]

Hi! not really sure if there is anything missing, have not had really time to think about it. I guess we could try and start implementing in Nimib and see how it goes! Cheers from Brussels (Fosdem)!

[HugoGranstrom]

Oh right it Fosdem! Have a great time and good luck! 😄

Yes I'll finish nbJs up and give it a try and we'll see if I get caught up somewhere. Imagine if we got this working before 27th Febuary, only a year from opening the PR 😎

[HugoGranstrom]

There's one thing I remember that we have discussed, but I can't remember what our conclusion was. Namely, target backends. The sugar has toHtml but do we want to support other formats as well in the future (like markdown) or are we putting our full focus on html?

We have the backend: NbRender field on Nb so we can shift backends, but I have a hard time seeing how this arbitrary backend would work with our sugar syntax 🤔

[pietroppeter]

Well, as long as we can still support custom backends, we do not need the sugar to support them. I guess the markdown backend could be an example on how to add a custom backend. I think it make sense to have the html backend as the only one to be supported by the sugar (although json is also kind of a backend of a special kind)

[HugoGranstrom]

Okay, that sounds reasonable 👍 We could always expand the sugar in the future if the demand for it starts to exist (for example by adding a toMd: section in the sugar). But focusing on html for now sounds like a good idea 😄

[HugoGranstrom]

And oh, backwards compatibility with the old mustache-backend must be handled somehow. It should be possible to implement as a custom backend during a transition period and then in the next major release we drop support for it? The things we'd like to remove are these fields from NbDoc and move them to the mustache-render object:

partials*: Table[string, string]
templateDirs*: seq[string]
renderPlans*: Table[string, seq[string]]
renderProcs*: Table[string, NbRenderProc]

[HugoGranstrom]

Or should we just throw backwards-compatibility out the door and say that users can use Nimib 3.x if they want to use the old mustache backend? That would simplify things for us and we could make a more thorough refactoring of code now. It would mostly be libraries like nimibook and nimislides that would need to adapt their code to work with Nimib 4.0.

[pietroppeter]

I would say to start implementing without thinking about backwards compatibility with old backend. Then at the end we can evaluate how much effort is to introduce some backward compatibility (with warning on usage and dropping on next version)

[HugoGranstrom]

That sounds like a good plan 👍