GC overhead limit exceeded while generating a pickler for an AST

[densh]

Following code is a representation of some AST that starts its branching at Root class. Generation of pickling for it fails to compile after minutes of full cpu utilisation with "GC overhead limit exceeded" error. This is not your average "Out Of Memory" failure as it can't really be aided by adding more memory.

[jsuereth]

I'd like to have a minimal test case, but I think your problem is here:

https://gist.github.com/densh/d8419136ad47c74b2638#file-picklinggcoverhofeadlimitexceeded-scala-L71

When you have a circular reference of this fashion, pickling will explode (see: #317).

That is, when Pickling tries to generate a pickler for abstract type T it needs a pickler for ALL of its subtypes. If computing these subtypes somehow leads to requiring a pickler for [T] again, you have an infinite loop.

As a workaround, you'll need to build your picklers by hand for that structure. Also, I'd love to see if you could minimize the example, as I think this one is a duplicate of #317 .

[densh]

I've refactored the code to manually generate picklers in each and every sub-hierarchy and now it compiles.

https://gist.github.com/densh/d8419136ad47c74b2638#file-workaround-scala

It really seems to be the problem with how pickling handles co-recursion between types.

[jsuereth]

@densh Yeah. That's exactly it. We're somewhat limited in how we handle the recursion currently. IIRC, there's something about how we generate our picklers such that certain chains of pickler lookup cannot find the previously generated Pickler, since we haven't fixed it into any old namespace, but instead capture it locally.

Given the new source code generator architecture, we could attempt to:

1. Order all the "external" pickler lookups in one spot
2. Attempt to allow all these picklers to see each other (implicitly) so that we only generate them once.

This is a bit of a (necessary) optimisatoin I'd like to do when I get more time to play with the macros. There's all sorts of improvements we can do in the macro-generated picklers for compile-time speed AND runtime speed. Glad you have a workaround though!