Does decoding process must memory the whole compute graph?

[baoy-nlp]

while decoding process without any need of back propagation, can i set some parameter reduce the memory we need allocate to the dynet?

[neubig]

Conceptually, no, this should not be necessary. In the forward pass of neural networks we can re-use memory for variable that are not going to be referenced anymore. The following page from MxNet has a good explanation of this:
http://mxnet.io/architecture/note_memory.html

However, in the current implementation in DyNet this is not supported. I'll put this as an enhancement, as it would be a very nice feature to have.

[baoy-nlp]

OK. Thanks a lot. I'm very looking forward to it.

[davidweichiang]

We're interested in helping with this. What would it entail? Since the memory pools can't free blocks, it seems not trivial.

[davidweichiang]

@aargueta2

[neubig]

Yes, this is non-trivial. Creating a new DyNet memory allocator that just wraps a system call to malloc is easy. It would result in a speed reduction, but if it is not the default, but an option I think this is reasonable.

The second thing to do would be ensuring that garbage collection is done when there are no existing remaining pointers to the memory block in use. This could basically be done by having an overridable delete operation for memory blocks, where the current behavior of "no-op" could be maintained, but if we were using the new memory efficient allocator then it does the appropriate reference counting and deletion when the reference count reaches zero.

One other concern is complexity: one of the nice things about DyNet is that it is not overly complex and bloated. We'd have to make sure that the changes can be made without making too many compromises with respect to simplicity.

So yes, this is non-trivial, but if it seems to be workable I'm happy to help you guys with the design @davidweichiang @aargueta2 (@redpony might also be interested)

As work-arounds that you could do with the existing code (both not so pretty):

1. you could also use multiple GPUs for search
2. search for a while, save the search state on CPU, clear the graph, and re-start search again

[davidweichiang]

How about multiple memory pools, is there support for that already? That would allow a kind of manual management (which is sort of what we're doing already):

1. encoder and attention create nodes in pool E
2. decoder initial state in pool D
3. decoder adds one input word, putting intermediate nodes in pool T and the new hidden state/cell in pool D'
4. free pools D and T
5. decoder adds one input word, putting intermediate nodes in pool T and the new hidden state/cell in pool D
6. free pools D' and T
   goto step 3

[davidweichiang]

Do you have an estimate of how much slowdown there would be from using malloc?

And, do you envision any complications arising from nodes having their argument node pointers invalidated?

[neubig]

Note: the following PR will allow for this #1064