[IR] Evaluate at

[justinchuby]

Given an ir.Value, it would be nice to have a method to evaluate the constant value for it.

tensors: Sequence[ir.TensorProtocol] = some_evaluator.evaluate_at([value1, value2, ...])

We can build the constant folder based on this, where we

1. Analyze which value should be constant folded (taking into account tensor sizes etc.
2. Replace the values with initializers after calling the function above

cc @gramalingam

[leshabirukov]

I have a question about learnable parameters. If processed net supposed to be trainable, such an optimization can change training behaviour, here speculative example:

X = input
W = const
Y = Conv(X,W) + Conv(X, Abs(W))

will become

X = input
Y = Conv(X,W1) + Conv(X, W2)

This is not the same in the perspective of training.

[justinchuby]

Yes. Constant folding is currently only used for inference models. For trainable models, it should be possible to mark parts of the model influenced by the trainable weights as not foldable.

[leshabirukov]

Replace the values with initializers after calling the function above

Is there a way to create initializers explicitly now? (I have searched up to ir._core.Graph.initializers)

---

EDIT:
Made it with

    inzer_name = f'{in_W.name}_pd'
    inzer_help = helper.make_tensor("",
                                    TensorProto.INT64, (8,),
                                    [0, 0, 1, 1, 0, 0, 0, 0])
    inzer_irir = TensorProtoTensor( inzer_help )
    inzer = op.initializer( inzer_irir, inzer_name)
    X = op.Pad(in_X, inzer )

@justinchuby, is this code ok?

[justinchuby]

Only the rewriter supports this, if that's what you meant

[leshabirukov]

Well, I meant both ways, but for now it is about rewriter.

[justinchuby]

Yes. You can use the numpy array semantics for it