How to enable fp16 with torch-tensorrt backend?

[vedantroy]

When trying to use fp16, I get the following error:

[2024-06-17 17:25:08,351][torch_tensorrt [TensorRT Conversion Context]][ERROR] - 4: [network.cpp::validate::3399] Error Code 4: Internal Error (fp16 precision has been set for a layer or layer output, but fp16 is not configured in the builder)

I'm configuring this as follows:

    with torch.no_grad(), torch.cuda.amp.autocast(dtype=torch.float16, enabled=True):
        model = torch.compile(model, fullgraph=True, backend="tensorrt")
        outputs = model(inputs)

[peri044]

You should use enabled_precisions as follows

model = torch.compile(model, fullgraph=True, 
                                       backend="tensorrt", 
                                       options={
                                         "enabled_precisions": {torch.float16},
                                       },)