TRTorch v0.4.1

TRTorch v0.4.1

Bug Fixes for Module Ignorelist for Partial Compilation, trtorch.Device, Version updates for PyTorch, TensorRT, cuDNN

Target Platform Changes

This is the first patch of TRTorch v0.4. It now targets by default PyTorch 1.9.1, TensorRT 8.0.3.4 and cuDNN 8.2.4.15 and CUDA 11.1. Older versions of PyTorch, TensorRT, cuDNN are still supported in the same manner as TRTorch v0.4.0

Module Ignorelist for Partial Compilation

There was an issue with the pass marking modules to be ignored during compilation where it unsafely assumed that methods are named forward all the way down the module tree. While this was fine for 1.8.0, with PyTorch 1.9.0, the TorchScript codegen changed slightly to sometimes use methods of other names for modules which reduce trivially to a functional api. This fix now will identify method calls as the recursion point and then use those method calls to select modules to recurse on. It will also check to verify existence of these modules and methods before recursing. Finally this pass was run by default even if the ignore list was empty causing issues for users not using the feature. Therefore this pass is now disabled unless explicitly enabled

trtorch.Device

Some of the constructors for trtorch.Device would not work or incorrectly configure the device. This patch will fix those issues.

Dependencies

- Bazel 4.0.0
- LibTorch 1.9.1
- CUDA 11.1 (on x86_64, by default, newer CUDA 11 supported with compatible PyTorch Build), 10.2 (on aarch64)
- cuDNN 8.2.3.4
- TensorRT 8.0.3.4

0.4.1 (2021-10-06)

Bug Fixes

• //core/lowering: Fixes module level fallback recursion (2fc612d)
• Move some lowering passes to graph level logging (0266f41)
• //py: Fix trtorch.Device alternate contructor options (ac26841)

Operators Supported

Operators Currently Supported Through Converters

• aten::_convolution(Tensor input, Tensor weight, Tensor? bias, int[] stride, int[] padding, int[] dilation, bool transposed, int[] output_padding, int groups, bool benchmark, bool deterministic, bool cudnn_enabled, bool allow_tf32) -> (Tensor)
• aten::_convolution.deprecated(Tensor input, Tensor weight, Tensor? bias, int[] stride, int[] padding, int[] dilation, bool transposed, int[] output_padding, int groups, bool benchmark, bool deterministic, bool cudnn_enabled) -> (Tensor)
• aten::abs(Tensor self) -> (Tensor)
• aten::acos(Tensor self) -> (Tensor)
• aten::acosh(Tensor self) -> (Tensor)
• aten::adaptive_avg_pool1d(Tensor self, int[1] output_size) -> (Tensor)
• aten::adaptive_avg_pool2d(Tensor self, int[2] output_size) -> (Tensor)
• aten::adaptive_max_pool2d(Tensor self, int[2] output_size) -> (Tensor, Tensor)
• aten::add.Scalar(Tensor self, Scalar other, Scalar alpha=1) -> (Tensor)
• aten::add.Tensor(Tensor self, Tensor other, Scalar alpha=1) -> (Tensor)
• aten::add_.Tensor(Tensor(a!) self, Tensor other, *, Scalar alpha=1) -> (Tensor(a!))
• aten::asin(Tensor self) -> (Tensor)
• aten::asinh(Tensor self) -> (Tensor)
• aten::atan(Tensor self) -> (Tensor)
• aten::atanh(Tensor self) -> (Tensor)
• aten::avg_pool1d(Tensor self, int[1] kernel_size, int[1] stride=[], int[1] padding=[0], bool ceil_mode=False, bool count_include_pad=True) -> (Tensor)
• aten::avg_pool2d(Tensor self, int[2] kernel_size, int[2] stride=[], int[2] padding=[0, 0], bool ceil_mode=False, bool count_include_pad=True, int? divisor_override=None) -> (Tensor)
• aten::avg_pool3d(Tensor self, int[3] kernel_size, int[3] stride=[], int[3] padding=[], bool ceil_mode=False, bool count_include_pad=True, int? divisor_override=None) -> (Tensor)
• aten::batch_norm(Tensor input, Tensor? gamma, Tensor? beta, Tensor? mean, Tensor? var, bool training, float momentum, float eps, bool cudnn_enabled) -> (Tensor)
• aten::bmm(Tensor self, Tensor mat2) -> (Tensor)
• aten::cat(Tensor[] tensors, int dim=0) -> (Tensor)
• aten::ceil(Tensor self) -> (Tensor)
• aten::clamp(Tensor self, Scalar? min=None, Scalar? max=None) -> (Tensor)
• aten::clamp_max(Tensor self, Scalar max) -> (Tensor)
• aten::clamp_min(Tensor self, Scalar min) -> (Tensor)
• aten::constant_pad_nd(Tensor self, int[] pad, Scalar value=0) -> (Tensor)
• aten::cos(Tensor self) -> (Tensor)
• aten::cosh(Tensor self) -> (Tensor)
• aten::cumsum(Tensor self, int dim, *, int? dtype=None) -> (Tensor)
• aten::div.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::div.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::div_.Scalar(Tensor(a!) self, Scalar other) -> (Tensor(a!))
• aten::div_.Tensor(Tensor(a!) self, Tensor other) -> (Tensor(a!))
• aten::elu(Tensor self, Scalar alpha=1, Scalar scale=1, Scalar input_scale=1) -> (Tensor)
• aten::embedding(Tensor weight, Tensor indices, int padding_idx=-1, bool scale_grad_by_freq=False, bool sparse=False) -> (Tensor)
• aten::eq.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::eq.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::erf(Tensor self) -> (Tensor)
• aten::exp(Tensor self) -> (Tensor)
• aten::expand(Tensor(a) self, int[] size, *, bool implicit=False) -> (Tensor(a))
• aten::expand_as(Tensor(a) self, Tensor other) -> (Tensor(a))
• aten::fake_quantize_per_channel_affine(Tensor self, Tensor scale, Tensor zero_point, int axis, int quant_min, int quant_max) -> (Tensor)
• aten::fake_quantize_per_tensor_affine(Tensor self, float scale, int zero_point, int quant_min, int quant_max) -> (Tensor)
• aten::flatten.using_ints(Tensor self, int start_dim=0, int end_dim=-1) -> (Tensor)
• aten::floor(Tensor self) -> (Tensor)
• aten::floor_divide(Tensor self, Tensor other) -> (Tensor)
• aten::floor_divide.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::ge.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::ge.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::gelu(Tensor self) -> (Tensor)
• aten::gru_cell(Tensor input, Tensor hx, Tensor w_ih, Tensor w_hh, Tensor? b_ih=None, Tensor? b_hh=None) -> (Tensor)
• aten::gt.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::gt.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::hardtanh(Tensor self, Scalar min_val=-1, Scalar max_val=1) -> (Tensor)
• aten::hardtanh_(Tensor(a!) self, Scalar min_val=-1, Scalar max_val=1) -> (Tensor(a!))
• aten::instance_norm(Tensor input, Tensor? weight, Tensor? bias, Tensor? running_mean, Tensor? running_var, bool use_input_stats, float momentum, float eps, bool cudnn_enabled) -> (Tensor)
• aten::layer_norm(Tensor input, int[] normalized_shape, Tensor? gamma, Tensor? beta, float eps, bool cudnn_enabled) -> (Tensor)
• aten::le.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::le.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::leaky_relu(Tensor self, Scalar negative_slope=0.01) -> (Tensor)
• aten::leaky_relu_(Tensor(a!) self, Scalar negative_slope=0.01) -> (Tensor(a!))
• aten::linear(Tensor input, Tensor weight, Tensor? bias=None) -> (Tensor)
• aten::log(Tensor self) -> (Tensor)
• aten::lstm_cell(Tensor input, Tensor[] hx, Tensor w_ih, Tensor w_hh, Tensor? b_ih=None, Tensor? b_hh=None) -> (Tensor, Tensor)
• aten::lt.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::lt.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::masked_fill.Scalar(Tensor self, Tensor mask, Scalar value) -> (Tensor)
• aten::matmul(Tensor self, Tensor other) -> (Tensor)
• aten::max(Tensor self) -> (Tensor)
• aten::max.other(Tensor self, Tensor other) -> (Tensor)
• aten::max_pool1d(Tensor self, int[1] kernel_size, int[1] stride=[], int[1] padding=[], int[1] dilation=[], bool ceil_mode=False) -> (Tensor)
• aten::max_pool2d(Tensor self, int[2] kernel_size, int[2] stride=[], int[2] padding=[0, 0], int[2] dilation=[1, 1], bool ceil_mode=False) -> (Tensor)
• aten::max_pool3d(Tensor self, int[3] kernel_size, int[3] stride=[], int[3] padding=[], int[3] dilation=[], bool ceil_mode=False) -> (Tensor)
• aten::mean(Tensor self, *, int? dtype=None) -> (Tensor)
• aten::mean.dim(Tensor self, int[] dim, bool keepdim=False, *, int? dtype=None) -> (Tensor)
• aten::min(Tensor self) -> (Tensor)
• aten::min.other(Tensor self, Tensor other) -> (Tensor)
• aten::mul.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::mul.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::mul_.Tensor(Tensor(a!) self, Tensor other) -> (Tensor(a!))
• aten::narrow(Tensor(a) self, int dim, int start, int length) -> (Tensor(a))
• aten::narrow.Tensor(Tensor(a) self, int dim, Tensor start, int length) -> (Tensor(a))
• aten::ne.Scalar(Tensor self, Scalar other) -> (Tensor)
• aten::ne.Tensor(Tensor self, Tensor other) -> (Tensor)
• aten::neg(Tensor self) -> (Tensor)
• aten::norm.ScalarOpt_dim(Tensor self, Scalar? p, int[1] dim, bool keepdim=False) -> (Tensor)
• aten::permute(Tensor(a) self, int[] dims) -> (Tensor(a))
• aten::pixel_shuffle(Tensor self, int upscale_factor) -> (Tensor)
• aten::pow.Tensor_Scalar(Tensor self, Scalar exponent) -> (Tensor)
• aten::pow.Tensor_Tensor(Tensor self, Tensor exponent) -> (Tensor)
• aten::prelu(Tensor self, Tensor weight) -> (Tensor)
• aten::prod(Tensor self, *, int? dtype=None) -> (Tensor)
• aten::prod.dim_int(Tensor self, int dim, bool keepdim=False, *, int? dtype=None) -> (Tensor)
• aten::reciprocal(Tensor self) -> (Tensor)
• aten::relu(Tensor input) -> (Tensor)
• aten::relu_(Tensor(a!) self) -> (Tensor(a!))
• aten::repeat(Tensor self, int[] repeats) -> (Tensor)
• aten::replication_pad1d(Tensor self, int[2] padding) -> (Tensor)
• aten::replication_pad2d(Tensor self, int[4] padding) -> (Tensor)
• aten::replication_pad3d(Tensor self, int[6] padding) -> (Tensor)
• aten::reshape(Tensor self, int[] shape) -> (Tensor)
• aten::rsub.Scalar(Tensor self, Scalar other, Scalar alpha=1) -> (Tensor)
• aten::rsub.Tensor(Tensor self, Tensor other, Scalar alpha=1) -> (Tensor)
• aten::select.int(Tensor(a) self, int dim, int index) -> (Tensor(a))
• aten::sigmoid(Tensor input) -> (Tensor)
• aten::sigmoid_(Tensor(a!) self) -> (Tensor(a!))
• aten::sin(Tensor self) -> (Tensor)
• aten::sinh(Tensor self) -> (Tensor)
• aten::slice.Tensor(Tensor(a) self, int dim=0, int? start=None, int? end=None, int step=1) -> (Tensor(a))
• aten::softmax.int(Tensor self, int dim, int? dtype=None) -> (Tensor)
• aten::split(Tensor self, int[] split_sizes, int dim=0) -> (Tensor[])
• aten::split.Tensor(Tensor(a) self, int split_size, int dim=0) -> (Tensor[])
• aten::split_with_sizes(Tensor(a) self, int[] split_sizes, int dim=0) -> (Tensor[])
• aten::sqrt(Tensor self) -> (Tensor)
• aten::squeeze.dim(Tensor(a) self, int dim) -> (Tensor(a))
• aten::stack(Tensor[] tensors, int dim=0) -> (Tensor)
• aten::sub.Scalar(Tensor self, Scalar other, Scalar alpha=1) -> (Tensor)
• aten::sub.Tensor(Tensor self, Tensor other, Scalar alpha=1) -> (Tensor)
• aten::sub_.Tensor(Tensor(a!) self, Tensor other, *, Scalar alpha=1) -> (Tensor(a!))
• aten::sum(Tensor self, *, int? dtype=None) -> (Tensor)
• aten::sum.dim_IntList(Tensor self, int[1] dim, bool keepdim=False, *, int? dtype=None) -> (Tensor)
• aten::t(Tensor self) -> (Tensor)
• aten::tan(Tensor self) -> (Tensor)
• aten::tanh(Tensor input) -> (Tensor)
• aten::tanh_(Tensor(a!) self) -> (Tensor(a!))
• aten::to.dtype(Tensor self, int dtype, bool non_blocking=False, bool copy=False, int? memory_format=None) -> (Tensor)
• aten::to.other(Tensor self, Tensor other, bool non_blocking=False, bool copy=False, int? memory_format=None) -> (Tensor)
• aten::to.prim_Device(Tensor(a) self, Device? device, int? dtype=None, bool non_blocking=False, bool copy=False) -> (Tensor(a|b))
• aten::topk(Tensor self, int k, int dim=-1, bool largest=True, bool sorted=True) -> (Tensor values, Tensor indices)
• aten::transpose.int(Tensor(a) self, int dim0, int dim1) -> (Tensor(a))
• aten::unsqueeze(Tensor(a) self, int dim) -> (Tensor(a))
• aten::upsample_bilinear2d(Tensor self, int[2] output_size, bool align_corners, float? scales_h=None, float? scales_w=None) -> (Tensor)
• aten::upsample_bilinear2d.vec(Tensor input, int[]? output_size, bool align_corners, float[]? scale_factors) -> (Tensor)
• aten::upsample_linear1d(Tensor self, int[1] output_size, bool align_corners, float? scales=None) -> (Tensor)
• aten::upsample_linear1d.vec(Tensor input, int[]? output_size, bool align_corners, float[]? scale_factors) -> (Tensor)
• aten::upsample_nearest1d(Tensor self, int[1] output_size, float? scales=None) -> (Tensor)
• aten::upsample_nearest1d.vec(Tensor input, int[]? output_size, float[]? scale_factors) -> (Tensor)
• aten::upsample_nearest2d(Tensor self, int[2] output_size, float? scales_h=None, float? scales_w=None) -> (Tensor)
• aten::upsample_nearest2d.vec(Tensor input, int[]? output_size, float[]? scale_factors) -> (Tensor)
• aten::upsample_nearest3d(Tensor self, int[3] output_size, float? scales_d=None, float? scales_h=None, float? scales_w=None) -> (Tensor)
• aten::upsample_nearest3d.vec(Tensor input, int[]? output_size, float[]? scale_factors) -> (Tensor)
• aten::upsample_trilinear3d(Tensor self, int[3] output_size, bool align_corners, float? scales_d=None, float? scales_h=None, float? scales_w=None) -> (Tensor)
• aten::upsample_trilinear3d.vec(Tensor input, int[]? output_size, bool align_corners, float[]? scale_factors) -> (Tensor)
• aten::view(Tensor(a) self, int[] size) -> (Tensor(a))
• trt::const(Tensor self) -> (Tensor)

Operators Currently Supported Through Evaluators

• aten::Bool.float(float b) -> (bool)
• aten::Bool.int(int a) -> (bool)
• aten::Float.Scalar(Scalar a) -> float
• aten::Float.bool(bool a) -> float
• aten::Float.int(int a) -> float
• aten::Int.Scalar(Scalar a) -> int
• aten::Int.bool(bool a) -> int
• aten::Int.float(float a) -> int
• aten::Int.int(int a) -> int
• aten::and(int a, int b) -> (bool)
• aten::getitem.t(t list, int idx) -> (t(*))
• aten::is(t1 self, t2 obj) -> bool
• aten::isnot(t1 self, t2 obj) -> bool
• aten::not(bool self) -> bool
• aten::or(int a, int b) -> (bool)
• aten::__round_to_zero_floordiv(int a, int b) -> (int)
• aten::xor(int a, int b) -> (bool)
• aten::add.float(float a, float b) -> (float)
• aten::add.int(int a, int b) -> (int)
• aten::add_.t(t self, t[] b) -> (t[])
• aten::append.t(t self, t(c -> *) el) -> (t)
• aten::arange(Scalar end, *, int? dtype=None, int? layout=None,
  Device? device=None, bool? pin_memory=None) -> (Tensor)
• aten::arange.start(Scalar start, Scalar end, *, ScalarType? dtype=None,
  Layout? layout=None, Device? device=None, bool? pin_memory=None) -> (Tensor)
• aten::arange.start_step(Scalar start, Scalar end, Scalar step, *, ScalarType? dtype=None,
  Layout? layout=None, Device? device=None, bool? pin_memory=None) -> (Tensor)
• aten::clone(Tensor self, *, int? memory_format=None) -> (Tensor)
• aten::copy_(Tensor(a!) self, Tensor src, bool non_blocking=False) -> (Tensor(a!))
• aten::dim(Tensor self) -> int
• aten::div.float(float a, float b) -> (float)
• aten::div.int(int a, int b) -> (float)
• aten::eq.bool(bool a, bool b) -> (bool)
• aten::eq.float(float a, float b) -> (bool)
• aten::eq.float_int(float a, int b) -> (bool)
• aten::eq.int(int a, int b) -> (bool)
• aten::eq.int_float(int a, float b) -> (bool)
• aten::floor.float(float a) -> (int)
• aten::floor.int(int a) -> (int)
• aten::floordiv.float(float a, float b) -> (int)
• aten::floordiv.int(int a, int b) -> (int)
• aten::ge.bool(bool a, bool b) -> (bool)
• aten::ge.float(float a, float b) -> (bool)
• aten::ge.float_int(float a, int b) -> (bool)
• aten::ge.int(int a, int b) -> (bool)
• aten::ge.int_float(int a, float b) -> (bool)
• aten::gt.bool(bool a, bool b) -> (bool)
• aten::gt.float(float a, float b) -> (bool)
• aten::gt.float_int(float a, int b) -> (bool)
• aten::gt.int(int a, int b) -> (bool)
• aten::gt.int_float(int a, float b) -> (bool)
• aten::is_floating_point(Tensor self) -> (bool)
• aten::le.bool(bool a, bool b) -> (bool)
• aten::le.float(float a, float b) -> (bool)
• aten::le.float_int(float a, int b) -> (bool)
• aten::le.int(int a, int b) -> (bool)
• aten::le.int_float(int a, float b) -> (bool)
• aten::len.t(t[] a) -> (int)
• aten::lt.bool(bool a, bool b) -> (bool)
• aten::lt.float(float a, float b) -> (bool)
• aten::lt.float_int(float a, int b) -> (bool)
• aten::lt.int(int a, int b) -> (bool)
• aten::lt.int_float(int a, float b) -> (bool)
• aten::mul.float(float a, float b) -> (float)
• aten::mul.int(int a, int b) -> (int)
• aten::ne.bool(bool a, bool b) -> (bool)
• aten::ne.float(float a, float b) -> (bool)
• aten::ne.float_int(float a, int b) -> (bool)
• aten::ne.int(int a, int b) -> (bool)
• aten::ne.int_float(int a, float b) -> (bool)
• aten::neg.int(int a) -> (int)
• aten::numel(Tensor self) -> int
• aten::size(Tensor self) -> (int[])
• aten::size.int(Tensor self, int dim) -> (int)
• aten::slice.t(t[] l, int start, int end=9223372036854775807, int step=1) -> (t[])
• aten::sqrt.float(float a) -> (float)
• aten::sqrt.int(int a) -> (float)
• aten::sub.float(float a, float b) -> (float)
• aten::sub.int(int a, int b) -> (int)
• aten::tensor(t[] data, *, int? dtype=None, Device? device=None, bool requires_grad=False) -> (Tensor)
• prim::dtype(Tensor a) -> (int)
• prim::max.bool(bool a, bool b) -> (bool)
• prim::max.float(float a, float b) -> (bool)
• prim::max.float_int(float a, int b) -> (bool)
• prim::max.int(int a, int b) -> (bool)
• prim::max.int_float(int a, float b) -> (bool)
• prim::max.self_int(int[] self) -> (int)
• prim::min.bool(bool a, bool b) -> (bool)
• prim::min.float(float a, float b) -> (bool)
• prim::min.float_int(float a, int b) -> (bool)
• prim::min.int(int a, int b) -> (bool)
• prim::min.int_float(int a, float b) -> (bool)
• prim::min.self_int(int[] self) -> (int)
• prim::shape(Tensor a) -> (int[])