Using torch.compile for speedup

[omar-abdelgawad]

Is your feature request related to a problem? Please describe.
Pytorch 2.0 introduced the torch.compile function and decorator that could speed up models by adding only a single line of code. It is still new though. Here is a tutorial from official docs:
https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html#conclusion

Write now we don't know what type of gpus does it work with and need more details about what it actually does and is it worth using?

Describe the solution you'd like
Please collect more data about the function (and topic in general) and if you conclude that it should be used make a pull request with the minimal corresponding changes after testing that it works on all of modified models (and actually made them faster).

Describe alternatives you've considered
This is experimental but important as we currently don't have a workstation so we are really low on computing resources.