Remove workaround for old pytorch. (#12480) #590

Job	Run time
test-unix-nightly (linux, 3.11, 12.1, nightly)	1d 0h 0m 0s
test-stable (linux, 3.11, 12.1, stable)	1s
test-stable (linux, 3.10, 12.1, stable)	1d 0h 0m 0s
test-stable (linux, 3.12, 12.1, stable)	1s
	2d 0h 0m 2s

Provide feedback