[Inference] Implement a "1 class = 1 provider<>task pair" logic to isolate provider-specific code

[Inference] Implement a "1 class = 1 provider<>task pair" logic to isolate provider-specific code #3819