Tracking the metrics with performance gaps on XLA

[sdesrozis]

🚀 Feature

Since this commit, the metrics have been disabled with xla device due to a performance regression.

ignite/ignite/metrics/metric.py

Lines 224 to 225 in 7bdf923

	if torch.device(device).type == "xla":
	raise ValueError("Cannot create metric on an XLA device. Use device='cpu' instead.")

To solve this, it would be interesting to track which metrics work fine and which have lack of performance. Thus, we could think about more specific enable / disable process.

[Moh-Yakoub]

@sdesrozis I am interested in this task. I have few questions though:

1. Do we have a plan about measuring those? How did we measure the performance regression in the first place?
2. Do you recommend we enable them one by one till we hit the regression or do some sort of performance measure prior to see which metrics would benefit from the xla? Do you suggest having this as a part of our examples repo?

[sdesrozis]

Sorry for the late answer, I completely missed your message !!

1. Do we have a plan about measuring those? How did we measure the performance regression in the first place?
2. Do you recommend we enable them one by one till we hit the regression or do some sort of performance measure prior to see which metrics would benefit from the xla? Do you suggest having this as a part of our examples repo?

Firstly, we could compare the performance gap between runs on CPU, GPU and TPU. The idea is to get an overview of the performance of all metrics. We could use the tests as a first step, try identifying the time consuming metrics on xla and look at them more closely.

What do you think ?