Evaluating Nuisance Model Performance:

[bindugupta]

I am currently using Double Machine Learning (DML) to estimate the causal impact of a treatment (T) on an outcome variable (Y), with a given set of confounders (X). Below is the context of my data.

Size: 300K rows
Class Imbalance:

• Treatment (T=1) appears in only 5% of the data.

• Outcome (Y=1) appears in only 7% of the data.

Variables:

• Treatment (T) & Outcome (Y): Both are binary (0/1).

• Confounders (X): 20 variables (16 binary, 4 numerical).

My doubt is,

1. If the nuisance models perform poorly, can I still trust the ATE estimates from the library ? Since DML relies on nuisance models, is it crucial to evaluate their performance ? Is there any suggested threshold performance for these models?

2. Interestingly, despite poor nuisance model performance, the four refutation tests (placebo treatment, data subset, random common cause, and dummy outcome) still pass. Is there an explanation for how these refutation tests could pass even when the nuisance models are weak?

[SvenKlaassen]

Thanks @bindugupta,

Regarding 1.:
Generally, the nuisance estimates are used to adjust for confounding. E.g. if treatment assignment strongly influenced by you covariates $X$ then the propensity score should should reflect that.
As the classifiers (for ml_m and ml_g) are used to fit conditional expectations

$$ m_0(X) := P(D=1|X) = E[D|X],$$

I would recommend to evaluate especially the logloss or similar measures and compare to a simple average (similar as $R^2$-measures). If the treatment probability is quite low, the model can be able to adjust for counfounding but still classifiy 0 (or not treated) for all units (e.g. if all probabilites are still predicted smaller than 0.5).
If the model does not predict better than an simple average (percentage of treated), then the estimate is mainly plausible if the assignment is very close to random (as in such case the propensity score would be $m_0(X) \approx 0.05$, for any unit).

Regarding 2.:
The placebo treatment and dummy outcome only verify if the implementation works. In such a case there exists no confounding in the data and the nuisance adjustments will only fail if they overfit too much.
A random common cause actually does not add a common cause (due to the randomness) and therefore also does not really refers to identification tests can mainly detect severe overfitting.

In my personal opinion these three tests will nearly never fail and can be more considered as "sanity checks", but not more. Using different data subsets are a nice option to see if the estimates vary much more than the estimated standard deviation, but serve a similar purpose as measures of out-of-sample performance of the nuisance estimates. None of these tests addresses identification issues.

[bindugupta]

Thank you so much @SvenKlaassen. This brings a lot of clarity.