Uplift modelling

,
1

There are several YouTube videos like these and I am struggling to understand whether this is a valid approach. It bothers me that I am hearing nothing about the principles of design of experiments I learned long ago. Replication, blocking, degrees of freedom. Machine learning models are being used to replace the usual linear models but there don’t seem to be any constraints on which predictors can go in and which should be kept out.

What sorts of designs should be used here if one doesn’t want to use machine learning? Response Surface Methodology perhaps?

Blaise F Egan
British Telecommunications PLC

1 Like
2

What do you mean by valid?

Uplift modelling requires causal-inference framework, but I don’t see anything wrong with traditional linear models under this framework.

1 Like
3

The variables that go into statistical models are carefully considered based on where they are in a DAG causal graph. Variables that are ‘downstream’ of other pedictors are left out, but all confounders are kept in. Backdoors paths caused by colliders are broken. ML approaches seem to just throw in all the predictors irrespective any such considerations.

1 Like
4

I can’t speak for all analyses or YouTube videos, and my experience might not match yours. But I can say, that

  1. I agree that most of these uplift+ML talks tend to focus more on the analytic phase rather than the design phase. Therefore, focusing more on the ML/statistical models employed rather than the study design.
  2. However, my personal experience is that most of these uplift for marketing/business settings are in digital products and therefore often consider experimental data. Which, in my opinion, creates less of an issue with confounding and variable selection. Covariate adjustment in experiments are often done to explain more of the outcome variability so that the treatment estimates are more precise, not to debias factors of self-selection into treatment groups. Therefore, as long as they are careful not to include post-treatment variables, applying a kitchen-sink with a flexible statistical estimator is not that outrages.

More generally, I agree the “A/B testing” school could have synergize better with the RCT school.
But I feel these YouTube videos are aimed for a more computer-science audience (“data science”) than a statistics one, which might explain the strong focus on analytical approaches, rather than design ones.