PhD student Heqiao Ruan's dissertation defense titled “Understanding Out-of-Distribution: Hypothesis Testing and Causal Inference”.  

In modern machine learning systems, out-of-distribution (OOD) data, or distribution shift, is a primary source of performance degradation, as the underlying data-generating process is rarely stationary. Factors such as product updates, evolving user behavior, seasonality, and changes in logging or instrumentation can alter both the distribution of observed variables and the data collection mechanism. Consequently, models trained on the historical batch D[Exist] are deployed under a different distribution in D[New] leading to over-optimistic risk estimates, degraded calibration, unstable decision thresholds, and reduced generalization performance. This highlights the need for reliable, efficient, and robust methods for detecting distribution shift.
  In this thesis, we develop two classes of statistical methodologies for detecting distribution shift. The first framework is based on comparing validation and test error, while the second leverages causal inference to characterize extrapolation through variable importance. We propose three hypothesis testing procedures with rigorous theoretical guarantees. The first, PermOOB, compares out-of-bag (OOB) error with predictive error in random forests via a permutation test. The second extends this approach to Gradient Boosting Decision Trees (GBDT), supported by a martingale central limit theorem. The third permutes batch assignments and evaluates distribution shift through variable importance induced by a causal machine learning model, yielding an efficient, interpretable, and attribution-oriented test. Extensions to a broad class of variable importance measures are developed. A direct and granular characterization of distribution shift from a causal inference perspective is also discussed.
  Applications of the proposed methods to Sentiment Analysis demonstrate their ability to capture diverse forms of distributional shift in text data. Beyond static settings, we further extend the methodology to streaming environments through the OnlinePermOOB procedure, which continuously monitors predictive performance over time. This extension naturally connects to online anomaly detection, enabling real-time detection of distribution shift in evolving data streams.
  Practical Significance: The proposed methodologies are efficient, robust, and broadly applicable. They not only detect changes in the data-generating process but also provide flexible tools for diagnosing and monitoring model reliability in deployed systems. In particular, the streaming extension enables proactive, statistically principled early-warning signals for predictive degradation, offering actionable guidance for timely model updating.

Advisor: Dr. Lucas Mentch