Model-Driven Root Cause Analysis for Trustworthy AI: A Data-and-Model-Centric Explanation Framework

Building trust in AI systems requires not only accurate models but also mechanisms to diagnose why machine learning (ML) pipelines succeed or fail. In this work, we propose a model-driven Root Cause Analysis (RCA) framework that attributes pipeline performance to interpretable factors spanning both data properties and model configurations. Unlike post-hoc explainers that approximate black-box behavior, our approach learns a faithful, inherently interpretable meta-model using Explainable Boosting Machines (EBM) to capture the mapping from data complexity and hyperparameters to predictive accuracy. To evaluate this framework, we curated a large-scale meta-dataset comprising 81,000 Decision Tree pipeline runs generated from 270 OpenML datasets combined with 300 hyperparameter configurations. The RCA meta-model achieved high predictive fidelity (R2 = 0.90, MAE = 0.030), far outperforming a meanregressor baseline (R2 ≈ 0). This fidelity ensures that feature attributions reflect genuine performance determinants rather than artifacts of an ill-fitting surrogate. Beyond predictive accuracy, we assessed the attribution validity of our RCA framework through observational analysis of ten representative pipelines—five high-performing and five low-performing—drawn from the test set. Results show that the attributions are concise, with typically fewer than three dominant contributors per case, making them easy to interpret. In success cases, low class overlap and balanced distributions dominate attributions, while failure cases are driven by severe imbalance, harmful interactions, and, in some cases, context-dependent effects such as redundant dense features. Hyperparameter effects emerge as secondary but aggravating under challenging conditions. These findings demonstrate that our RCA framework provides theoretically grounded yet empirically adaptive explanations, enabling robust root cause analysis for trustworthy AI.