The debate on the accuracy-interpretability tradeoff

In this talk we look into the debate on the alleged trade-off between accuracy and interpretability. We will discuss the literature on the implications of enforcing interpretability as an optimization constraint, navigating the contention of opting for interpretable models over black box models.

We will explore simple and powerful models which you can use to understand your data better, while we ask ourselves whether it is worth trying to use these or not, and what new problems interpretability brings to the table.

References

[Alu21D]

Does explainable artificial intelligence improve human decision-making?, Yasmeen Alufaisan, Laura R. Marusich, Jonathan Z. Bakdash, Yan Zhou, Murat Kantarcioglu.

2021

Publication

[Ang18L]

Learning Certifiably Optimal Rule Lists for Categorical Data, Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, Cynthia Rudin.

2018

We present the design and implementation of a custom discrete optimization technique for building rule lists over a categorical feature space. Our algorithm produces rule lists with optimal training performance, according to the regularized empirical risk, with a certificate of optimality. By leveraging algorithmic bounds, efficient data structures, and computational reuse, we achieve several …

Publication

[Bel22I]

It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy, Andrew Bell, Ian Solano-Kamaiko, Oded Nov, Julia Stoyanovich.

Jun 2022

To achieve high accuracy in machine learning (ML) systems, practitioners often use complex “black-box” models that are not easily understood by humans. The opacity of such models has resulted in public concerns about their use in high-stakes contexts and given rise to two conflicting arguments about the nature — and even the existence — of the accuracy-explainability trade-off. One side postulates …

[Bro23T]

Toward a taxonomy of trust for probabilistic machine learning, Tamara Broderick, Andrew Gelman, Rachael Meager, Anna L. Smith, Tian Zheng.

Feb 2023

Probabilistic machine learning increasingly informs critical decisions in medicine, economics, politics, and beyond. To aid the development of trust in these decisions, we develop a taxonomy delineating where trust in an analysis can break down: (i) in the translation of real-world goals to goals on a particular set of training data, (ii) in the translation of abstract goals on the training data …

Publication

[Che18I]

An Interpretable Model with Globally Consistent Explanations for Credit Risk, Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang, Tong Wang.

Nov 2018

We propose a possible solution to a public challenge posed by the Fair Isaac Corporation (FICO), which is to provide an explainable model for credit risk assessment. Rather than present a black box model and explain it afterwards, we provide a globally interpretable model that is as accurate as other neural networks. Our "two-layer additive risk model" is decomposable into subscales, where each …

[Che19T]

This Looks Like That: Deep Learning for Interpretable Image Recognition, Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, Jonathan K Su.

2019

When we are faced with challenging image classification tasks, we often explain our reasoning by dissecting the image, and pointing out prototypical aspects of one class or another. The mounting evidence for each of the classes helps us make our final decision. In this work, we introduce a deep network architecture -- prototypical part network (ProtoPNet), that reasons in a similar way: the …

[Che23U]

Understanding and Exploring the Whole Set of Good Sparse Generalized Additive Models, Zhi Chen, Chudi Zhong, Margo Seltzer, Cynthia Rudin.

Mar 2023

In real applications, interaction between machine learning model and domain experts is critical; however, the classical machine learning paradigm that usually produces only a single model does not facilitate such interaction. Approximating and exploring the Rashomon set, i.e., the set of all near-optimal models, addresses this practical challenge by providing the user with a searchable space …

Publication

[Dzi20E]

Enforcing Interpretability and its Statistical Impacts: Trade-offs between Accuracy and Interpretability, G. Dziugaite, S. Ben-David, Daniel M. Roy.

Oct 2020

To date, there has been no formal study of the statistical cost of interpretability in machine learning. As such, the discourse around potential trade-offs is often informal and misconceptions abound. In this work, we aim to initiate a formal study of these trade-offs. A seemingly insurmountable roadblock is the lack of any agreed upon definition of interpretability. Instead, we propose a shift in …

Publication

[Esp22C]

Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off, Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lió, Mateja Jamnik.

Dec 2022

Publication

[Gel13B]

Bayesian Data Analysis, Third Edition, Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, Donald B. Rubin.

Nov 2013

Now in its third edition, this classic book is widely considered the leading text on Bayesian methods, lauded for its accessible, practical approach to analyzing data and solving research problems. Bayesian Data Analysis, Third Edition continues to take an applied approach to analysis using up-to-date Bayesian methods. The authors—all leaders in the statistics community—introduce basic concepts …

Publication

[Hof21T]

This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networks, Adrian Hoffmann, Claudio Fanconi, Rahul Rade, Jonas Kohler.

Jun 2021

Deep neural networks that yield human interpretable decisions by architectural design have lately become an increasingly popular alternative to post hoc interpretation of traditional black-box models. Among these networks, the arguably most widespread approach is so-called prototype learning, where similarities to learned latent prototypes serve as the basis of classifying an unseen data point. In …

[Hu19O]

Optimal Sparse Decision Trees, Xiyang Hu, Cynthia Rudin, Margo Seltzer.

2019

Decision tree algorithms have been among the most popular algorithms for interpretable (transparent) machine learning since the early 1980's. The problem that has plagued decision tree algorithms since their inception is their lack of optimality, or lack of guarantees of closeness to optimality: decision tree algorithms are often greedy or myopic, and sometimes produce unquestionably suboptimal …

[Jac21H]

How machine-learning recommendations influence clinician treatment selections: The example of antidepressant selection, Maia Jacobs, Melanie F. Pradier, Thomas H. McCoy, Roy H. Perlis, Finale Doshi-Velez, Krzysztof Z. Gajos.

Feb 2021

Decision support systems embodying machine learning models offer the promise of an improved standard of care for major depressive disorder, but little is known about how clinicians’ treatment decisions will be influenced by machine learning recommendations and explanations. We used a within-subject factorial experiment to present 220 clinicians with patient vignettes, each with or without a …

[Lin20G]

Generalized and Scalable Optimal Sparse Decision Trees, Jimmy Lin, Chudi Zhong, Diane Hu, Cynthia Rudin, Margo Seltzer.

Nov 2020

Decision tree optimization is notoriously difficult from a computational perspective but essential for the field of interpretable machine learning. Despite efforts over the past 40 years, only recently have optimization breakthroughs been made that have allowed practical algorithms to find optimal decision trees. These new techniques have the potential to trigger a paradigm shift, where, it is …

Publication

[Mct22F]

Fast Sparse Decision Tree Optimization via Reference Ensembles, Hayden McTavish, Chudi Zhong, Reto Achermann, Ilias Karimalis, Jacques Chen, Cynthia Rudin, Margo Seltzer.

Jun 2022

Sparse decision tree optimization has been one of the most fundamental problems in AI since its inception and is a challenge at the core of interpretable machine learning. Sparse decision tree optimization is computationally hard, and despite steady effort since the 1960's, breakthroughs have been made on the problem only within the past few years, primarily on the problem of finding optimal …

Publication

[Pas22O]

Overreliance on AI: Literature review, Samir Passi, Mihaela Vorvoreanu.

Dec 2022

Overreliance on AI: Literature review Overreliance on AI occurs when users start accepting incorrect AI outputs. This can lead to issues and errors that can ultimately make people lose trust in AI systems. This report explains what overreliance on AI is, how it happens, and how we can mitigate it. An important goal of AI system design is to empower users to develop appropriate reliance on AI. This …

Publication

[Pou21M]

Manipulating and Measuring Model Interpretability, Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, Hanna Wallach.

May 2021

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely …

Publication

[Rib20A]

Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh.

Jul 2020

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors. Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP …

Publication

[Rud19S]

The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to Be More Effective at Data Analysis, Cynthia Rudin, David Carlson.

Oct 2019

Despite the widespread usage of machine learning throughout organizations, there are some key principles that are commonly missed. In particular: 1) There are at least four main families for supervised learning: logical modeling methods, linear combination methods, case-based reasoning methods, and iterative summarization methods. 2) For many application domains, almost all machine learning …

Publication

[Rud19S]

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Cynthia Rudin.

May 2019

Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first …

[Sem22E]

On the Existence of Simpler Machine Learning Models, Lesia Semenova, Cynthia Rudin, Ronald Parr.

Jun 2022

It is almost always easier to find an accurate-but-complex model than an accurate-yet-simple model. Finding optimal, sparse, accurate models of various forms (linear models with integer coefficients, decision sets, rule lists, decision trees) is generally NP-hard. We often do not know whether the search for a simpler model will be worthwhile, and thus we do not go to the trouble of searching for …

Publication

[Ust19L]

Learning Optimized Risk Scores, Berk Ustun, Cynthia Rudin.

2019

Risk scores are simple classification models that let users make quick risk predictions by adding and subtracting a few small numbers. These models are widely used in medicine and criminal justice, but are difficult to learn from data because they need to be calibrated, sparse, use small integer coefficients, and obey application-specific constraints. In this paper, we introduce a machine learning …

Publication

[Ust13S]

Supersparse linear integer models for predictive scoring systems, Berk Ustun, Stefano Tracà, Cynthia Rudin.

Jan 2013

Scoring systems are classification models that make predictions using a sparse linear combination of variables with integer coefficients. Such systems are frequently used in medicine because they are interpretable; that is, they only require users to add, subtract and multiply a few meaningful numbers in order to make a prediction. See, for instance, these commonly used scoring systems: (Gage et …

[Xin22E]

Exploring the Whole Rashomon Set of Sparse Decision Trees, Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, Cynthia Rudin.

Oct 2022

In any given machine learning problem, there may be many models that could explain the data almost equally well. However, most learning algorithms return only one of these models, leaving practitioners with no practical way to explore alternative models that might have desirable properties beyond what could be expressed within a loss function. The Rashomon set is the set of these all …

[Yu20O]

Optimal Decision Lists using SAT, Jinqiang Yu, Alexey Ignatiev, Pierre Le Bodic, Peter J. Stuckey.

Oct 2020

Decision lists are one of the most easily explainable machine learning models. Given the renewed emphasis on explainable machine learning decisions, this machine learning model is increasingly attractive, combining small size and clear explainability. In this paper, we show for the first time how to construct optimal "perfect" decision lists which are perfectly accurate on the training data, and …

Publication

[Zha23O]

Optimal Sparse Regression Trees, Rui Zhang, Rui Xin, Margo Seltzer, Cynthia Rudin.

Jun 2023

Regression trees are one of the oldest forms of AI models, and their predictions can be made without a calculator, which makes them broadly useful, particularly for high-stakes applications. Within the large literature on regression trees, there has been little effort towards full provable optimization, mainly due to the computational hardness of the problem. This work proposes a dynamic …

References

In this series →