Interpretable or explainable? Not the same thing
Cette note n'existe qu'en anglais.
"Interpretable" and "explainable" are often used as synonyms, but they describe two different ways of understanding a model. In finance, where decisions must be justified to clients, auditors and regulators, the difference matters.
Two ways to understand a model
An interpretable model is one you can understand by reading it. A credit scorecard is the classic example: each factor adds or removes a fixed number of points, and the score is the sum. Linear and logistic regression, small decision trees, rule lists and additive models belong to this family. You can see the whole logic before a single decision is made.
An explainable model is usually a black box, such as a large gradient-boosted model or a neural network, that is explained after the fact. Tools such as SHAP and LIME estimate how much each input pushed one prediction up or down. These explanations are useful, but they are approximations of the model, not the model itself.
The terms are not used consistently in the research literature, and some authors treat them as synonyms. The split above follows the most common usage, for example in Christoph Molnar's book on the topic.
Example 1: credit decisions
A lender that rejects an application must tell the applicant the main reasons. With a scorecard, the reasons come straight from the model: the factors that cost the most points. With a black box, the lender needs a post-hoc explanation and has to trust that it reflects what the model actually did.
Regulators have noticed. In 2022 the US Consumer Financial Protection Bureau stated that creditors using complex algorithms must still give accurate and specific reasons for a rejection, and that not understanding one's own model is no defence.
Example 2: trading signals
A rule such as "price above its 50-day average and volume twice its usual level" explains itself every time it fires: you can check each condition against the data. A deep learning model may produce a better signal, but its explanation is an estimate. One attribution method might credit momentum, another volatility, for the same prediction. Research on this "disagreement problem" finds that popular explanation methods often conflict, and that practitioners resolve the conflicts with improvised rules of thumb.
Does interpretability cost accuracy?
Less often than people assume. In FICO's public challenge on explainable credit risk models, a team led by Cynthia Rudin reported a globally interpretable model as accurate as the black-box models it was compared with. Rudin has argued more broadly that for high-stakes decisions, building interpretable models is safer than explaining black boxes after the fact.
A practical rule
- Start with an interpretable model as the baseline.
- Move to a black box only if it is clearly better on data it has never seen, after costs and in every market regime you care about.
- If you do, test the explanations: check that they are stable, and that different methods tell the same story.
References
- Cynthia Rudin, Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, Nature Machine Intelligence, 2019.
- Chaofan Chen et al., An Interpretable Model with Globally Consistent Explanations for Credit Risk, 2018.
- Satyapriya Krishna et al., The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective, 2022.
- Scott Lundberg and Su-In Lee, A Unified Approach to Interpreting Model Predictions (SHAP), NIPS 2017.
- Zachary C. Lipton, The Mythos of Model Interpretability, 2016.
- Consumer Financial Protection Bureau, Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms, May 2022.
- Christoph Molnar, Interpretable Machine Learning.