Explainable AI (XAI)
Also known as: XAI, model explainability, AI transparency, interpretable AI
Explainable AI (XAI) refers to a set of methods, techniques, and design principles that make the outputs and decision-making processes of AI systems understandable to the humans who use, oversee, or are affected by them.
Traditional machine learning models (particularly deep learning) are often 'black boxes': they produce accurate predictions but cannot easily explain why they reached a particular conclusion. XAI techniques produce explanations that accompany AI outputs, such as 'this loan application was declined because the debt-to-income ratio was the most important factor' or 'this image was classified as a cat primarily because of features in this region of the image'.
Explainable AI (XAI), explained at three levels
At home
Explainable AI is about making sure people can understand why an AI made a particular decision. If an AI model rejects your loan application, you have a right to know why, and 'because the AI said so' is not an acceptable answer. Explainable AI adds the ability for the system to tell you: 'Your application was declined mainly because of a missed payment in the last 12 months and a high debt-to-income ratio.' The explanation gives you something concrete to understand and potentially challenge.
At work
Explainable AI is important in any context where AI decisions affect people and need to be auditable. Regulated industries (healthcare, finance, insurance, and criminal justice) are increasingly requiring AI systems to be able to explain their outputs. Beyond compliance, explanations help system operators detect when a model is making decisions for the wrong reasons (e.g., using a proxy variable correlated with a protected characteristic), and help end users understand and trust AI recommendations.
At home
XAI techniques work at different levels of explanation. Global explanations describe how the model behaves overall, which features are most important to the model across all predictions. Local explanations describe why the model made a specific prediction for a specific input, which features most influenced this particular loan decision or this particular diagnosis. SHAP (Shapley Additive exPlanations) is one of the most widely used methods, providing consistent local and global feature importance values derived from cooperative game theory.
At work
There is an inherent tension between model accuracy and explainability. Simpler, more interpretable models (linear regression, decision trees) are naturally explainable but may be less accurate. Complex models (gradient boosting, deep neural networks) are often more accurate but harder to explain. XAI techniques like LIME and SHAP address this tension by providing post-hoc explanations for complex models (approximating the model's behaviour locally with a simpler, interpretable model) rather than replacing the complex model with a simpler one.
At home
SHAP borrows Shapley values from game theory: it treats each input feature as a player and measures how much the prediction changes, on average, when that feature joins every possible group of the others. Exact calculation is too slow for big models, so tools approximate it by sampling, while LIME fits a simple model to slightly altered copies of one input. Both describe behaviour near one case, not the model's true internals, and correlated features can split credit oddly. In my view, treat these explanations as evidence, not proof.
At work
From a practitioner standpoint, the biggest XAI implementation risk is explanation washing: providing technically compliant explanations that are so vague or complex that they do not actually help people understand the decision. 'The model considered 47 features weighted by their Shapley values' is technically an explanation but is meaningless to most users. Good explanations are designed for the specific audience: a loan officer needs different detail than a consumer receiving a decline letter than a regulator reviewing the model. Explanation design should be treated as a user experience problem, not just a technical one.
Real-world examples
At home
Marcus applied for a mortgage and was declined. The lender's AI model was required to provide an explanation. He received a letter stating that the three most important factors were: his employment length (under 2 years at current employer), his loan-to-value ratio (above 90%), and a missed utility payment from 14 months ago. The explanation allowed him to understand what to address before reapplying.
At work
A health insurer deployed an AI model to support underwriting decisions. Their compliance team required every model output to include a SHAP-based explanation identifying the top three contributing factors, displayed in plain language alongside the decision in the underwriter's workflow. A model audit found the top factor in 12% of high-risk classifications was postcode, which correlated with demographic characteristics. The model was retrained to remove postcode as an input.
How it compares
| Feature | Explainable AI (XAI) | Interpretable AI | Black-Box AI |
|---|---|---|---|
| What it means | Complex models with post-hoc explanation methods added | Inherently simple models whose logic can be read directly | Complex models with no explanation capability |
| Examples | Neural network + SHAP, gradient boosting + LIME | Decision tree, linear regression, rule-based system | Large deep learning model with no explanation layer |
| Accuracy | Typically high (complex base model) | May be lower for complex tasks | Typically highest |
| Explainability quality | Approximate: explanation is an interpretation of the model | Exact: the model's own logic is the explanation | None |
| Regulatory use | Satisfies many XAI requirements in practice | Preferred where perfect explainability is required | Increasingly restricted in regulated decision contexts |
Interpretable models are transparent by design but may sacrifice accuracy. Explainable AI adds explanation capability to complex models but explanations are approximations. Black-box AI maximises accuracy but faces growing regulatory and ethical barriers in consequential decision contexts.
For analysts
In BA artefacts, XAI requirements appear wherever AI makes or supports decisions that affect people. Specify the explanation audience (consumer-facing vs. regulator-facing vs. model-operator-facing) and the required explanation format and granularity for each. GDPR NFR: 'For any automated decision with significant effect on a data subject, the system must generate a human-readable explanation identifying the top three factors influencing the decision, expressed in plain language without technical jargon, and must provide a mechanism for the data subject to contest the decision.' Note: explanation and contestation are separate requirements: specify both.
History & origin
The desire to explain AI decisions predates modern machine learning: rule-based expert systems (1970s-1990s) were inherently explainable because every decision could be traced to an explicit rule. The problem became acute as deep neural networks replaced interpretable models in the 2010s. DARPA launched a formal XAI research program in 2016, funding projects to develop explanation capabilities for deep learning systems. LIME (Local Interpretable Model-Agnostic Explanations, 2016) and SHAP (SHapley Additive exPlanations, 2017) emerged as widely adopted explanation techniques. GDPR (2018) added regulatory urgency: people subject to significant automated decisions are entitled to meaningful information about the logic involved, often described as a 'right to explanation'.