What is explainable AI?

The black box AI problem

Black box AI is a system whose output can be seen but whose reasons cannot. A neural network learns by adjusting a huge number of internal weights, and no single weight maps to a reason a person would recognise.

That is acceptable when the stakes are low and the output is easy to check. It becomes a problem when a system decides who gets a loan, which patient is seen first or which accounts a sales team ignores, because nobody can say whether the decision rested on a sound signal or a quirk in the data.

Interpretability vs explainability

Interpretability is about seeing how a model works inside; explainability is about giving a useful account of a particular decision. The terms overlap and are often used interchangeably, but the split is practical.

An interpretable model, such as a short decision tree or a scoring formula, can be read directly. An explainable system may still be a black box inside but produces an account a person can act on: the factors that weighed most, a similar past case, or the sources it relied on.

How explanations are produced

Explanations come either from building a system that is understandable by design or from adding a method that describes a complex model's behaviour after the fact. Research programmes on explainable AI, such as the one run by the US defence research agency DARPA, aimed for systems that can justify their decisions, describe their own strengths and weaknesses, and show how they are likely to behave, while noting a trade-off between explainability and raw performance.

Symbolic parts are easier to explain than learned ones. A rule that fired, a citation that was used or a list of steps can be shown to a person directly, which is one reason research on neurosymbolic AI pairs neural networks with explicit rules and knowledge.

• Simpler models that can be read directly.
• Feature attributions that show which inputs mattered most.
• Example-based explanations that point to similar past cases.
• Rules, citations and step lists that show the path to an answer.

Explainable AI examples in sales

A lead score is a simple illustration. A bare number tells a rep that a lead is hot, with no way to judge whether to agree. An explained score lists its reasons: the company fits the target industry, someone visited the pricing page twice, and no budget owner has been identified yet.

The explained version is more useful even when the number is the same. The rep can see that the missing budget owner is the next job, and a sales manager can spot when a reason is wrong, such as a visit from the company's own staff.

Why AI transparency matters for buyers of AI tools

AI transparency matters because a team cannot govern what it cannot see. Explanations let people trust an output for the right reasons, find the cause when something goes wrong, and show regulators or customers how a decision was made; the NIST AI Risk Management Framework names explainability among the qualities of trustworthy AI.

There is a limit to watch. An explanation produced by the system itself can be plausible without being faithful to what really drove the output. Treat explanations as evidence to check, not as proof.