Ivona Cickovic and Andrea Serafino

Machine learning models are increasingly used in organisational decision-making, yet their inner workings often remain opaque. When these systems influence real world outcomes, knowing what they predict is not enough – we also need to understand why. Explainability methods aim to illuminate this ‘black box,’ and feature attribution tools that link predictions to individual inputs are especially popular. They feel intuitive but rely on strict data assumptions that rarely hold, making their outputs unreliable. The 2019 Apple Card case illustrates why this matters: despite gender not being an explicit input, women appeared to receive lower credit limits than men with similar profiles – an outcome attribution methods struggle to explain. This post examines a key assumption underpinning these tools and how it distorts explanations.

Continue reading “Explainability in machine learning: do popular methods deliver on their promises?” →