What I learned from a model that confidently failed.

The validation curve looked excellent. The confusion matrix looked almost suspiciously tidy. Then the model met data from a different acquisition setup and failed with complete confidence.

The useful part was not that performance dropped. That was predictable. The useful part was seeing how little the confidence scores noticed the change.

The metric I was not measuring

I had measured accuracy, sensitivity, specificity, and calibration on familiar data. I had not measured whether the model knew when the world around it had changed.

Lab note: “unknown” is a legitimate output. A system that cannot say it is outside its competence is not ready to carry real responsibility.

The failed run changed the next experiment. We stopped asking only how often the model was correct and started asking when its confidence deserved to be believed.