跳到正文
DEV Community· Solomon Owusu-Ansah·· 2 天前AI 評分40

AI 分類器自信答錯,卻無法判斷輸入是否陌生

Your AI Is Confidently Wrong — and It Can't Even Tell

AI 導讀

一般分類器遇到訓練資料以外的輸入,仍會將它判為某個已知類別;softmax 信心分數只反映輸入與已知類別的相似程度。OOD detection 會為輸入評分,低於門檻便標記,而非直接採信預測。近分佈外輸入仍難識別,常用 AUROC 和 FPR95 評估分數表現。

正文

Your AI Is Confidently Wrong — and It Can't Even Tell

Train a model to tell cats from dogs. Show it thousands of photos until it gets nearly every one right. Then show it a car.

It will not hesitate. It will not say "that's not an animal." It will say "dog, 92% confident" — and it will be wrong with a straight face.

That is not a bug in one model. It is how classifiers work.

The forced choice

A normal classifier has no way to say "I don't know." Its final layer, the softmax, turns its raw scores into probabilities that always add up to 1. So whatever you show it — a car, a handwritten digit, a medical scan it was never trained on — it must hand the win to one of the classes it knows.

The confidence number it prints looks like self-awareness. It isn't. It only says which known class the input resembles most, not whether the input belongs to any known class at all. Researchers have known this since at least 2017, when Hendrycks and Gimpel showed that a model's top softmax probability is a weak guard against unfamiliar inputs — but it was the first guard anyone had, and it is still the baseline everything else is compared against.

Diagram: a car goes into a dog classifier — the softmax must split 100% between cat and dog, so it answers "dog, 92% confident"

So what is OOD detection?

OOD stands for out-of-distribution: data that comes from outside whatever the model was trained on. OOD detection is the guard you bolt on next to the classifier. It does two things:

  1. It scores every input for strangeness. A high score means "this looks familiar." A low score means "this looks strange."
  2. It draws a line — a threshold. Score below the line, and the system flags the input instead of trusting the prediction.

That is the whole idea. Everything else in the field is a debate about what the score should be, and where the line should go.

Infographic: in-distribution (a dog), near-OOD (a wolf — looks like a dog but isn't), and far-OOD (a handwritten digit)

Near vs far: the case that actually matters

Not all strange inputs are equally strange.

  • Far-OOD is obvious: a handwritten digit shown to a dog classifier. The features look nothing like the training data, so almost any score catches it.
  • Near-OOD is the hard one: a wolf shown to a dog classifier. It shares fur, ears, a snout — most of the features overlap. The model is at its most confident exactly where it should be most careful.

Far-OOD is a solved-ish problem. Near-OOD is where the field still struggles, and it is the case that shows up in real deployments — a new disease variant, a road sign in a different country, a product photo taken in different light.

The scores, in plain English

If you read OOD papers, four families of scores come up again and again:

  • MSP — maximum softmax probability. Just take the model's top confidence. Simple, free, and the baseline from Hendrycks and Gimpel (2017). Weak, but everything is measured against it.
  • Energy. Built from the model's raw logits instead of the normalised probabilities (Liu et al., 2020). It separates familiar from strange more cleanly than MSP, and costs nothing extra to compute.
  • Feature distance. Ignore the final answer and look at where the input lands inside the network. How far is it from the training examples — measured with a Mahalanobis distance, or by its nearest neighbours (kNN)? If it sits far from everything the model has seen, flag it.
  • ViM — virtual-logit matching. Part feature distance, part residual: it measures the part of the input's features that the classifier's own directions can't explain (Wang et al., 2022). The unexplained leftover is the signal.

Figure 1 from Liu et al., "Energy-Based Out-of-Distribution Detection" (NeurIPS 2020): the network's output goes through an energy function, and the energy score separates familiar from strange inputs around a threshold

Diagram: ViM splits a feature into the part training explains (principal space) and the leftover residual — the residual, scaled, becomes a virtual "OOD" logit

None of these retrains the model. They all read a frozen model and ask, in different ways, does this input belong here?

How do you know if a score is any good?

Two numbers do most of the work:

  • AUROC — how well the score separates familiar inputs from strange ones. 50% means it is guessing; 100% means perfect separation.
  • FPR95 — when you catch 95% of the strange inputs, what fraction of normal inputs do you wrongly flag? Lower is better. This is the number that hurts in practice, because every false flag is a real input you refused to answer.

Infographic: two score distributions — familiar inputs score high, strange inputs score low — with a threshold line between them

Why it matters

A wrong answer is not always a harmless answer. A medical model that confidently misreads a scan from a new hospital, a self-driving system that confidently misreads an unfamiliar road sign, a moderation filter that confidently passes something it has never seen — in all of these, the cost of a confident mistake is far higher than the cost of saying "not sure, ask a human."

OOD detection is not about making models smarter. It is about making them honest.

What it doesn't solve — yet

Being honest about the limits is part of understanding the idea:

  • Near-OOD is still hard. The best scores separate far-OOD almost perfectly and near-OOD only partially.
  • The threshold needs real data to set. Pick it on the wrong data and you either flag everything or nothing.
  • No single score wins everywhere. A score that is best on one dataset and model can be mediocre on another. That is why the field benchmarks so many of them side by side.

I'm Solomon, a master's student at North Dakota State University. I write about machine learning in plain English — follow along if that sounds useful.

Discussion prompt for readers: Have you ever caught a model being confidently wrong on something outside its training data? What was it, and what did you do about it?

來源:DEV Community · dev.to