Calibrated probabilities: what the number from Jev means

A model is calibrated when the probabilities it gives line up with reality over many answers. TypeSafe's AI primer puts it this way: across many predictions from a well-calibrated model, outcomes assigned a probability of 0.2 should occur about 20% of the time, those at 0.8 about 80% of the time, and those at 1.0 all of the time.

These rates describe groups of predictions, not a promise about any single answer. A model can be well calibrated and still be wrong on one case, just as a forecast of an 80% chance of rain is not wrong when it stays dry.

• What is a System One model?

TypeSafe's argument is that calibration is what makes uncertainty usable by software. If a number really reflects how often the model is right, your code can act on high values, check the middle ones and send low ones to a person, instead of treating every answer as equally reliable.

It is also why Jev is trained to return decisions and probabilities, not text. TypeSafe describes the contrast with chat models, which are trained to produce answers people prefer, and says that can reward confident-sounding answers that are not reliable enough for unattended automation. That is the company's own view.

• Confidence routing

Choice and Score answers include a probability for every option or level, and a confidence value computed from how peaked those probabilities are. All the probability on one outcome gives 1, and an even spread gives 0. For a Choice with n options, TypeSafe's formula is the top probability minus 1/n, divided by 1 minus 1/n.

A Noul answer is a single probability and has no separate confidence value. If you want one, TypeSafe suggests the distance from 0.5, which is the absolute value of 2p minus 1. TypeSafe also says its confidence is one reasonable summary, not the only one, and because the full probabilities come back with every answer you can compute whichever measure suits you.

• Jev question types

Calibration is a property measured across many answers, so check it the same way. Run a labelled sample of your own cases through Jev, group the answers by confidence, and see how often each group was right. TypeSafe's own test on 60 company reports found that answers at 0.9 confidence or above were right 90% of the time and the rest 40%, which is the kind of split to look for.

Then set your thresholds from what you see, with higher bars for riskier actions, and leave a path to a person for the in-between answers. TypeSafe's documentation says the right threshold depends on your domain and recommends starting conservative.

• Jev benchmarks

Calibration is the quiet reason this category is useful: a number you can trust on average lets you automate the easy cases and keep people for the hard ones. The word is easy to over-read, though. It says nothing about any one answer, and it only holds where your data looks like what the model was trained to judge.

So treat the probability as a way to rank cases by how safe they are to automate, and prove the ranking on your own sample before you rely on it.

• What Jev can't do