[ Learn ] # Calibrated probabilities: what the number from Jev means What it means for a decision model such as Jev to be calibrated: a 0.8 should be right about 80% of the time across many answers, not every time. Checked 1 November 2026. Published 1 November 2026By Jev Hub [Back to Learn](https://jevhub.io/learn) [ TLDR ] - A calibrated model's probabilities match how often it is right across many answers. - According to TypeSafe, outcomes given a probability of 0.2 should occur about 20% of the time, 0.8 about 80% of the time and 1.0 every time. - That describes groups of predictions, not a guarantee for any single answer. - TypeSafe trains Jev for this on purpose, which is why its answers carry probabilities and a confidence value your code can threshold on. - The practical point is that a probability is useful for deciding when to act and when to hand a case to a person, but only if you check it on your own data first. ## What a calibrated probability means | | What it means | What it does not mean | | --- | --- | --- | | A probability of 0.8 | Across many answers given 0.8, about 80% should be right | That this particular answer is right | | A probability of 1.0 | Across many answers given 1.0, all should be right | That the model cannot be wrong on your data without testing | | A flat spread of probabilities | The model sees no clear winner, so confidence is low | That the answer is wrong | | Confidence | A 0 to 1 summary of how peaked the probabilities are | A separate guarantee of accuracy | The meaning of calibration is as TypeSafe's AI primer and Confidence page describe it, read on 1 November 2026. TypeSafe says calibration is measured across groups of predictions and does not guarantee that an individual answer is correct. ## What does calibrated mean? A model is calibrated when the probabilities it gives line up with reality over many answers. TypeSafe's AI primer puts it this way: across many predictions from a well-calibrated model, outcomes assigned a probability of 0.2 should occur about 20% of the time, those at 0.8 about 80% of the time, and those at 1.0 all of the time. These rates describe groups of predictions, not a promise about any single answer. A model can be well calibrated and still be wrong on one case, just as a forecast of an 80% chance of rain is not wrong when it stays dry. How often each probability should be right. From TypeSafe's AI primer, read 1 November 2026. These rates describe groups of answers, not any single one. - [What is a System One model?](https://jevhub.io/what-is-a-system-one-model) ## Why does it matter for software? TypeSafe's argument is that calibration is what makes uncertainty usable by software. If a number really reflects how often the model is right, your code can act on high values, check the middle ones and send low ones to a person, instead of treating every answer as equally reliable. It is also why Jev is trained to return decisions and probabilities, not text. TypeSafe describes the contrast with chat models, which are trained to produce answers people prefer, and says that can reward confident-sounding answers that are not reliable enough for unattended automation. That is the company's own view. - [Confidence routing](https://jevhub.io/confidence-routing) ## How does confidence relate to it? Choice and Score answers include a probability for every option or level, and a confidence value computed from how peaked those probabilities are. All the probability on one outcome gives 1, and an even spread gives 0. For a Choice with n options, TypeSafe's formula is the top probability minus 1/n, divided by 1 minus 1/n. A Noul answer is a single probability and has no separate confidence value. If you want one, TypeSafe suggests the distance from 0.5, which is the absolute value of 2p minus 1. TypeSafe also says its confidence is one reasonable summary, not the only one, and because the full probabilities come back with every answer you can compute whichever measure suits you. - [Jev question types](https://jevhub.io/jev-question-types) ## How should you check it on your own data? Calibration is a property measured across many answers, so check it the same way. Run a labelled sample of your own cases through Jev, group the answers by confidence, and see how often each group was right. TypeSafe's own test on 60 company reports found that answers at 0.9 confidence or above were right 90% of the time and the rest 40%, which is the kind of split to look for. Then set your thresholds from what you see, with higher bars for riskier actions, and leave a path to a person for the in-between answers. TypeSafe's documentation says the right threshold depends on your domain and recommends starting conservative. - [Jev benchmarks](https://jevhub.io/jev-benchmarks) ## Our read Calibration is the quiet reason this category is useful: a number you can trust on average lets you automate the easy cases and keep people for the hard ones. The word is easy to over-read, though. It says nothing about any one answer, and it only holds where your data looks like what the model was trained to judge. So treat the probability as a way to rank cases by how safe they are to automate, and prove the ranking on your own sample before you rely on it. - [What Jev can't do](https://jevhub.io/jev-limitations) ## Common questions That its probabilities match how often it is right across many answers: about 20% of the outcomes it rates 0.2 happen, about 80% of those it rates 0.8, and so on, according to TypeSafe. Not for the single answer. TypeSafe says calibration is measured across groups of predictions and does not guarantee that an individual answer is correct. Because TypeSafe trains it to return decisions and calibrated probabilities instead of text, so software can act on high values and send uncertain ones to a person. Probabilities give the share for each option or level. Confidence collapses how peaked they are into one 0 to 1 number. It is 1 when all probability is on one outcome and 0 when it is spread evenly. No. Its single probability describes both outcomes, and a value near 0.5 is the model saying it is unsure. Run a labelled sample through the model, group the answers by confidence and compare each group's accuracy with what you expected, then set thresholds from that. ## Sources Written by Jev Hub from the sources below and checked against them on 1 November 2026. Where a figure is a company's own claim, the article says so. - [TypeSafe docs: AI primer (read 1 November 2026)](https://docs.typesafe.ai/introduction/machine-learning-primer) - [TypeSafe docs: Confidence (read 1 November 2026)](https://docs.typesafe.ai/confidence) - [TypeSafe docs: System One (read 1 November 2026)](https://docs.typesafe.ai/concepts/system-one) - [TypeSafe docs: Choice (read 1 November 2026)](https://docs.typesafe.ai/primitives/choice) - [TypeSafe docs: Noul (read 1 November 2026)](https://docs.typesafe.ai/primitives/noul) ## Related content - [What is Jev?](https://jevhub.io/what-is-jev) - [How to use Jev](https://jevhub.io/how-to-use-jev) - [Jev pricing](https://jevhub.io/economics)