Jev benchmarks: what do independent tests say?

It depends on the task more than on anything else. Across the public tests, Jev ties the best models on some jobs and finishes last on others, so there is no single accuracy figure that means much.

The pattern is consistent, though. Jev does well on narrow questions with a few clear answers and a short, clean input: checking a claim against a passage, spotting spam, rating relevance, moderating hate speech. It does worse on tasks that need specialist knowledge, many fine-grained labels or several steps of reasoning. TypeSafe itself describes Jev as a decision model, not a frontier reasoner.

• Where Jev struggles

TypeSafe publishes workflow evals on four business tasks. Across them Jev scores 67.8% agreement with reference answers, the same as Sonnet 5 and a little behind Opus 5 (73.1%). The best model on the page scores 74.1%. Jev's cost on the page is $0.0004 per case and its time 0.4 seconds, against $0.1174 and 78.1 seconds for Sonnet 5.

Per workflow, Jev scores 61.7% on security incidents, 71.6% on agent trace review, 61.8% on invoice processing and 76.0% on customer service. The invoice result is the weakest: the best model there reaches 79.1%.

Read these as the maker's own numbers. TypeSafe says the reference answers are an average of two other frontier models (GPT-6 Astra and Claude Fable 5.1), so accuracy here means agreement with those two, not proof of being right. TypeSafe's own people built the workflows, and the page gives no date.

• Jev vs Claude

Independent here means the tester is not TypeSafe. None of the sources below disclose a financial link to TypeSafe, though not all of them say either way. Where Jev did well:

• Checking claims against a document (Vals AI). On 400 claims drawn from company annual filings, Jev scored 0.975 (390 of 400). All twelve systems scored between 0.953 and 0.995, and Jev tied three GPT models. Vals measured Jev's cost at about $0.02 per 1,000 cases.
• Many public datasets (preprint). On 37 datasets Jev beat the open model Qwen3.8-27B on 27 and Gemma-4-E4B on all 37, with 95% to 99% on IMDB, SST-2, HellaSwag and ARC. The authors ran 346,009 requests for under USD 10.
• Hard science and knowledge questions (JevResearch). 82.7% on MMLU-Pro and 76.5% on GPQA Diamond. The authors say Jev is not a frontier system and compare it with twelve cheap models.
• Moderation and prompt injection. 98.2% on the HateCheck set (Esbati; the models he tested scored between 95.1% and 100%), and 87.0% on prompt-injection detection (AY Automate), level with GPT-5.6 Terra at 86.8%.

The same tests also show where it falls behind:

• Legal questions (Vals AI). On a 396-question slice of LegalBench, Jev ranked last of twelve. Its class-balanced score was 0.730 against 0.932 for the best system. It answered "yes" to all 33 questions in one contract task. Vals notes the slice is not representative of the whole benchmark.
• Fine-grained routing. On 77 banking intents Jev scored 78.8% against 84.0% for GPT-5.6 Terra, a gap the AY Automate author found statistically significant. On the 8-class version Jev scored 83.8% against 89.4% for Terra and 90.0% for GPT-5.4 nano. Esbati found 81.0% against 85.6% and 86.1% for two GPT models on intent routing.
• Asking one broad question (Beri.net). On phishing emails, one yes/no question gave 62.6% against 81.3% for Claude Haiku 4.5.
• Out-of-the-box thresholds (Vals AI). With a 1% error budget, Jev reached 1.6% error on held-out data, over the budget, as did five other systems. Vals adds that, with hindsight, Jev could have automated 91% of cases within the budget.

Jev returns a probability with every answer, and TypeSafe describes it as calibrated. The tests suggest that holds on some tasks and not others. Vals found the lowest calibration error of all twelve systems on claim checking (0.011) and the highest on LegalBench (0.108). Beri.net measured 0.107 on synthetic support tickets.

The practical lesson from several authors is the same: use the probability to decide which answers to trust, and set the threshold on a sample of your own labelled data instead of using a fixed 0.5. In the preprint, tuning thresholds on training data raised one task from a micro-F1 of 0.50 to 0.75.

Every tester that measured speed or cost found Jev far ahead, though the size of the gap depends on the setup.

• Speed. Vals measured about 0.10 seconds for a single call and a median 194 times lower than GPT-6 Astra at 32 parallel judgments. AY Automate measured a median 0.33 seconds against 1.17 for GPT-5.6 Terra. JevResearch measured a fixed delay of about 73 ms plus about 6 ms per 1,000 input tokens.
• Cost. AY Automate measured about $0.015 per 1,000 eight-way decisions against $0.609 for Terra. Beri.net measured $0.038 per 1,000 emails against $0.462 for Haiku 4.5 on a single question.
• TypeSafe's headline claims. TypeSafe's launch claims of 193.6 times faster and 444.6 times cheaper could not be reproduced by AY Automate against its baselines, and Vals found the speed claim held at 32 parallel judgments but not for a single call under load. A DEV Community review put the ratios across eight setups at 97.8 times and 149.2 times.

In two separate tests, yes. Beri.net split phishing detection into five narrow questions and combined the answers with a simple model trained on 1,000 labelled emails. Accuracy rose from 62.6% to 95.0%, ahead of Claude Haiku 4.5 at 93.2%, though the author found that gap not statistically significant.

Using Jev as a first pass also works. AY Automate sent only the low-confidence cases to GPT-5.6 Terra and matched its accuracy at about a quarter of the cost (90.0% against 89.4% on the 8-class task). Vals estimated that routing 1% of its claim cases to GPT-6 Astra would match Astra's held-out accuracy for about $0.27 per 1,000 cases against about $12 for Astra alone. Both authors note that the cutoffs were set on the same data they were measured on, so treat the savings as estimates.

• Question design guide
• Jev vs ChatGPT

JevBench is an open benchmark for decision models like Jev, created by Florian Standhartinger and contributors. On version 1.5.4, shown by BenchLM on 15 October 2026, Jev 1.13.0 is third with 72.13, behind Cygnet (73.70) and Winnow-12B Q8 (73.23). The score is an index that combines accuracy, calibration, speed and cost, not a percentage of right answers. BenchLM lists JevBench for reference and leaves it out of its own scoring, and scores from different JevBench versions cannot be compared.

Featherless ran Jev on the 231 public JevBench items and reports 200 right (86.58%) in the published baseline and 199 (86.15%) in its own rerun. It notes that the prompts used for the open models were tuned on that same set, so those results are not held-out.

Almost every apparent contradiction has one of these causes:

• Different tasks. Jev tied the best models on claim checking and finished last on legal questions in the same Vals report.
• Different wording. Beri.net's phishing result moved by 32 points depending on whether the job was one question or five.
• Different reference answers. TypeSafe's accuracy is agreement with two other models. Others use human labels, public datasets or rules.
• Small samples. AY Automate says its confidence intervals are 6 to 13 points wide, so smaller differences mean nothing.
• Different setups. Latency depends on how many requests run at once, where the tester sits and which endpoint is used. One test ran on an early-access endpoint.
• Different versions. Every test above used Jev 1.13. Jev 1.14 became available as a preview on 14 October and none of the tests cover it yet.

Every careful source ends with the same advice. Vals recommends 200 to 400 audited cases and setting thresholds on a first slice (it used 30%) before testing on the rest. The DEV Community review suggests labelling 50 to a few hundred of your own decisions.

In practice: pin a specific version such as jev-1.13.0 so results do not change under you, write each question as one narrow judgment, run Jev and one alternative on the same cases, and compare accuracy, cost and time on your own numbers. The cost calculator turns the price into a monthly figure for your volume.

• Cost calculator
• How to use Jev

Our read: the tests agree more than the headlines suggest. Jev is a fast, cheap and well-behaved tool for narrow decisions, close to mid-priced language models on average and behind the best ones, with results that depend heavily on the task. It is a good first stage and a poor last word on anything that needs deep knowledge or careful reasoning.

The gaps in the evidence matter too. Most tests are small, several use public datasets that models may have seen, and nobody has published results for Jev 1.14 yet. This page will be updated when they appear.

• What is Jev?
• Jev vs open models