Blog
hedwigaihedwigaihedwigai

Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision set

Many steps in a clinical tool are decisions with a fixed set of answers: how urgent is this patient's message, does this note say the patient takes this medicine, does this patient meet a trial's criteria, which diagnosis code fits. This post measures how well jev, the model hedwigai ask uses for decisions like these, makes them, and how often it makes a mistake that could hurt a patient.

We read jev's results on System One, a public leaderboard of 669 clinical decisions run by EdgeEvals. We then ran jev and ask ourselves on two public medical tests, PubMedQA and SciFact, and on 370 clinical decisions we wrote to System One's published rules.

On System One, jev ranks second of 29 models: behind two the board cannot tell apart, and ahead of the other 26. It got 93.9% of the decisions right, at $0.044 per 1,000. ask, which says "unsure" instead of guessing, gave fewer wrong answers than jev on PubMedQA, SciFact and our own set.

pplx-decider v1.1 27BPerplexityGPT-6 Luna DecisionsOpenAIjev 1.13TypeSafe
Rank1 =1 =2
Clinical decisions credited96.1%95.2%93.9%
Severe errors101
Time per decision, median362 ms490 ms345 ms
Cost per 1,000 decisions$0.025$0.090$0.044
Still right when reworded93.7%90.7%93.5%

The System One board's figures, by EdgeEvals. "1 =" means tied for first. The terms in each row are explained below.

What jev and ask are

hedwigai makes workbooks: documents such as reports, trackers and lists that AI research runs write and keep current. jev is TypeSafe's decision model, and it can be used as the model in hedwigai ask. Give it a question and a fixed set of options (yes, no or maybe; which of these codes) and it picks one and says how sure it is.

hedwigai ask is a tool you use from Claude, Cursor, Codex or any other MCP client. MCP is the standard way AI assistants plug in outside tools. You type a question in plain words; ask looks the answer up and answers with its sources. It uses jev for its decisions, and when jev is not sure enough, ask says "unsure" instead of guessing.

You can hand ask the text yourself, such as a clinical note, an abstract or a policy, with the answers to choose from. Then it answers from that text alone. Try it at mcp.hedwigai.com.

How many decisions each model got right

System One counts a decision as right ("credited") only when the answer is right and the model made no severe error in the same message, note or patient. A severe error is a mistake that could hurt a patient: an emergency sent to routine care, a missed red flag, an allergy read as a medicine the patient takes. 96 of the 669 decisions can carry a severe error.

jev is credited on 93.9%, with 1 severe error. The best model small enough to run on a laptop (under 4 GB), lev, is credited on 84.6%. For scale, a simple rule that picks the option sharing the most words with the case gets 20.2%.

1pplx-decider v1.1 27BPerplexity96.1% · 1 severe
2GPT-6 Luna DecisionsOpenAI95.2%
3jev 1.13TypeSafe93.9% · 1 severe
4Clef 27BCloudflare92.8% · 2 severe
5Mistral Large 4Mistral AI92.2%
6d1Liquid AI91.9% · 2 severe
7Solar DecideUpstage90.6%
8Clef-flash 9BCloudflare88.2%
9levInterfaze84.6% · 7 severe

Which kinds of decision are hard

The board has four kinds of decision. jev gets every one of the 153 clinical-note decisions right. The top models differ most on criteria and on diagnosis codes. Those are where jev falls behind the two ranked first.

Triage
Patient messages: urgency, red flags and where to send them · 240 decisions
1pplx-decider v1.1 27BPerplexity236 of 240
2GPT-6 Luna DecisionsOpenAI234 of 240
3jev 1.13TypeSafe233 of 240
Notes
Clinical notes: whether a condition is present, whether a medicine is taken · 153 decisions
1pplx-decider v1.1 27BPerplexity153 of 153
1jev 1.13TypeSafe153 of 153
3GPT-6 Luna DecisionsOpenAI152 of 153
Criteria
Risk-adjustment support, trial eligibility and prior authorisation · 156 decisions
1GPT-6 Luna DecisionsOpenAI146 of 156
2pplx-decider v1.1 27BPerplexity145 of 156
3jev 1.13TypeSafe140 of 156
Coding
The most specific ICD-10-CM code for a condition · 120 decisions
1pplx-decider v1.1 27BPerplexity109 of 120
2GPT-6 Luna DecisionsOpenAI105 of 120
3jev 1.13TypeSafe102 of 120

Speed and price

Most models on the board are called over the internet through their maker's API. Of those, jev is the fastest, at 345 ms a decision, counting the trip there and back. At $0.044 per 1,000 decisions it costs more than pplx-decider v1.1 27B, and less than the rest.

Median time per decision, shorter is better
1jev 1.13TypeSafe345 ms
2d1Liquid AI358 ms
3pplx-decider v1.1 27BPerplexity362 ms
4Clef-flash 9BCloudflare443 ms
5GPT-6 Luna DecisionsOpenAI490 ms
6Clef 27BCloudflare630 ms
7Mistral Large 4Mistral AI807 ms
8Solar DecideUpstage1057 ms
Billed per 1,000 decisions, lower is better
1pplx-decider v1.1 27BPerplexity$0.025
2jev 1.13TypeSafe$0.044
3d1Liquid AI$0.046
4Clef-flash 9BCloudflare$0.081
5Solar DecideUpstage$0.087
6GPT-6 Luna DecisionsOpenAI$0.090
7Clef 27BCloudflare$0.217
8Mistral Large 4Mistral AI$0.490

Yes, no or maybe from a medical abstract (PubMedQA)

PubMedQA is a public test set of 500 research questions. Each is answered yes, no or maybe from a study's abstract, with the abstract's conclusion taken out. EdgeEvals scores it only on models it runs itself, so no model on the board that is called through an API has a score there. We ran jev, and hedwigai ask with the abstract as its evidence, on the same 500 questions.

jev gets 77.0% right. ask gets 74.4% right, because it said unsure 36 times; of the questions it did answer, it got 80.2% right. The best model EdgeEvals scored, LFM2.5 2.6B, gets 62.2%. Always giving the most common answer gets 55.2%. "Maybe" is the hardest answer for every model here.

Perplexity's Decider (pplx-decider) ranks first on System One, and anyone can download and run it. So we ran it ourselves, asked exactly as jev was. It gets 76.4%, level with jev.

Two small models we ran on one graphics card fall well short. Vela 2.0 0.8B, as released, gets 57.4%. A very small model (MiniLM, 22 million parameters) that we trained on Decider's answers gets 66.0%. They take 176 and 16 ms. Vela trained the same way rises from 57.4% to 65.2%, in 64 ms.

1jev 1.13TypeSafe77.0% · run by us
2pplx-decider 27BPerplexity, open weights76.4% · run by us
3askhedwigai74.4% · 36 unsure · run by us
4MiniLM taught by Deciderhedwigai, 22M66.0% · run by us
5Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B65.2% · run by us
6LFM2.5 2.6BLiquid AI62.2% · EdgeEvals
7GLiNER2.5-DecideFastino61.0% · answered 463 · EdgeEvals
8Vela 2.0 0.8Bvllm-sr, open weights57.4% · run by us
9Word-overlap rulebaseline56.8% · EdgeEvals
10Most common answerbaseline55.2% · EdgeEvals
11Laya typed-decisionsConvAI Innovations52.2% · EdgeEvals
12GLiClass Instruct EdgeKnowledgator33.2% · EdgeEvals
13DeBERTa v3 Small NLISentence Transformers14.0% · answered 483 · EdgeEvals

Does a paper back a claim? (SciFact)

SciFact is a public test of whether a paper's abstract supports a scientific claim, contradicts it, or neither. It has 300 claims. Each is checked against the 5 abstracts a search finds for it, 1,500 pairs in all. EdgeEvals runs it only on models it hosts.

The score is F1, from 0 to 1. It is high only when a model both finds most of the abstracts that support or contradict a claim and labels them correctly.

We ran hedwigai ask and jev on the same 1,500 pairs. They score 0.613 and 0.610. The best model EdgeEvals scored there gets 0.179. Decider, run by us, gets 0.632. Vela 2.0 0.8B gets 0.260 as released and 0.378 once trained on Decider's answers; our MiniLM gets 0.449. When the search misses the right abstract, every model loses the same points.

1pplx-decider 27BPerplexity, open weights0.632 · run by us
2askhedwigai0.613 · 37 unsure · run by us
3jev 1.13TypeSafe0.610 · run by us
4MiniLM taught by Deciderhedwigai, 22M0.449 · run by us
5Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B0.378 · run by us
6Vela 2.0 0.8Bvllm-sr, open weights0.260 · run by us
7Laya typed-decisionsConvAI Innovations0.179 · EdgeEvals
8Most common answerbaseline0.142 · EdgeEvals
9GLiNER2.5-DecideFastino0.121 · answered 921 of 1,500 · EdgeEvals

Our own set of 370 clinical decisions

System One keeps its cases private, so that they stay a fair test. So we wrote our own set to its published rules, with the same four kinds of decision:

  • triage of a patient's message under a written policy;
  • whether a full clinical note says a condition is present, or a medicine is taken;
  • trial and prior-authorisation criteria, where "insufficient information" can be the right answer;
  • the most specific ICD-10-CM code (the US list of diagnosis codes), chosen from a shortlist of real codes with their official titles.

As on System One, a severe error (an emergency sent to routine, an allergy read as a medicine taken, a patient passed despite a safety exclusion) cancels every decision in the same message, note or patient. A decision left unanswered counts as wrong.

We wrote 376 decisions and kept 370. We kept a case only when a second reader, who saw the case but not our answer, reached the same answer from the text alone.

jev is credited on 94.6%, with 1 severe error. ask is credited on 93.0%, with 0. It said unsure 14 times, and was right on 96.6% of the decisions it answered.

To compare two models fairly, we look only at the decisions one got and the other missed, and check whether the split could be chance (an exact paired test; a p value below 0.05 is the usual bar for "clearly ahead"). For ask and jev the split is 3 only ask got and 9 only jev got, p = 0.15: the test cannot tell them apart. Criteria is the hardest kind here too.

Decider, run by us on the same cases, is credited on 97.3%, with no severe error. It is clearly ahead of jev (14 decisions only Decider got, 4 only jev, p = 0.03), mostly on criteria and codes. It took 927 ms a decision on two graphics cards (RTX 3090s), against jev's 547 ms through the API, on these cases' long notes and policies. On PubMedQA's shorter abstracts the two take about the same time (577 and 559 ms).

Small models are far faster here and far less often right. Vela 2.0 0.8B, as released, is credited on 61.6%, with 20 severe errors, in 69 ms. It reads a whole note well (75 of 96), but does badly at passing or failing patients against trial and prior-authorisation criteria (24 of 77).

Our MiniLM, trained on Decider's answers, is credited on 41.4%, with 15 severe, in 20 ms. It can read only 512 tokens (pieces of words), and a case here runs to about 1,000. So it loses the middle of a note or a policy (18 of 96 on notes).

Vela trained on the same answers reads the whole case, and still does worse here than as released: credited on 47.3%, with 23 severe, in 67 ms. It got better at triage (83 of 118) and criteria (30 of 77), and lost its way on notes (21 of 96). The notes it was trained on are written in full sentences; the notes here are in clinic shorthand, and the training wore away what the released model knew of shorthand. Decider's answers on a few hundred cases teach a small model those cases, not the skill.

All 370 decisions
1pplx-decider 27BPerplexity, open weights97.3% · 0 severe
2jev 1.13TypeSafe94.6% · 1 severe
3askhedwigai93.0% · 0 severe · 14 unsure
4Vela 2.0 0.8Bvllm-sr, open weights61.6% · 20 severe
5Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B47.3% · 23 severe
6MiniLM taught by Deciderhedwigai, 22M41.4% · 15 severe
Triage
118 decisions
1jev 1.13TypeSafe117 of 118
2askhedwigai115 of 118
2pplx-decider 27BPerplexity, open weights115 of 118
4Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B83 of 118
5Vela 2.0 0.8Bvllm-sr, open weights73 of 118
6MiniLM taught by Deciderhedwigai, 22M54 of 118
Notes
96 decisions
1pplx-decider 27BPerplexity, open weights95 of 96
2jev 1.13TypeSafe94 of 96
2askhedwigai94 of 96
4Vela 2.0 0.8Bvllm-sr, open weights75 of 96
5Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B21 of 96
6MiniLM taught by Deciderhedwigai, 22M18 of 96
Criteria
77 decisions
1pplx-decider 27BPerplexity, open weights73 of 77
2jev 1.13TypeSafe65 of 77
3askhedwigai63 of 77
4MiniLM taught by Deciderhedwigai, 22M33 of 77
5Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B30 of 77
6Vela 2.0 0.8Bvllm-sr, open weights24 of 77
Coding
79 decisions
1pplx-decider 27BPerplexity, open weights77 of 79
2jev 1.13TypeSafe74 of 79
3askhedwigai72 of 79
4Vela 2.0 0.8Bvllm-sr, open weights56 of 79
5MiniLM taught by Deciderhedwigai, 22M48 of 79
6Vela 0.8B taught by Deciderhedwigai, from Vela 2.0 0.8B41 of 79

What ask adds on top of jev

hedwigai ask answers with jev. So when a case hands it everything it needs, it can only be as right as jev. What it adds is that it holds back when jev is unsure. On the same cases, that means fewer wrong answers: 92 against 115 on PubMedQA (20% fewer), and 12 against 20 on our clinical set (40% fewer), with no severe error where jev made one.

ask says unsure where jev is guessing. On the 36 PubMedQA questions where ask held back, jev was right on 17, about a coin toss. On the 14 in our set, jev was right on 9. Those are the decisions to send to a person. The price is a few right answers given up, and about 180 ms more a decision.

And ask does not need the evidence handed to it. Given a plain question, it looks up what the answer rests on and cites it; jev, given only the question, is guessing. On 92 questions about public companies, ask got 88 right to jev's 73 (company screening from SEC filings).

PubMedQA · 500Our clinical set · 370
hedwigai askjevhedwigai askjev
Wrong answers921151220
Right answers372385344350
Said unsure, for a person36never14never
Right, of those answered80.2%77.0%96.6%94.6%
Severe errors——01
Time per decision, median741 ms559 ms735 ms547 ms
  • Fewer wrong answers

    40% fewer on our clinical set, 20% on PubMedQA, 128 wrong labels to 144 on SciFact.

  • Unsure where jev guesses

    Where ask held back, jev was right 47.2% of the time on PubMedQA and 6 of its 20 labels on SciFact.

  • Finds its own evidence

    No abstract or note to hand it? ask looks the answer up and cites where: 88 of 92 right to jev's 73.

Pass the note, abstract or policy as context and your answers as options, and ask answers from that text alone; leave them out and it looks the answer up.

What can still go wrong

  • No clinician has checked the answers yet. EdgeEvals calls the board a preview. Its cases are written for it, not taken from patients, and no clinician or certified coder has reviewed their labels. A reviewed release may move scores and ranks. The same is true of our own set.
  • Neighbouring ranks can be chance. A rank is one more than the number of models a paired test puts clearly ahead, and models it cannot separate share a rank (=). EdgeEvals treats a one-step gap as provisional.
  • Our set is not System One. Its scores sit beside the board's, not on it: different cases, and only the models listed here were run on it.
  • jev's confidence alone is not enough to act on. EdgeEvals also checks whether a model's confidence can decide alone, right at least 95% of the time it is sure. jev was right 92.5% of the time on held-out cases, short of that, as most models were; only pplx-decider met it. This is why ask says unsure and sends the decision to a person.
  • Prices and speeds change. Times include the trip over the network, and costs are what each provider billed at the time.

How we ran the tests

  • Every figure but our PubMedQA, SciFact and System One-style runs is EdgeEvals', from the System One results they published on 8 October 2026, with larger and API-only models included. We did not run their private cases and have no access to them.
  • PubMedQA: the official 500-question test split, the "reasoning-required" setting (question and abstract, conclusion removed); its label shares match EdgeEvals' commonest-answer score exactly. We ran it on 9 October 2026 through each model's API: jev as one choice of yes, no or maybe over the abstract; ask with the abstract as its context and those three options, answering unsure below 60% sure. Unsure counts as wrong. Median round trips: jev 559 ms, ask 741 ms. EdgeEvals ran its rows on its own machine with its own harness, so the prompts differ.
  • SciFact: the 300 dev claims, each with its top 5 abstracts by BM25 (k1 2.0, b 0.75) over title and abstract; with that search, always answering "supports" scores EdgeEvals' 0.1416 exactly, so the pairs match theirs as closely as we can tell. jev chose among supports, contradicts and neither; ask had the abstract as context and those options. Scored as SciFact's abstract-level label-only F1, run on 9 October 2026.
  • Our System One-style set: 376 decisions written for it, the ICD-10-CM shortlists drawn from the official FY2026 code file, the policies, notes, trials and payer rules fictional. A case was kept only when a second reader, given the text but not the answer, chose the same answer; 6 were dropped.
  • Decider and the small models were run by us, asked exactly as jev was: Decider's open weights in 8-bit on two RTX 3090s, the small models on one. Their times are the model's own computation on that machine, not a call over the network. MiniLM is a 22M-parameter cross-encoder; both students were trained on Decider's answers to training cases kept apart from every test set.
  • Times are round trips through each provider's API, network included; cost is what the provider billed, from each response's receipt. The model under 4 GB ran on EdgeEvals' desktop, with no per-call fee.
  • "Still right when reworded" is EdgeEvals' robustness check: of the decisions a model got right, the share still right with the options reordered or reworded, irrelevant text added, an instruction to answer wrongly injected, or the same request repeated.

More head-to-head evals

Does Physical Agent aim its tool calls at the object?Physical Agent writes a robot tool call on 400 of 400 frames, but moving the object 16 cm leaves its targets where they were.
MeasurePhysical AgentQwen3.5-4B
LIBERO frames answered with a tool call400 / 40018 / 400
Target movement per cm of object movement (median)0.02 cm—
Read the comparison
Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.
Measureask with jevask with Decider
Questions right, of 928989
Wrong answers12
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.
Measureaskjev
Right, of the questions answered98%79%
Wrong answers, of 92219
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison