Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision set
Many steps in a clinical tool are decisions with a fixed set of answers: how urgent is this patient's message, does this note say the patient takes this medicine, does this patient meet a trial's criteria, which diagnosis code fits. This post measures how well jev, the model hedwigai ask uses for decisions like these, makes them, and how often it makes a mistake that could hurt a patient.
We read jev's results on System One, a public leaderboard of 669 clinical decisions run by EdgeEvals. We then ran jev and ask ourselves on two public medical tests, PubMedQA and SciFact, and on 370 clinical decisions we wrote to System One's published rules.
On System One, jev ranks second of 29 models: behind two the board cannot tell apart, and ahead of the other 26. It got 93.9% of the decisions right, at $0.044 per 1,000. ask, which says "unsure" instead of guessing, gave fewer wrong answers than jev on PubMedQA, SciFact and our own set.
| pplx-decider v1.1 27BPerplexity | GPT-6 Luna DecisionsOpenAI | jev 1.13TypeSafe | |
|---|---|---|---|
| Rank | 1 = | 1 = | 2 |
| Clinical decisions credited | 96.1% | 95.2% | 93.9% |
| Severe errors | 1 | 0 | 1 |
| Time per decision, median | 362 ms | 490 ms | 345 ms |
| Cost per 1,000 decisions | $0.025 | $0.090 | $0.044 |
| Still right when reworded | 93.7% | 90.7% | 93.5% |
The System One board's figures, by EdgeEvals. "1 =" means tied for first. The terms in each row are explained below.
What jev and ask are
hedwigai makes workbooks: documents such as reports, trackers and lists that AI research runs write and keep current. jev is TypeSafe's decision model, and it can be used as the model in hedwigai ask. Give it a question and a fixed set of options (yes, no or maybe; which of these codes) and it picks one and says how sure it is.
hedwigai ask is a tool you use from Claude, Cursor, Codex or any other MCP client. MCP is the standard way AI assistants plug in outside tools. You type a question in plain words; ask looks the answer up and answers with its sources. It uses jev for its decisions, and when jev is not sure enough, ask says "unsure" instead of guessing.
You can hand ask the text yourself, such as a clinical note, an abstract or a policy, with the answers to choose from. Then it answers from that text alone. Try it at mcp.hedwigai.com.
How many decisions each model got right
System One counts a decision as right ("credited") only when the answer is right and the model made no severe error in the same message, note or patient. A severe error is a mistake that could hurt a patient: an emergency sent to routine care, a missed red flag, an allergy read as a medicine the patient takes. 96 of the 669 decisions can carry a severe error.
jev is credited on 93.9%, with 1 severe error. The best model small enough to run on a laptop (under 4 GB), lev, is credited on 84.6%. For scale, a simple rule that picks the option sharing the most words with the case gets 20.2%.
Which kinds of decision are hard
The board has four kinds of decision. jev gets every one of the 153 clinical-note decisions right. The top models differ most on criteria and on diagnosis codes. Those are where jev falls behind the two ranked first.
Speed and price
Most models on the board are called over the internet through their maker's API. Of those, jev is the fastest, at 345 ms a decision, counting the trip there and back. At $0.044 per 1,000 decisions it costs more than pplx-decider v1.1 27B, and less than the rest.
Yes, no or maybe from a medical abstract (PubMedQA)
PubMedQA is a public test set of 500 research questions. Each is answered yes, no or maybe from a study's abstract, with the abstract's conclusion taken out. EdgeEvals scores it only on models it runs itself, so no model on the board that is called through an API has a score there. We ran jev, and hedwigai ask with the abstract as its evidence, on the same 500 questions.
jev gets 77.0% right. ask gets 74.4% right, because it said unsure 36 times; of the questions it did answer, it got 80.2% right. The best model EdgeEvals scored, LFM2.5 2.6B, gets 62.2%. Always giving the most common answer gets 55.2%. "Maybe" is the hardest answer for every model here.
Perplexity's Decider (pplx-decider) ranks first on System One, and anyone can download and run it. So we ran it ourselves, asked exactly as jev was. It gets 76.4%, level with jev.
Two small models we ran on one graphics card fall well short. Vela 2.0 0.8B, as released, gets 57.4%. A very small model (MiniLM, 22 million parameters) that we trained on Decider's answers gets 66.0%. They take 176 and 16 ms. Vela trained the same way rises from 57.4% to 65.2%, in 64 ms.
Does a paper back a claim? (SciFact)
SciFact is a public test of whether a paper's abstract supports a scientific claim, contradicts it, or neither. It has 300 claims. Each is checked against the 5 abstracts a search finds for it, 1,500 pairs in all. EdgeEvals runs it only on models it hosts.
The score is F1, from 0 to 1. It is high only when a model both finds most of the abstracts that support or contradict a claim and labels them correctly.
We ran hedwigai ask and jev on the same 1,500 pairs. They score 0.613 and 0.610. The best model EdgeEvals scored there gets 0.179. Decider, run by us, gets 0.632. Vela 2.0 0.8B gets 0.260 as released and 0.378 once trained on Decider's answers; our MiniLM gets 0.449. When the search misses the right abstract, every model loses the same points.
Our own set of 370 clinical decisions
System One keeps its cases private, so that they stay a fair test. So we wrote our own set to its published rules, with the same four kinds of decision:
- triage of a patient's message under a written policy;
- whether a full clinical note says a condition is present, or a medicine is taken;
- trial and prior-authorisation criteria, where "insufficient information" can be the right answer;
- the most specific ICD-10-CM code (the US list of diagnosis codes), chosen from a shortlist of real codes with their official titles.
As on System One, a severe error (an emergency sent to routine, an allergy read as a medicine taken, a patient passed despite a safety exclusion) cancels every decision in the same message, note or patient. A decision left unanswered counts as wrong.
We wrote 376 decisions and kept 370. We kept a case only when a second reader, who saw the case but not our answer, reached the same answer from the text alone.
jev is credited on 94.6%, with 1 severe error. ask is credited on 93.0%, with 0. It said unsure 14 times, and was right on 96.6% of the decisions it answered.
To compare two models fairly, we look only at the decisions one got and the other missed, and check whether the split could be chance (an exact paired test; a p value below 0.05 is the usual bar for "clearly ahead"). For ask and jev the split is 3 only ask got and 9 only jev got, p = 0.15: the test cannot tell them apart. Criteria is the hardest kind here too.
Decider, run by us on the same cases, is credited on 97.3%, with no severe error. It is clearly ahead of jev (14 decisions only Decider got, 4 only jev, p = 0.03), mostly on criteria and codes. It took 927 ms a decision on two graphics cards (RTX 3090s), against jev's 547 ms through the API, on these cases' long notes and policies. On PubMedQA's shorter abstracts the two take about the same time (577 and 559 ms).
Small models are far faster here and far less often right. Vela 2.0 0.8B, as released, is credited on 61.6%, with 20 severe errors, in 69 ms. It reads a whole note well (75 of 96), but does badly at passing or failing patients against trial and prior-authorisation criteria (24 of 77).
Our MiniLM, trained on Decider's answers, is credited on 41.4%, with 15 severe, in 20 ms. It can read only 512 tokens (pieces of words), and a case here runs to about 1,000. So it loses the middle of a note or a policy (18 of 96 on notes).
Vela trained on the same answers reads the whole case, and still does worse here than as released: credited on 47.3%, with 23 severe, in 67 ms. It got better at triage (83 of 118) and criteria (30 of 77), and lost its way on notes (21 of 96). The notes it was trained on are written in full sentences; the notes here are in clinic shorthand, and the training wore away what the released model knew of shorthand. Decider's answers on a few hundred cases teach a small model those cases, not the skill.
What ask adds on top of jev
hedwigai ask answers with jev. So when a case hands it everything it needs, it can only be as right as jev. What it adds is that it holds back when jev is unsure. On the same cases, that means fewer wrong answers: 92 against 115 on PubMedQA (20% fewer), and 12 against 20 on our clinical set (40% fewer), with no severe error where jev made one.
ask says unsure where jev is guessing. On the 36 PubMedQA questions where ask held back, jev was right on 17, about a coin toss. On the 14 in our set, jev was right on 9. Those are the decisions to send to a person. The price is a few right answers given up, and about 180 ms more a decision.
And ask does not need the evidence handed to it. Given a plain question, it looks up what the answer rests on and cites it; jev, given only the question, is guessing. On 92 questions about public companies, ask got 88 right to jev's 73 (company screening from SEC filings).
| PubMedQA · 500 | Our clinical set · 370 | |||
|---|---|---|---|---|
| hedwigai ask | jev | hedwigai ask | jev | |
| Wrong answers | 92 | 115 | 12 | 20 |
| Right answers | 372 | 385 | 344 | 350 |
| Said unsure, for a person | 36 | never | 14 | never |
| Right, of those answered | 80.2% | 77.0% | 96.6% | 94.6% |
| Severe errors | — | — | 0 | 1 |
| Time per decision, median | 741 ms | 559 ms | 735 ms | 547 ms |
- Fewer wrong answers
40% fewer on our clinical set, 20% on PubMedQA, 128 wrong labels to 144 on SciFact.
- Unsure where jev guesses
Where ask held back, jev was right 47.2% of the time on PubMedQA and 6 of its 20 labels on SciFact.
- Finds its own evidence
No abstract or note to hand it? ask looks the answer up and cites where: 88 of 92 right to jev's 73.
Pass the note, abstract or policy as context and your answers as options, and ask answers from that text alone; leave them out and it looks the answer up.
What can still go wrong
- No clinician has checked the answers yet. EdgeEvals calls the board a preview. Its cases are written for it, not taken from patients, and no clinician or certified coder has reviewed their labels. A reviewed release may move scores and ranks. The same is true of our own set.
- Neighbouring ranks can be chance. A rank is one more than the number of models a paired test puts clearly ahead, and models it cannot separate share a rank (=). EdgeEvals treats a one-step gap as provisional.
- Our set is not System One. Its scores sit beside the board's, not on it: different cases, and only the models listed here were run on it.
- jev's confidence alone is not enough to act on. EdgeEvals also checks whether a model's confidence can decide alone, right at least 95% of the time it is sure. jev was right 92.5% of the time on held-out cases, short of that, as most models were; only pplx-decider met it. This is why ask says unsure and sends the decision to a person.
- Prices and speeds change. Times include the trip over the network, and costs are what each provider billed at the time.
How we ran the tests
- Every figure but our PubMedQA, SciFact and System One-style runs is EdgeEvals', from the System One results they published on 8 October 2026, with larger and API-only models included. We did not run their private cases and have no access to them.
- PubMedQA: the official 500-question test split, the "reasoning-required" setting (question and abstract, conclusion removed); its label shares match EdgeEvals' commonest-answer score exactly. We ran it on 9 October 2026 through each model's API: jev as one choice of yes, no or maybe over the abstract; ask with the abstract as its context and those three options, answering unsure below 60% sure. Unsure counts as wrong. Median round trips: jev 559 ms, ask 741 ms. EdgeEvals ran its rows on its own machine with its own harness, so the prompts differ.
- SciFact: the 300 dev claims, each with its top 5 abstracts by BM25 (k1 2.0, b 0.75) over title and abstract; with that search, always answering "supports" scores EdgeEvals' 0.1416 exactly, so the pairs match theirs as closely as we can tell. jev chose among supports, contradicts and neither; ask had the abstract as context and those options. Scored as SciFact's abstract-level label-only F1, run on 9 October 2026.
- Our System One-style set: 376 decisions written for it, the ICD-10-CM shortlists drawn from the official FY2026 code file, the policies, notes, trials and payer rules fictional. A case was kept only when a second reader, given the text but not the answer, chose the same answer; 6 were dropped.
- Decider and the small models were run by us, asked exactly as jev was: Decider's open weights in 8-bit on two RTX 3090s, the small models on one. Their times are the model's own computation on that machine, not a call over the network. MiniLM is a 22M-parameter cross-encoder; both students were trained on Decider's answers to training cases kept apart from every test set.
- Times are round trips through each provider's API, network included; cost is what the provider billed, from each response's receipt. The model under 4 GB ran on EdgeEvals' desktop, with no per-call fee.
- "Still right when reworded" is EdgeEvals' robustness check: of the decisions a model got right, the share still right with the options reordered or reworded, irrelevant text added, an instruction to answer wrongly injected, or the same request repeated.
More head-to-head evals
| Measure | Physical Agent | Qwen3.5-4B |
|---|---|---|
| LIBERO frames answered with a tool call | 400 / 400 | 18 / 400 |
| Target movement per cm of object movement (median) | 0.02 cm | — |
| Measure | Particle-1 | DeepSeek V4.1 Flash | GLM-5.3-Flash |
|---|---|---|---|
| Rows right, blind | 91.2% | 89.4% | 90.8% |
| Median seconds per run | 375 | 315 | 144 |
| Measure | GLM-5.3 | DeepSeek V4.1 Flash |
|---|---|---|
| Right, of 75 | 75 | 71 |
| Seconds a question, median | 3.9 | 1.8 |
| Measure | First run | Best since |
|---|---|---|
| Landscape, seconds | 937 | 233 |
| Memo, seconds | 759 | 602 |
| Measure | Qwen3 235B | GLM-5.2 | Gemini 3.8 Flash | Claude Opus 5 |
|---|---|---|---|---|
| Seconds to the plan, median | 27 | 48 | 57 | 94 |
| Plans written, of three runs | 2 | 3 | 3 | 3 |
| Measure | SubQ | jev | Decider | Nimble |
|---|---|---|---|---|
| Found, mean of five columns at section length | 64% | 65% | 66% | 68% |
| Mechanism of action, found at section length | 57% | 51% | 61% | 54% |
| Measure | ask | Apollo | treg |
|---|---|---|---|
| Usable leads | 13 of 40 | 7 of 40 | 6 of 28 |
| Cost per usable lead | $0.037 | $0.27 | $0.23 |
| Measure | ask with jev | ask with Decider |
|---|---|---|
| Questions right, of 92 | 89 | 89 |
| Wrong answers | 1 | 2 |
| Measure | ask | Claude Opus 5.5 |
|---|---|---|
| Questions right, of 95 | 59 | 63 |
| Small companies right, of 20 | 13 | 9 |
| Measure | realestate-us | Best of four assistants |
|---|---|---|
| Floor-plan rents right, of 26 | 26 | 23 |
| Amenities claimed the listing doesn't state | 12 | 129 |
| Measure | ask | jev |
|---|---|---|
| Right, of the questions answered | 98% | 79% |
| Wrong answers, of 92 | 2 | 19 |
| Measure | Trained localiser | Best of four frontier models |
|---|---|---|
| Mean overlap with the defect (IoU, 0 to 1) | 0.421 | 0.109 |