Blog
hedwigaihedwigaihedwigai

Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questions

hedwigai ask answers a question by looking things up and making small decisions as it goes: which lookups to run, which figure measures the amount in the question, and in the end whether the answer is yes or no. Those decisions are made by the model ask is set to use; here that was jev, TypeSafe's decision model.

We tried making every one of them instead with Decider, Perplexity's open decision model, running on two NVIDIA RTX 3090s (24 GB of VRAM each). Same 92 yes-or-no questions about public companies, every answer checked against SEC records.

Decider gets as many right as jev, 89 of 92, with one more wrong. It is about 9 times slower on those two cards, and its open weights carry no per-decision fee.

What ask, jev and Decider are

hedwigai makes workbooks: documents such as reports, trackers and lists that AI research runs write and keep current. ask is one of its tools, on hedwigai's MCP server. MCP is the standard way AI assistants plug in outside tools, so you use ask from Claude, Cursor, Codex or any other MCP client. You type a question in plain words; it looks the answer up, in web pages, SEC filings and other sources, and answers with its sources.

jev is TypeSafe's decision model, and it can be used as the model in hedwigai ask. Given a question and a fixed set of options (yes or no, or which one of several), it picks one and says how sure it is.

Decider (pplx-decider-v1-27b) is a model of the same kind from Perplexity, the AI search company, with its weights published so anyone can run it. Perplexity also serves it, billed per call, as its Decisions API. “Self-hosted” in the title means the published weights ran on the GPUs described below, not through that API.

To try ask, add hedwigai to your AI assistant: mcp.hedwigai.com.

The questions

They are the 92 questions of an earlier test. Each names a company, with its stock ticker and exchange, and asks whether it is a biotech above a given size. Some are traps: a made-up company, or a name and ticker that belong to different companies, or the wrong exchange. Some are close calls, within 6% of the size asked about. The right answers were worked out from SEC records, the filings public companies make with the US Securities and Exchange Commission, before either system ran.

Results

ask with jevask with Decider
Right8989
Wrong12
Unsure21
Right, of those answered99%98%
Time, median per question2.1 s19.1 s
Decisions per question81 in 2.2 calls81 in 2.2 calls
Where the decisions runTypeSafe's jev servicetwo NVIDIA RTX 3090s
Charge per decisionbilled per callnone (open weights)
  • Unsure means ask declined to answer because it was not sure enough. It is not counted as right or as wrong.
  • Right, of those answered leaves the unsure ones out: of the questions answered, the share that were right.
  • Time is the median wait for a whole question: half came back faster, half slower.
  • Decisions per question is how many small decisions ask made for one question, on average, and in how many requests it sent them to the decision model. It is the same with either model.

Question by question

The two agree on almost every kind of question. On the 15 biotechs that are clearly big enough, Decider answers all of them, where jev is unsure of 2. Decider says yes to one company that is not a biotech, because the lookup it chose never checked whether it was one. And it is unsure of one question that names the wrong stock exchange.

rightwrongsaid unsure
A made-up company
10 questions
1ask with jev10 of 10 right ✓
1ask with Decider10 of 10 right ✓
The name and the ticker belong to different companies
10 questions
1ask with jev10 of 10 right ✓
1ask with Decider10 of 10 right ✓
A real company, but the wrong stock exchange named
9 questions
1ask with jev9 of 9 right ✓
2ask with Decider8 of 9 right
A biotech, and clearly big enough
15 questions
1ask with Decider15 of 15 right ✓
2ask with jev13 of 15 right
A biotech, but clearly too small
15 questions
1ask with jev15 of 15 right ✓
1ask with Decider15 of 15 right ✓
A biotech, and within 6% of the size asked about
10 questions
1ask with jev9 of 10 right ✓
1ask with Decider9 of 10 right ✓
Not a biotech at all
23 questions
1ask with jev23 of 23 right ✓
2ask with Decider22 of 23 right
All 92 questions
1ask with jev89 right · 1 wrong
2ask with Decider89 right · 2 wrong
Time to answer
Median per question, shorter is better
1ask with jev2.1 s ✓
2ask with Decider19.1 s

Why Decider is slower

A question takes about 81 decisions, sent in 2.2 requests on average. Most of them come at once, when ask plans which lookups to run. jev answers a request like that in well under a second.

Decider is a large model, with 27 billion parameters (the numbers a model learns in training). Here it was split across two graphics cards of the kind sold for home PCs, and worked through 16 decisions at a time, so the planning request alone takes most of its 19.1 s. That is the cost of these two cards, not of the model: a single larger card, or more of them, cuts it.

One sentence decided 22 answers

Every question here is about a company's size: its market cap (the value of all its shares) or its public float (the value of the shares held by the public). Only one lookup returns either: the one for shares outstanding and the public float.

To check an amount, ask first decides which lookup measures it, or comes closest. It decides from a short description of each lookup. Whatever it picks is the only number it has to compare against. If it picks none, the amount cannot be checked, and the answer stays unsure.

We ran both models with that lookup's description worded two ways, shown below. Worded as “NOT a market cap”, Decider chose “none of these” on 59 of 92 questions and left 23 answers unsure. It was a close call each time. On one question it gave “none of these” a probability of 0.55 and the lookup 0.39; a probability here is a number from 0 to 1 for how likely the model thinks an option is the right one.

Worded as the closest stand-in for a market cap, Decider chose the lookup at 0.96 and got 89 right. jev chose it either way, and its answers did not change. Decider takes a description literally, so the descriptions ask now chooses between say what each lookup is good for, not only what it is not.

rightwrongsaid unsure
Decidersaid as not a market cap67 right · 23 unsure
Decidersaid as the stand-in89 right · 1 unsure
jevsaid as not a market cap89 right · 2 unsure
jevsaid as the stand-in89 right · 2 unsure
Not a market cap
shares outstanding and the public float, dated and citable — NOT a market cap, which needs a share price nothing here has
The stand-in (used now)
shares outstanding and the public float, dated and citable — the public float is the closest stand-in for a market cap here; a market cap itself needs a share price nothing here has

What can still go wrong

  • This is a small test. One kind of question, one account, 92 questions, and one run with each model. A tie at 89 right says the two are close here, not that they are equal everywhere.
  • On these two cards, Decider is slow. A median of 19.1 s a question against 2.1 s with jev.
  • Decider takes descriptions literally. A description that only says what a lookup is not can lead it to pick none of them, as one sentence did above.
  • “Biotech” is a line someone has to draw. Here it means SEC industry codes 2833 to 2836 or 8731.
  • The questions, their kinds and the SEC answers are the 92 of jev, Vela 2.0 and ask on 92 yes/no questions.
  • Both runs used the same build of ask, on the same day (10 October 2026), with the same lookups; only the service answering the decisions differed. Each decision was put to Decider the way it is put to jev, and Decider's answer was handed back in jev's form and read by the same rules.
  • Decider is pplx-decider-v1-27b, Perplexity's open weights (Apache 2.0), run in 8-bit weights on two NVIDIA RTX 3090s (24 GB of VRAM each), 16 decisions at a time.
  • Times are medians of the whole question, lookups included, which are the same for both; the difference between the two is the decisions.

More head-to-head evals

Does Physical Agent aim its tool calls at the object?Physical Agent writes a robot tool call on 400 of 400 frames, but moving the object 16 cm leaves its targets where they were.
MeasurePhysical AgentQwen3.5-4B
LIBERO frames answered with a tool call400 / 40018 / 400
Target movement per cm of object movement (median)0.02 cm—
Read the comparison
Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.
Measurejevask
Our 370 clinical decisions, right94.6%93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)77.0%74.4%
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.
Measureaskjev
Right, of the questions answered98%79%
Wrong answers, of 92219
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison