Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questions
hedwigai ask answers a question by looking things up and making small decisions as it goes: which lookups to run, which figure measures the amount in the question, and in the end whether the answer is yes or no. Those decisions are made by the model ask is set to use; here that was jev, TypeSafe's decision model.
We tried making every one of them instead with Decider, Perplexity's open decision model, running on two NVIDIA RTX 3090s (24 GB of VRAM each). Same 92 yes-or-no questions about public companies, every answer checked against SEC records.
Decider gets as many right as jev, 89 of 92, with one more wrong. It is about 9 times slower on those two cards, and its open weights carry no per-decision fee.
What ask, jev and Decider are
hedwigai makes workbooks: documents such as reports, trackers and lists that AI research runs write and keep current. ask is one of its tools, on hedwigai's MCP server. MCP is the standard way AI assistants plug in outside tools, so you use ask from Claude, Cursor, Codex or any other MCP client. You type a question in plain words; it looks the answer up, in web pages, SEC filings and other sources, and answers with its sources.
jev is TypeSafe's decision model, and it can be used as the model in hedwigai ask. Given a question and a fixed set of options (yes or no, or which one of several), it picks one and says how sure it is.
Decider (pplx-decider-v1-27b) is a model of the same kind from Perplexity, the AI search company, with its weights published so anyone can run it. Perplexity also serves it, billed per call, as its Decisions API. “Self-hosted” in the title means the published weights ran on the GPUs described below, not through that API.
To try ask, add hedwigai to your AI assistant: mcp.hedwigai.com.
The questions
They are the 92 questions of an earlier test. Each names a company, with its stock ticker and exchange, and asks whether it is a biotech above a given size. Some are traps: a made-up company, or a name and ticker that belong to different companies, or the wrong exchange. Some are close calls, within 6% of the size asked about. The right answers were worked out from SEC records, the filings public companies make with the US Securities and Exchange Commission, before either system ran.
Results
| ask with jev | ask with Decider | |
|---|---|---|
| Right | 89 | 89 |
| Wrong | 1 | 2 |
| Unsure | 2 | 1 |
| Right, of those answered | 99% | 98% |
| Time, median per question | 2.1 s | 19.1 s |
| Decisions per question | 81 in 2.2 calls | 81 in 2.2 calls |
| Where the decisions run | TypeSafe's jev service | two NVIDIA RTX 3090s |
| Charge per decision | billed per call | none (open weights) |
- Unsure means ask declined to answer because it was not sure enough. It is not counted as right or as wrong.
- Right, of those answered leaves the unsure ones out: of the questions answered, the share that were right.
- Time is the median wait for a whole question: half came back faster, half slower.
- Decisions per question is how many small decisions ask made for one question, on average, and in how many requests it sent them to the decision model. It is the same with either model.
Question by question
The two agree on almost every kind of question. On the 15 biotechs that are clearly big enough, Decider answers all of them, where jev is unsure of 2. Decider says yes to one company that is not a biotech, because the lookup it chose never checked whether it was one. And it is unsure of one question that names the wrong stock exchange.
Why Decider is slower
A question takes about 81 decisions, sent in 2.2 requests on average. Most of them come at once, when ask plans which lookups to run. jev answers a request like that in well under a second.
Decider is a large model, with 27 billion parameters (the numbers a model learns in training). Here it was split across two graphics cards of the kind sold for home PCs, and worked through 16 decisions at a time, so the planning request alone takes most of its 19.1 s. That is the cost of these two cards, not of the model: a single larger card, or more of them, cuts it.
One sentence decided 22 answers
Every question here is about a company's size: its market cap (the value of all its shares) or its public float (the value of the shares held by the public). Only one lookup returns either: the one for shares outstanding and the public float.
To check an amount, ask first decides which lookup measures it, or comes closest. It decides from a short description of each lookup. Whatever it picks is the only number it has to compare against. If it picks none, the amount cannot be checked, and the answer stays unsure.
We ran both models with that lookup's description worded two ways, shown below. Worded as “NOT a market cap”, Decider chose “none of these” on 59 of 92 questions and left 23 answers unsure. It was a close call each time. On one question it gave “none of these” a probability of 0.55 and the lookup 0.39; a probability here is a number from 0 to 1 for how likely the model thinks an option is the right one.
Worded as the closest stand-in for a market cap, Decider chose the lookup at 0.96 and got 89 right. jev chose it either way, and its answers did not change. Decider takes a description literally, so the descriptions ask now chooses between say what each lookup is good for, not only what it is not.
- Not a market cap
- shares outstanding and the public float, dated and citable — NOT a market cap, which needs a share price nothing here has
- The stand-in (used now)
- shares outstanding and the public float, dated and citable — the public float is the closest stand-in for a market cap here; a market cap itself needs a share price nothing here has
What can still go wrong
- This is a small test. One kind of question, one account, 92 questions, and one run with each model. A tie at 89 right says the two are close here, not that they are equal everywhere.
- On these two cards, Decider is slow. A median of 19.1 s a question against 2.1 s with jev.
- Decider takes descriptions literally. A description that only says what a lookup is not can lead it to pick none of them, as one sentence did above.
- “Biotech” is a line someone has to draw. Here it means SEC industry codes 2833 to 2836 or 8731.
- The questions, their kinds and the SEC answers are the 92 of jev, Vela 2.0 and ask on 92 yes/no questions.
- Both runs used the same build of ask, on the same day (10 October 2026), with the same lookups; only the service answering the decisions differed. Each decision was put to Decider the way it is put to jev, and Decider's answer was handed back in jev's form and read by the same rules.
- Decider is pplx-decider-v1-27b, Perplexity's open weights (Apache 2.0), run in 8-bit weights on two NVIDIA RTX 3090s (24 GB of VRAM each), 16 decisions at a time.
- Times are medians of the whole question, lookups included, which are the same for both; the difference between the two is the decisions.
More head-to-head evals
| Measure | Physical Agent | Qwen3.5-4B |
|---|---|---|
| LIBERO frames answered with a tool call | 400 / 400 | 18 / 400 |
| Target movement per cm of object movement (median) | 0.02 cm | — |
| Measure | Particle-1 | DeepSeek V4.1 Flash | GLM-5.3-Flash |
|---|---|---|---|
| Rows right, blind | 91.2% | 89.4% | 90.8% |
| Median seconds per run | 375 | 315 | 144 |
| Measure | GLM-5.3 | DeepSeek V4.1 Flash |
|---|---|---|
| Right, of 75 | 75 | 71 |
| Seconds a question, median | 3.9 | 1.8 |
| Measure | First run | Best since |
|---|---|---|
| Landscape, seconds | 937 | 233 |
| Memo, seconds | 759 | 602 |
| Measure | Qwen3 235B | GLM-5.2 | Gemini 3.8 Flash | Claude Opus 5 |
|---|---|---|---|---|
| Seconds to the plan, median | 27 | 48 | 57 | 94 |
| Plans written, of three runs | 2 | 3 | 3 | 3 |
| Measure | SubQ | jev | Decider | Nimble |
|---|---|---|---|---|
| Found, mean of five columns at section length | 64% | 65% | 66% | 68% |
| Mechanism of action, found at section length | 57% | 51% | 61% | 54% |
| Measure | ask | Apollo | treg |
|---|---|---|---|
| Usable leads | 13 of 40 | 7 of 40 | 6 of 28 |
| Cost per usable lead | $0.037 | $0.27 | $0.23 |
| Measure | jev | ask |
|---|---|---|
| Our 370 clinical decisions, right | 94.6% | 93.0% |
| PubMedQA, right (best EdgeEvals scored: 62.2%) | 77.0% | 74.4% |
| Measure | ask | Claude Opus 5.5 |
|---|---|---|
| Questions right, of 95 | 59 | 63 |
| Small companies right, of 20 | 13 | 9 |
| Measure | realestate-us | Best of four assistants |
|---|---|---|
| Floor-plan rents right, of 26 | 26 | 23 |
| Amenities claimed the listing doesn't state | 12 | 129 |
| Measure | ask | jev |
|---|---|---|
| Right, of the questions answered | 98% | 79% |
| Wrong answers, of 92 | 2 | 19 |
| Measure | Trained localiser | Best of four frontier models |
|---|---|---|
| Mean overlap with the defect (IoU, 0 to 1) | 0.421 | 0.109 |