Blog
hedwigaihedwigaihedwigai

Company screening from SEC filings: jev, Vela 2.0 and hedwigai ask on 92 yes/no questions

Screening companies often comes down to a yes or a no: is this company a biotech, is it big enough, is it listed where you think it is? We put 92 questions like that, about public companies, to three systems. Every answer was checked against SEC records, the filings public companies make with the US Securities and Exchange Commission.

The three were hedwigai ask, which looks the answer up before it decides; jev, TypeSafe's decision model, which ask used for its decisions here, answering on its own; and Vela 2.0, an open decision model anyone can download, given exactly the same request as jev.

hedwigai ask got 88 right, jev 73 and Vela 2.0 43. The two models answer from what they already know; ask looks the numbers up, which makes it slower on a question it has not seen before (1.07 s against jev's 525 ms).

What hedwigai ask and jev are

hedwigai makes workbooks: documents such as reports, trackers and lists that AI research runs write and keep current. ask is one of its tools. It lives on hedwigai's MCP server, and MCP is the standard way AI assistants plug in outside tools, so you use ask from Claude, Cursor, Codex or any other MCP client.

You type a question in plain words. ask looks the answer up, in web pages, SEC filings and other sources, and answers with the sources it used.

jev is TypeSafe's decision model, and it can be used as the model in hedwigai ask. Given a question and a fixed set of options (yes or no, or which one of several), it picks one and says how sure it is. In this test ask made its decisions with jev, after it had looked the facts up. In this test jev also answered alone, with nothing looked up, to show what the looking up adds.

The questions

Each question names a company, with its stock ticker and exchange, and asks whether it is a biotech above a given size. Size here is the float: the value of the company's shares held by the public.

Some questions are traps. The company is made up, or the name and the ticker belong to two different companies, or the exchange is the wrong one. A good answer notices, and says the question itself is wrong. Others are close calls: a biotech whose float is within 6% of the size in the question.

The right answer to every question was worked out from SEC records before any of the three systems ran.

Results

hedwigai askjevVela 2.0
Right887343
Wrong21949
Unsure200
Right, of those answered98%79%47%
Time, a new question1.07 s525 ms617 ms, Apple M1 Pro GPU
Time, a question seen before466 ms525 ms617 ms, Apple M1 Pro GPU
Work, a new question1.3 small-model + 1.8 jev calls, 2.2 lookups1 jev call1 Vela call
Work, a question seen beforenothing (kept)1 jev call1 Vela call
Shows where the answer came fromyesnono
  • Unsure means the system declined to answer because it was not sure enough. It is not counted as right or as wrong.
  • Right, of those answered leaves the unsure ones out: of the questions it did answer, the share it got right.
  • Time is the median wait, from sending the question to getting the answer: half the answers came faster, half slower. ask keeps the answers it has worked out, so a question it has seen before comes back faster and needs no new work.
  • Work counts what one answer needed, on average: calls to a small helper model, calls to jev, and lookups in SEC records. jev and Vela 2.0 each answer in a single call, with nothing looked up.

Where each one wins

jev does well when the words of the question are enough, as with a made-up name or a ticker that belongs to someone else. ask wins wherever the answer depends on a number, such as whether a biotech's float is above the size in the question. It looks the number up instead of guessing.

Looking up takes time. On a new question jev answers first. On a question it has seen before, ask does, because it kept the answer.

Vela 2.0 gets 43 right. It never answers that the question itself is wrong, so it misses every made-up company and every ticker that belongs to another company. It also calls 10 of 23 companies that are not biotechs biotech. On the close calls it beats jev, 7 to 3.

rightwrongsaid unsure
A made-up company
10 questions
1ask10 of 10 right ✓
2jev9 of 10 right
3Vela 2.00 of 10 right
The name and the ticker belong to different companies
10 questions
1jev10 of 10 right ✓
2ask9 of 10 right
3Vela 2.00 of 10 right
A real company, but the wrong stock exchange named
9 questions
1ask9 of 9 right ✓
1Vela 2.09 of 9 right ✓
3jev7 of 9 right
A biotech, and clearly big enough
15 questions
1ask13 of 15 right ✓
2Vela 2.010 of 15 right
3jev8 of 15 right
A biotech, but clearly too small
15 questions
1ask15 of 15 right ✓
2jev14 of 15 right
3Vela 2.04 of 15 right
A biotech, and within 6% of the size asked about
10 questions
1ask9 of 10 right ✓
2Vela 2.07 of 10 right
3jev3 of 10 right
Not a biotech at all
23 questions
1ask23 of 23 right ✓
2jev22 of 23 right
3Vela 2.013 of 23 right
All 92 questions
1ask88 right · 2 wrong
2jev73 right · 19 wrong
3Vela 2.043 right · 49 wrong
Time to answer
Median wait, shorter is better
1askasked before466 ms ✓
2jev525 ms
3Vela 2.0Apple M1 Pro GPU617 ms
4asknew question1.07 s

What can still go wrong

  • “Biotech” is a line someone has to draw. Here it means SEC industry codes 2833 to 2836 or 8731, which counts specialty drug makers in. Most of ask's misses are about where that line falls.
  • This is a small test. One kind of question, one account, 92 questions, measured on 6 and 7 October 2026. Other kinds of questions could come out differently.
  • A brand new account starts slower. The times for new questions were measured with the SEC data already loaded, so a new account's first questions take longer.
  • Vela 2.0's time flatters it a little. It ran on a laptop with no network in between, while the times for jev and ask include the trip over the internet.
  • jev's and ask's times are medians of the round trip from the same Apple M1 Pro over the internet; on the server a new question took 612 ms and a repeat 1 ms.
  • Vela 2.0 is the 0.8B model from vllm-sr, given exactly the request jev gets. It ran locally on an Apple M1 Pro: 10-core CPU (8 performance, 2 efficiency), 16-core GPU (used, through PyTorch's Metal backend), 16 GB unified memory, in 32-bit floats. The 0.3B got 23 right in 77 ms. The 4B and 9B need more than the 16 GB of memory and were not run.

More head-to-head evals

Does Physical Agent aim its tool calls at the object?Physical Agent writes a robot tool call on 400 of 400 frames, but moving the object 16 cm leaves its targets where they were.
MeasurePhysical AgentQwen3.5-4B
LIBERO frames answered with a tool call400 / 40018 / 400
Target movement per cm of object movement (median)0.02 cm—
Read the comparison
Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.
Measureask with jevask with Decider
Questions right, of 928989
Wrong answers12
Read the comparison
Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.
Measurejevask
Our 370 clinical decisions, right94.6%93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)77.0%74.4%
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison