Company screening from SEC filings: jev, Vela 2.0 and hedwigai ask on 92 yes/no questions
Screening companies often comes down to a yes or a no: is this company a biotech, is it big enough, is it listed where you think it is? We put 92 questions like that, about public companies, to three systems. Every answer was checked against SEC records, the filings public companies make with the US Securities and Exchange Commission.
The three were hedwigai ask, which looks the answer up before it decides; jev, TypeSafe's decision model, which ask used for its decisions here, answering on its own; and Vela 2.0, an open decision model anyone can download, given exactly the same request as jev.
hedwigai ask got 88 right, jev 73 and Vela 2.0 43. The two models answer from what they already know; ask looks the numbers up, which makes it slower on a question it has not seen before (1.07 s against jev's 525 ms).
What hedwigai ask and jev are
hedwigai makes workbooks: documents such as reports, trackers and lists that AI research runs write and keep current. ask is one of its tools. It lives on hedwigai's MCP server, and MCP is the standard way AI assistants plug in outside tools, so you use ask from Claude, Cursor, Codex or any other MCP client.
You type a question in plain words. ask looks the answer up, in web pages, SEC filings and other sources, and answers with the sources it used.
jev is TypeSafe's decision model, and it can be used as the model in hedwigai ask. Given a question and a fixed set of options (yes or no, or which one of several), it picks one and says how sure it is. In this test ask made its decisions with jev, after it had looked the facts up. In this test jev also answered alone, with nothing looked up, to show what the looking up adds.
The questions
Each question names a company, with its stock ticker and exchange, and asks whether it is a biotech above a given size. Size here is the float: the value of the company's shares held by the public.
Some questions are traps. The company is made up, or the name and the ticker belong to two different companies, or the exchange is the wrong one. A good answer notices, and says the question itself is wrong. Others are close calls: a biotech whose float is within 6% of the size in the question.
The right answer to every question was worked out from SEC records before any of the three systems ran.
Results
| hedwigai ask | jev | Vela 2.0 | |
|---|---|---|---|
| Right | 88 | 73 | 43 |
| Wrong | 2 | 19 | 49 |
| Unsure | 2 | 0 | 0 |
| Right, of those answered | 98% | 79% | 47% |
| Time, a new question | 1.07 s | 525 ms | 617 ms, Apple M1 Pro GPU |
| Time, a question seen before | 466 ms | 525 ms | 617 ms, Apple M1 Pro GPU |
| Work, a new question | 1.3 small-model + 1.8 jev calls, 2.2 lookups | 1 jev call | 1 Vela call |
| Work, a question seen before | nothing (kept) | 1 jev call | 1 Vela call |
| Shows where the answer came from | yes | no | no |
- Unsure means the system declined to answer because it was not sure enough. It is not counted as right or as wrong.
- Right, of those answered leaves the unsure ones out: of the questions it did answer, the share it got right.
- Time is the median wait, from sending the question to getting the answer: half the answers came faster, half slower. ask keeps the answers it has worked out, so a question it has seen before comes back faster and needs no new work.
- Work counts what one answer needed, on average: calls to a small helper model, calls to jev, and lookups in SEC records. jev and Vela 2.0 each answer in a single call, with nothing looked up.
Where each one wins
jev does well when the words of the question are enough, as with a made-up name or a ticker that belongs to someone else. ask wins wherever the answer depends on a number, such as whether a biotech's float is above the size in the question. It looks the number up instead of guessing.
Looking up takes time. On a new question jev answers first. On a question it has seen before, ask does, because it kept the answer.
Vela 2.0 gets 43 right. It never answers that the question itself is wrong, so it misses every made-up company and every ticker that belongs to another company. It also calls 10 of 23 companies that are not biotechs biotech. On the close calls it beats jev, 7 to 3.
What can still go wrong
- “Biotech” is a line someone has to draw. Here it means SEC industry codes 2833 to 2836 or 8731, which counts specialty drug makers in. Most of ask's misses are about where that line falls.
- This is a small test. One kind of question, one account, 92 questions, measured on 6 and 7 October 2026. Other kinds of questions could come out differently.
- A brand new account starts slower. The times for new questions were measured with the SEC data already loaded, so a new account's first questions take longer.
- Vela 2.0's time flatters it a little. It ran on a laptop with no network in between, while the times for jev and ask include the trip over the internet.
- jev's and ask's times are medians of the round trip from the same Apple M1 Pro over the internet; on the server a new question took 612 ms and a repeat 1 ms.
- Vela 2.0 is the 0.8B model from vllm-sr, given exactly the request jev gets. It ran locally on an Apple M1 Pro: 10-core CPU (8 performance, 2 efficiency), 16-core GPU (used, through PyTorch's Metal backend), 16 GB unified memory, in 32-bit floats. The 0.3B got 23 right in 77 ms. The 4B and 9B need more than the 16 GB of memory and were not run.
More head-to-head evals
| Measure | Physical Agent | Qwen3.5-4B |
|---|---|---|
| LIBERO frames answered with a tool call | 400 / 400 | 18 / 400 |
| Target movement per cm of object movement (median) | 0.02 cm | — |
| Measure | Particle-1 | DeepSeek V4.1 Flash | GLM-5.3-Flash |
|---|---|---|---|
| Rows right, blind | 91.2% | 89.4% | 90.8% |
| Median seconds per run | 375 | 315 | 144 |
| Measure | GLM-5.3 | DeepSeek V4.1 Flash |
|---|---|---|
| Right, of 75 | 75 | 71 |
| Seconds a question, median | 3.9 | 1.8 |
| Measure | First run | Best since |
|---|---|---|
| Landscape, seconds | 937 | 233 |
| Memo, seconds | 759 | 602 |
| Measure | Qwen3 235B | GLM-5.2 | Gemini 3.8 Flash | Claude Opus 5 |
|---|---|---|---|---|
| Seconds to the plan, median | 27 | 48 | 57 | 94 |
| Plans written, of three runs | 2 | 3 | 3 | 3 |
| Measure | SubQ | jev | Decider | Nimble |
|---|---|---|---|---|
| Found, mean of five columns at section length | 64% | 65% | 66% | 68% |
| Mechanism of action, found at section length | 57% | 51% | 61% | 54% |
| Measure | ask | Apollo | treg |
|---|---|---|---|
| Usable leads | 13 of 40 | 7 of 40 | 6 of 28 |
| Cost per usable lead | $0.037 | $0.27 | $0.23 |
| Measure | ask with jev | ask with Decider |
|---|---|---|
| Questions right, of 92 | 89 | 89 |
| Wrong answers | 1 | 2 |
| Measure | jev | ask |
|---|---|---|
| Our 370 clinical decisions, right | 94.6% | 93.0% |
| PubMedQA, right (best EdgeEvals scored: 62.2%) | 77.0% | 74.4% |
| Measure | ask | Claude Opus 5.5 |
|---|---|---|
| Questions right, of 95 | 59 | 63 |
| Small companies right, of 20 | 13 | 9 |
| Measure | realestate-us | Best of four assistants |
|---|---|---|
| Floor-plan rents right, of 26 | 26 | 23 |
| Amenities claimed the listing doesn't state | 12 | 129 |
| Measure | Trained localiser | Best of four frontier models |
|---|---|---|
| Mean overlap with the defect (IoU, 0 to 1) | 0.421 | 0.109 |