SubQ against jev on biotech diligence memos: finding the sentences a column is about

Both are decision models. Neither writes text. Each is given something to read and a question with fixed answers (here: does this passage state the mechanism of action of one of the company's own drugs, yes or no) and answers with a probability for each, a number from 0 to 1 for how likely it thinks that answer is right.

jev is TypeSafe's, and it already makes the small decisions inside hedwigai's research tools. SubQ's Decisions API takes the same kind of request, so the same 8,357 questions went to both, word for word.

Why judge one sentence at a time? Because the answer to a column is rarely in one place. A drug's mechanism shows up in the overview, the pipeline table and the paragraph on each product. A tool that picks the best whole section of the memo, as hedwigai's section search does, gets one or two of those places and misses the rest.

Ten diligence memos on biotech companies, written by hedwigai research runs. A memo runs to 228 sentences on average (148 to 322), counting each table row and list item as one: finances, recent events, the drug pipeline, the science, the market, competitors and patents.

For each of five columns, a person marked the sentences that state it for the company's own drugs, before either model saw a memo. None of the memos has a heading for mechanism of action, so no section title gives it away.

Each model's probabilities rank a memo's sentences. We took as many of its top sentences as the whole sections held, so the three are compared at the same length. Found is the share of marked sentences among those chosen; on target is the share of chosen sentences that were marked.

A filled cell should not need a fixed number of sentences. So we also kept every sentence each model gave better-than-even odds. On mechanism of action, SubQ kept 30.3 sentences a memo and found 95% of the marked ones; jev kept 27.9 and found 78%.

We also put 6,296 questions to both on columns no one marked (indication, development stage, efficacy results, safety, dosing and others) on three more memos. They gave the same answer to 98% of them, mostly the easy no's. Of the sentences either one said yes to, they both did on 253 of 383.

jev read 2,263,852 tokens of input for the scored run, and its responses reported a cost of $0.10. SubQ read 1,697,123; at its preview price of $0.05 a million input tokens that is $0.08. A token is a piece of a word; the two count them differently, which is why the same questions came to different totals.