Blog
hedwigaihedwigaihedwigai

Reference parsing · open models

Can a 32M-parameter model parse references better than GROBID?

A reference string holds a paper’s authors, title, journal, year, volume, issue, pages and DOI. A reference parser has to pull each one out, whatever citation style the author used. GROBID has done this job in open source for years. It is accurate, and it runs as a Java server.

We trained two small models for the same job and named them citeparse. They read one reference at a time and return CSL-JSON or BibTeX. They run on a CPU, and both are open under Apache-2.0.

What it does

What does citeparse return for a reference?

Install it from GitHub, then pass a file with one reference on each line.

pip install git+https://github.com/un-seen/citeparse
citeparse refs.txt            # CSL-JSON, one record per line
citeparse refs.txt --bibtex   # BibTeX

This reference goes in.

K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition,"
in Proc. CVPR, 2016, pp. 770-778. doi:10.1109/CVPR.2016.90

This BibTeX comes out. The model labels each word, and the package splits the names into family and given parts.

@inproceedings{He2016_1,
  author = {He, K. and Zhang, X. and Ren, S. and Sun, J.},
  title = {Deep residual learning for image recognition},
  booktitle = {CVPR},
  year = {2016},
  pages = {770-778},
  doi = {10.1109/CVPR.2016.90}
}

How we scored it

How do you score a parser without labelling references by hand?

Publishers deposit their papers’ reference lists with Crossref. Many of those references link to the DOI of the work they cite. That cited work has its own Crossref record, with its title, journal, volume, issue and first page. So the record is the answer key, and nobody has to label anything.

We took 700 such references and called them the wild set. None of the cited DOIs appear in our training data. A field counts as right only when it matches the record exactly, after both are lowercased and cut to letters and digits. A journal may also match its short title.

We ran GROBID 0.8.1 with its default CRF models on the same 700 strings. We gave it the same DOI rule that citeparse uses, so the DOI column is equal and the gap left comes from the models.

Results

Which parser got more real references right?

citeparse-150m got 584 of the 700 references fully right. citeparse-32m got 562, and GROBID got 498. The smaller model beat GROBID on every field except the DOI, where the two are within 0.006.

References with every field right, of the 700 in the wild set
citeparse-150m584 of 700
citeparse-32m562 of 700
GROBID 0.8.1498 of 700
Share of references with this field that got it right, where 1.000 means all of them

Title (543 of the 700 references have it)

citeparse-150m0.934
citeparse-32m0.921
GROBID 0.8.10.882

Container (journal or proceedings) (400 of the 700 references have it)

citeparse-150m0.878
citeparse-32m0.858
GROBID 0.8.10.805

Volume (596 of the 700 references have it)

citeparse-150m0.966
citeparse-32m0.956
GROBID 0.8.10.908

Issue (208 of the 700 references have it)

citeparse-150m0.861
citeparse-32m0.841
GROBID 0.8.10.803

First page (569 of the 700 references have it)

citeparse-150m0.982
citeparse-32m0.968
GROBID 0.8.10.953
The 700 references in the wild set, scored field by field. GROBID was given the same DOI rule as citeparse.
ParserAll fieldsReferences fully rightTitleContainerVolumeIssueFirst pageDOI
GROBID 0.8.1 (CRF)0.91171.1%0.8820.8050.9080.8030.9530.994
citeparse-32m0.93980.3%0.9210.8580.9560.8410.9680.988
citeparse-150m0.95283.4%0.9340.8780.9660.8610.9820.994

Where GROBID wins

Does citeparse beat GROBID on GROBID's own data?

It does not. We also held out citations from GROBID’s own training corpus. On those, GROBID scores 0.943 over all fields. citeparse-150m scores 0.900 and citeparse-32m scores 0.882.

Those citations come from the same papers and styles that GROBID was trained on. The wild set comes from the reference lists publishers deposit today. We report both results, and we think the wild set is closer to what a user hands a parser.

DOIs

Why does a regular expression read the DOI?

A DOI has a fixed shape. It starts with 10., then a number, a slash and a suffix. The model often labelled only a piece of a doi.org link, such as the “https” or the “10.”. On the 162 references in the wild set that carry a DOI, the 150M model alone scored 0.833.

citeparse now reads the DOI with a regular expression and keeps the model’s span only when the text has no well-formed DOI. That lifts the DOI score to 0.994.

Speed

How fast is it on a CPU?

On 8 CPU threads, citeparse-32m parses about 118 references a second. citeparse-150m parses about 20 a second. A thesis bibliography of 200 references takes under 2 seconds with the smaller model.

Training

What was it trained on?

Both models start from the Ettin encoders, jhu-clsp/ettin-encoder-32m and jhu-clsp/ettin-encoder-150m. We trained each one for two epochs to label every word with one of 16 fields.

The data has two parts. The first is GROBID’s citation corpus, released under Apache-2.0. We removed its evaluation files from training, along with every training reference whose text also appears in them. The second is Crossref records, released under CC0, written out in 11 citation styles with typing noise.

The code to rebuild the data, train both models and rerun both benchmarks is in the GitHub repository.

A parser reads what a reference says. It does not tell you whether the work exists. To check that every reference in a draft is real, use the free reference checker.

  • All figures were measured on 19 November 2026.
  • The wild set lists 700 references. Of them, 660 have a year, 596 a volume, 569 a first page, 543 a title, 400 a container, 208 an issue and 162 a DOI.
  • “All fields” is the share of the 3,138 fields in the set that a parser got right. A container also counts as right when it matches the journal’s short title in its Crossref record.
  • Both models label English-language citation styles. “et al.” comes out as a literal author.

More head-to-head evals

Does Physical Agent aim its tool calls at the object?Physical Agent writes a robot tool call on 400 of 400 frames, but moving the object 16 cm leaves its targets where they were.
MeasurePhysical AgentQwen3.5-4B
LIBERO frames answered with a tool call400 / 40018 / 400
Target movement per cm of object movement (median)0.02 cm—
Read the comparison
Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.
Measureask with jevask with Decider
Questions right, of 928989
Wrong answers12
Read the comparison
Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.
Measurejevask
Our 370 clinical decisions, right94.6%93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)77.0%74.4%
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.
Measureaskjev
Right, of the questions answered98%79%
Wrong answers, of 92219
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison