Reference parsing · open models
Can a 32M-parameter model parse references better than GROBID?
A reference string holds a paper’s authors, title, journal, year, volume, issue, pages and DOI. A reference parser has to pull each one out, whatever citation style the author used. GROBID has done this job in open source for years. It is accurate, and it runs as a Java server.
We trained two small models for the same job and named them citeparse. They read one reference at a time and return CSL-JSON or BibTeX. They run on a CPU, and both are open under Apache-2.0.
references fully right, citeparse-150m
More real references parsed whole
GROBID 0.8.1 got 498 of the same 700 references fully right. citeparse-32m got 562.
Read how we measured itreferences parsed on a CPU, citeparse-32m
Small enough for a laptop
The 32M-parameter model parses about 118 references a second on 8 CPU threads. The 150M model parses about 20.
Read how we measured itGROBID on its own held-out citations
GROBID wins on its own data
On citations held out from GROBID's training corpus, GROBID scores 0.943 and citeparse-150m scores 0.900.
Read how we measured itDOI field, citeparse-150m
A regex reads the DOI
The model alone scored 0.833 on DOIs. A regular expression lifts that to 0.994, the same as GROBID with the same rule.
Read how we measured itWhat it does
What does citeparse return for a reference?
Install it from GitHub, then pass a file with one reference on each line.
pip install git+https://github.com/un-seen/citeparse citeparse refs.txt # CSL-JSON, one record per line citeparse refs.txt --bibtex # BibTeX
This reference goes in.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. CVPR, 2016, pp. 770-778. doi:10.1109/CVPR.2016.90
This BibTeX comes out. The model labels each word, and the package splits the names into family and given parts.
@inproceedings{He2016_1,
author = {He, K. and Zhang, X. and Ren, S. and Sun, J.},
title = {Deep residual learning for image recognition},
booktitle = {CVPR},
year = {2016},
pages = {770-778},
doi = {10.1109/CVPR.2016.90}
}How we scored it
How do you score a parser without labelling references by hand?
Publishers deposit their papers’ reference lists with Crossref. Many of those references link to the DOI of the work they cite. That cited work has its own Crossref record, with its title, journal, volume, issue and first page. So the record is the answer key, and nobody has to label anything.
We took 700 such references and called them the wild set. None of the cited DOIs appear in our training data. A field counts as right only when it matches the record exactly, after both are lowercased and cut to letters and digits. A journal may also match its short title.
We ran GROBID 0.8.1 with its default CRF models on the same 700 strings. We gave it the same DOI rule that citeparse uses, so the DOI column is equal and the gap left comes from the models.
Results
Which parser got more real references right?
citeparse-150m got 584 of the 700 references fully right. citeparse-32m got 562, and GROBID got 498. The smaller model beat GROBID on every field except the DOI, where the two are within 0.006.
Title (543 of the 700 references have it)
Container (journal or proceedings) (400 of the 700 references have it)
Volume (596 of the 700 references have it)
Issue (208 of the 700 references have it)
First page (569 of the 700 references have it)
| Parser | All fields | References fully right | Title | Container | Volume | Issue | First page | DOI |
|---|---|---|---|---|---|---|---|---|
| GROBID 0.8.1 (CRF) | 0.911 | 71.1% | 0.882 | 0.805 | 0.908 | 0.803 | 0.953 | 0.994 |
| citeparse-32m | 0.939 | 80.3% | 0.921 | 0.858 | 0.956 | 0.841 | 0.968 | 0.988 |
| citeparse-150m | 0.952 | 83.4% | 0.934 | 0.878 | 0.966 | 0.861 | 0.982 | 0.994 |
Where GROBID wins
Does citeparse beat GROBID on GROBID's own data?
It does not. We also held out citations from GROBID’s own training corpus. On those, GROBID scores 0.943 over all fields. citeparse-150m scores 0.900 and citeparse-32m scores 0.882.
Those citations come from the same papers and styles that GROBID was trained on. The wild set comes from the reference lists publishers deposit today. We report both results, and we think the wild set is closer to what a user hands a parser.
DOIs
Why does a regular expression read the DOI?
A DOI has a fixed shape. It starts with 10., then a number, a slash and a suffix. The model often labelled only a piece of a doi.org link, such as the “https” or the “10.”. On the 162 references in the wild set that carry a DOI, the 150M model alone scored 0.833.
citeparse now reads the DOI with a regular expression and keeps the model’s span only when the text has no well-formed DOI. That lifts the DOI score to 0.994.
Speed
How fast is it on a CPU?
On 8 CPU threads, citeparse-32m parses about 118 references a second. citeparse-150m parses about 20 a second. A thesis bibliography of 200 references takes under 2 seconds with the smaller model.
Training
What was it trained on?
Both models start from the Ettin encoders, jhu-clsp/ettin-encoder-32m and jhu-clsp/ettin-encoder-150m. We trained each one for two epochs to label every word with one of 16 fields.
The data has two parts. The first is GROBID’s citation corpus, released under Apache-2.0. We removed its evaluation files from training, along with every training reference whose text also appears in them. The second is Crossref records, released under CC0, written out in 11 citation styles with typing noise.
The code to rebuild the data, train both models and rerun both benchmarks is in the GitHub repository.
A parser reads what a reference says. It does not tell you whether the work exists. To check that every reference in a draft is real, use the free reference checker.
- All figures were measured on 19 November 2026.
- The wild set lists 700 references. Of them, 660 have a year, 596 a volume, 569 a first page, 543 a title, 400 a container, 208 an issue and 162 a DOI.
- “All fields” is the share of the 3,138 fields in the set that a parser got right. A container also counts as right when it matches the journal’s short title in its Crossref record.
- Both models label English-language citation styles. “et al.” comes out as a literal author.
More head-to-head evals
| Measure | Physical Agent | Qwen3.5-4B |
|---|---|---|
| LIBERO frames answered with a tool call | 400 / 400 | 18 / 400 |
| Target movement per cm of object movement (median) | 0.02 cm | — |
| Measure | Particle-1 | DeepSeek V4.1 Flash | GLM-5.3-Flash |
|---|---|---|---|
| Rows right, blind | 91.2% | 89.4% | 90.8% |
| Median seconds per run | 375 | 315 | 144 |
| Measure | GLM-5.3 | DeepSeek V4.1 Flash |
|---|---|---|
| Right, of 75 | 75 | 71 |
| Seconds a question, median | 3.9 | 1.8 |
| Measure | First run | Best since |
|---|---|---|
| Landscape, seconds | 937 | 233 |
| Memo, seconds | 759 | 602 |
| Measure | Qwen3 235B | GLM-5.2 | Gemini 3.8 Flash | Claude Opus 5 |
|---|---|---|---|---|
| Seconds to the plan, median | 27 | 48 | 57 | 94 |
| Plans written, of three runs | 2 | 3 | 3 | 3 |
| Measure | SubQ | jev | Decider | Nimble |
|---|---|---|---|---|
| Found, mean of five columns at section length | 64% | 65% | 66% | 68% |
| Mechanism of action, found at section length | 57% | 51% | 61% | 54% |
| Measure | ask | Apollo | treg |
|---|---|---|---|
| Usable leads | 13 of 40 | 7 of 40 | 6 of 28 |
| Cost per usable lead | $0.037 | $0.27 | $0.23 |
| Measure | ask with jev | ask with Decider |
|---|---|---|
| Questions right, of 92 | 89 | 89 |
| Wrong answers | 1 | 2 |
| Measure | jev | ask |
|---|---|---|
| Our 370 clinical decisions, right | 94.6% | 93.0% |
| PubMedQA, right (best EdgeEvals scored: 62.2%) | 77.0% | 74.4% |
| Measure | ask | Claude Opus 5.5 |
|---|---|---|
| Questions right, of 95 | 59 | 63 |
| Small companies right, of 20 | 13 | 9 |
| Measure | realestate-us | Best of four assistants |
|---|---|---|
| Floor-plan rents right, of 26 | 26 | 23 |
| Amenities claimed the listing doesn't state | 12 | 129 |
| Measure | ask | jev |
|---|---|---|
| Right, of the questions answered | 98% | 79% |
| Wrong answers, of 92 | 2 | 19 |
| Measure | Trained localiser | Best of four frontier models |
|---|---|---|
| Mean overlap with the defect (IoU, 0 to 1) | 0.421 | 0.109 |