Aircraft records completeness

Two numbers, never one, which is the same rule the workbook benchmarks already run on.

Coverage is how much of a method the run carried out: the share of that method's criteria it met, counted only over the criteria that had something to judge. A criterion with nothing to check is neither met nor failed and sits in neither half of the fraction, or an empty package collects free marks.

Quality is how much of what it claimed is really there: the share of its findings whose cited evidence resolves to an actual page in the package. A finding with no resolvable citation is not a finding, it is a sentence.

They fail in opposite directions, which is the whole reason for keeping them apart. A run that flags every possible defect scores high coverage and low quality. A run that flags only what it is certain of scores the reverse. Averaging them into one figure destroys the only information either one carries. A run that claimed nothing gets a null quality, not a zero: a run that made no claims made no false ones, and marking an honest empty run below a boastful wrong one teaches the wrong thing.

For this domain the two land as recall and precision over seeded defects, which is what a records team actually needs to know. A missed gap costs money at redelivery. An invented gap costs you the analyst, who stops opening the tool after the third phantom finding.

Two rows carry the rules. lease-delta has all ten of its criteria inapplicable, because no lease was supplied: it scores neither zero nor one, it does not score, and writing a zero there would report a run as having failed a discipline nobody asked it to apply. sb-effectivity made no claims, so its quality is null rather than zero.

The package numbers are over the whole denominator, not an average of the eight. A mean of fractions would weight a six-criterion discipline the same as a twenty-two-criterion one, and the reader would have no way of knowing it had.

Real records packages are confidential, enormous and unpublishable. So the corpus is built the other way round: clean synthetic packages assembled from public primitives — directive texts, standard form layouts, logbook and utilisation formats — into which defects are injected from a fixed taxonomy.

That buys perfect ground truth, unlimited volume, a difficulty dial, and no NDA. It also buys the thing that makes coverage exactly measurable: the seeded defect list is known before the run, so the denominator is not up for negotiation afterwards.

Alongside it sits a small real gold set from a design partner, held under NDA and never published. The synthetic corpus measures reasoning about compliance logic. The real set measures the whole pipeline, including the part where the document is a photograph of a handwritten entry in a third language and the OCR has mangled the serial.