Aircraft records completeness · the method
Aircraft records completeness: an evaluation method (no models scored yet)
When an aircraft changes hands, or goes back to its owner at the end of a lease, it comes with a records package: the paperwork that proves the aircraft's history. A records team reads it for gaps, such as a part whose history cannot be traced or a required safety fix with no proof it was done. A gap they miss costs money when the aircraft is handed back.
Unlike most document work, this job has right and wrong answers. A part's history is either unbroken or it has a gap. That means an AI system doing it can be scored. This page sets out how: build practice packages, plant known mistakes in them, and count how many a system catches and how many it makes up.
No model has been scored yet. This page is the method. The numbers on it come from one worked example, to show what a score will look like: on that practice package, 70 of 82 planted mistakes were caught (85%), and 10 findings pointed at mistakes that were not there.
The two numbers we score
A few terms first. A life-limited part is one, such as an engine disc, that must be retired after a set number of flights or hours, so its full history from manufacture has to be on paper. An airworthiness directive is a safety fix the regulator makes compulsory; the record must show how it was done, who signed for it, and the work order that closed it. A run is one go by an AI system over one package.
Every run gets two numbers, never one. hedwigai, which makes workbooks (documents such as reports, trackers and lists that AI research runs write and keep current), scores its own workbook tests the same way.
Coverage is how much of the checking the run actually did: of the checks that had something to judge, the share it got right. A check with nothing to judge counts as neither passed nor failed. Otherwise an empty package would earn free marks.
Quality is how much of what the run claimed is really there: the share of its findings that point to an actual page in the package. A finding that cannot be traced to a page is not a finding, it is a sentence.
The two fail in opposite directions, which is why they are kept apart. A run that flags every possible problem scores high coverage and low quality. A run that flags only what it is sure of scores the reverse. Averaging them into one figure hides both. A run that claimed nothing gets no quality score at all, not a zero: it made no false claims, and ranking an honest empty run below a confident wrong one would teach the wrong lesson.
In practice the two become two plain questions. Recall: of the mistakes we planted, how many did it catch? Precision: of the mistakes it reported, how many were real? A missed gap costs money when the aircraft is handed back. A made-up gap costs you the analyst, who stops opening the tool after the third false alarm.
The eight areas a run checks
A records review splits into eight areas, listed below. Each area has a set of individual checks, and any check can drop out when there is nothing for it to judge.
Much of what counts as complete is set by the lease, not by the regulator. A lease may ask for a longer history, a set digital format, English translations, proof of who owned a part, statements about past incidents, the owner's written consent, or records kept for longer. Those are terms of one contract, not rules for every aircraft, and a test that treated them as universal would mark a correct package down.
| Method | What it asks | Inapplicable when |
|---|---|---|
| llp-continuity | Every life-limited part traced back to birth; removals matched to installations; cycle counts reconcile | No LLPs in scope for the package |
| ad-compliance | Each directive carries an applicability check, a method of compliance, a signatory, and a linked closed work order | Directive does not apply to this serial |
| sb-effectivity | Service bulletins checked against the actual serial and configuration | No service bulletins embodied |
| repair-approval | Every repair carries its approval basis; no undocumented dirty fingerprint | No repairs in the package |
| release-certification | Release documents present and valid; dual release where the lease requires it | Lease requires single release only |
| supersedure-integrity | Superseded revisions marked as superseded rather than deleted | No superseded documents present |
| utilisation-reconciliation | Technical log, utilisation records and CAMO data agree on times and cycles | Only one time source in the package |
| lease-delta | Package measured against the return conditions of the governing lease | No lease document supplied |
Some words in the table: LLP is a life-limited part. A service bulletin is a manufacturer's recommended change, which matters only if it fits this exact aircraft. A repair approval is the engineering sign-off that says a repair is allowed; the table's “undocumented dirty fingerprint” is a repair whose approval the records do not show. A release certificate is the signed document that returns a part or aircraft to service after work. CAMO is the organisation that manages the aircraft's ongoing airworthiness, and a cycle is one take-off and landing.
A folder of files for one aircraft cannot, on its own, show that anything is complete. What gets judged is each individual record, with its identifiers, kind of document, date, issuer, revision, hours and cycles, related work order, review status and exceptions. A check that cannot name the record it is talking about has not found anything.
What a score will look like
A worked example, not a measurement. The figures below are one practice package, built by us rather than taken from a real aircraft, scored area by area. They show the shape of a score and how the rules above apply. No model is named because none has been scored on these packages yet.
In the chart, each bar is one area's coverage, from 0 (no checks passed) to 1 (all passed). The dashed line is the coverage for the package as a whole. In the table, “n/a” is checks with nothing to judge, “claims” is findings the run reported, and “quality” is the share of those that point to a real page.
| method | criteria | n/a | scored | met | coverage | claims | quality |
|---|---|---|---|---|---|---|---|
| llp-continuity | 14 | — | 14 | 13 | 0.929 | 21 | 0.952 |
| ad-compliance | 22 | 3 | 19 | 17 | 0.895 | 26 | 0.923 |
| sb-effectivity | 11 | 2 | 9 | 8 | 0.889 | — | — |
| repair-approval | 9 | — | 9 | 7 | 0.778 | 14 | 0.857 |
| release-certification | 7 | 1 | 6 | 6 | 1.000 | 5 | 1.000 |
| supersedure-integrity | 6 | — | 6 | 5 | 0.833 | 6 | 0.833 |
| utilisation-reconciliation | 12 | 1 | 11 | 8 | 0.727 | 13 | 0.846 |
| lease-delta | 10 | 10 | 0 | — | — | — | — |
| the package | 91 | 17 | 74 | 64 | 0.865 | 85 | 0.906 |
Two rows show the rules at work. lease-delta has all ten of its checks set aside, because no lease was supplied. It does not score at all. Writing a zero there would say the run failed at something nobody asked it to do. sb-effectivity made no claims, so it has no quality score rather than a zero.
The package numbers count every check together, rather than averaging the eight areas. An average would give an area with six checks the same weight as one with twenty-two, and the reader would have no way of knowing.
The planted mistakes, and how many were caught
Each practice package has seeded defects: mistakes deliberately planted to see if they are caught, drawn from a fixed list of kinds. Because we planted them, we know exactly what should be found before the run starts. Recall is counted against that list, never against whatever the run happened to report, so a run cannot look better by finding less.
In the chart, the outline is how many of each kind were planted and the solid part is how many were caught, with a citation that points to a real page. Made-up findings are not part of what was planted, so they are drawn in the lighter shade outside the bar.
| defect | seeded | found | missed | invented | recall |
|---|---|---|---|---|---|
| Birth certificate missing for an LLP | 12 | 11 | 1 | 1 | 0.92 |
| Back-to-birth gap after a shop visit | 9 | 7 | 2 | — | 0.78 |
| Directive with no method of compliance | 14 | 13 | 1 | 2 | 0.93 |
| Directive with no signatory | 8 | 8 | — | 1 | 1.00 |
| Bulletin embodied and not recorded | 6 | 4 | 2 | — | 0.67 |
| Repair with no approval basis | 10 | 9 | 1 | 3 | 0.90 |
| Single release where the lease requires dual | 5 | 5 | — | — | 1.00 |
| Superseded revision deleted rather than marked | 7 | 5 | 2 | 1 | 0.71 |
| Technical log and CAMO disagree on cycles | 11 | 8 | 3 | 2 | 0.73 |
| all kinds | 82 | 70 | 12 | 10 | 0.85 |
On this package that is 85% recall and 88% precision: 12 planted mistakes were missed and 10 findings named something that was not there. The two failures are not interchangeable. The 12 missed gaps are the ones that surface when the aircraft is handed back, with it sitting on the ground; the 10 made-up ones are the ones that cost you the analyst.
Where the test packages come from
Real records packages are confidential, enormous and cannot be published. So the test packages are built the other way round. We start from clean practice (synthetic) packages put together from public material: directive texts, standard form layouts, logbook formats and flight-hour records. Then we plant mistakes in them from the fixed list of kinds.
That gives us an exact answer key, as many packages as we want, a dial for how hard they are, and no NDA (a non-disclosure agreement, the confidentiality contract real records come under). It also makes coverage exact: the list of planted mistakes is known before the run, so what counts as complete cannot be argued about afterwards.
Alongside them sits a small gold set: real packages from a design partner, checked by people so the right answers are known, held under NDA and never published. The practice packages test whether a system can reason about the rules. The real ones test the whole job, including the case where a page is a photograph of a handwritten entry in a third language and the text recognition (OCR) has misread the serial number.
| Corpus | What it is | Volume | Ground truth | Published |
|---|---|---|---|---|
| Synthetic packages | Assembled from public primitives, with defects injected from the fixed taxonomy | unlimited | exact | yes |
| Design partner gold set | Real packages, held under NDA and never published. Only the scores leave it | small | reviewed, with judgement calls marked | scores only |
These produce two different numbers and must never be quoted as one. A figure at 0.9 on synthetic packages can sit far below that on real ones, and saying so is the difference between a benchmark and a brochure. Every score published here will name which corpus it came from in the same breath as the number.
What can still go wrong
- Nothing has been scored yet. Every figure on this page is a worked example on one practice package. It shows the shape of a score, not how good any model is.
- Practice packages are easier than real ones. A score of 0.9 on practice packages can sit far below that on real ones, where pages are handwritten, photographed and in other languages.
- The real gold set is small, and its answer key includes judgement calls, which are marked as such.
- Leases differ. Much of what counts as complete comes from one contract, so a check that applies to one package may not apply to the next. Checks with nothing to judge are set aside rather than scored.