How Instruments Are Developed

Why promise is not enough

Building an instrument is the fast part. Proving where it can be trusted is the slow one, and this page is about that work: characterization.

An instrument earns promotion by being characterized. Looking promising is grounds to investigate it, and grounds for nothing beyond that.

Promise is a reason to investigate an instrument. Relying on one requires more.

This page describes how instruments move through the Realis laboratory: why stages exist, what has to be true for an instrument to advance, why some instruments spend a long time in development, and why some are retired instead. The Current Capabilities page shows where each instrument stands today. This page explains the discipline behind that placement.

Why Stages Exist

A laboratory that offered every promising instrument the moment it looked useful would be indistinguishable from one that offered nothing but confidence.

The stages exist to separate what an instrument appears to do from what it has been shown to do. An instrument in early development often produces results that look correct, and the appearance of correctness is exactly what a laboratory cannot take as proof of it. The distance between looking right and being shown right is where a laboratory either earns trust or forfeits it, and the stages exist to keep that distance in plain view, so no confident presentation can paper over it.

Each stage answers one question about the instrument: how much is known about where its readings can be relied upon, and where they cannot. An available instrument has a characterized boundary; a developing instrument does not yet have one. A stage is not a marketing tier but a statement about the current state of knowledge concerning the instrument itself.

Why Characterization Comes Before Adoption

A laboratory answers to a different responsibility from an implementation organization, and that difference is the reason this discipline exists.

An implementation organization succeeds by putting a capability into use. A laboratory succeeds by knowing what a capability can and cannot do before anyone relies on it. Conflating those two obligations is how instruments come to be trusted before they have earned it, so Realis takes the laboratory's side of that line: it delays promotion until characterization is sufficient, even when an instrument would be useful sooner.

Two decisions, two partiesThe laboratory decides whether it has earned the right to offer an instrument. The user decides whether to adopt it. Separating those two decisions is what lets the user trust the offer, because the laboratory has not made the adoption decision on the user's behalf by overstating what the instrument is ready to do.

A laboratory draws that separation where a vendor collapses it. The laboratory's task is to be honest about the instrument, and what the user does with an honestly described instrument is theirs to decide.

What Characterization Means

CharacterizationTo characterize an instrument is to establish where it works, where it fails, and how to tell the difference from the outside. An instrument becomes understood not when it produces a result but when the conditions under which that result can be trusted have themselves been mapped.

This is the work that occupies most of an instrument's life in the laboratory. It goes slower than development and shows less, and it is the part that separates an instrument from a good idea.

Characterizing an instrument means finding the edge of its competence deliberately, rather than waiting for the edge to reveal itself in use. The purpose is to discover the instrument's true operating envelope, which means constructing the cases most likely to break it, not the cases most likely to flatter it. It means treating a result that disappoints the instrument's developer as more informative than one that confirms what the developer hoped. An instrument that has only ever been shown succeeding has not been characterized. It has been advertised.

Why an Instrument Advances

An instrument advances from development to availability when its boundary is known well enough that its readings can be relied upon within stated limits, and those limits can be communicated honestly to the person using it.

Evidence advances an instrument. Time in development does not, demand does not, and the laboratory's own wish for the instrument to be ready does not. An instrument that has spent a long time in development has not stalled; the laboratory is holding it at the stage its characterization supports. When pressure builds for an instrument to be ready before its characterization justifies it, the correct response is to leave it where it is.

The promotion ruleAn instrument moves to availability only when the laboratory can state, honestly and specifically, what the instrument can do, what it cannot yet do, and how a user can tell which case they are in. Until that statement can be made, the instrument waits in development, however useful it would be if it were ready.

Why Some Instruments Are Retired

Not every instrument that enters the laboratory advances. Some are retired.

An instrument is retired when characterization shows its boundary is too narrow to be useful, or that its readings cannot be reliably distinguished from the failure it was meant to detect, or that a different instrument does the same work with a better-understood boundary. Retirement is the laboratory working as intended. The discipline is doing its job. An instrument that survives characterization has earned its place; an instrument that does not survive it was never going to be trustworthy in the field, and finding that out in the laboratory is the outcome the laboratory exists to produce.

A laboratory that never retires an instrument is not succeeding at everything. It is failing to test.

What Characterization Involves

The philosophy is built on specific practice. An instrument's characterization runs through frozen protocols fixed before the instrument is applied, so the standard cannot be adjusted to fit the result. Predictions go on record before observation. Negative controls are built in deliberately, so that an instrument reporting structure everywhere, including where none exists, gets exposed instead of credited. Where an instrument's reliability depends on being usable by more than its developer, it faces independent and blind evaluation, because an instrument only one person can operate is not yet a laboratory capability.

The governing disciplineThe apparatus exists to prevent the laboratory from fooling itself. Every element of it, the frozen protocol, the preregistered prediction, the negative control, the blind evaluation, answers one question: has this instrument been given a real chance to fail, and did it survive?

The full treatment of this apparatus, including how each protection is built and why, is set out in MR-001, the laboratory's methodological reference, for readers who want to examine the method itself.

For the method in full: The complete development and characterization apparatus is specified in MR-001 — The Development and Characterization of Institutional Instruments, which documents each protection, why it is necessary, and a worked example of the apparatus applied to one of the laboratory's own instruments.

For where each instrument stands today: See Current Capabilities for the current stage of every instrument in the laboratory.

For the conditions these instruments evaluate: See The Corpus by Concept.

Banner: A profilometer draws its stylus across a machined surface, measuring the microscopic irregularities that a visual inspection cannot register. Chosen for structural resonance with the laboratory's method: a surface that looks finished is not thereby shown to be, and the difference is legible only to an instrument that measures it directly. The stylus reads what the eye reports as smooth.