Metadata

Metadata is structured information describing data: what the variables are, in what units, over what extent, produced by what, under what licence. It is what allows a file to be interpreted by someone with no access to the person who made it — which, after a few years, includes the person who made it.

The framing I find most useful is that metadata is not documentation of the data. It is part of the data. An array of numbers without units and coordinates is not usable data with poor documentation; it is not usable data.

The layers, which are often confused

• Structural — dimensions, variable names, types, shapes. netCDF handles this at the format level.
• Semantic — what the variables mean. t2 is structure; “air temperature at 2 m above the surface, in kelvin” is semantics. CF conventions exist for exactly this layer, through standard names.
• Discovery — title, authors, abstract, keywords, DOI. What a catalogue indexes.
• Provenance — what produced this, from what, with which version and settings.
• Quality — uncertainty, known issues, validation status.

Compliance work usually addresses the first three. The last two are where the gaps are, and they are the layers that determine whether data can actually be reused rather than merely opened.

Why it decays

Metadata is written once, at the point where the producer knows everything and therefore finds most of it too obvious to record. It is read years later by someone who knows nothing. The asymmetry guarantees under-specification, and no amount of exhortation fixes it — which is why the mechanisms that work are the ones removing the judgment call: required fields, automated validation, tooling that writes provenance without being asked.

This is the reasoning behind community standards. ATMODAT and the UC2 data standard both work by making a specific list of attributes mandatory and machine-checkable. The list is more valuable than any individual item on it, because it converts “describe your data well” — advice nobody can act on consistently — into a check that passes or fails.

The second-order benefit is that machine-checkable metadata enables things beyond human reading: automated aggregation across sources, knowledge graph construction, and the FAIR goal of data reusable without human negotiation. None of that works on prose in a README.

See also: FAIR for the principles, CF conventions for the semantic layer, provenance, and datasets for the unit being described.