Hugging Face proposes model integration contracts for Transformers

Hugging Face has proposed Model Integration Contracts, or MICs, as a standard for model-specific regression checks in Transformers’ continuous-integration system. They are designed to catch architecture-specific failures on pull requests without running the library’s full GPU-heavy integration suite for every code change.

A contract has three parts: a pinned, reduced-size model fixture; a runnable Markdown example that exercises a task such as generation or a training step; and human-reviewed output baselines. For each pull request, the check selects contracts based on the files changed, runs their usage paths against the fixtures and compares the results with the recorded baselines.

The fixture comes from a real checkpoint whose configuration is reduced while preserving the architecture’s different layer types. Hugging Face says the process leaves tokenizer and processor files unchanged, keeps the original weight-file format, validates finite outputs, loss and gradients, and publishes a pinned snapshot so CI does not rebuild it for every pull request. A fleet dataset separately pins each contract revision and the package versions used to record its baselines.

The proposal grew out of a UMT5 attention bug. A July 30 Transformers change enabled scaled dot-product attention for T5 and related models. According to Hugging Face’s retrospective, the SDPA path left UMT5 decoder self-attention non-causal during training, letting a token attend to later tokens that should have been hidden.

A September 28 fix explicitly marked the UMT5 decoder path as causal and added regression tests. When Hugging Face replayed frozen contracts, it found differences from the reviewed UMT5 baseline across 597 commits between the bug’s introduction and its fix. The company reported a 1.22 difference in label log-probability and about a 0.20 difference in training loss.

Hugging Face also reverted the fix on a proof branch to test the mechanism. The selected UMT5 check failed with a 1.22 label-log-probability difference and a 0.204 loss difference. Excluding setup, the selected T5 runner step took 13.4 seconds on a CPU in the company’s measurement.

These are Hugging Face’s backfill and runtime figures, not an independent benchmark. In a separate warm-cache CPU test using experimental builds and fresh processes, the company measured 66 MB of weights for the UMT5 fixture versus 1,227 MB for the real checkpoint. Work times were 0.76 seconds and 1.15 seconds, respectively. Hugging Face cautioned that process startup dominates and that small cached checkpoints may show little local time saving.

MICs do not replace full integration tests. Hugging Face describes fixtures as a fast pull-request regression signal, while tests with real checkpoints remain necessary after merge, in nightly runs and before releases. The company also documented an assisted-decoding defect that both fixture-weight strategies missed but a real Whisper checkpoint exposed.

The proposal targets a library that Hugging Face says supports more than 750 model types and runs more than 88,000 CPU tests on its main branch, plus more than 200,000 GPU tests daily. At publication, the contracts job reported failures without blocking pull requests, coverage remained partial, and Hugging Face had not measured real pull-request catches, false alarms or the cost of reviewing baselines.