A silent Transformers bug let UMT5 see future tokens for two months; Hugging Face’s fix is tiny test models

Transformers is plumbing for a huge share of open AI work, supporting more than 750 model types. On July 30, a change switching T5 to PyTorch’s scaled dot-product attention let UMT5’s decoder see future tokens during training. Nothing crashed, and the bug stayed in main until September 28. A silent training bug is worse than a crash: people train on it, ship on it, and don’t know. The test suite already runs more than 88,000 CPU tests on main and over 200,000 GPU tests daily, but full integration tests on real checkpoints are too expensive for every change.

Replaying five months of commits, the UMT5 fixture, 66 MB against a 1,227 MB original, flagged the regression on every one in about 13 seconds on CPU, cheap enough for pull requests rather than nightly GPU runs. It caught the bug only because it used the same default attention path as real users; the older eager path missed it, a reminder that tests off the real code path give false comfort. The post is candid about limits: the job doesn’t block pull requests yet, the false-alarm rate and baseline review burden are unknown, and an assisted-decoding bug in Whisper was missed by random and overfit fixtures and appeared only with real weights. Hugging Face has been pushing cheap, made-to-order model building (ML Intern), which makes reliable evaluation more important.

Library upgrades can change training behavior silently. Pinning versions and checking loss against baselines is worth the effort, especially for fine-tuning recipes built on Transformers and TRL (TRL watercolours) or domain models like Carbon-X (Carbon-X). Hub model pages may eventually link to tested usage examples. More in our research coverage and tools coverage.