We train linear probes on the hidden states of Apertus-8B, Llama-3.1-8B and Mistral-7B to warn, token by token, that a model is about to fall into a repetitive loop, before it shows in the text. We propose ONWARD, an evaluation that fixes the false-alarm rate on healthy responses and scores alarms against a human-verified failure onset. We show that AUROC saturates on this task and cannot tell an early warning from a late detection, and that LoRA adaptation below the probe more than doubles warning coverage on all three models.
@inproceedings{denegri2026degeneration,title={Degeneration Detection Is Not Early Warning},author={De Negri, Marco and Sartori, Luca and Baggi, Lorenzo and Reihs, Moritz and {\v{D}}urech, Eduard F. and El-Assady, Mennatallah and Pyatkin, Valentina and Hedstr{\"o}m, Anna},booktitle={NeurIPS 2026 Workshop: Interpretability as a Science},year={2026},}
arXiv
From Open Text Models to Open Multimodal Reasoners: Apertus v1.5 Technical Report
Project Apertus
arXiv preprint arXiv:2610.15542. Contributor, Interpretability team , 2026
@article{apertus2026v15,title={From Open Text Models to Open Multimodal Reasoners: Apertus v1.5 Technical Report},author={{Project Apertus}},journal={arXiv preprint arXiv:2610.15542},year={2026},}