StateBench finds long state-tracking challenges for language models
AI · First reported 3 Nov, 17:15 · Updated 17:49
StateBench tests whether models can update information across long sequences, from simple lookups to chains of dependent changes. In the reported evaluations, frontier models struggled as harder sequences grew, while recurrent and hybrid models trained directly on the benchmark handled several task types at much greater lengths.
- StateBench compares four levels of state tracking, keeping the variables and update positions consistent while changing the type of assignment.
- The four evaluated frontier models did well on simpler tasks, but accuracy dropped on longer soluble and general finite sequences; on the longest general finite tasks it fell to 13% or lower.
- In experiments training 275-million-parameter models from scratch, recurrent and hybrid models generalized well on the first three levels; transformers degraded on longer sequences and did not learn the general finite level.
Summary written by Lux AI from 2 articlesUnverified
Key facts
- Benchmark levels
- Four: R-trivial, aperiodic, soluble, and general finite.
- Frontier models
- Four models were tested: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna.
- Longest general finite results
- Frontier-model accuracy fell to 13% or lower.
- Direct-training evaluation length
- Models trained on sequences up to 256 assignments were tested on sequences up to 64,000 assignments.
- Architecture result
- The more flexible Gated DeltaNet variant stayed near perfect on the hardest level to roughly 10 times its longest training sequences.
Source spectrum
Far left
Left
Lean left
Centre
Lean right
Right
Far right
Unrated
Left 0 · Centre 0 · Right 0 of 0 rated sources · +1 unrated
All coverage
Coverage over time
LeftCentreRight
Articles per hour since first reported, by side.
Ownership
- Unknown 50% (1)
- Corporation 50% (1)
Share of the 2 sources by owner type.