METR estimates Anthropic’s AI capabilities are advancing faster

AI · First reported 5 Nov, 19:41 · Updated 19:49

Image: Wikimedia Commons · NASA Johnson Space Center · Public domain

A nonprofit’s review of Anthropic data and staff interviews estimated that the company’s models are improving at roughly twice last year’s rate on well-specified tasks. METR noted caveats, while Anthropic co-founder Sam McCandlish said the report’s conclusions largely matched his own view.

METR estimated that Anthropic is improving model performance on well-specified tasks around twice as quickly as a year earlier.
The assessment drew on about 11 weeks of company data and employee interviews; METR said its estimate could be overstated and might apply mainly to tasks that are easier to grade.
Anthropic co-founder Sam McCandlish said the report’s conclusions largely aligned with his own view, while noting uncertainty about how much AI is speeding up research.

Summary written by Lux AI from 2 articles

Key facts

Assessment group: Model Evaluation and Threat Research (METR)
Estimated pace: Roughly twice as fast as a year earlier for well-specified tasks
Data period: About 11 weeks, from mid-August through late October
Company response: Co-founder Sam McCandlish said many conclusions matched his own, while the amount of research acceleration was hard to determine