Measuring Thinking Efficiency in Reasoning Models (Nous Research)

Author: Nous Research URL: https://nousresearch.com/measuring-thinking-efficiency-in-reasoning-models-the-missing-benchmark Published: August 14, 2025 (data and code: LRMTokenEconomy on GitHub) One-line: Open-weight reasoning models spend 1.5–4x more tokens than closed ones on the same problems, up to 10x on trivial questions, which can erase their per-token price advantage.

Summary

The missing axis. Cost comparisons usually plot intelligence against price per million tokens. Nous argues token efficiency, meaning how many tokens a model burns reasoning toward an answer, matters as much. It drives the real cost per query, latency, and how fast the context window fills up.

Measuring hidden reasoning. Most closed labs don’t expose raw chain of thought; they summarize or transcribe it. So the study uses billed completion tokens as the proxy for effort. Plotting characters returned against tokens billed reveals who transcribes what:

• Open models: about 3–3.4 characters per token, meaning the chain of thought is intact.
• Claude 3.7 Sonnet: about 2.8, lightly filtered.
• Gemini: about 1, a roughly 3:1 densified transcript.
• OpenAI and newer Claude: about 0.3, only a summary.

Dataset. Three domains, N=5 samples per prompt and model, with a 30,000-token cap, run through OpenRouter in July 2025:

• Knowledge questions (e.g., the capital of Australia) that need no reasoning at all. Reasoning models still spent hundreds of tokens on them. Open models used about 4x the tokens of the most efficient closed models, and Magistral up to 10x. Closed models’ efficiency often offset their higher per-token price.
• Math (AIME 2023/2025 problems plus variable-swapped variants). Paired originals and variants took almost identical token counts, which suggests the models solve them algorithmically rather than from memory. o4-mini-high used about 3x fewer tokens than other commercial models. Grok-4 was also optimized. Nvidia’s llama-3.3-nemotron-super-49b-v1 was the most efficient open model.
• Logic puzzles (bridge-and-torch and Monty Hall variants, including impossible and inverted versions). Default versions took the fewest tokens, because training-data bias gives the model a shortcut. Modified versions took more, and impossible ones the most. Many models failed the modified versions.

Inspecting gpt-oss. OpenAI’s open-weight gpt-oss-120b exposes its reasoning, which shows how the compression works: terse, almost telegraphic reasoning, minimal formatting, few re-checks, and long reasoning only on unfamiliar variants. On one AIME prompt DeepSeek-R1-0528 averaged 3,104 reasoning tokens against gpt-oss-120b’s 268.

Trends. Closed labs keep cutting token use across versions. DeepSeek and Qwen increased it in newer releases, possibly trading efficiency for benchmark scores. The overall gap is about 3x on knowledge prompts and under 2x on math and logic. Nous suggests gpt-oss as a reference target for other open models.

Full text

Archived privately against link rot, with all 22 figures: ../attachments/measuring-thinking-efficiency-in-reasoning-models/measuring-thinking-efficiency-in-reasoning-models.md

Connections

• Nous Research — publisher; its DeepHermes model was run through the Nous API in the study.
• OpenAI — its o-series and gpt-oss models set the efficiency frontier in this study.
• Open Source — the study’s central finding is a hidden cost of open-weight models.
• Nvidia — its Nemotron model was the most efficient open-weight entrant.
• Anthropic — Claude’s transcribed chain of thought and low reasoning-token counts on logic puzzles.