GPU allocation / Report summary

Long Context, Broader AI Capabilities

Longer context expands what AI applications can attempt, but useful capacity depends on reasoning reliability and the cost of repeatedly processing evidence.

November 2026 · 7 pages · 815 KB · Tolaga Research

Download full report (PDF) ↓

Executive summary

Large context windows let applications examine document collections, software repositories and multimodal material in fewer fragmented requests. They also let agents retain plans, tool results and intermediate findings across longer workflows. However, the ability to accept more tokens does not establish that a model can use them reliably or economically.

The report distinguishes nominal context capacity from effective reasoning capacity and economic capacity. Its document-investigation study shows how repeatedly carrying original evidence into later stages can sharply increase completion time, even when the final answer remains short.

Key findings

  • Nominal, effective and economic context capacity are different measures and need not improve together.
  • Dense-attention prefill work grows approximately quadratically with sequence length; KV-cache capacity grows approximately linearly with context and concurrency.
  • In the modeled investigation, increasing retained evidence from 0% to 100% raises completion time from roughly 13 to 239 minutes.
  • Selective retrieval, compression and reuse can reduce redundant work, but their value depends on implementation and evidence needs.

The document-investigation study

A scripted agent plans an investigation, assigns four workers, consolidates their findings, performs two verification checks, completes an evidence-review revision and produces a 512-token final response. The source collection contains two million tokens, approximately 75% visual content from 1,095 page images and 25% text and delimiters.

The synthetic approximately one-trillion-parameter model uses 64 Rubin GPUs in the study assumptions. Later processing stages retain 0%, 10%, 25%, 50% or 100% of the original evidence. The 0% case uses summary-based reasoning, with a 5% evidence allowance per verification and revision call. The four workers still process their original source partitions. Retained evidence is supplied again to later calls rather than preserved as reusable KV state.

Infrastructure and application implications

Larger context places pressure on both compute and memory. Prefill must process relationships within the input, while decode repeatedly accesses accumulated KV state. Distributing context can increase available capacity and spread attention work, but introduces communication and synchronization costs.

Application design is therefore an infrastructure decision. Developers choose what to retain, retrieve, summarize or discard at each stage. Repeatedly processing the same collection can consume far more input tokens than its original size suggests. Prefix caching, retrieval and compression address different parts of that problem and should be evaluated against the actual workflow.

Reasoning quality remains a separate requirement. Finding one fact in a long input does not demonstrate reliable synthesis across dispersed or conflicting evidence. The practical target is useful information processed within acceptable quality, latency and cost constraints.

Assumptions and limitations

The timing comparison assumes one investigation with no competing requests. Calls execute sequentially on one 64-GPU replica, without cross-call KV or prefix-cache reuse. The four workers divide the investigation logically but execute sequentially. The simulation estimates execution costs; it does not assess answer quality or whether retaining more evidence improves the conclusions. At 100% retention, later-stage inputs exceed two million tokens, an analytical assumption rather than a validated model capability. Concurrent scheduling and compatible prefix caching may change completion times and the relative penalty of retaining more evidence. Hardware and model settings are scenario assumptions.

Source: Long Context, Broader AI Capabilities, PDF pages 2–8 (printed pages 1–7). This summary describes the published report; consult the PDF for figures and references.