Skip to content

Inference

Research cut-off date: 2026-07-22. Status: Phase 2, not yet written. This file states the scope of the work, the arguments it owns, and the research required to complete it. It contains no findings, because none have been sourced. See the changelog for phase status.

Scope

Covers what happens between a request and a response: prefill and decode, decoding strategies and their effect on reproducibility, batching policy, prefix and prompt caching, speculative decoding, quantization at serving time, and retrieval augmentation as an inference-time architecture. Explains why the serving stack, not the weights alone, determines observed behaviour.

Arguments this chapter owns

Other files link here rather than restating these. Duplicated argumentation is a defect under the writing standards, not redundancy for the reader's convenience.

  • The prefill and decode decomposition, and the different resources each is bound by.
  • The argument that identical weights served by two stacks are not the same system.
  • The reproducibility consequences of sampling parameters and non-deterministic kernels.

Developed elsewhere

Research checklist

  • Cite the primary sources for paged attention, speculative decoding, and the quantization methods discussed.
  • State the reproducibility conditions required for any evaluation run reported in this repository, and cross-reference evaluation/reproducibility.md.
  • Record which profiled providers document prompt caching, its granularity, and its retention period, with sources.
  • Add Mermaid diagrams of the inference pipeline and of retrieval augmentation.

Completion criteria

This chapter is complete when every checklist item above is closed, when every numerical claim carries a footnote resolving to data/sources.csv, when every claim about a current model carries an absolute date, and when the twelve-point quality-control checklist in the research methodology passes.