pipette
ENEnglish

Inference for Reasoning Models: Practical Lessons from Evaluation and Serving

Luv Verma

Preprint con versión publicada

En palabras de los autores

Running a language model involves more than its trained weights. The same model can receive very different benchmark scores, produce different answers, or cost more to serve depending on how it is evaluated and run. In one experiment in this book, changing only the maximum response length moved accuracy on a math benchmark from 0.08% to 93.18%. This book explains how inference settings shape the results we report and the systems we deploy. We begin with evaluation, including generation limits, answer extraction, scoring methods, and quantization. We then examine why outputs can change across batches, attention backends, and caching, even when sampling is disabled. The next chapters cover serving performance, including batching, memory capacity, latency, throughput, and cost. We also study structured outputs, tool calls, reasoning budgets, and long-context tasks, showing what these controls and benchmarks do and do not establish. The book concludes with practical guidance for recording inference runs, checking evaluations, estimating capacity, and comparing systems fairly. Written for researchers and engineers working with language models, it aims to make inference results easier to understand, reproduce, and trust.

Resultado principalEl resumen no menciona limitaciones.

Apareció: martes, 10 de noviembre. arXiv. Preprint con versión publicada.

DOI: 10.5281/zenodo.21754285

Versión publicada: 10.5281/zenodo.21754285

Comentario de los autores: 217 pages. Book, first edition, version 1.0 (August 2026). Published on Zenodo