A small benchmark for local language models

Measure response latency and memory with a repeatable workload.

Start with five short prompts that match what you want the machine to do. Use public or made-up inputs. Include one long input and one task with an answer you can check mechanically. Write down the exact model file, quantization, context size, output limit and runtime version before collecting any numbers.

Separate waiting from generation

Use a streaming response and record two intervals: send to first generated token, and first token to last token. Count output tokens using the runtime's own statistics. Long inputs can make the first interval grow even when generation speed stays flat.

Run one cold request and at least three warm requests at each setting. Compare medians for the warm runs and keep the cold result separate. Record peak process memory and GPU memory separately when the runtime offloads layers. Hold concurrency at one before running a second round with realistic parallel traffic.

Pick a workload limit

Check the outputs against a few explicit criteria before comparing speed. Select a context limit that still leaves room for spikes on your smallest machine. Keep the prompts and results together so a runtime upgrade can be checked against the same workload.

Did this work on your setup, or did you spot a mistake? Send the editor a private note or leave a public note for the next reader.

Related edition