Choose the Best Local Model for Your Hardware

What you will build

A small comparison sheet that combines task quality, response timing, and observed memory use. The result is a justified model choice for your own work, not a universal ranking.

Allow 1-3 hours plus downloads. Use only models that fit your hardware and whose licenses suit your use. Start with two candidates instead of filling the disk with every popular release.

Before you start

Complete the local server guide. Use Python 3.11+, a loopback Ollama server, and an otherwise quiet machine. Record the OS, CPU, RAM, GPU and driver if present, and Ollama version.

Pick two installed local models. For a small lab, candidates might include llama3.2:3b and qwen2.5:3b; model availability and suitability must be checked before downloading. Record IDs and quantization rather than treating names as immutable versions.

ollama --version
ollama list
ollama ps

Download and inspect benchmark.py. It uses Python's standard library, sends non-streaming requests to a fixed loopback endpoint, and writes a JSON report without overwriting an existing file.

1. Define tasks before measuring

Create a small private evaluation set representing your actual needs. Include factual extraction, a code change with a test, following a strict output format, a question requiring admission of missing information, and a longer input.

Write the expected behavior before running either model. Score correctness and instruction following separately from writing style. A persuasive answer that ignores the requested constraint is a failure.

The downloaded script provides a tiny timing workload. It is not an automatic quality benchmark; review its saved responses and use your own tasks for the final decision.

2. Run warm measurements

python3 benchmark.py llama3.2:3b qwen2.5:3b --runs 3 --output comparison.json

The script sends one warm-up request before recording each model's runs. It records client end-to-end latency, runtime timing fields, and generated answers. Keep generation limits, context settings, and sampling parameters fixed across candidates.

Checkpoint: the report contains every requested model and run. A timeout or API error must count as a failed run, not disappear from the comparison.

3. Measure cold behavior separately

Unload a candidate using ollama stop MODEL_NAME, then time its next request while watching model loading. Do not call a warm test cold merely because you opened a new terminal.

The script's warm timings do not measure first-token latency because it requests a complete non-streaming response. To evaluate interactive responsiveness, measure the arrival of streamed chunks with a monotonic clock and distinguish protocol metadata from actual generated text.

4. Observe memory and sustained load

Watch free -h, ollama ps, and the appropriate GPU monitor during runs. Record peak RAM or VRAM, swapping, offload, and thermal throttling. Larger context budgets and simultaneous sequences increase memory requirements independently of weight size.

Repeat a longer prompt representative of real use. A model that fits with a short prompt can still run out of memory with a long conversation. Avoid increasing concurrency until the single-request path is stable.

5. Make the decision explicit

Use a table with model identity, quality score, median latency, generation rate, observed memory, failed requests, and a brief reviewer note. Report sample count. Three runs are a smoke test, not a reliable p95 service-level estimate.

Choose the smallest candidate that reliably meets your needs. A slower model may still be necessary for difficult tasks; route those deliberately rather than silently changing the model during a failure.

When it breaks

If the second model changes the first model's results, check memory pressure and eviction. If repeated prompts get unusually fast, note caching. If model output differs despite temperature zero, do not assume bitwise determinism across hardware or runtime versions.

For a code task, run the actual tests. For extraction, compare with reference fields. LLM judges can help triage but should not be the only authority on correctness.

Cleanup and next steps

Keep reports with software versions and workload definitions. Remove only models you intentionally downloaded and no longer need; other applications may share them. Use the failover guide only after deciding which backends are genuinely interchangeable.

Explore theLAB