Your First Local AI Chat Server

What you will build

A local Ollama service that answers a question through both its command-line client and HTTP API. You will also check the listening interface, record a baseline, and learn how to stop the service without deleting models.

This is a Linux reference build, not an instruction to change an existing production model server. Use a spare machine or VM. Allow 45-90 minutes plus downloads; software is free, but models consume storage, electricity, and download bandwidth.

Before you start

Use Ubuntu 24.04 with a terminal, curl, administrator access, and at least 8 GB of system RAM. Sixteen GB provides more room; model and context choices determine actual requirements. A GPU is optional for this small-model exercise. Check supported GPU drivers in the official Ollama documentation before expecting acceleration.

Record the OS, available RAM, free disk space, and any existing listener:

lsb_release -ds
free -h
df -h "$HOME"
ss -ltn

If port 11434 is already in use, identify the existing service before proceeding. Do not stop an unknown process or overwrite its configuration.

1. Install and verify Ollama

Download the official installer, inspect it, then run it only if you accept its system changes:

curl --fail --location https://ollama.com/install.sh -o ollama-install.sh
less ollama-install.sh
sh ollama-install.sh
ollama --version
systemctl status ollama --no-pager

An installer can create users, install binaries, and register a system service. If your OS or installation policy differs, use the official platform-specific installation instead.

Checkpoint: ollama --version reports an installed version, and the service is active. Record that exact version in your notes.

2. Pull one model that fits

ollama pull llama3.2:3b
ollama list
ollama run llama3.2:3b "Explain what a reverse proxy does in three sentences."

The first request includes loading time. If memory is tight, start with a smaller compatible model and use its name consistently in later commands. A model name is not an immutable version: record its ID from ollama list as well.

Checkpoint: the response is understandable and the process does not exhaust RAM or enter sustained swapping. GPU owners can inspect ollama ps to see where the model is loaded; an installed GPU alone does not establish GPU execution.

3. Call the API

curl --fail-with-body http://127.0.0.1:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.2:3b","stream":false,"messages":[{"role":"user","content":"What is a backup? Use one sentence."}]}'

Look for message.content in the JSON response. An HTTP success with an unrelated model or empty response is not a successful functional check.

Repeat the request after the model is warm. Record elapsed time and the runtime's output-token and generation-duration fields. Those internal timings are not the same as client-observed first-token latency. The model comparison guide provides a small measurement tool.

4. Check privacy and exposure

ss -ltn | grep 11434
ollama ps

For this single-machine build, the API should listen on loopback, not every network interface. Do not expose an unauthenticated model-management API to the public Internet. If accessing it from your laptop, prefer an authenticated SSH tunnel:

ssh -N -L 11435:127.0.0.1:11434 user@your-authorized-server

Your laptop can then request http://127.0.0.1:11435/api/tags. This does not change the server's bind address. Use real authorized account details, not the example hostname literally.

Local inference keeps the prompt on the machine running that model, but installation and model downloads still use the network. Hosted providers and cloud-routed models have different data boundaries.

When it breaks

Symptom	First check
Connection refused	`systemctl status ollama` and the listening port
Model not found	Exact model name in `ollama list`
Very slow first answer	Model load time versus warm request time
Every answer is slow	RAM, swapping, CPU offload, drivers, context size
Laptop works but a container fails	Container loopback is not host loopback

Read recent logs with journalctl -u ollama -n 100 --no-pager. Redact prompts and identifiers before sharing logs.

Stop, recover, and continue

Use ollama stop llama3.2:3b to unload the model. On this system-service installation, sudo systemctl stop ollama stops the server without deleting model files. Do not delete an existing model store as a troubleshooting shortcut.

Continue with a private LibreChat workspace.

Explore theLAB