Connect Two AI Servers with Safe Failover

What you will build

A loopback HAProxy endpoint that routes new local inference requests to a healthy primary or backup server. The lab disables retries: an interrupted request fails visibly, while a later new request can use the backup.

This is failover, not transparent generation continuation. It does not deduplicate agent side effects, guarantee equal model outputs, or authorize arbitrary callers. Allow 2-3 hours and use two disposable or approved private inference servers.

Before you start

Install HAProxy 2.8 or newer using the OS's supported packages. Both backend servers must have the same intended local model and compatible API behavior. Record model digests, runtime versions, context limits, and tool support.

Reach each server through an authorized private network or authenticated tunnel. Restrict backend access to approved clients. Plain HTTP across an untrusted network is not acceptable; use verified TLS or a protected transport there.

For a self-contained exercise, arrange two SSH tunnels in separate terminals:

ssh -N -L 11435:127.0.0.1:11434 user@first-authorized-server
ssh -N -L 11436:127.0.0.1:11434 user@second-authorized-server

These local ports are examples. Verify each with /api/tags and a direct request before adding a proxy. Do not guess backend identity based on an open port alone.

1. Write a bounded routing policy

For this exercise, both backends must satisfy the same local-only data policy. Neither may route the selected model to a hosted provider. All test prompts are synthetic, and no agent tools are enabled.

The API endpoint itself can accept many model names, so this simple proxy is not a model-policy enforcement gateway. Restrict its callers to the lab and send only the agreed model. A shared production gateway needs explicit model and capability validation.

2. Configure the proxy

Create haproxy-lab.cfg in a fresh lab directory:

global
    log stdout format raw local0

defaults
    mode http
    log global
    option httplog
    timeout connect 5s
    timeout client 180s
    timeout server 180s
    retries 0

frontend local_ai
    bind 127.0.0.1:18081
    default_backend local_models

backend local_models
    option httpchk GET /api/tags
    http-check expect status 200
    default-server inter 3s fall 2 rise 2
    server primary 127.0.0.1:11435 check
    server secondary 127.0.0.1:11436 check backup

Validate and run it in the foreground:

haproxy -c -f haproxy-lab.cfg
haproxy -db -f haproxy-lab.cfg

HTTP 200 on /api/tags checks basic API availability, not model readiness or inference quality. Before use, verify the agreed model and run representative direct requests against both servers.

3. Send a request and inspect evidence

In another terminal:

curl --fail-with-body http://127.0.0.1:18081/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.2:3b","stream":false,"messages":[{"role":"user","content":"Reply with one sentence about backups."}]}'

Use the exact model installed on both backends. Check HAProxy's access log for the selected backend server; the answer text alone does not establish which machine responded.

4. Exercise failure and recovery

Stop only the first lab tunnel, wait for two failed health checks, and send a new request. It should reach the secondary. Reestablish the primary tunnel, wait for successful checks, and confirm that later requests use the primary again.

Then interrupt the selected backend during a longer request. Expect an error or truncated connection, not a seamless continuation. Because retries are disabled, the proxy does not automatically replay the POST against another backend. The caller must decide whether a new attempt is safe.

When it breaks

If both servers appear down, test their tunnels independently. If a health check succeeds but inference fails, inspect model inventory, memory, and request compatibility. If long generation times out, measure the actual stage before broadening timeouts indiscriminately.

Never retry an entire tool-using agent run automatically just because inference failed. Notes, emails, payments, or deployments may already have occurred.

Cleanup and next steps

Stop foreground HAProxy and close both tunnels. No system configuration needs replacing for this exercise. A maintained deployment also needs authentication, model restrictions, bounded queues, cancellation, metrics, and a deliberately approved fallback policy.

Explore theLAB