Engineer's Notes

Disable Tailscale Key Expiry on Always-On Servers

Tailscale node keys expire by default (~180 days). When a server's key expires it silently drops off the tailnet — still powered on, still on your LAN, but unreachable by its 100.x address, and any service that connects to it by tailnet IP just times out.

For headless boxes that should always be reachable (database hosts, DNS, anything other services depend on), open the Tailscale admin console and disable key expiry for that node. Otherwise you'll eventually chase a “service is down” ghost that's really just an expired key.

Cloudflare Tunnel Beats Port-Forwarding + Let's Encrypt

Exposing a home service the old way means opening router ports, running certbot, and renewing certificates. A Cloudflare Tunnel replaces all of it: the cloudflared daemon dials out to Cloudflare, so there are no inbound ports to open, and TLS terminates at the edge — no Let's Encrypt, no renewals.

Point a hostname at http://localhost:PORT in /etc/cloudflared/config.yml and it's live over HTTPS in seconds. Bonus: put Cloudflare Access in front and you get SSO auth on any internal app without touching its code.

Add Features to Apps You Can't Fork

You don't need the source to customize a self-hosted app. Put Apache in front as a reverse proxy and use mod_substitute to inject your own <script> before </body>. Proxy your private backends under the same origin (e.g. /api/ → a tailnet box) so the browser never hits CORS or mixed-content.

Hard-won caveat: if the app is a React/Vue SPA, mount your injected UI on document.body — never insert nodes inside the framework's managed DOM. Splicing a node into React's child list corrupts its virtual DOM and throws removeChild errors that break the whole app's interactivity.

systemd --user Services Need Lingering Enabled

Running a service as a systemd --user unit is clean — until you log out and it dies. User services are tied to your login session unless you enable lingering:

sudo loginctl enable-linger <user>

With lingering on, the user manager starts at boot and your services (MCP servers, workers, agents) survive reboots and logouts. Forget this and you'll see “works when I'm SSH'd in, dead otherwise.”

Pin transformers for Brand-New Model Architectures

New model families ship faster than the tooling stabilizes. A fresh architecture often needs a specific transformers version: too old and it doesn't recognize the arch, too new and a downstream tool (Unsloth, a GGUF converter) breaks.

Find the one version that satisfies every tool in the chain and pin it hard (e.g. transformers==5.5.0). Watch out for requirements.txt installs that silently downgrade it out from under you — reinstall the pinned version afterward.

Bind Services to localhost or the Tailnet, Not 0.0.0.0

By default many servers listen on 0.0.0.0 — every interface, including whatever's exposed to the LAN. For anything that only needs to be reached locally or by other fleet boxes, bind it explicitly: publish Docker ports as 127.0.0.1:PORT, or bind to the Tailscale IP.

Your reverse proxy (which is reachable) can still front it. This keeps databases, vector stores, and admin ports off the open network by construction, not by firewall afterthought.

Timestamp Backups Before Touching Prod Config

Before editing any live config, drop a dated copy next to it:

cp file.conf file.conf.bak.$(date +%Y%m%d-%H%M%S)

It costs nothing and turns a bad edit into a one-line rollback. For reverse-proxy or tunnel changes, also test locally before cutting over — hit the origin with a forced Host header (curl -H "Host: site.com" http://127.0.0.1/) and confirm it's right before flipping DNS or the tunnel.

RAG Grounded in Your Own Data

To make an LLM answer from your notes instead of its training data, build a retrieval pipeline: chunk documents into overlapping passages, embed each chunk (a small model like nomic-embed-text is plenty), and store the vectors in Qdrant. At query time, embed the question, retrieve the top-k nearest chunks, and hand them to the model with a firm instruction: answer only from the provided context, and say so if it isn't there.

That one instruction is what kills hallucination — the model stops guessing and starts citing. Return the source documents alongside the answer so every claim is traceable back to a note.

Pull Ollama Models via the API, and Warm Them First

You don't need shell access to a model host to add a model — POST /api/pull with the model name and Ollama fetches it over HTTP. Perfect for provisioning a remote box you can only reach on the tailnet.

And when you benchmark: the first request pays a cold-start tax while the model loads into VRAM (an embedding call can look like it ‘hung’ for 30 seconds). Fire one throwaway request to warm the model, then time the real ones.

Match CUDA to the GPU Architecture

A new GPU generation needs new toolkit. Blackwell cards (compute capability sm_120) require CUDA 12.8+ and matching PyTorch wheels — older images silently fail or fall back to CPU. If a container targets an older arch, edit its Dockerfile base image and the PyTorch install lines to the CUDA 12.8 build.

Symptom to recognize: training or inference ‘works’ but is mysteriously slow, or throws no kernel image is available for execution. That's an arch/toolkit mismatch, not a bug in your code.

Watch Your Chunker for Off-by-One Loops

A subtle text-chunking bug will quietly wreck a vector index. If your loop advances the cursor by len(piece) - overlap and a trailing piece is shorter than the overlap, the step collapses to one character — and you emit hundreds of near-duplicate tail chunks per document.

I watched an index balloon to 9,000+ junk chunks before catching it; switching to a fixed step = size - overlap that always moves forward dropped it to ~1,450 clean chunks. Always advance by a positive constant, and break when the window reaches the end of the text.

Reasoning Models Spewing Bad JSON? Turn Off Thinking

When you need strict JSON from a reasoning-capable model, its ‘thinking’ trace leaks into the output and breaks your parser. The fix isn't a lower reasoning-effort setting — it's Ollama's native /api/chat with "think": false, which disables the reasoning channel entirely.

Pair that with a low temperature and an explicit ‘respond with JSON only’ instruction and you get clean, parseable output every time.

Reach Unreachable Boxes with an SSH Jump Host

Some machines aren't directly reachable — behind NAT, on another subnet, or only exposed to one other box. Instead of VPN gymnastics, hop through a reachable neighbor: ssh -J jump-host target, or a ProxyJump entry in ~/.ssh/config.

SSH tunnels the connection through the jump host transparently — you land on the target as if it were direct, and scp/rsync work the same way.

Keep Inference Local-First on the Tailnet

Run the whole loop — wake word, speech-to-text, the LLM, and text-to-speech — on your own boxes, reachable over Tailscale. Local-first means zero per-token cost, no rate limits, real privacy (your notes and audio never leave your network), and it keeps working when the internet doesn't.

The tailnet gives every service a stable private address, so components on different physical machines talk to each other as if they shared one LAN. Cloud models become an optional upgrade, not a dependency.

Run AI Coding Agents Linux-to-Linux, Not Windows-to-Linux

Agentic coding tools like Claude Code and ChatGPT Codex are happiest when the machine they run on matches the machine they target — Linux driving Linux. Point one at a Linux server from Windows and you fight a constant impedance mismatch.

The root issue is two different shells and conventions colliding in one workflow. On Windows the agent's local shell is PowerShell/cmd, but every remote command is bash — so it has to juggle two quoting and escaping dialects at once. Unix tools it instinctively reaches for locally (grep, sed, awk, cat, heredocs) don't exist or behave differently, nested ssh quoting gets brutal, and single-quoted heredocs collide with piped-in sudo passwords.

On top of that: CRLF vs LF line endings (a Windows-authored script breaks on a stray \r in the shebang), path separators (\ vs /), and SSH key-permission quirks. These models are trained overwhelmingly on POSIX workflows, so their default commands assume Linux — every Windows deviation is friction they have to notice and work around.

Takeaway: when your targets are Linux, run the agent from Linux, WSL2, or macOS. A homogeneous Linux-to-Linux path deletes the translation layer — same shell, same quoting, same line endings, same tools — and the agent stops tripping over its own commands.