wimpy-setup: a dual-GPU inference box you can rebuild from scripts

This is not a guide. It’s the actual, ordered, idempotent script set that builds my inference server — shared because rebuilding from memory is how homelabs die. Wimpy is a Ryzen 7 7700 with two GPUs serving LLM inference at the same time: an AMD Radeon AI PRO R9700 (32GB, ROCm) and an NVIDIA RTX 5060 Ti (16GB, CUDA), behind one llama-swap endpoint.

The dual-GPU part is the reason this repo exists. HIP and CUDA backends of llama.cpp share library filenames, so they can’t coexist in one prefix — there are two isolated builds (/usr/local for ROCm, /opt/llama-cuda for CUDA) and one llama-swap config with two non-exclusive groups. A -cuda model entry is the same GGUF as its AMD twin with a different binary and device pin, so either card can serve it. GPUs are pinned by UUID (the Ryzen iGPU also enumerates as a ROCm device and will happily steal your workloads if you let it).

Features:

• Nine ordered idempotent host scripts (run-all.sh --from/--only) covering base system through KVM
• Dual-GPU llama.cpp: ROCm gfx1201 + CUDA sm_120 builds in isolated prefixes, one llama-swap over both
• fetch-model.sh — full GGUF acquisition pipeline: inspect, validate, GPU smoke-test, register, auto-deploy with rollback; handles HF repos, URLs, and local files
• Automatic mmproj projector resolution for vision-language models, renamed to prevent collisions between repos
• Mid-pipeline failures write a resume dossier instead of leaving you stranded
• Parameterized Hermes VM provisioning (hermesvm-setup.sh --hostname X), clone-and-go for every future VM
• Bidirectional SSH trust scripts with merge-never-replace sshd dropins — re-running can’t lock anyone out
• Stubbed test suite that runs without network or GPU

Source: github.com/rahlquist/wimpy-setup