https://aislackin.com/tags/cuda/Recent content in Cuda on AiSlackinHugoen-usSun, 28 Jun 2026 00:00:00 +0000https://aislackin.com/posts/2026-06-28-wimpy-dual-gpu-inference/Sun, 28 Jun 2026 00:00:00 +0000https://aislackin.com/posts/2026-06-28-wimpy-dual-gpu-inference/<p>This is not a guide. It&rsquo;s the actual, ordered, idempotent script set that builds my inference server — shared because rebuilding from memory is how homelabs die. Wimpy is a Ryzen 7 7700 with two GPUs serving LLM inference <em>at the same time</em>: an AMD Radeon AI PRO R9700 (32GB, ROCm) and an NVIDIA RTX 5060 Ti (16GB, CUDA), behind one llama-swap endpoint.</p> <p>The dual-GPU part is the reason this repo exists. HIP and CUDA backends of llama.cpp share library filenames, so they can&rsquo;t coexist in one prefix — there are two isolated builds (<code>/usr/local</code> for ROCm, <code>/opt/llama-cuda</code> for CUDA) and one llama-swap config with two non-exclusive groups. A <code>-cuda</code> model entry is the same GGUF as its AMD twin with a different binary and device pin, so either card can serve it. GPUs are pinned by UUID (the Ryzen iGPU also enumerates as a ROCm device and will happily steal your workloads if you let it).</p>