quill-of-hermes: LAN OCR that eats PDFs and multi-page TIFFs

Thu, 30 Jul 2026 00:00:00 +0000

OCR as a service usually means cloud APIs, per-page billing, and your documents on someone else’s disk. This is the local counterargument: a FastAPI wrapper around OvisOCR2 running GGUF via llama.cpp’s llama-mtmd-cli, serving your LAN. POST an image, a PDF, or a multi-page TIFF; get back clean Markdown.

The design decisions worth noting: each page is sent to the model independently and results are joined with a horizontal rule, so a 50-page PDF degrades gracefully instead of failing whole. Jobs are serialized to keep the GPU sane, there’s a five-minute timeout per page so one bad scan can’t wedge the queue, and temp files are cleaned unconditionally. The service deliberately stays separate from llama-swap because multimodal llama-server support for this model wasn’t proven — llama-mtmd-cli was.

wimpy-setup: a dual-GPU inference box you can rebuild from scripts

Sun, 28 Jun 2026 00:00:00 +0000

This is not a guide. It’s the actual, ordered, idempotent script set that builds my inference server — shared because rebuilding from memory is how homelabs die. Wimpy is a Ryzen 7 7700 with two GPUs serving LLM inference at the same time: an AMD Radeon AI PRO R9700 (32GB, ROCm) and an NVIDIA RTX 5060 Ti (16GB, CUDA), behind one llama-swap endpoint.

The dual-GPU part is the reason this repo exists. HIP and CUDA backends of llama.cpp share library filenames, so they can’t coexist in one prefix — there are two isolated builds (/usr/local for ROCm, /opt/llama-cuda for CUDA) and one llama-swap config with two non-exclusive groups. A -cuda model entry is the same GGUF as its AMD twin with a different binary and device pin, so either card can serve it. GPUs are pinned by UUID (the Ryzen iGPU also enumerates as a ROCm device and will happily steal your workloads if you let it).

Llama-Cpp on AiSlackin

quill-of-hermes: LAN OCR that eats PDFs and multi-page TIFFs

wimpy-setup: a dual-GPU inference box you can rebuild from scripts