Setting Up llama.cpp + llama-swap on Ubuntu 26.04 with an RTX 5070 Ti

A from-scratch walkthrough of installing llama.cpp with CUDA support, downloading multiple GGUF models, and running them behind llama-swap as a single OpenAI-compatible endpoint — alongside an existing Ollama install — for use with Open WebUI.

Hardware used: Ubuntu 26.04 LTS ("Resolute Raccoon"), NVIDIA RTX 5070 Ti (16GB VRAM), Intel i7-14700F, ~31GB system RAM.

1. Build llama.cpp with CUDA support

Install build dependencies

sudo apt update
sudo apt install -y build-essential cmake curl libcurl4-openssl-dev pciutils git

Clone the repo

cd ~
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Watch your working directory. It's easy to accidentally cd into the repo twice (e.g. running the clone from inside an already-existing llama.cpp folder), producing a nested ~/llama.cpp/llama.cpp. Run pwd && ls before building — you want CMakeLists.txt, src/, ggml/ etc. directly under ~/llama.cpp, not one level deeper.

Install the CUDA Toolkit

Check your driver and GPU first:

nvidia-smi

Look at the CUDA Version field in the top-right of the output — that's the maximum version your driver supports, not what's installed. The RTX 50-series (Blackwell) requires CUDA 12.8 or later.

On Ubuntu 26.04 specifically: this is the first Ubuntu release to ship CUDA natively in its own repositories (via the multiverse component), so you don't need NVIDIA's external apt repo at all:

sudo apt install -y cuda-toolkit

If that fails to find the package, enable multiverse first:

sudo add-apt-repository multiverse
sudo apt update
sudo apt install -y cuda-toolkit

On older Ubuntu (24.04/22.04): use NVIDIA's own apt repo instead, since the native OS package won't exist:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-8

Adjust ubuntu2404 to match your actual release.

Add CUDA to your PATH:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Verify:

nvcc --version

You should see a real version string. If nvcc still isn't found after this, the toolkit didn't actually install — don't proceed to the build step until this works, or CMake will silently fall back to a CPU-only build.

Build

cd ~/llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --clean-first --target llama-cli llama-server llama-gguf-split
cp build/bin/llama-* .

This compiles CUDA kernels and will take 10-20+ minutes depending on your CPU. Verify it worked:

./llama-cli --version

2. Set a permanent model cache location

By default, llama-cli/llama-server cache downloaded GGUFs under ~/.cache/llama.cpp. If you want a clearer, dedicated location:

echo 'export LLAMA_CACHE="$HOME/llama-models"' >> ~/.bashrc
source ~/.bashrc
mkdir -p "$LLAMA_CACHE"

Do this before downloading any models. If LLAMA_CACHE isn't set consistently across every terminal session and every systemd service that touches llama.cpp, models will scatter across multiple cache locations, and processes (like llama-swap) may not find files you already downloaded — leading to confusing duplicate downloads or "missing model" errors that aren't actually missing.

3. Download models

llama.cpp can pull GGUFs directly from Hugging Face with -hf <repo>:<quant_tag>. There's a gotcha worth calling out:

--version exits before processing -hf. If you try to "quickly test" a download with llama-cli -hf some/repo --version, it will print the build version and exit immediately — no download happens. To actually trigger a download, you need a real generation request:

./llama-cli -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M -p "hi" -n 10

Example downloads used in this setup:

cd ~/llama.cpp
 
./llama-cli -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M -p "hi" -n 10
./llama-cli -hf unsloth/Qwen3.5-27B-GGUF:Q4_K_M -p "hi" -n 10
./llama-cli -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M -p "hi" -n 10
./llama-cli -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M -p "hi" -n 10
./llama-cli -hf ggml-org/embeddinggemma-300M-GGUF -p "hi" -n 10
./llama-cli -hf Qwen/Qwen3-Embedding-0.6B-GGUF -p "hi" -n 10

Run these one at a time and let each fully finish before starting the next, especially on a single GPU — see the VRAM warning in the next section for why.

Confirm what's actually cached:

ls -la ~/llama-models

Hugging Face-style cache folders look like models--unsloth--Qwen3.6-27B-GGUF — that naming is normal.

4. A critical gotcha: stuck/suspended processes hold VRAM forever

This was the single biggest time-sink in this setup, so it gets its own section.

If you Ctrl+Z out of an interactive llama-cli session (instead of /exit or Ctrl+C), the process suspends but does not release GPU memory. It'll show up in ps aux with state Tl (stopped, traced) and continue holding however many GB it had loaded — invisible unless you specifically check.

This causes a very confusing downstream symptom: a completely unrelated attempt to start llama-server later fails with:

cudaMalloc failed: out of memory
GGML_ASSERT(buffer) failed

The fix is simple once you know to look for it:

ps aux | grep llama
nvidia-smi

If nvidia-smi shows VRAM in use but Processes: says "No running processes found," check for stopped/suspended jobs:

jobs -l

And kill any stray PIDs:

kill -9 <pid>

Lesson: always exit llama-cli interactive sessions cleanly (type /exit, or Ctrl+C once), and get in the habit of checking nvidia-smi before debugging a fresh OOM error — the cause is very often a zombie process from an earlier test, not the current command.

5. Running multiple models: llama-swap

Plain llama-server only serves one model per process. If you have several large models and limited VRAM (in this case, 16GB — not enough to hold two 27B+ models at once), you need something that loads/unloads models on demand based on incoming requests. llama-swap does exactly this: a lightweight Go proxy that sits in front of llama-server, reads the model field of each OpenAI-style request, and starts/stops backend processes accordingly.

Install

Don't guess the release asset filename — get it from the GitHub API directly, since the version number is embedded in the filename and changes frequently:

curl -s https://api.github.com/repos/mostlygeek/llama-swap/releases/latest | grep browser_download_url

Copy the URL ending in linux_amd64.tar.gz and download it:

cd ~
wget <the-exact-url-from-above>
tar -xzf llama-swap_*_linux_amd64.tar.gz
sudo mv llama-swap /usr/local/bin/
llama-swap --version

Configure

mkdir -p ~/llama-swap
nano ~/llama-swap/config.yaml

healthCheckTimeout: 1800 # 30 min — large GGUFs can take a while to download on first load
 
macros:
  llamaCmd: /home/majed/llama.cpp/llama-server
 
models:
  "qwen3.6-27b":
    cmd: |
      ${llamaCmd}
      -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M
      --port ${PORT}
      --ctx-size 16384
      -ngl 99
    ttl: 600
 
  "qwen3.5-27b":
    cmd: |
      ${llamaCmd}
      -hf unsloth/Qwen3.5-27B-GGUF:Q4_K_M
      --port ${PORT}
      --ctx-size 16384
      -ngl 99
    ttl: 600
 
  "qwen3-coder-30b":
    cmd: |
      ${llamaCmd}
      -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M
      --port ${PORT}
      --ctx-size 16384
      -ngl 99
    ttl: 600
 
  "gemma4-26b-a4b":
    cmd: |
      ${llamaCmd}
      -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
      --port ${PORT}
      --ctx-size 16384
      -ngl 99
    ttl: 600
 
  "embeddinggemma-300m":
    cmd: |
      ${llamaCmd}
      -hf ggml-org/embeddinggemma-300M-GGUF
      --port ${PORT}
      --embedding
      --ctx-size 2048
      -ngl 99
    ttl: 0
 
  "qwen3-embedding-0.6b":
    cmd: |
      ${llamaCmd}
      -hf Qwen/Qwen3-Embedding-0.6B-GGUF
      --port ${PORT}
      --embedding
      --ctx-size 2048
      -ngl 99
    ttl: 0
 
groups:
  "chat":
    swap: true # unload current model before loading the next
    exclusive: true # only one member of this group runs at a time
    members:
      - "qwen3.6-27b"
      - "qwen3.5-27b"
      - "qwen3-coder-30b"
      - "gemma4-26b-a4b"
 
  "embeddings":
    swap: false # both can stay loaded simultaneously — they're tiny
    exclusive: false
    members:
      - "embeddinggemma-300m"
      - "qwen3-embedding-0.6b"

Key points:

• ${PORT} is filled in automatically by llama-swap per model instance — don't hardcode a port.
• ttl is the idle-unload timer in seconds. 0 means never auto-unload (used here for the small embedding models, which barely use VRAM).
• The groups block is what enforces "only one big chat model loaded at a time" — without it, llama-swap would try to load whatever's requested without unloading the previous one, and you'd OOM with two 16+GB models in 16GB of VRAM.
• healthCheckTimeout matters more than it looks. The default (120s / 2 minutes) is too short for a first-time download of an 18-27GB model — llama-swap will give up and return a 500 error (health check timed out after 2m0s) before the download even finishes, even though the model would have loaded fine given more time. Set this high (e.g. 1800s) or pre-download every model with llama-cli first so llama-swap only ever loads from local cache.

Run as a systemd service

sudo nano /etc/systemd/system/llama-swap.service

[Unit]
Description=llama-swap model router
After=network.target
 
[Service]
Type=simple
User=majed
Environment="LLAMA_CACHE=/home/majed/llama-models"
WorkingDirectory=/home/majed/llama-swap
ExecStart=/usr/local/bin/llama-swap --config /home/majed/llama-swap/config.yaml --listen 0.0.0.0:8080
Restart=on-failure
RestartSec=5
 
[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable llama-swap
sudo systemctl start llama-swap
sudo systemctl status llama-swap

Watch logs live:

journalctl -u llama-swap -f

Verify

curl http://localhost:8080/v1/models

This lists every model defined in the config, regardless of whether it's been downloaded yet — it's not a "what's currently loaded" list. To actually trigger a load (and a download, if not cached):

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.6-27b", "messages": [{"role": "user", "content": "hi"}]}'

6. Coexisting with Ollama on the same GPU

You don't need to uninstall Ollama to run llama.cpp/llama-swap alongside it. The only real conflict is VRAM, not ports (Ollama defaults to 11434, llama-swap here uses 8080).

Before testing llama.cpp/llama-swap, make sure Ollama isn't holding GPU memory:

ollama ps
ollama stop <model-name>   # if anything's loaded

Or stop the Ollama service entirely while testing:

sudo systemctl stop ollama

Ollama also unloads idle models automatically after a few minutes by default, so in practice this is rarely an issue once everything is running steady-state — it mainly bites during active back-to-back testing of both stacks.

7. Models that won't fit, and why

Worth documenting since it's easy to assume any GGUF will run if you wait long enough.

• Architecture not yet in mainline llama.cpp. Some very new model releases (e.g. ones using a fresh MoE architecture not yet merged into ggml-org/llama.cpp) require building from an open pull request branch instead of main. If llama-server errors with an unrecognized architecture, check the model card on Hugging Face — it'll usually say explicitly which fork/branch/PR is required.
• Raw size exceeds available memory, full stop. Frontier-scale MoE models in the hundreds-of-billions-to-trillion parameter range (e.g. DeepSeek-V4-Pro at ~1.6T params, ~860GB+ even quantized) are not a "pick a smaller quant" problem on consumer hardware — there is no quant level that fits a 16GB GPU + ~31GB RAM. Going below Q4 on models whose experts are already trained at low precision (FP4-native MoE weights) tends to produce broken or incoherent output rather than a usable smaller model. If a model's smallest practical quant is still 10-50x your available memory, it's not a configuration problem — different hardware is the only fix.

8. Connecting to Open WebUI

In Open WebUI's admin settings:

1. Go to Settings → Connections
2. Add a new OpenAI API connection (not Ollama)
3. Base URL: http://<server-ip>:8080/v1
4. API key: any placeholder string — llama-swap doesn't require auth by default

Your configured model names (qwen3.6-27b, qwen3.5-27b, etc.) should appear in Open WebUI's model dropdown, and llama-swap will transparently load/unload backends as you switch between them in the UI.

Summary checklist

• Install CUDA toolkit matching your GPU/driver, verify with nvcc --version
• Build llama.cpp with -DGGML_CUDA=ON
• Set LLAMA_CACHE permanently, before downloading anything
• Download each model with a real prompt (-p "hi" -n 10), not --version
• Always exit interactive llama-cli sessions cleanly — check nvidia-smi for zombie VRAM holders before debugging OOM errors
• Install llama-swap, set healthCheckTimeout high enough for first-time downloads
• Use groups with exclusive: true to prevent multiple large models loading simultaneously
• Point Open WebUI's OpenAI connection at llama-swap's /v1 endpoint