Skip to content
SkyKeephelp

Using a GPU for the models

Optional, and off by default on every installation. SkyKeep ships CPU-only because that is the configuration that runs on any machine you install it on; a GPU is a speed-up you opt into on the hosts that have one. Turning it on is a single setting here and a one-time root-level change to the host, in that order of importance.

This page quotes no timings on purpose. What a model costs depends on your hardware, and this deployment measures its own rather than inheriting a figure from the machine SkyKeep was built on — see the model page in the administration console.

  1. What this changes, and what it does not

    Only where the answering and embedding models run. SkyKeep itself never learns what a GPU is: it speaks to the model runtime over HTTP, and the runtime does its own hardware detection. Nothing about compartments, encryption, audit or retrieval behaves differently — the same question returns the same answer from the same evidence, sooner. If you have no GPU, you need nothing on this page: the shipped default is CPU and it is a supported configuration, not a degraded one.

  2. Step 1 — confirm the host can actually do this

    Two separate things must be true and they are commonly confused. The first is a working NVIDIA driver on the host, which is what the first command reports. The second is that the Docker daemon has NVIDIA's container runtime, which is a different piece of software and is what the second command reports. A host can pass the first and fail the second — that is the usual case on a fresh machine, and it is the case this page exists for.

    nvidia-smi
    docker info --format '{{range $k, $v := .Runtimes}}{{$k}} {{end}}'

  3. Step 2 — install the container runtime (needs root, once)

    If the second command above did not list 'nvidia', install NVIDIA's container toolkit and register it with Docker. This is a one-time change to the HOST, not to SkyKeep, and it needs root and outbound network access to NVIDIA's package repository — so on an air-gapped installation it must be done from your own mirror, or not at all. These are NVIDIA's published steps for an apt-based distribution; check their current installation guide if your distribution differs, because this is their software and not ours.

    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
    sudo nvidia-ctk runtime configure --runtime=docker
    sudo systemctl restart docker

  4. Step 3 — prove it before changing SkyKeep

    Run the model runtime's own image against the GPU with no SkyKeep involved. If this prints your card, the host is ready and any later failure is SkyKeep's configuration rather than the driver — which is a much shorter thing to debug. If it fails, stop here: enabling the setting below will only move the same failure into your vault's startup.

    docker run --rm --runtime=nvidia --gpus all --entrypoint nvidia-smi ollama/ollama:0.32.14

  5. Step 4 — tell the deployment to use it

    One setting in your environment file. 'runc' is the default and means CPU; 'nvidia' hands the model service the host's GPUs. Nothing else changes — the same compose file, the same services, the same ports, the same data volumes. Then restart the deployment; your documents and audit history are untouched, because only the model service's runtime changed.

    SKYKEEP_OLLAMA_RUNTIME=nvidia
    scripts/skykeep.sh restart

  6. If the runtime is missing, SkyKeep refuses to start

    Deliberately, and it is worth saying why, because refusing to start looks unhelpful. The failure being prevented is not a crash — it is a vault that comes up healthy, answers every question, and does it all on the CPU while the configuration says otherwise. Nobody notices that for weeks. A performance setting that silently does nothing is the same class of defect as a scanner that silently passes, so this one says so and stops. The message names the runtimes your daemon does have, and setting SKYKEEP_OLLAMA_RUNTIME back to 'runc' always starts.

  7. What a GPU does not fix

    Two things, and both matter more than the speed-up. A question whose evidence is large can still take longer than the front door will wait, on any hardware — a faster machine moves that boundary rather than removing it. And the amount of memory on the card is a real ceiling: a model larger than the card holds is served partly from system memory and can be SLOWER than the CPU path. Choose the answering model against the card you have, on the administration console's model page, which states each model's cost against this deployment's own measurements rather than against anyone else's.