# ggml CUDA backend built against the Nvidia-provided repositories ###

The ggml CUDA backend in the official Debian/Ubuntu APT repositories, package
[libggml0-backend-cuda](https://packages.debian.org/sid/libggml0-backend-cuda),
is built using the Nvidia driver and CUDA packages from the same official 
repositories.

In some cases, these Nvidia packages (and possibly also the Linux kernel) might
be too old to support the most recent hardware.
These Nvidia packages might also not be appropriate when running a cloud kernel 
(since the Debian-provided Nvidia driver requires a full-fledged kernel).

To address this, the Debian/Ubuntu ggml maintainers provide versions of the
ggml CUDA backend built against the CUDA packages in Nvidia's APT repository.
Currently, the following versions of this backend are provided:

  - Package `libggml0-backend-vendor-cuda-13`, built against CUDA 13

The maintainers ship these packages from their private APT repository, for the
following releases:

  - `trixie-backports`, for Debian 13 (trixie)
  - `noble-backports`, for Ubuntu 24.04 (noble)
  - `resolute-backports`, for Ubuntu 26.04 (resolute)


## Prerequisites

### Enable official backports

To enable backports on Debian, follow the official instructions at
[Add Backports to sources.list](https://backports.debian.org/Instructions/#index2h2).

Ubuntu enables backports by default, so no action is required there.

### Enable Nvidia's repository

The full instructions to enable the vendor-provided repository and to install
the Nvidia driver are provided by Nvidia at:

 - [NVIDIA Driver Installation Guide (Debian)](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/debian.html)
 - [NVIDIA Driver Installation Guide (Ubuntu)](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/debian.html)


## Enable the Debian/Ubuntu ggml maintainers' private repository

To make use of the maintainers' private APT repository, you'll first need to
install that repository's archive key:

```
$ sudo wget -O /usr/share/keyrings/ai-archive-keyring.gpg \
        https://apt.ai.debian.net/debian/ai-archive-keyring.gpg
```

Then, you need to create a `.sources` file for that repository. Pick the
version of this file applicable to your distribution from below: 

```shell
# Debian 13 trixie
$ sudo tee /etc/apt/sources.list.d/llama.cpp.sources <<EOF
Types: deb deb-src
URIs: https://apt.ai.debian.net/llama.cpp/debian
Suites: trixie-backports
Components: vendor-nvidia
Signed-By: /usr/share/keyrings/ai-archive-keyring.gpg
EOF

# Ubuntu 24.04 noble
$ sudo tee /etc/apt/sources.list.d/llama.cpp.sources <<EOF
Types: deb deb-src
URIs: https://apt.ai.debian.net/llama.cpp/ubuntu
Suites: noble-backports
Components: vendor-nvidia
Signed-By: /usr/share/keyrings/ai-archive-keyring.gpg
EOF

# Ubuntu 26.04 resolute
$ sudo tee /etc/apt/sources.list.d/llama.cpp.sources <<EOF
Types: deb deb-src
URIs: https://apt.ai.debian.net/llama.cpp/ubuntu
Suites: resolute-backports
Components: vendor-nvidia
Signed-By: /usr/share/keyrings/ai-archive-keyring.gpg
EOF
```

Finally, run `sudo apt update`.


## Installing the backend

With everything configured as per the instructions above, all you need to do is
install the backend (it will also install the required CUDA libraries):

```shell
# For CUDA 13. Replace <name> with trixie, noble, or resolute
$ sudo apt install libggml0-backend-vendor-cuda-13/<name>-backports
```

If you still have
[libggml0-backend-cuda](https://packages.debian.org/sid/libggml0-backend-cuda)
installed, the above command should automatically uninstall it. This is
expected, as both versions cannot be installed at the same time.
