Skip to main content

Building the GPU Kubernetes cluster

📚 Documentation home

GShare deploys onto an existing GPU Kubernetes cluster. This page covers building that prerequisite cluster from scratch. The automation lives in hack/cluster-bootstrap.sh.

The stack it produces:

ComponentChoice
Kuberneteskubeadm 1.36 — one control plane, N GPU workers, optionally M CPU workers
Runtimecontainerd with the NVIDIA runtime (RuntimeClass nvidia)
CNIflannel, pod CIDR 10.244.0.0/16
GPU partitioningHAMi — device plugin and scheduler, nvidia.com/gpumem and gpucores
Ingressingress-nginx on NodePort 30080/30443, plain HTTP
TLSTerminated by a reverse proxy in front; the cluster stays HTTP internally
Storagelocal-path as the default StorageClass; optionally a storage node (ZFS + NFS behind democratic-csi) for user volumes with a real, df-visible quota

Write cluster-info on a control machine — your laptop or a bastion — and let the script set up every node over SSH.

Prerequisites: from the control machine to each node, (1) SSH key authentication (ssh-copy-id) and (2) passwordless remote sudo must already work.

cp hack/cluster-info.example hack/cluster-info   # fill in MASTER_NODE and WORKER_NODES
./hack/cluster-bootstrap.sh up                   # prereqs → init → join → addons → label → verify

cluster-info format:

MASTER_NODE=ubuntu@10.0.0.10
# Workers are user@ip:MODE, where MODE is exclusive or fractional (default: fractional).
WORKER_NODES=ubuntu@10.10.0.196:exclusive,ubuntu@10.10.0.197:fractional
# Optional GPU-less workers, comma separated. Labeled `cpu`; the NVIDIA runtime is not installed.
# CPU_WORKERS=ubuntu@10.10.0.200,ubuntu@10.10.0.201
# Optional storage node: a ZFS pool on the given (empty) block device, exported over NFS through
# democratic-csi as StorageClass gshare-data. Labeled and tainted `storage`.
# STORAGE_NODE=ubuntu@10.10.0.194:/dev/vdb
# Optional plain-HTTP registry on the LAN, configured for containerd on every node.
# LOCAL_REGISTRY=10.10.0.191:5001
# SSH_OPTS="-i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=accept-new"   # optional

up copies the script to every node and then runs, in order: prerequisites (control plane, CPU workers, and GPU workers in parallel) → reboot and reconnect any GPU worker that just got a driver → kubeadm init on the control plane, harvesting the join command → workers join (GPU and CPU in parallel) → addons on the control plane → node labeling (control plane and CPU workers as cpu, GPU workers with their configured mode) → verification.

hack/cluster-info contains real addresses and is git-ignored; only cluster-info.example is tracked. hack/storage-csi-key (the SSH key pair the script generates for democratic-csi when a storage node is configured) is git-ignored too — a fresh clone simply generates a new pair on the next up.

To run the steps by hand on each node, use the same subcommands directly — see below.

Prerequisites​

  • Node OS: Ubuntu 22.04 or 24.04 (apt-based). One control plane, N GPU workers, and optionally M GPU-less CPU workers.
  • NVIDIA driver: if nvidia-smi does not work on a GPU worker, prereqs --gpu installs one — NVIDIA_DRIVER=auto uses the ubuntu-drivers recommendation. Pin a version with NVIDIA_DRIVER=nvidia-driver-550-server, or skip installation entirely with NVIDIA_DRIVER=skip. A fresh install needs a reboot for the kernel module to load. up reboots the node itself and waits for it to come back (a changed boot_id) with a working nvidia-smi before continuing; the timeout is REBOOT_WAIT_TRIES × 5 s, ten minutes by default. Run standalone, prereqs --gpu only prints the instruction unless you pass NVIDIA_DRIVER_REBOOT=1. Nodes with a working driver are left alone.
  • helm v3: installed automatically during the addons step if missing (via get-helm-3). Disable with HELM_SKIP_INSTALL=1.
  • Network: between nodes, open 6443 (API server), 10250 (kubelet), 8472/udp (flannel VXLAN), and the NodePort range 30000–32767.

Step by step​

1. Every node — prerequisites​

# Control plane and CPU-only nodes
sudo ./hack/cluster-bootstrap.sh prereqs

# GPU workers — also installs nvidia-container-toolkit and the runtime,
# and the driver itself if nvidia-smi is missing.
sudo ./hack/cluster-bootstrap.sh prereqs --gpu

This installs containerd with SystemdCgroup, the required kernel modules and sysctls, disables swap, and installs kubeadm, kubelet, and kubectl 1.36. When a driver is newly installed the script asks for a reboot; add NVIDIA_DRIVER_REBOOT=1 to have it reboot itself.

2. Control plane — initialize​

sudo ./hack/cluster-bootstrap.sh init

Runs kubeadm init, applies flannel, and prints the kubeadm join … command. Copy it.

3. Each worker — join​

sudo ./hack/cluster-bootstrap.sh join "kubeadm join 10.x.x.x:6443 --token ... --discovery-token-ca-cert-hash sha256:..."

4. Control plane — addons​

./hack/cluster-bootstrap.sh addons

Installs RuntimeClass nvidia, HAMi, ingress-nginx on a NodePort, and local-path as the default StorageClass.

5. Control plane — label the nodes​

HAMi needs gpu=on and GShare scheduling needs gshare.io/gpu-mode. The up path has already applied the modes from cluster-info — this step is only for the manual path.

./hack/cluster-bootstrap.sh label <gpu-node-1> exclusive
./hack/cluster-bootstrap.sh label <gpu-node-2> fractional
./hack/cluster-bootstrap.sh label <control-plane> cpu

6. Verify​

./hack/cluster-bootstrap.sh verify

Checks that nodes are Ready, that nvidia.com/gpu and nvidia.com/gpumem capacity is advertised, that the HAMi, ingress, flannel, and local-path pods are Running, and that a default StorageClass exists.

Storage node (optional): volumes with a real quota​

local-path serves RWO only and enforces nothing: a 50 GiB volume shows the node's whole disk in df and can fill it. A storage node fixes both. STORAGE_NODE=user@ip:/dev/<device> in cluster-info makes up run, after the addons:

  1. storage <device> on that node — installs zfsutils-linux and nfs-kernel-server, creates the pool gshare on the device (taken whole; several devices as a mirror or raidz via STORAGE_VDEVS), the datasets gshare/volumes and gshare/snapshots, caps the ARC at ZFS_ARC_MAX_MB (3 GiB), and creates the gshare-csi account: SSH key only, sudo restricted to the zfs/exportfs/chown commands the driver runs.
  2. storage-csi <ip> <key> on the control plane — installs democratic-csi (zfs-generic-nfs, values in deploy/storage/democratic-csi-values.yaml) and the StorageClass gshare-data: one dataset per PVC with refquota = the claim, so inside a session df shows exactly the quota and writes past it fail with ENOSPC; RWO, RWX, and ROX all come from this one class; allowVolumeExpansion for approved quota increases.
  3. label <node> storage — gshare.io/role=storage plus a NoSchedule taint, so sessions and the control plane never land on it.

Every node gets nfs-common from prereqs. Then point the operator at the class: --set operator.volumeStorageClass=gshare-data (or uncomment it in the values overlay). The operator reports each volume PVC to the control plane every operator.volumeSyncInterval (5m): used bytes for the ledger, approved quota growth applied to the claim, and PVCs of volumes deleted more than api.volumeReclaimGraceHours (24h) ago reclaimed — dataset included.

By hand, in that order: sudo STORAGE_CSI_PUBKEY="$(cat key.pub)" bash cluster-bootstrap.sh storage /dev/vdb on the storage node, then bash cluster-bootstrap.sh storage-csi <ip> key and bash cluster-bootstrap.sh label <node> storage on the control plane.

The pool is a single point of failure by design (no replication); take ZFS snapshots on the node and replicate with zfs send off-box for durability.

Local registry (optional)​

Session images are large (5–15 GiB); on a LAN a local registry beats pulling from Docker Hub on every node. Two pieces, both optional:

  1. The registry itself — deploy/registry/registry.yaml runs two plain-HTTP registries on the storage node (gshare.io/role=storage): a push target on :5000 for locally built session images, and a docker.io pull-through mirror on :5001 so public catalog images are fetched from the LAN after the first pull. kubectl apply -f deploy/registry/registry.yaml (adjust the nodeSelector / storageClassName in the header if your layout differs).

  2. Node trust — containerd on every node must be told about them:

    sudo LOCAL_REGISTRY=<storage-ip>:5000 LOCAL_REGISTRY_MIRROR=<storage-ip>:5001 \
         bash cluster-bootstrap.sh registry     # per node; idempotent

    This writes /etc/containerd/certs.d/<host:port>/hosts.toml (and a docker.io entry routing pulls through the mirror, upstream as fallback) and sets config_path in config.toml — containerd 2.x silently ignores certs.d without it, and every pull ends in "server gave HTTP response to HTTPS client". The up path runs this on all nodes when LOCAL_REGISTRY is set in cluster-info. Moving to TLS later means dropping the CA next to hosts.toml; the layout stays.

Build and push the catalog images to it from any docker host that lists the registry under "insecure-registries":

REG=<storage-ip>:5000 build/images/build.sh push   # builds all catalog images, pushes to the LAN

then register them in the console (Images → the registry reference is <storage-ip>:5000/gshare-session:<tag>); nodes pull straight from the LAN.

Growing or shrinking a running cluster​

The steps above describe a first build, but each subcommand is idempotent and stands on its own, so the same pieces add a machine to a cluster that is already serving sessions. Nothing here interrupts running work: existing nodes are untouched.

Adding a node​

  1. On the new node — prerequisites. Swap off, kernel modules and sysctls, containerd, nfs-common, and — for a GPU node — the NVIDIA runtime (and the driver, if nvidia-smi is missing):

    sudo ./hack/cluster-bootstrap.sh prereqs        # add `gpu` on a GPU node

  2. On the control plane — mint a join command, then run it on the new node:

    kubeadm token create --print-join-command       # control plane
    sudo ./hack/cluster-bootstrap.sh join "kubeadm join 10.x.x.x:6443 --token ... --discovery-token-ca-cert-hash sha256:..."

  3. On the control plane — label it. Skipping this is the usual reason a new GPU node advertises no capacity: HAMi only claims cards on nodes labeled gpu=on, and placement reads gshare.io/gpu-mode.

    ./hack/cluster-bootstrap.sh label <node> fractional    # or: exclusive | cpu | storage

  4. On the new node — the LAN registry, if the cluster uses one. prereqs writes this from LOCAL_REGISTRY, so a node joined by hand misses it and every session image pull fails with http: server gave HTTP response to HTTPS client — the node has no reason to know the registry speaks plain HTTP. Re-running prereqs with the variable set is the safe way:

    sudo LOCAL_REGISTRY=<registry-ip>:5000 LOCAL_REGISTRY_MIRROR=<registry-ip>:5001 \
         ./hack/cluster-bootstrap.sh prereqs

    Doing it by hand instead? Copy /etc/containerd/certs.d/ from a working node and set config_path = "/etc/containerd/certs.d" in /etc/containerd/config.toml. Two rules that cost an outage once: back the file up first (cp config.toml config.toml.bak), and write the value with an editor rather than a sed/regex substitution — the file also contains plugin_config_path, and a pattern that matches the shorter key corrupts the longer one, after which containerd refuses to start. Restart containerd from the node's own console (sudo systemctl restart containerd), never from a pod on that node: stopping containerd kills the pod issuing the command, the start half never runs, and the node then has no working kubectl exec to repair it with.

  5. Verify — nodes Ready, nvidia.com/gpu and nvidia.com/gpumem advertised, addons Running:

    ./hack/cluster-bootstrap.sh verify

  6. Record it in hack/cluster-info (WORKER_NODES, CPU_WORKERS, …) so a later up run keeps the file and the cluster in step.

Then, in GShare itself:

  1. Nothing to register. The operator's inventory controller reports the node and its cards on its next tick; the control plane upserts them by (cluster, hostname) and the node appears under Nodes on its own. node_liveness moves it to ready as the heartbeats arrive (and back to offline if they stop for NODE_STALE_SEC, 5 minutes by default).

  2. Check the offering for that GPU model. Admission matches offering.gpu_model to the device model by string equality, so a card whose model has no active offering is visible to administrators but unusable — session creation answers 409 unserviceable. The catalog row's "in cluster" tag under Resources → GPU offerings tells you which models are backed by real cards; create or activate an offering for a new model before announcing the capacity.

  3. Optional, in the console:

    • Node pool — assign the node to a dedicated pool if a department should get it first; unassigned nodes are shared by everyone.
    • Pre-pull the session images on the node so the first session does not wait on a 5–15 GiB pull (with the LAN registry configured in step 4 this is fast, but not instant).
    • Lossless pause — label the node gshare.io/criu=ready and gshare.io/cuda-checkpoint=ready once CRIU and cuda-checkpoint are in place, or sessions there fall back to a cold pause.

Removing a node​

Drain first, delete second: the ledger must not be left holding a session on hardware that is gone.

  1. In the console — terminate or pause the sessions still on the node (Sessions, filtered by node), so their credits settle.

  2. On the control plane:

    kubectl cordon <node>
    kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
    kubectl delete node <node>

    DaemonSets (calico, kube-proxy, the HAMi device plugin, dcgm-exporter, node-exporter, the CSI node plugin) are expected to stay; that is what --ignore-daemonsets is for.

  3. On the removed machine, if it is leaving for good: sudo kubeadm reset -f and clear /etc/cni/net.d.

  4. In the console — Nodes → Delete. The button appears once the node reads offline (node_liveness marks it after the heartbeat stops). Deletion is refused with 409 node_busy while any live allocation or non-terminal session remains on it, and it removes the node and its card records while keeping the billing history — past allocations survive with their gpu_uuid, detached from the card that no longer exists.

  5. Tidy up — a dedicated pool that just lost its last node stays behind as an empty pool; delete it or assign another node. Sessions of a tenant whose pool went empty keep running and fall back to shared nodes, provided the tenant's policy allows the spill (shared_pool).

Deleting a node that is still in the cluster is pointless rather than dangerous: the operator's next inventory report recreates the row.

Next — deploy GShare​

With the cluster ready, the all-in-one install is the simplest path. The chart brings up the data tier, secrets, CRDs, namespaces, the operator's internal JWT, and the local cluster registration, with no manual prerequisites:

make deploy-incluster                          # deploy/values/incluster.yaml
ADMIN_JWT=$(./hack/bootstrap-superadmin.sh)    # optional: a super_admin token for admin API calls

Front it with your reverse proxy, terminating TLS for the console domain and forwarding to the ingress-nginx NodePort (:30080, HTTP). For a production install against external CloudNativePG, Redis, and external-secrets, use make prod-deploy with deploy/values/dockerhub.yaml.

Caveats​

  • This script reproduces one known-good stack. It is not guaranteed to work unmodified everywhere — driver versions, kernels, corporate proxies and registries, and firewall policy all force adjustments. Run verify after each stage.
  • Re-running is safe. prereqs, init, join, and addons are idempotent: keyrings are overwritten, the swap line is not commented twice, and addons use apply or helm upgrade -i. init probes the API server's /healthz to decide what to do — if it is healthy the step is skipped; if only leftovers from a previous init are present (occupied ports, manifests, etcd data) it runs kubeadm reset -f and re-initializes. Set INIT_FORCE_RESET=0 to be told about it instead. join is skipped when kubelet.conf exists. One exception: the containerd configuration is regenerated from defaults every time, so hand edits there are not preserved.
  • The HAMi scheduler image tag is set to match the cluster version automatically (scheduler.kubeScheduler.imageTag), but check it against the HAMi compatibility matrix.
  • Production hardening — multiple control planes, CephFS or NFS for RWX volumes, external-secrets with Vault or a KMS, and cosign verification with Kyverno — is out of scope for this script. See the reference configuration in the root README.md.