> ## Documentation Index
> Fetch the complete documentation index at: https://felimet-hub.jmcores.com/llms.txt
> Use this file to discover all available pages before exploring further.

# GPUStack inference service deployment

> Run GPUStack server and worker on one GPU host with Docker Compose and expose an OpenAI-compatible LLM inference API via Cloudflare Tunnel.

export const ArchFlow = ({nodes = [], lang = "zh"}) => {
  const t = lang === "en" ? {
    flow: "Data flow",
    hint: "Click any stage to see what it does",
    stage: "Stage"
  } : {
    flow: "資料流",
    hint: "點任一階段看它做什麼",
    stage: "階段"
  };
  const safe = Array.isArray(nodes) ? nodes : [];
  const [active, setActive] = useState(0);
  if (safe.length === 0) return null;
  const idx = Math.min(active, safe.length - 1);
  const current = safe[idx];
  const css = `
  .af-root{--af-bg:#FAF8F3;--af-surface:rgba(191,117,81,0.06);--af-surface2:rgba(0,0,0,0.025);--af-border:rgba(0,0,0,0.09);--af-text:#2b2722;--af-dim:#6f6a62;--af-faint:#8a8378;--af-rail:rgba(0,0,0,0.12);--af-accent:#bf7551;border:1px solid var(--af-border);border-radius:14px;background:var(--af-bg);color:var(--af-text);overflow:hidden;}
  .dark .af-root{--af-bg:#1b1a18;--af-surface:rgba(207,138,104,0.09);--af-surface2:rgba(255,255,255,0.03);--af-border:rgba(255,255,255,0.08);--af-text:#e7e3da;--af-dim:#a8a299;--af-faint:#8a8378;--af-rail:rgba(255,255,255,0.13);--af-accent:#cf8a68;}
  .af-head{padding:13px 18px 11px;display:flex;align-items:center;gap:9px;flex-wrap:wrap;border-bottom:1px solid var(--af-border);}
  .af-head-ic{color:var(--af-accent);flex-shrink:0;}
  .af-head-t{font-size:12px;font-weight:700;letter-spacing:.6px;color:var(--af-dim);text-transform:uppercase;}
  .af-head-h{font-size:12px;color:var(--af-faint);}
  .af-grid{display:flex;gap:0;align-items:stretch;}
  .af-list{flex:1 1 0;min-width:0;padding:12px 12px 16px;}
  .af-node{display:flex;align-items:stretch;gap:10px;width:100%;text-align:left;background:transparent;border:none;cursor:pointer;padding:0;color:inherit;font:inherit;-webkit-tap-highlight-color:transparent;}
  .af-railcol{width:22px;flex-shrink:0;display:flex;flex-direction:column;align-items:center;}
  .af-dot{width:9px;height:9px;border-radius:50%;margin-top:14px;background:var(--af-faint);opacity:.5;flex-shrink:0;transition:background .15s,box-shadow .15s,opacity .15s;}
  .af-line{width:2px;flex:1 1 0;background:var(--af-rail);min-height:10px;}
  .af-body{flex:1 1 0;min-width:0;padding:9px 12px;margin:2px 0;border-radius:9px;border:1px solid transparent;transition:background .15s,border-color .15s;}
  .af-node:hover .af-body{background:var(--af-surface2);}
  .af-node-on .af-body{background:var(--af-surface);border-color:rgba(191,117,81,.30);}
  .af-node-on .af-dot{background:var(--af-accent);opacity:1;box-shadow:0 0 0 4px rgba(191,117,81,.15);}
  .af-label{font-size:14.5px;font-weight:550;line-height:1.45;}
  .af-node-on .af-label{color:var(--af-accent);}
  .af-detail{width:300px;flex-shrink:0;border-left:1px solid var(--af-border);padding:16px 18px;background:var(--af-surface2);}
  .af-detail-k{font-size:11px;font-weight:700;letter-spacing:.5px;text-transform:uppercase;color:var(--af-accent);margin-bottom:7px;}
  .af-detail-l{font-size:16px;font-weight:600;margin-bottom:9px;line-height:1.35;}
  .af-detail-d{font-size:13.5px;line-height:1.65;color:var(--af-dim);}
  @media (max-width:640px){.af-grid{flex-direction:column;}.af-detail{width:auto;border-left:none;border-top:1px solid var(--af-border);}}
  `;
  return <div className="af-root">
      <style>{css}</style>
      <div className="af-head">
        <svg className="af-head-ic" xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round"><rect width="8" height="8" x="3" y="3" rx="2" /><path d="M7 11v4a2 2 0 0 0 2 2h4" /><rect width="8" height="8" x="13" y="13" rx="2" /></svg>
        <span className="af-head-t">{t.flow}</span>
        <span className="af-head-h">{t.hint}</span>
      </div>
      <div className="af-grid">
        <div className="af-list">
          {safe.map((n, i) => <button key={i} type="button" className={"af-node" + (i === idx ? " af-node-on" : "")} onClick={() => setActive(i)} onMouseEnter={() => setActive(i)}>
              <div className="af-railcol">
                <div className="af-dot" />
                {i < safe.length - 1 && <div className="af-line" />}
              </div>
              <div className="af-body">
                <div className="af-label">{n.label}</div>
              </div>
            </button>)}
        </div>
        <div className="af-detail">
          <div className="af-detail-k">{t.stage} {idx + 1} / {safe.length}</div>
          <div className="af-detail-l">{current.label}</div>
          <div className="af-detail-d">{current.detail}</div>
        </div>
      </div>
    </div>;
};

`GPUStack` · `LLM inference` · `Cloudflare Tunnel` · `Docker` · `NVIDIA`

## Overview

This turns a single GPU host into an inference service for LLM, VLM, embedding, and rerank models. The GPUStack v2.1.2 server and worker run as Docker Compose containers, exposed through a Cloudflare Tunnel so the host opens no inbound ports. The public interface is OpenAI-compatible, so existing code using the openai SDK connects with almost no changes.

## Core features

* server and worker as two containers, brought up with a single `docker compose up`
* Cloudflare Tunnel for public access; the host firewall opens no inbound ports
* Two hostnames, split by purpose: the API uses a CF Access Service Token, the Web UI uses an email allowlist
* OpenAI-compatible endpoint (`/v1/chat/completions`), usable directly with the openai SDK
* Models deployed from the Web UI, sourced from HuggingFace, Ollama, or a local path

## Architecture

<ArchFlow
  lang="en"
  nodes={[
{ label: "External client", detail: "Reaches Cloudflare over HTTPS and never touches the host directly." },
{ label: "Cloudflare Edge", detail: "TLS termination and Access checks: the API hostname uses a Service Token, the Web UI hostname uses an email allowlist." },
{ label: "Cloudflare Tunnel", detail: "Outbound-only: cloudflared dials out to Cloudflare from inside, so the host firewall opens no inbound ports." },
{ label: "gpustack server", detail: "Web UI and gateway; receives inference requests and routes them to the worker." },
{ label: "gpustack worker", detail: "Runs the NVIDIA runtime and manages the lifecycle and scheduling of inference containers." },
{ label: "Inference container", detail: "A llama.cpp or vLLM container spawned on demand by GPUStack, running on host networking with direct GPU access." }
]}
/>

The server and worker talk over an internal bridge network, and inference containers attach to the worker via host networking, so the outside only ever reaches the Cloudflare layer.

## Quick start

<Steps>
  <Step title="Prerequisites (fresh machine only)">
    ```bash theme={null}
    sudo bash scripts/prereq.sh
    ```

    Checks the GPU, installs Docker Engine and the NVIDIA Container Toolkit, and creates the data directories.
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    cp .env.example .env
    ```

    Fill in the admin password, the shared server/worker token, the HuggingFace token, and the Cloudflare Tunnel token.
  </Step>

  <Step title="Configure the Cloudflare Tunnel">
    In Zero Trust → Tunnels, create a tunnel and point the Public Hostname service URL at the container name `http://gpustack:80`, not localhost.
  </Step>

  <Step title="Start and verify">
    ```bash theme={null}
    docker compose up -d
    docker compose logs -f gpustack
    ```
  </Step>
</Steps>

## Notes

<Warning>
  For the API hostname that uses a Service Token, leave the Public Hostname Access field unbound to any Application. If you bind one, cloudflared re-validates at the origin, drops the Service Token JWT, and the CF edge returns 502. The `.env` file holds passwords and tokens, so keep it out of version control.
</Warning>

## In practice

This fits a lab machine or any host without a public IP where you want self-hosted LLM/VLM inference, a programmatic API for other services to call, and no need to open inbound firewall ports or rent cloud GPUs.

## Links

* GitHub: [felimet/gpustack-api](https://github.com/felimet/gpustack-api)
* GPUStack docs: [docs.gpustack.ai](https://docs.gpustack.ai)