On this page

A model served inside the cluster#

A recipe for running the install's model on your own hardware, so that no prompt leaves the network: an OpenAI-compatible server (vLLM, Ollama, llama.cpp server and the like) in a pod of the cluster, reached by core over its Service.

1. Serve the model#

Any server that speaks POST /v1/chat/completions with tool calling. It is not part of this chart; run it in the namespace of your choice with a ClusterIP Service, for example http://model.llm.svc:8000/v1 (vLLM) or http://ollama.llm.svc:11434/v1 (Ollama).

2. Values to set#

yaml

core:
  env:
    AI_PROVIDER: openai-compatible
    OPENAI_BASE_URL: http://model.llm.svc:8000/v1      # with the version path; core adds /chat/completions
    AI_OPENAI_MODEL: <the model id the server serves>
    OPENAI_API_KEY: none                               # any word if the server asks for no key
    OUTBOUND_ALLOWED_HOSTS: model.llm.svc              # an address inside the cluster is refused unless named here
    AI_TIMEOUT: 300s                                   # a local model is slower than a hosted one
networkPolicy:
  egress:
    extraCidrs:
      - 10.42.0.0/16                                   # the pod range of the cluster, or of the model's nodes

OUTBOUND_ALLOWED_HOSTS lets core's own check accept the host; extraCidrs lets the network policy carry the packet. The internet rule of the chart excludes the private ranges, so without the second one core cannot reach the pod even though the first accepts the name. Give the range the model pod's address falls in. In a closed install also set networkPolicy.egress.core.internet=false (air-gapped.md); the image of the model server then comes from your own registry.

3. Or bring it per organisation#

The same model can be saved by an administrator in the settings screen as an OpenAI-compatible endpoint, per tier (models.md). The address must still be named in OUTBOUND_ALLOWED_HOSTS.

4. Test it#

A saved configuration is not used until it has passed a test: the screen sends one short question and shows the result. The agents also need the model to answer in JSON and to call tools when offered; a small model that cannot will pass the plain test but the AI features that need those will not work well.