Connect a Kubernetes cluster#

The cluster Hyvop runs on is read in every plan, the Community Edition included, wherever it runs (on premises too), and it is not counted as a connection. In the UI it is This cluster: the one Hyvop runs on. Any other cluster needs a licence that includes it (which plan has it is in core's table, see editions.md): core refuses it without one. The Kubernetes ingestor is the same for both, and it is off until you ask for it:

helm upgrade hyvop ./hyvop -n hyvop --reuse-values --set ingestors.kubernetes.enabled=true

Core is told the ingestor's address (KUBERNETES_INGESTOR_URL) only when it is on. A cluster is then connected by an administrator, on the Connections page, under Kubernetes and on-premises.

The cluster Hyvop runs on, in every plan#

1. Turn the ingestor on with the command above. The chart makes the ingestor's service account, a read-only ClusterRole and its binding (The cluster Hyvop runs in): nothing is applied by hand. The Kubernetes card of the Connections page then says that its ingestor is on.

2. With the network policy on, which is the chart's default, the ingestor reaches this cluster's API server only at the addresses you give, and the API server of most clusters (kind, k3s, a managed cluster's private endpoint) is on a private address. Give its addresses:

   shCopy

   kubectl get endpoints kubernetes        # the API server's address(es)
   helm upgrade hyvop ./hyvop -n hyvop --reuse-values --set 'ingestors.kubernetes.inCluster.apiServerCidrs={10.0.0.5/32}'

   Without them the connection test stops at its first check, Reach the API server.

3. Open the Kubernetes card (Configure), give the cluster an id (the name Hyvop knows it by: it is the account of every resource it collects), and choose This cluster: the one Hyvop runs on as the way Hyvop reaches it. It is the only choice where the plan has no other clusters. Test the connection, then save: the first collection starts.

What is read is the table of What Hyvop may read. Its nodes and workloads count towards the resource limit of the plan, as those of any cluster do (What counts); its pods, namespaces and the rest are listed under Ownership › Resources and not counted, which the licence screen says beside the figure.

On a cluster on premises, Hyvop reads OpenCost only under its default service name and port, the service opencost in the namespace opencost on 9003 (the one service the role lets it reach): an OpenCost installed under another name is not read, the connection test says "No OpenCost service at opencost/opencost:9003. The cost of each project and workload is modelled from its nodes' cost.", and the rest of the cluster is still collected.

Hyvop reads only. It lists nodes, workloads (Deployments, StatefulSets, DaemonSets and Jobs), pods, namespaces, resource quotas, limit ranges and persistent volume claims, and reads the usage of nodes and pods from the metrics API (the cluster needs metrics-server for the usage). On OpenShift it also reads the cluster version and the routes, and where OpenCost runs, what it allocated. It reads no Secret, no ConfigMap, no log, and it cannot change anything.

What Hyvop may read#

Everything it is allowed is in one read-only role, and every rule of it is get and list, but the one to OpenCost, which is get alone:

On a cluster that serves no OpenShift API and runs no OpenCost, the last three rules allow nothing. The role the chart makes and the ingestor's deploy/hyvop-reader.yaml hold the same rules.

Not in the role, and so not readable: Secrets, ConfigMaps, logs, exec, ReplicaSets, CronJobs, anything that writes, and watch.

What counts, and what a workload costs#

A node and a workload (a Deployment, a StatefulSet, a DaemonSet, a Job) are resources and count towards the limit of a plan; a pod, a namespace, a quota, a limit range, a persistent volume claim and the cluster itself do not. A workload scaled to zero is still a workload. A Job that has finished is not one: a cluster that keeps thousands of finished Jobs counts the ones that run.

Kubernetes has no bill and no price list: what a node costs is collected from the cloud that runs it. What a workload costs is therefore modelled, never billed. What is collected with each workload is the model's input, named as modelled: the workload's share of each node it runs on. Per pod, the larger of the CPU it requests over the CPU the node allocates and the memory it requests over the memory the node allocates; where the pods of a node ask for more than the node, the shares are scaled to add up to one; what no pod requests is the node's idle share and belongs to no workload. A workload whose pods request nothing has no share, which is not a cost of zero. The amount is that share times what the node costs; a screen that shows it per workload is not built yet.

Two ways in#

The cluster Hyvop runs in#

With the ingestor on, the chart makes a ServiceAccount hyvop-ingestor-kubernetes, a read-only ClusterRole and a ClusterRoleBinding, and mounts the account's token in that pod only. The role is exactly the one of the reader manifest: the table of What Hyvop may read.

A ClusterRole and its binding belong to the cluster, not to the namespace, and this chart is otherwise namespaced. They are named hyvop-reader-<release namespace> so that two installs in two namespaces do not collide; two releases in the same namespace would. They go with the release (helm uninstall).

With the network policy on, the ingestor may reach this cluster's API server only at the addresses you give. A policy sees the address after the Service is resolved, and the internet rule leaves private ranges out:

kubectl get endpoints kubernetes        # the API server's address(es)
helm upgrade hyvop ./hyvop -n hyvop --reuse-values --set 'ingestors.kubernetes.inCluster.apiServerCidrs={10.0.0.5/32}'

Those addresses are opened on 443 and 6443.

If you only connect other clusters by token, switch the in-cluster access off:

helm upgrade hyvop ./hyvop -n hyvop --reuse-values --set ingestors.kubernetes.inCluster.enabled=false

The chart then makes no service account, no ClusterRole and no binding, the pod has no token of the Kubernetes API, and the ingestor is told (KUBERNETES_IN_CLUSTER_ALLOWED=false) that every connection must carry its own API server and token.

Another cluster#

The ingestor's project ships the manifest to apply in the cluster to be read, deploy/hyvop-reader.yaml: the same read-only role, a service account hyvop-reader in the namespace hyvop-system and a long-lived token Secret for it.

kubectl apply -f hyvop-reader.yaml
kubectl -n hyvop-system get secret hyvop-reader-token -o jsonpath='{.data.token}' | base64 -d      # the token
kubectl -n hyvop-system get secret hyvop-reader-token -o jsonpath='{.data.ca\.crt}' | base64 -d     # the certificate authority

What Hyvop needs of that cluster is its API server address (https), the token and the certificate authority. A kubeconfig works only if the credentials are inside it: one that names a file, an exec plugin (aws eks get-token, gke-gcloud-auth-plugin, kubelogin) or turns certificate checks off is refused, so for those clusters use the token of the service account. To revoke Hyvop's access, delete the Secret or the service account.

A cluster that applied the manifest before Hyvop read workloads has a role without apps and batch: apply the manifest again. Until then the connection test names the four reads that are refused, the workloads of that cluster are not collected, and everything else is read as before.

Several clusters#

An organisation connects as many clusters as its plan holds, each a connection of its own on the Kubernetes page of the UI (Add a cluster), collected on its own, with its own last read, and every row Hyvop keeps names the cluster it came from (its id). Each cluster other than the one Hyvop runs in counts as one connection of the plan; the one Hyvop runs in is free in every plan and is connected once. The same cluster twice is refused: the same id in any letter case, the same API server under another id, or a second connection that reads the cluster Hyvop runs in (a connection that names no API server, token or kubeconfig is read by the ingestor as that cluster). A cluster reached through a kubeconfig is told apart by its id alone. Removing a cluster takes its resources and its findings out of every figure at once, and the other clusters are untouched.

What the network allows#

The API server of a cluster you connect is reached on 443 or 6443 from the ingestor: both ports are open to the internet by default (networkPolicy.egress.ingestors.internet), and a server on a private address needs its network in ingestors.kubernetes.extraCidrs. ingestors.kubernetes.extraPorts (default [6443]) lists the ports besides 443. See air-gapped.md for a closed install.