Hyvop

Get 40% offSee the offer

0.9.0 betaDocumentation of the Community Edition, copied on .
On this page

Connect a GCP project#

The Community Edition connects one cloud account, on AWS, Azure or GCP (editions.md). The GCP ingestor is off until you ask for it:

sh
helm upgrade hyvop ./hyvop -n hyvop --reuse-values --set ingestors.gcp.enabled=true

An administrator then connects the project under Integrations, GCP.

Hyvop reads only. It signs in as a service account and reads resources and metrics in one project.

What is collected#

Inventory, metrics, prices and billing. Services the ingestor collects: Compute Engine, Persistent Disks, Cloud SQL, Cloud Storage, Cloud Functions (v1 and v2), Load Balancing, Cloud Run and BigQuery. Pricing is the public Cloud Billing catalog, on-demand prices only.

Billing#

Core asks the GCP ingestor for billing only when the connection names a billing export table. Turn on the Cloud Billing standard usage cost export to BigQuery (Billing, Billing export, BigQuery export), wait for the first rows to arrive, and enter the table as project.dataset.table in the billing table field of the wizard. Leave the field empty and the project is scanned without billed cost; add the table later by editing the connection. The ingestor reads the table one invoice month at a time (the current one and the two before it).

What to create in GCP#

  1. A service account in the project (IAM & Admin, Service Accounts); the screen suggests the name hyvop-reader, with the email hyvop-reader@<PROJECT_ID>.iam.gserviceaccount.com.

  2. One custom role holding the permissions below, granted to the service account on the project. No predefined role is used: the viewer roles hold more than the calls need. No permission reads inside a bucket or a function; the billing export is read by a query.

    Needed for Permissions
    The project and its regions (always) resourcemanager.projects.get, compute.regions.list
    Compute Engine compute.instances.list, compute.machineTypes.list
    Persistent Disks compute.disks.list
    Load Balancing compute.forwardingRules.list
    Cloud SQL cloudsql.instances.list
    Cloud Storage storage.buckets.list, storage.buckets.getIamPolicy
    Cloud Functions cloudfunctions.functions.list
    Cloud Run run.services.list
    BigQuery bigquery.datasets.get, bigquery.jobs.list, bigquery.jobs.listAll
    Metrics (Compute Engine, Cloud SQL, Functions, Load Balancing, Cloud Run) monitoring.timeSeries.list
    Billing export bigquery.tables.get, bigquery.tables.getData, bigquery.jobs.create

    As commands, with every service and billing:

    sh
    PROJECT="<PROJECT_ID>"
SA="hyvop-reader@$PROJECT.iam.gserviceaccount.com"
gcloud iam roles create hyvopReader --project=$PROJECT --title="Hyvop reader" --stage=GA \
  --permissions=bigquery.datasets.get,bigquery.jobs.create,bigquery.jobs.list,bigquery.jobs.listAll,bigquery.tables.get,bigquery.tables.getData,cloudfunctions.functions.list,cloudsql.instances.list,compute.disks.list,compute.forwardingRules.list,compute.instances.list,compute.machineTypes.list,compute.regions.list,monitoring.timeSeries.list,resourcemanager.projects.get,run.services.list,storage.buckets.getIamPolicy,storage.buckets.list
gcloud projects add-iam-policy-binding $PROJECT \
  --member=serviceAccount:$SA --role=projects/$PROJECT/roles/hyvopReader

    Leave out the billing row if you do not use the billing export, and the rows of services you do not select. Where a permission is missing the resource is stored as unknown and named in the result, never as private or idle. If the billing export table is in another project, grant the role there too (or on its dataset).

  3. A JSON key for the service account (Keys, Add key, JSON).

What to enter in Hyvop#

Field Meaning
Project ID The project to collect
Service account email The account above
Service account key (JSON) Paste the whole key file. It is stored encrypted under the install's data key and is never sent to the browser again
Services and regions What to collect; none selected for regions means all commercial regions
Billing table Optional: the standard usage cost export, project.dataset.table. Without it no billing is collected

Test the connection, then save. The project is collected at once, then on the same schedules as an AWS account (inventory every 4 hours, metrics every 2 hours, billing nightly when a billing table is named).

The ingestor needs to reach Google's APIs on port 443 (networkPolicy.egress.ingestors.internet, true by default).

How much history the first collection reads#

The first collection of a newly connected account reads 30 days of metrics, so that the rules that need history can report once it is there. The ingestor setting GCP_METRICS_BACKFILL_DAYS (30 by default) sets it: ingestors.gcp.env.GCP_METRICS_BACKFILL_DAYS=14. A longer one is limited by what the cloud keeps (Cloud Monitoring keeps 6 weeks at full resolution).

Demo mode#

ingestors.gcp.env.GCP_DATA_SOURCE=dummy serves fixture data instead of reading GCP. The default is live.