Overview
RPI can use Google BigQuery as a data warehouse. Query execution against BigQuery (for example, from a SQL Database Definition or a selection rule) goes through the Simba Open Database Connectivity (ODBC) driver, which supports two ways to authenticate:
-
Service-account key file: a downloaded Google credential file, mounted into the RPI containers. Works everywhere and supports every BigQuery operation RPI performs.
-
Workload Identity (keyless): available from RPI v7.8 on Google Kubernetes Engine (GKE) only. RPI doesn't store or mount a credential file; the pod authenticates as a Google service account through GKE's own identity mechanism.
A separate set of operations always requires the service-account key file, regardless of which option you choose for queries. RPI performs the following operations directly against the BigQuery API rather than through the ODBC driver, and this path doesn't support Workload Identity today: adding or removing a column on a BigQuery-backed table, bulk-loading data into BigQuery, and reading the structure of nested/repeated (RECORD/STRUCT) columns. If your use of this connection includes any of those, configure the key file (below) even if you also enable Workload Identity for querying.
This topic covers configuring the BigQuery connection via the Helm chart.
Which authentication method do I need?
|
If your BigQuery connection... |
Use |
|---|---|
|
Only runs SQL queries (SQL Database Definitions, selection rules, reporting) |
Workload Identity, if you're on GKE (simplest option, no credential to manage) |
|
Also needs schema changes, bulk loads, or has nested/repeated columns |
A service-account key file (required for those operations either way) |
|
Is not on GKE (AWS, Azure, self-hosted) |
A service-account key file (Workload Identity is GKE-only) |
Configure the connection: Kubernetes Helm chart (v7.8+)
Option A: Service-account key file
Provide the key file as a ConfigMap and reference it per connection:
global:
deployment:
platform: google
cloudIdentity:
google:
configMapName: my-bigquery-creds # ConfigMap holding the key file
configMapFilePath: /app/google-creds
keyName: my-bigquery-creds.json
databases:
datawarehouse:
bigquery:
enabled: true
connections:
- name: gbq-tenant1
projectId: my-google-project
credentialsType: serviceAccount
serviceAccountEmail: gbq-tenant1@my-google-project.iam.gserviceaccount.com
The chart mounts the key file and points the ODBC connection at it automatically. Create the ConfigMap from your downloaded key before deploying:
kubectl create configmap my-bigquery-creds \
--namespace <rpi-namespace> \
--from-file=my-bigquery-creds.json=/path/to/downloaded-key.json
Option B: Workload Identity (keyless, GKE only)
Query traffic authenticates as a Google service account bound to the pod's Kubernetes identity. RPI doesn't store a key file anywhere. This reuses the same Workload Identity binding used for Google Cloud SQL with IAM authentication; if you've already set that up for your operational database, the binding step is the same mechanism, applied for BigQuery access instead.
global:
deployment:
platform: google
cloudIdentity:
enabled: true
google:
serviceAccountEmail: gbq-tenant1@my-google-project.iam.gserviceaccount.com
configMapName: "" # empty = keyless
databases:
datawarehouse:
bigquery:
enabled: true
connections:
- name: gbq-tenant1
projectId: my-google-project
credentialsType: workloadIdentity
OAuthMechanism: 3
Google Cloud prerequisites (once per service account, provisioned on the Google side; RPI doesn't create these):
-
Grant the service account the ability to read BigQuery data and run BigQuery jobs, for example:
Bashgcloud projects add-iam-policy-binding my-google-project \ --member="serviceAccount:gbq-tenant1@my-google-project.iam.gserviceaccount.com" \ --role="roles/bigquery.dataViewer" gcloud projects add-iam-policy-binding my-google-project \ --member="serviceAccount:gbq-tenant1@my-google-project.iam.gserviceaccount.com" \ --role="roles/bigquery.jobUser"Adjust to the specific BigQuery datasets and least-privilege policy your organization requires.
-
Bind the Kubernetes service account to the Google service account (Workload Identity):
Bashgcloud iam service-accounts add-iam-policy-binding \ gbq-tenant1@my-google-project.iam.gserviceaccount.com \ --role="roles/iam.workloadIdentityUser" \ --member="serviceAccount:my-google-project.svc.id.goog[<namespace>/<ksa-name>]"With
cloudIdentity.serviceAccount.mode: shared, RPI uses one Kubernetes service account (redpoint-rpi) for the binding. Withper-service(the default), bind each RPI service's own account that needs BigQuery access (typically Execution Service, Node Manager, Interaction API, Integration API, Queue Reader, and Callback API).
Remember: Workload Identity covers SQL queries only. If this connection will also need schema changes, bulk loads, or nested-type lookups, also configure Option A's key file for the same connection.
Register the connection in RPI
Once the chart has the ODBC Data Source Name (DSN) configured (Option A or B above), point RPI at it from the client:
-
In the RPI client, open Administration.
-
Locate the BigQuery data warehouse (auxiliary database) definition, or create one.
-
Set Database Type to Google BigQuery.
-
Enter the DSN Name you used in
connections[].name. -
In Additional Configuration, enter:
JSON{"ProjectId":"my-google-project","ServiceAccountJsonPath":"/app/google-creds/my-bigquery-creds.json"}-
ProjectIdis your Google Cloud project ID. -
ServiceAccountJsonPathis the in-pod path to the mounted key file. This path must be present and must match where the chart mounts it (cloudIdentity.google.configMapFilePath/keyName, or the connection's ownConfigMapFilePath/keyNameif set) for schema changes, bulk loads, and nested-type lookups to work. -
If you're using Workload Identity only for querying and haven't mounted a key file, leave
ServiceAccountJsonPathunset. Queries will work, but attempting a schema change, bulk load, or nested-type lookup on this connection will fail until you add a key file.
-
-
Save your changes and, if you changed chart values, run
helm upgrade.
Verify the connection
Confirm the DSN rendered correctly and the driver can connect:
kubectl exec -n <rpi-namespace> deploy/rpi-executionservice -- isql <connection-name> -v