Central control plane

Run your own control plane for NeuralTrust data planes spread across several clusters.

Central control plane is the topology for organisations that need more than one data-plane cluster but only one place to administer them. You run the control plane, the console, and the analytics stack; data planes in other clusters enrol into your control plane instead of into NeuralTrust SaaS. Select it with global.deploymentMode: saas.

Do not confuse this with the hosted SaaS model, where NeuralTrust operates both planes and there is nothing to install. Here the chart renders a control plane that behaves like the hosted one, but it is yours and it runs in your environment.

Choose it when a single External install cannot work because data has to stay where it was produced — separate business units, jurisdictions, or environments — but the console, alerting, and cross-cluster reporting have to be in one place. If every workload fits in one cluster, External is simpler. If NeuralTrust hosts the control plane, use Hybrid.

Architecture

What runs in your central cluster

This mode is a superset of External: everything External deploys, plus three components and one behaviour change. PostgreSQL and Redis are recommended managed, outside the cluster; ClickHouse stays in-cluster on the central side only. Each remote cluster has its own recommended PostgreSQL and Redis — raw payloads never leave the cluster that produced them.

External baseline (also here)

Component	Purpose
TrustGate proxy + MCP, TrustGuard data plane, Firewall	Request path (also on every remote cluster)
AgentGateway admin, TrustGuard control plane	In-cluster product control planes
control-plane-app + control-plane-api	Console
ClickStack collector + ClickHouse	In-cluster analytics
DataCore	Residency query API
AlertEngine API + worker	Alert rules / SIEM forwarding
PostgreSQL (five databases)	Recommended managed, outside the cluster
Redis	Recommended managed, outside the cluster

Additions that make it a control plane for other clusters

Addition	Purpose
DataBridge	Remote DataAgents hold a long-lived bidirectional gRPC stream here; DataCore queries across them from the cluster-internal side. Stores nothing.
ClickStack ingest gateway	The public OTLP ingest endpoint. Verifies DataCore-issued RS256 JWTs and stamps the tenant from the verified claim, so an untrusted sender cannot write another tenant’s telemetry.
Published config-sync Services	Products in the remote clusters pull their configuration from your central control planes.

The behaviour change is in DataCore: it runs with a hybrid residency backend rather than reading the local ClickHouse for everything, so entitled reads go out through DataBridge to the cluster that holds the data.

The console, ClickHouse, AlertEngine, the bootstrap administrator, and the datastore split all behave exactly as in External — this page covers only what a control plane for other clusters adds on top.

Select the topology

Two values, on the central cluster:

global:
  deploymentMode: saas
  platform: kubernetes      # aws | gcp | azure | openshift | kubernetes
  domain: platform.example.com
  controlPlane:
    domain: nt.example.com  # bare DNS suffix — no scheme, port, or path

global.controlPlane.domain is what remote clusters dial. Leave it empty to keep using NeuralTrust SaaS through global.saasRegion.

Cross-cluster endpoints

The chart derives all four endpoints from that one suffix:

Endpoint	Served by	Dialled by
databridge.<domain>:443	DataBridge southbound Service	DataAgent
https://telemetry.<domain>	ClickStack ingest gateway Ingress	ClickStack egress sidecar
agentgateway-configsync.<domain>:443	AgentGateway config-sync Service	TrustGate data plane
trustguard-configsync.<domain>:443	TrustGuard config-sync Service	TrustGuard data plane

One value drives all four deliberately. A remote cluster that reached DataBridge on your domain but still dialled NeuralTrust for config-sync would half-work, and the half that broke would be silent.

Set the same global.controlPlane.domain on the central cluster and on every remote cluster. DNS, certificates, and load-balancer provisioning for these names are operator prerequisites.

There are no NeuralTrust hostnames or IPs anywhere in this topology, and no NeuralTrust inbound source IP — you own both ends.

Network rules

From each remote cluster (egress): allow TCP 443 to all four endpoints above on your own domain, plus the container registry (or mirror), the LLM upstreams, and that cluster’s own PostgreSQL and Redis.

Remote planes keep raw payloads in that cluster’s PostgreSQL and export metadata over OTLP to https://telemetry.<domain> (TELEMETRY_EXPORTERS_METADATA=otlp, TELEMETRY_EXPORTERS_RAW=postgres), so the central ClickHouse holds metadata and the console reaches raw payloads on demand through DataBridge.

Into the central cluster (ingress): it publishes those four endpoints, so it accepts inbound TCP 443 on each. Restrict them to the egress ranges of your data-plane clusters rather than leaving them open, and prefer an internal load-balancer scheme when the callers are on a private network — see Cloud notes. Where a link is private, you can skip publishing a config-sync Service at all with <product>.configSync.expose.enabled: false.

Nothing here has to traverse the internet. VPC peering, a Transit Gateway or Direct Connect all work, and a private path is what makes the chart’s own certificates a sound choice — see TLS.

Two of these carry long-lived streams. If an egress proxy or middlebox in the path reaps idle connections, config-sync and DataBridge drop and reconnect on that interval — check its idle timeout. A TLS-intercepting proxy breaks certificate verification unless its CA reaches the client: put it in the same bundle you point dataagent.databridge.tlsCa, <product>.configSync.tlsCa and global.clickstack.egress.tlsCaSecretName at, since each of those replaces the system roots rather than adding to them.

Prerequisites

Have these ready before installing. Everything else the chart does for you.

	What	Notes
1	Four DNS records, from the table above, pointing at the central cluster’s load balancers	Create them after the first install, once the load balancers have addresses. A private zone is fine.
2	A decision on certificates	See TLS. The chart-generated option needs nothing from you up front.
3	Network reachability from every remote cluster to all four endpoints	VPC peering, Transit Gateway, Direct Connect or the public internet — the chart does not care which.
4	An ingress controller in the central cluster	Only the telemetry endpoint uses one. The other three are layer-4 Services.
5	One enrolment token per remote data plane, issued from your console	See Telling data planes apart.

The central cluster is also a full External install, so the External prerequisites apply to it: capacity for roughly 4–5 workers at 8 vCPU / 16–32 GiB, the gcr-secret image pull Secret, an onprem-superadmin Secret for the first console administrator, and a decision on the console hostname before install. Each is covered in External → Prerequisites.

Images

A central cluster pulls two images beyond the External set, both from the NeuralTrust registry and both covered by the gcr-secret pull secret you already have — databridge, and opentelemetry-collector-contrib for the ingest gateway. The collector is the same image and tag your Hybrid clusters run as their egress sidecar, so a mirror that already carries it needs nothing new.

Mirroring for an air-gapped install works the same as everywhere else, and global.imageRegistry covers both of these. See Images and registries.

TLS

All four endpoints are dialled from other clusters, and each terminates TLS itself. So each needs a certificate covering its hostname, and each remote cluster needs to accept it. There are three ways to get there, and you can mix them per endpoint.

Endpoint	Bring your own	cert-manager	Chart-generated
databridge.<domain>	databridge.tls.existingSecret	databridge.tls.certManager.enabled	databridge.tls.autoGenerate
agentgateway-configsync.<domain>	agentgateway.configSync.grpcTls.existingSecret	—	agentgateway.configSync.expose.selfSignedTls
trustguard-configsync.<domain>	trustguard.configSync.grpcTls.existingSecret	—	trustguard.configSync.expose.selfSignedTls
telemetry.<domain>	clickstack-ingest-gateway.ingress.tls.secretName	via ingress.annotations	clickstack-ingest-gateway.ingress.tls.autoGenerate

The chart refuses to render an endpoint with no certificate at all, rather than publishing one that nothing outside the cluster can verify. If you see a validation error naming DataBridge or one of the config-sync listeners, this is why, and the message lists the options above.

Which option fits depends on how your remote clusters reach the control plane.

A certificate the data planes already trust

If the data planes traverse the public internet, or you run an internal PKI whose root is already in their trust stores, supply the certificates:

databridge:
  tls:
    existingSecret: databridge-southbound-tls   # or tls.certManager.enabled: true
agentgateway:
  configSync:
    grpcTls:
      existingSecret: agentgateway-configsync-tls
trustguard:
  configSync:
    grpcTls:
      existingSecret: trustguard-configsync-tls
clickstack-ingest-gateway:
  ingress:
    tls:
      secretName: telemetry-tls

Each certificate must cover its own hostname from the table above. Nothing further is needed on the remote side — the default trust store already accepts them.

On AWS, an ACM certificate cannot serve the first three: TLS terminates inside the pod and ACM does not export private keys. It can serve the telemetry endpoint, which terminates at the ALB. Use cert-manager or your own PKI for the rest.

Chart-generated, for a control plane on a private network

When remote clusters arrive over VPC peering, Direct Connect or a private link, no public trust store is involved and there is nothing to buy. Let the chart mint every certificate and distribute the CA as configuration:

databridge:
  tls:
    autoGenerate: true
agentgateway:
  configSync:
    expose:
      selfSignedTls: true
trustguard:
  configSync:
    expose:
      selfSignedTls: true
clickstack-ingest-gateway:
  ingress:
    enabled: true
    tls:
      autoGenerate: true

Each component mints its own CA, so a remote cluster needs all four. Export them as a single bundle from the central cluster:

./scripts/export-controlplane-ca.sh -n neuraltrust -o controlplane-ca.yaml

Apply that Secret in every remote cluster and point the dialling legs at it, as in Remote data planes.

Minting a certificate does not make anyone trust it. Until the CA bundle is installed on a remote cluster, every handshake from it fails.

Keypairs survive upgrades, because agents hold long-lived streams that a reissue would drop, and are reissued only when the names they cover change — which is how retargeting global.controlPlane.domain reaches the certificates. Rerun the export script after any such change.

Keeping an endpoint off a load balancer

To reach a config-sync listener over peering without publishing a Service at all, set <product>.configSync.expose.enabled: false and route to the ClusterIP yourself.

For endpoints that do get a load balancer, prefer an internal scheme when the callers are on a private network — see Cloud notes.

Telling data planes apart

DataBridge has to know which data plane is which. Two authentication modes do that:

databridge.auth.mode	How it works
introspect (default)	DataBridge asks your in-cluster DataCore about each enrolment token
jwt	DataBridge verifies DataCore-signed enrolment JWTs locally, with no round trip

token and dev are rejected in this mode. Both authenticate every data plane with one shared credential and then trust whichever tenant an agent claims for itself, which would let any enrolled data plane read another’s data.

Mint one enrolment token per remote data plane, each with its own instance ID. Reusing a single token across clusters collapses them into one identity in every query result and audit trail, and nothing later signals that it happened.

Secrets

Five credentials come from the shared platform Secret and are generated for you when the chart owns secrets:

Key	Used by
ENROLMENT_INTROSPECTION_TOKEN	DataCore — compares what DataBridge presents
DATACORE_SERVICE_TOKEN	DataBridge — alias of the above, must hold the identical value
ENROLMENT_SIGNING_SECRET	DataCore — signs enrolment tokens
CONFIG_SYNC_SIGNING_SECRET	DataCore — signs private-gateway install tokens; the product control planes verify it as CONFIG_SYNC_JWT_SECRET
TELEMETRY_JWT_PRIVATE_KEY_PEM	DataCore — RS256 key for the OTLP tokens the ingest gateway verifies

If you pre-provision secrets yourself — global.preserveExistingSecrets, global.autoGenerateSecrets: false, or global.platformSecret.existingSecret — all five must be present, and the two token keys must hold one identical value. When they drift, every agent connection returns 401 with nothing visibly wrong on either side. Omitting CONFIG_SYNC_SIGNING_SECRET fails later and further away: DataCore answers POST /v1/admin/credentials with HTTP 501 not implemented, so the console’s private-gateway wizard breaks after it has already created the gateway row. Running ./create-secrets.sh with DEPLOYMENT_MODE=saas writes all five correctly, including the alias.

Taking the platform Secret out of play in this mode is rejected at render time rather than at runtime, so a mistake here surfaces during helm upgrade instead of on first traffic.

See Secrets for how the shared platform Secret works generally.

Remote data planes

Each remote cluster is an ordinary Hybrid install pointed at your domain instead of at NeuralTrust:

global:
  deploymentMode: hybrid
  controlPlane:
    domain: nt.example.com
  products:
    trustgate: true
    trustguard: true

Its enrolment and config-sync tokens are issued by your console, not by app.neuraltrust.ai. Everything else — the four operator-supplied Secrets, the values file, verification, exposing both TrustGate entry points — is the ordinary Hybrid install, with global.controlPlane.domain set and no NeuralTrust hostnames to allow.

Trusting a chart-generated control plane

Only needed when the central cluster serves chart-generated certificates. Apply the bundle produced by scripts/export-controlplane-ca.sh, then point all three dialling legs at it:

global:
  customCaCert:
    enabled: true
    secretName: controlplane-ca      # mounts ca.crt at /etc/ssl/certs/custom-ca.crt
  clickstack:
    egress:
      tlsCaSecretName: controlplane-ca
dataagent:
  databridge:
    tlsCa: /etc/ssl/certs/custom-ca.crt
agentgateway:
  configSync:
    tlsCa: /etc/ssl/certs/custom-ca.crt
trustguard:
  configSync:
    tlsCa: /etc/ssl/certs/custom-ca.crt

Three separate settings because the clients read their trust store differently. DataAgent and the config-sync clients take a file path, so they use the mount that global.customCaCert provides. The telemetry collector configures TLS from its own config file and ignores that mount, so it takes a Secret name instead.

tlsCa replaces the system roots on that connection rather than adding to them. If a leg also has to trust something else — a TLS-intercepting corporate proxy, for instance — put every CA in one bundle.

Before rolling out, confirm from inside each remote cluster that it can reach all four central endpoints. A data plane whose egress is blocked does not crash — it starts cleanly, serves its last-known-good configuration, and quietly stops receiving updates. See Network rules.

Cloud notes

Nothing in the chart is cloud-specific; the LoadBalancer Services take free-form annotations. On EKS with the AWS Load Balancer Controller:

databridge:
  service:
    southbound:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        # internal for peered or Direct Connect callers; internet-facing only
        # when the data planes genuinely traverse the internet.
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
      # NAT egress ranges of your remote clusters. Without this the endpoint
      # accepts connections from anywhere the load balancer is reachable.
      loadBalancerSourceRanges: ["10.20.0.0/16"]

agentgateway:
  configSync:
    expose:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
      loadBalancerSourceRanges: ["10.20.0.0/16"]

clickstack-ingest-gateway:
  ingress:
    enabled: true
    annotations:
      alb.ingress.kubernetes.io/scheme: internal

Use the same shape for trustguard.configSync.expose. GKE already gives a layer 4 passthrough load balancer for a plain LoadBalancer Service; its private equivalent is networking.gke.io/load-balancer-type: "Internal", and on AKS it is service.beta.kubernetes.io/azure-load-balancer-internal: "true".

Three things worth knowing on any cloud:

• Use a layer 4 load balancer for DataBridge and config-sync. Both carry long-lived gRPC streams with TLS terminated in the pod. A layer 7 load balancer would have to re-terminate, and its idle timeout will cut streams that are healthy but quiet.
• The ingest gateway is plain HTTP, so it goes through an Ingress and a layer 7 load balancer is fine. It is the only one of the four that is not layer 4.
• An internal scheme keeps the whole topology off the public internet, which is what makes chart-generated certificates a sound production choice rather than a shortcut.

Central datastores — managed PostgreSQL, Redis, and ClickHouse — follow the normal External guidance.

Install

Start from the maintained neuraltrust-platform chart, then layer your platform, domain, ingress, TLS, and datastore choices:

helm upgrade --install neuraltrust-platform \
  oci://europe-west1-docker.pkg.dev/neuraltrust-app-prod/helm-charts/neuraltrust-platform \
  --version <VERSION> \
  --namespace neuraltrust --create-namespace \
  --set global.deploymentMode=saas \
  --set global.controlPlane.domain=nt.example.com \
  -f your-values.yaml

Install and verify the central cluster first, then bring up one remote cluster as a canary before the rest.

Verify

Verify the central cluster on its own before any remote cluster dials it. A remote plane that enrols against a half-configured control plane fails in ways that look like network faults.

kubectl get pods -n neuraltrust
kubectl get svc,ingress -n neuraltrust

Everything in the External baseline should be running, plus the two workloads that make this a control plane for other clusters: databridge and clickstack-ingest-gateway. If either is absent, global.deploymentMode is not saas — and nothing else about the install will look wrong, because every other component is shared with External.

Then confirm the four cross-cluster endpoints have addresses and that DNS resolves to them. Three are layer-4 Services and one is an Ingress, so they appear in different places:

kubectl get svc -n neuraltrust databridge-southbound \
  agentgateway-admin-configsync trustguard-control-plane-configsync
kubectl get ingress -n neuraltrust clickstack-ingest-gateway

The Service names are not the hostnames. Those three Services must each report an external address, and the DNS records from Cross-cluster endpoints — databridge.<domain>, agentgateway-configsync.<domain>, and trustguard-configsync.<domain> — are what point at them.

Those DNS records are an operator prerequisite the chart cannot check. Until each name resolves, remote clusters cannot enrol; and because config-sync and DataBridge retry, the symptom is a remote data plane that stays Running but never becomes Ready, with nothing wrong on the central side.

Sign in to the console and create the organisation exactly as in External → Verify; the console is the same install here.

Then enrol one remote cluster

Bring up a single remote data plane as a canary. Two log lines on the remote side prove both directions of the topology, and both are worth reading before the rest of the clusters follow:

kubectl logs -n neuraltrust deploy/dataagent | grep -i databridge
kubectl logs -n neuraltrust deploy/trustguard-data-plane | grep -i 'config snapshot'

Log line	What it proves
registered with databridge with your addr and tenant_id	The DataAgent reached databridge.<domain>:443, its enrolment token verified, and residency queries can now reach this cluster
applied config snapshot with a version hash	Config-sync pulled configuration from your central control plane

Check the addr in that first line names your domain. A remote cluster that still reads databridge.<saas region> has an empty global.controlPlane.domain and has enrolled into NeuralTrust SaaS instead of into your control plane.

Finish with the validation checklist, which covers the parts no kubectl command reaches: TLS from the remote clusters, one enrolment identity per data plane, LLM and MCP traffic, and telemetry arriving in the central ClickHouse.

High availability

This topology has two availability questions, and they are not equally urgent.

The data planes matter most, because they are on the request path. Each remote cluster climbs the same ladder as in Hybrid → High availability: one cluster across zones first, then twin clusters, and only for regional loss an active/passive pair on one writable PostgreSQL primary with region-local Redis, with just the active cluster running DataAgent.

The central cluster is not on the request path. If it goes down, every remote data plane keeps serving traffic from its last synchronized configuration. What stops is configuration updates, telemetry ingest, cross-cluster reporting, and the console. That is an availability target measured in minutes of operator inconvenience rather than dropped requests — as long as remote clusters keep their last-known-good configuration on persistent storage.

The central cluster

Most central clusters should sit on tier 1: one cluster across three zones with managed stores. It is not on the request path, so a promotion runbook is rarely worth its own operational cost.

If a regional requirement forces tier 3, run it as an External active/passive pair on one writable PostgreSQL primary with a cross-region read replica and region-local Redis — see External → High availability for the shape, the ClickHouse caveat, and the promotion steps. Two things are specific to a central cluster:

The four published endpoints have to follow the promotion. Remote clusters dial databridge.<domain>, telemetry.<domain>, and the two config-sync hostnames. Those names must resolve to the promoted cluster’s load balancers, so plan the DNS switch as part of the runbook. DataBridge streams and config-sync reconnect on their own once DNS converges; there is nothing to restart in the remote clusters.

Certificates must be valid from both central clusters. Chart-generated certificates are minted per cluster with a per-cluster CA, so a remote cluster that trusts only the primary’s bundle fails every handshake after a promotion. Either export and install both clusters’ CA bundles in every remote cluster, or issue the four certificates from your own PKI or cert-manager so one bundle covers either cluster. The second option is much easier to operate — see TLS.

The ENROLMENT_SIGNING_SECRET and TELEMETRY_JWT_PRIVATE_KEY_PEM in the platform Secret must be identical in both central clusters. If each cluster generates its own, enrolment tokens and OTLP tokens minted by one are rejected by the other, and every remote agent returns 401 after a promotion. Pre-provision the platform Secret and apply the same one to both clusters — see Secrets.

Before you rely on it

• Every remote cluster keeps its last-known-good configuration on persistent storage, so a central outage cannot take the request path down.
• Central PostgreSQL is managed, with one primary and a cross-region read replica; each central region has its own Redis.
• The four endpoint DNS records can be repointed, and you know how long convergence takes.
• Both central clusters serve certificates every remote cluster trusts.
• Both central clusters hold the same platform Secret values.
• You have rehearsed a central promotion and confirmed remote data planes reconnect without intervention.

Next steps

Hybrid

The install every remote data-plane cluster runs.

External

The baseline your central cluster is built on.

Config sync

How remote products pull configuration from your control plane.

Secrets

Credential handling, including the four cross-cluster keys.

Validation checklist

What to prove before the install counts as done.