> ## Documentation Index
> Fetch the complete documentation index at: https://neuraltrust-92b43583-develop.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Central control plane
> Run your own control plane for NeuralTrust data planes spread across several clusters.
**Central control plane** is the topology for organisations that need more than
one data-plane cluster but only one place to administer them. You run the control
plane, the console, and the analytics stack; data planes in other clusters enrol
into your control plane instead of into NeuralTrust SaaS. Select it with
`global.deploymentMode: saas`.
Do not confuse this with the hosted **SaaS** model, where NeuralTrust operates
both planes and there is nothing to install. Here the chart renders a control
plane that behaves like the hosted one, but it is yours and it runs in your
environment.
Choose it when a single [External](/neuraltrust/deployment/external) install
cannot work because data has to stay where it was produced — separate business
units, jurisdictions, or environments — but the console, alerting, and
cross-cluster reporting have to be in one place. If every workload fits in one
cluster, External is simpler. If NeuralTrust hosts the control plane, use
[Hybrid](/neuraltrust/deployment/hybrid).
## Architecture
## What runs in your central cluster
This mode is a superset of External: everything External deploys, plus three
components and one behaviour change. PostgreSQL and Redis are **recommended
managed, outside the cluster**; ClickHouse stays in-cluster on the **central**
side only. Each remote cluster has its own recommended PostgreSQL and Redis —
raw payloads never leave the cluster that produced them.
### External baseline (also here)
| Component | Purpose |
| ------------------------------------------------------ | ------------------------------------------- |
| TrustGate proxy + MCP, TrustGuard data plane, Firewall | Request path (also on every remote cluster) |
| AgentGateway admin, TrustGuard control plane | In-cluster product control planes |
| control-plane-app + control-plane-api | Console |
| ClickStack collector + ClickHouse | In-cluster analytics |
| DataCore | Residency query API |
| AlertEngine API + worker | Alert rules / SIEM forwarding |
| PostgreSQL (five databases) | Recommended managed, outside the cluster |
| Redis | Recommended managed, outside the cluster |
### Additions that make it a control plane for other clusters
| Addition | Purpose |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| DataBridge | Remote DataAgents hold a long-lived bidirectional gRPC stream here; DataCore queries across them from the cluster-internal side. Stores nothing. |
| ClickStack ingest gateway | The public OTLP ingest endpoint. Verifies DataCore-issued RS256 JWTs and stamps the tenant from the verified claim, so an untrusted sender cannot write another tenant's telemetry. |
| Published config-sync Services | Products in the remote clusters pull their configuration from your central control planes. |
The behaviour change is in DataCore: it runs with a hybrid residency backend
rather than reading the local ClickHouse for everything, so entitled reads go
out through DataBridge to the cluster that holds the data.
The console, ClickHouse, AlertEngine, the bootstrap administrator, and the
datastore split all behave exactly as in
[External](/neuraltrust/deployment/external) — this page covers only what a
control plane for other clusters adds on top.
## Select the topology
Two values, on the central cluster:
```yaml theme={null}
global:
deploymentMode: saas
platform: kubernetes # aws | gcp | azure | openshift | kubernetes
domain: platform.example.com
controlPlane:
domain: nt.example.com # bare DNS suffix — no scheme, port, or path
```
`global.controlPlane.domain` is what remote clusters dial. Leave it empty to keep
using NeuralTrust SaaS through `global.saasRegion`.
## Cross-cluster endpoints
The chart derives all four endpoints from that one suffix:
| Endpoint | Served by | Dialled by |
| -------------------------------------- | --------------------------------- | ------------------------- |
| `databridge.:443` | DataBridge southbound Service | DataAgent |
| `https://telemetry.` | ClickStack ingest gateway Ingress | ClickStack egress sidecar |
| `agentgateway-configsync.:443` | AgentGateway config-sync Service | TrustGate data plane |
| `trustguard-configsync.:443` | TrustGuard config-sync Service | TrustGuard data plane |
One value drives all four deliberately. A remote cluster that reached DataBridge
on your domain but still dialled NeuralTrust for config-sync would half-work, and
the half that broke would be silent.
Set the same `global.controlPlane.domain` on the central cluster **and** on every
remote cluster. DNS, certificates, and load-balancer provisioning for these names
are operator prerequisites.
There are no NeuralTrust hostnames or IPs anywhere in this topology, and no
NeuralTrust inbound source IP — you own both ends.
### Network rules
**From each remote cluster (egress):** allow TCP 443 to all four endpoints above
on your own domain, plus the container registry (or mirror), the LLM upstreams,
and that cluster's own PostgreSQL and Redis.
Remote planes keep raw payloads in that cluster's PostgreSQL and export
metadata over OTLP to `https://telemetry.`
(`TELEMETRY_EXPORTERS_METADATA=otlp`, `TELEMETRY_EXPORTERS_RAW=postgres`), so
the central ClickHouse holds metadata and the console reaches raw payloads on
demand through DataBridge.
**Into the central cluster (ingress):** it publishes those four endpoints, so it
accepts inbound TCP 443 on each. Restrict them to the egress ranges of your
data-plane clusters rather than leaving them open, and prefer an internal
load-balancer scheme when the callers are on a private network — see
[Cloud notes](#cloud-notes). Where a link is private, you can skip publishing a
config-sync Service at all with
`.configSync.expose.enabled: false`.
Nothing here has to traverse the internet. VPC peering, a Transit Gateway or
Direct Connect all work, and a private path is what makes the chart's own
certificates a sound choice — see [TLS](#tls).
Two of these carry long-lived streams. If an egress proxy or middlebox in the
path reaps idle connections, config-sync and DataBridge drop and reconnect on
that interval — check its idle timeout. A TLS-intercepting proxy breaks
certificate verification unless its CA reaches the client: put it in the same
bundle you point `dataagent.databridge.tlsCa`, `.configSync.tlsCa` and
`global.clickstack.egress.tlsCaSecretName` at, since each of those replaces the
system roots rather than adding to them.
## Prerequisites
Have these ready before installing. Everything else the chart does for you.
| | What | Notes |
| - | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| 1 | Four DNS records, from the table above, pointing at the central cluster's load balancers | Create them after the first install, once the load balancers have addresses. A private zone is fine. |
| 2 | A decision on certificates | See [TLS](#tls). The chart-generated option needs nothing from you up front. |
| 3 | Network reachability from every remote cluster to all four endpoints | VPC peering, Transit Gateway, Direct Connect or the public internet — the chart does not care which. |
| 4 | An ingress controller in the central cluster | Only the telemetry endpoint uses one. The other three are layer-4 Services. |
| 5 | One enrolment token per remote data plane, issued from your console | See [Telling data planes apart](#telling-data-planes-apart). |
The central cluster is also a full External install, so the External
prerequisites apply to it: capacity for roughly **4–5** workers at 8 vCPU /
16–32 GiB, the `gcr-secret` image pull Secret, an `onprem-superadmin` Secret for
the first console administrator, and a decision on the console hostname before
install. Each is covered in
[External → Prerequisites](/neuraltrust/deployment/external#prerequisites).
### Images
A central cluster pulls two images beyond the External set, both from the
NeuralTrust registry and both covered by the `gcr-secret` pull secret you already
have — `databridge`, and `opentelemetry-collector-contrib` for the ingest
gateway. The collector is the same image and tag your Hybrid clusters run as
their egress sidecar, so a mirror that already carries it needs nothing new.
Mirroring for an air-gapped install works the same as everywhere else, and
`global.imageRegistry` covers both of these. See
[Images and registries](/neuraltrust/deployment/images).
## TLS
All four endpoints are dialled from other clusters, and each terminates TLS
itself. So each needs a certificate covering its hostname, and each remote cluster
needs to accept it. There are three ways to get there, and you can mix them per
endpoint.
| Endpoint | Bring your own | cert-manager | Chart-generated |
| ---------------------------------- | -------------------------------------------------- | ------------------------------------ | ---------------------------------------------------- |
| `databridge.` | `databridge.tls.existingSecret` | `databridge.tls.certManager.enabled` | `databridge.tls.autoGenerate` |
| `agentgateway-configsync.` | `agentgateway.configSync.grpcTls.existingSecret` | — | `agentgateway.configSync.expose.selfSignedTls` |
| `trustguard-configsync.` | `trustguard.configSync.grpcTls.existingSecret` | — | `trustguard.configSync.expose.selfSignedTls` |
| `telemetry.` | `clickstack-ingest-gateway.ingress.tls.secretName` | via `ingress.annotations` | `clickstack-ingest-gateway.ingress.tls.autoGenerate` |
The chart refuses to render an endpoint with no certificate at all, rather than
publishing one that nothing outside the cluster can verify. If you see a
validation error naming DataBridge or one of the config-sync listeners, this is
why, and the message lists the options above.
Which option fits depends on how your remote clusters reach the control plane.
### A certificate the data planes already trust
If the data planes traverse the public internet, or you run an internal PKI whose
root is already in their trust stores, supply the certificates:
```yaml theme={null}
databridge:
tls:
existingSecret: databridge-southbound-tls # or tls.certManager.enabled: true
agentgateway:
configSync:
grpcTls:
existingSecret: agentgateway-configsync-tls
trustguard:
configSync:
grpcTls:
existingSecret: trustguard-configsync-tls
clickstack-ingest-gateway:
ingress:
tls:
secretName: telemetry-tls
```
Each certificate must cover its own hostname from the table above. Nothing further
is needed on the remote side — the default trust store already accepts them.
On AWS, an ACM certificate cannot serve the first three: TLS terminates inside the
pod and ACM does not export private keys. It can serve the telemetry endpoint,
which terminates at the ALB. Use cert-manager or your own PKI for the rest.
### Chart-generated, for a control plane on a private network
When remote clusters arrive over VPC peering, Direct Connect or a private link, no
public trust store is involved and there is nothing to buy. Let the chart mint
every certificate and distribute the CA as configuration:
```yaml theme={null}
databridge:
tls:
autoGenerate: true
agentgateway:
configSync:
expose:
selfSignedTls: true
trustguard:
configSync:
expose:
selfSignedTls: true
clickstack-ingest-gateway:
ingress:
enabled: true
tls:
autoGenerate: true
```
Each component mints its own CA, so a remote cluster needs all four. Export them
as a single bundle from the central cluster:
```bash theme={null}
./scripts/export-controlplane-ca.sh -n neuraltrust -o controlplane-ca.yaml
```
Apply that Secret in every remote cluster and point the dialling legs at it, as in
[Remote data planes](#remote-data-planes).
Minting a certificate does not make anyone trust it. Until the CA bundle is
installed on a remote cluster, every handshake from it fails.
Keypairs survive upgrades, because agents hold long-lived streams that a reissue
would drop, and are reissued only when the names they cover change — which is how
retargeting `global.controlPlane.domain` reaches the certificates. Rerun the export
script after any such change.
### Keeping an endpoint off a load balancer
To reach a config-sync listener over peering without publishing a Service at all,
set `.configSync.expose.enabled: false` and route to the ClusterIP
yourself.
For endpoints that do get a load balancer, prefer an internal scheme when the
callers are on a private network — see [Cloud notes](#cloud-notes).
## Telling data planes apart
DataBridge has to know which data plane is which. Two authentication modes do
that:
| `databridge.auth.mode` | How it works |
| ---------------------- | ------------------------------------------------------------------------------ |
| `introspect` (default) | DataBridge asks your in-cluster DataCore about each enrolment token |
| `jwt` | DataBridge verifies DataCore-signed enrolment JWTs locally, with no round trip |
`token` and `dev` are rejected in this mode. Both authenticate every data plane
with one shared credential and then trust whichever tenant an agent claims for
itself, which would let any enrolled data plane read another's data.
Mint one enrolment token per remote data plane, each with its own instance ID.
Reusing a single token across clusters collapses them into one identity in every
query result and audit trail, and nothing later signals that it happened.
## Secrets
Five credentials come from the shared platform Secret and are generated for you
when the chart owns secrets:
| Key | Used by |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `ENROLMENT_INTROSPECTION_TOKEN` | DataCore — compares what DataBridge presents |
| `DATACORE_SERVICE_TOKEN` | DataBridge — alias of the above, must hold the identical value |
| `ENROLMENT_SIGNING_SECRET` | DataCore — signs enrolment tokens |
| `CONFIG_SYNC_SIGNING_SECRET` | DataCore — signs private-gateway install tokens; the product control planes verify it as `CONFIG_SYNC_JWT_SECRET` |
| `TELEMETRY_JWT_PRIVATE_KEY_PEM` | DataCore — RS256 key for the OTLP tokens the ingest gateway verifies |
If you pre-provision secrets yourself — `global.preserveExistingSecrets`,
`global.autoGenerateSecrets: false`, or `global.platformSecret.existingSecret` —
all five must be present, and the two token keys must hold one identical value.
When they drift, every agent connection returns 401 with nothing visibly wrong on
either side. Omitting `CONFIG_SYNC_SIGNING_SECRET` fails later and further away:
DataCore answers `POST /v1/admin/credentials` with HTTP 501 `not implemented`, so
the console's private-gateway wizard breaks *after* it has already created the
gateway row. Running `./create-secrets.sh` with `DEPLOYMENT_MODE=saas` writes all
five correctly, including the alias.
Taking the platform Secret out of play in this mode is rejected at render time
rather than at runtime, so a mistake here surfaces during `helm upgrade` instead
of on first traffic.
See [Secrets](/neuraltrust/deployment/secrets) for how the shared platform Secret
works generally.
## Remote data planes
Each remote cluster is an ordinary Hybrid install pointed at your domain instead
of at NeuralTrust:
```yaml theme={null}
global:
deploymentMode: hybrid
controlPlane:
domain: nt.example.com
products:
trustgate: true
trustguard: true
```
Its enrolment and config-sync tokens are issued by **your** console, not by
[app.neuraltrust.ai](https://app.neuraltrust.ai/en/v2/). Everything else — the
four operator-supplied Secrets, the values file, verification, exposing both
TrustGate entry points — is the ordinary
[Hybrid install](/neuraltrust/deployment/hybrid#install), with
`global.controlPlane.domain` set and no NeuralTrust hostnames to allow.
### Trusting a chart-generated control plane
Only needed when the central cluster serves chart-generated certificates. Apply the
bundle produced by `scripts/export-controlplane-ca.sh`, then point all three
dialling legs at it:
```yaml theme={null}
global:
customCaCert:
enabled: true
secretName: controlplane-ca # mounts ca.crt at /etc/ssl/certs/custom-ca.crt
clickstack:
egress:
tlsCaSecretName: controlplane-ca
dataagent:
databridge:
tlsCa: /etc/ssl/certs/custom-ca.crt
agentgateway:
configSync:
tlsCa: /etc/ssl/certs/custom-ca.crt
trustguard:
configSync:
tlsCa: /etc/ssl/certs/custom-ca.crt
```
Three separate settings because the clients read their trust store differently.
DataAgent and the config-sync clients take a file path, so they use the mount that
`global.customCaCert` provides. The telemetry collector configures TLS from its own
config file and ignores that mount, so it takes a Secret name instead.
`tlsCa` **replaces** the system roots on that connection rather than adding to
them. If a leg also has to trust something else — a TLS-intercepting corporate
proxy, for instance — put every CA in one bundle.
Before rolling out, confirm from inside each remote cluster that it can reach all
four central endpoints. A data plane whose egress is blocked does not crash — it
starts cleanly, serves its last-known-good configuration, and quietly stops
receiving updates. See [Network rules](#network-rules).
## Cloud notes
Nothing in the chart is cloud-specific; the LoadBalancer Services take free-form
annotations. On EKS with the AWS Load Balancer Controller:
```yaml theme={null}
databridge:
service:
southbound:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
# internal for peered or Direct Connect callers; internet-facing only
# when the data planes genuinely traverse the internet.
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
# NAT egress ranges of your remote clusters. Without this the endpoint
# accepts connections from anywhere the load balancer is reachable.
loadBalancerSourceRanges: ["10.20.0.0/16"]
agentgateway:
configSync:
expose:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
loadBalancerSourceRanges: ["10.20.0.0/16"]
clickstack-ingest-gateway:
ingress:
enabled: true
annotations:
alb.ingress.kubernetes.io/scheme: internal
```
Use the same shape for `trustguard.configSync.expose`. GKE already gives a layer 4
passthrough load balancer for a plain `LoadBalancer` Service; its private
equivalent is `networking.gke.io/load-balancer-type: "Internal"`, and on AKS it is
`service.beta.kubernetes.io/azure-load-balancer-internal: "true"`.
Three things worth knowing on any cloud:
* **Use a layer 4 load balancer for DataBridge and config-sync.** Both carry
long-lived gRPC streams with TLS terminated in the pod. A layer 7 load balancer
would have to re-terminate, and its idle timeout will cut streams that are
healthy but quiet.
* **The ingest gateway is plain HTTP**, so it goes through an Ingress and a layer 7
load balancer is fine. It is the only one of the four that is not layer 4.
* **An internal scheme keeps the whole topology off the public internet**, which is
what makes chart-generated certificates a sound production choice rather than a
shortcut.
Central datastores — managed PostgreSQL, Redis, and ClickHouse — follow the normal
[External guidance](/neuraltrust/deployment/external#datastores).
## Install
Start from the maintained
[`neuraltrust-platform`](https://github.com/NeuralTrust/neuraltrust-platform)
chart, then layer your platform, domain, ingress, TLS, and datastore choices:
```bash theme={null}
helm upgrade --install neuraltrust-platform \
oci://europe-west1-docker.pkg.dev/neuraltrust-app-prod/helm-charts/neuraltrust-platform \
--version \
--namespace neuraltrust --create-namespace \
--set global.deploymentMode=saas \
--set global.controlPlane.domain=nt.example.com \
-f your-values.yaml
```
Install and verify the central cluster first, then bring up one remote cluster as
a canary before the rest.
## Verify
Verify the central cluster on its own before any remote cluster dials it. A
remote plane that enrols against a half-configured control plane fails in ways
that look like network faults.
```bash theme={null}
kubectl get pods -n neuraltrust
kubectl get svc,ingress -n neuraltrust
```
Everything in the [External baseline](#external-baseline-also-here) should be
running, plus the two workloads that make this a control plane for other
clusters: `databridge` and `clickstack-ingest-gateway`. If either is absent,
`global.deploymentMode` is not `saas` — and nothing else about the install will
look wrong, because every other component is shared with External.
Then confirm the four cross-cluster endpoints have addresses and that DNS
resolves to them. Three are layer-4 Services and one is an Ingress, so they
appear in different places:
```bash theme={null}
kubectl get svc -n neuraltrust databridge-southbound \
agentgateway-admin-configsync trustguard-control-plane-configsync
kubectl get ingress -n neuraltrust clickstack-ingest-gateway
```
The Service names are not the hostnames. Those three Services must each report an
external address, and the DNS records from
[Cross-cluster endpoints](#cross-cluster-endpoints) —
`databridge.`, `agentgateway-configsync.`, and
`trustguard-configsync.` — are what point at them.
Those DNS records are an operator prerequisite the chart cannot check. Until each
name resolves, remote clusters cannot enrol; and because config-sync and
DataBridge retry, the symptom is a remote data plane that stays `Running` but
never becomes `Ready`, with nothing wrong on the central side.
Sign in to the console and create the organisation exactly as in
[External → Verify](/neuraltrust/deployment/external#verify); the console is the
same install here.
### Then enrol one remote cluster
Bring up a single remote data plane as a canary. Two log lines on the remote
side prove both directions of the topology, and both are worth reading before
the rest of the clusters follow:
```bash theme={null}
kubectl logs -n neuraltrust deploy/dataagent | grep -i databridge
kubectl logs -n neuraltrust deploy/trustguard-data-plane | grep -i 'config snapshot'
```
| Log line | What it proves |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `registered with databridge` with your `addr` and `tenant_id` | The DataAgent reached `databridge.:443`, its enrolment token verified, and residency queries can now reach this cluster |
| `applied config snapshot` with a version hash | Config-sync pulled configuration from **your** central control plane |
Check the `addr` in that first line names your domain. A remote cluster that
still reads `databridge.` has an empty `global.controlPlane.domain`
and has enrolled into NeuralTrust SaaS instead of into your control plane.
Finish with the [validation checklist](/neuraltrust/deployment/validation),
which covers the parts no `kubectl` command reaches: TLS from the remote
clusters, one enrolment identity per data plane, LLM and MCP traffic, and
telemetry arriving in the central ClickHouse.
## High availability
This topology has two availability questions, and they are not equally urgent.
**The data planes matter most**, because they are on the request path. Each
remote cluster climbs the same ladder as in
[Hybrid → High availability](/neuraltrust/deployment/hybrid#high-availability):
one cluster across zones first, then twin clusters, and only for regional loss an
active/passive pair on one writable PostgreSQL primary with region-local Redis,
with just the active cluster running DataAgent.
**The central cluster is not on the request path.** If it goes down, every remote
data plane keeps serving traffic from its last synchronized configuration. What
stops is configuration updates, telemetry ingest, cross-cluster reporting, and
the console. That is an availability target measured in minutes of operator
inconvenience rather than dropped requests — as long as remote clusters keep
their last-known-good configuration on persistent storage.
### The central cluster
Most central clusters should sit on tier 1: one cluster across three zones with
managed stores. It is not on the request path, so a promotion runbook is rarely
worth its own operational cost.
If a regional requirement forces tier 3, run it as an External active/passive
pair on **one writable PostgreSQL primary** with a cross-region read replica and
**region-local Redis** — see
[External → High availability](/neuraltrust/deployment/external#high-availability)
for the shape, the ClickHouse caveat, and the promotion steps. Two things are
specific to a central cluster:
**The four published endpoints have to follow the promotion.** Remote clusters
dial `databridge.`, `telemetry.`, and the two config-sync
hostnames. Those names must resolve to the promoted cluster's load balancers, so
plan the DNS switch as part of the runbook. DataBridge streams and config-sync
reconnect on their own once DNS converges; there is nothing to restart in the
remote clusters.
**Certificates must be valid from both central clusters.** Chart-generated
certificates are minted per cluster with a per-cluster CA, so a remote cluster
that trusts only the primary's bundle fails every handshake after a promotion.
Either export and install both clusters' CA bundles in every remote cluster, or
issue the four certificates from your own PKI or cert-manager so one bundle
covers either cluster. The second option is much easier to operate — see
[TLS](#tls).
The `ENROLMENT_SIGNING_SECRET` and `TELEMETRY_JWT_PRIVATE_KEY_PEM` in the
platform Secret must be **identical** in both central clusters. If each cluster
generates its own, enrolment tokens and OTLP tokens minted by one are rejected by
the other, and every remote agent returns 401 after a promotion. Pre-provision
the platform Secret and apply the same one to both clusters — see
[Secrets](#secrets).
### Before you rely on it
* [ ] Every remote cluster keeps its last-known-good configuration on persistent
storage, so a central outage cannot take the request path down.
* [ ] Central PostgreSQL is managed, with one primary and a cross-region read
replica; each central region has its own Redis.
* [ ] The four endpoint DNS records can be repointed, and you know how long
convergence takes.
* [ ] Both central clusters serve certificates every remote cluster trusts.
* [ ] Both central clusters hold the same platform Secret values.
* [ ] You have rehearsed a central promotion and confirmed remote data planes
reconnect without intervention.
## Next steps
The install every remote data-plane cluster runs.
The baseline your central cluster is built on.
How remote products pull configuration from your control plane.
Credential handling, including the four cross-cluster keys.
What to prove before the install counts as done.