When Hybrid Manager (HM) is deployed across multiple locations, Transparent Data Encryption (TDE) behaves differently depending on the cluster architecture you provision. This page describes what happens when you supply a customer-managed key per location, and what to plan for when you scale a cluster from a single region to multiple regions. The guidance applies to every supported key management provider — HashiCorp Vault, AWS KMS, Azure Key Vault, Google Cloud KMS — and to the passphrase option.
Read this page in addition to the multi-DC deployment guide and the Enabling KMS for TDE index.
How TDE protects your data
TDE uses two keys:
A customer-managed key (KEK). You register this in HM as a customer key. It lives in your key management service (KMS) provider or for the passphrase option, is derived from a passphrase you supply. HM never sees the key material itself. HM only invokes wrap/unwrap operations against your KMS.
A data encryption key (DEK). HM generates this once per Postgres cluster at cluster creation. In a distributed high-availability deployment, each data group generates and holds its own DEK. The DEK is what actually encrypts your on-disk data and write-ahead log files. Postgres unwraps the DEK using your KMS at each start-up and keeps the plaintext DEK in memory to encrypt reads and writes. DEKs never leave the cluster (or, for PGD, the data group) that generated them.
Two consequences follow that matter for multi-DC planning:
- Once a cluster's on-disk encryption file is written, only a KEK that can decrypt that file will let the cluster start.
- Whichever KEK produced the file is the KEK that must remain available for the lifetime of that cluster's data.
How TDE and cross-data-center replication interact
Whether the customer key registered in a second location has to share key material with the source's key, or is free to be genuinely different, depends on how data reaches that location and on what the operator does with the encryption file after the copy:
Physical streaming replication (PSR). The source's data directory is copied byte-for-byte via Postgres base-backup, including the encryption file that stores the wrapped DEK. The replica keeps that file as-is, so only a KEK compatible with the source's KEK can unwrap it. This is the transport used by replica clusters, and it is why the replica's own customer key must resolve to compatible key material.
Adding a data group to a distributed high-availability (PGD) cluster. A physical join (
bdr_init_physical) copies the source group's data directory, but immediately after the copy completes the operator re-wraps the DEK on the target under the target group's own configured KEK, so the target's own KEK becomes the effective KEK for ongoing data at rest. The source group's KEK is only required to be reachable during the physical join itself, so the target pod's manager can read the DEK out of the file it just received. Once the physical join finishes, the target group is self-sufficient and itskey.binnames its own KEK.How PGD data groups stay in sync after they're up. Changes are decoded into row images inside the source's Postgres instance, after the storage layer has already decrypted the page, and streamed to the destination over mutually authenticated Transport Layer Security (TLS). The destination's Postgres instance receives the row images and its own storage layer encrypts them under its own DEK before writing to disk. Wrapped keys never leave a location, and the two locations don't exchange any key material. This is why a data group configured with a genuinely different KEK from every other group in the cluster keeps replicating normally.
Note
The takeaway for planning: a PGD data group always ends up under its own KEK, whether it was created as the seed or added later. Only a PSR replica cluster inherits the source's KEK for the life of its data at rest.
KMS permissions when adding a PGD data group in a new location
When you add a PGD data group into a new location, the target group needs KMS decrypt access on two keys at different times:
- During the join — the target location's workload identity must have decrypt access on the source group's KEK (the key that wrapped the
key.binbeing copied), in addition to encrypt and decrypt access on the target group's own configured KEK. The operator uses the source's KEK once to read the DEK out of the copied file, and the target's KEK to re-wrap the DEK before Postgres starts. - After the join completes — the target group only needs access to its own configured KEK. The source's KEK is no longer touched by the target and access to it can be revoked from the target's identity if your security policy requires it.
Example: seed group in region 1 is configured with region-key1, and you scale a new data group into region 2 configured with region-key2. Before starting the join, grant the region 2 workload identity decrypt access on both region-key1 and region-key2. Once the region 2 pods are running and reporting healthy, decrypt access on region-key1 is optional and can be removed from the region 2 identity.
Missing this permission is the most common cause of a joining pod stuck in the bootstrap phase with a KMS 403 / AccessDenied naming the source location's key — see PGD data group stuck with key unwrap command produced no output in the troubleshooting guide.
Replica clusters across data centers
A replica cluster (from the Replica Clusters tab) is a physical streaming replica of a primary cluster. When the replica is created, it's bootstrapped from the primary using Postgres's base-backup mechanism, which copies the primary's data directory in full, including the encryption file that stores the wrapped DEK.
This has an important implication:
- The KEK that protects a replica cluster's data at rest is the primary's KEK, not the customer key you configure on the replica. The replica's customer key is registered per location for access-control purposes, but the underlying key material must be compatible with the primary's KEK so that the replica's Postgres instance can start.
In practice, "compatible" means one of the following:
The same underlying key material referenced from the replica's location. Examples:
- AWS KMS: use a multi-region replica key whose replicas are present in every location that hosts a replica cluster.
- Azure Key Vault: use a key vault reachable from every location, or replicate the same key material into a per-region vault.
- Google Cloud KMS: use a multi-region or global key ring reachable from every location.
- HashiCorp Vault: replicate the Transit engine key across regions.
- Passphrase: use the same passphrase everywhere.
A single KEK reachable from all locations. If the same customer-managed key is accessible from every region that hosts a replica cluster, you can register it as one customer key per location in HM (each row is per-location metadata) and the replica clusters will be able to start.
Recommendations
Register a customer key in every location that will host a replica cluster, and make sure they resolve to compatible key material. HM lets you configure a different customer key per replica in the create request; use this to represent per-location access control, not to swap the underlying key material.
Don't expect the replica's own customer key to become the effective KEK after bootstrap. HM doesn't re-wrap the encryption file on the replica after base-backup. The primary's KEK remains the key that must be able to unwrap the file for the life of the replica cluster.
Plan KEK rotation as a per-cluster operation across all locations. Rotating the KEK on the primary rewrites the primary's on-disk encryption file, but this change doesn't propagate over streaming replication. If you rotate the primary's KEK, you must run an equivalent rotation on each replica so their Postgres instances can continue to start.
Cross-cloud multi-DC isn't supported. A multi-DC HM deployment requires the same provider or on-premises family across all locations, and TDE inherits that constraint. Your KMS must be reachable from every location that hosts a replica cluster.
Distributed high-availability (PGD) clusters across data centers
A distributed high-availability cluster is composed of independent data groups. Each data group is a separate Postgres cluster that participates in the cluster's cross-group replication. In Hybrid Manager, every data group added after the seed is joined by a physical join (bdr_init_physical), which copies the source group's data directory in full, including the wrapped-DEK file. Every data group ends up encrypted under its own configured KEK once it's running, regardless of whether it's the seed group or a group added later:
| Data group | How key.bin is created on the target | Effective KEK after the group is running |
|---|---|---|
| Seed (first) data group | Freshly generated at cluster creation, wrapped by the group's own configured KEK | The group's own configured KEK |
Any group added later (physical join via bdr_init_physical) | Copied from an existing group by base-backup, then re-wrapped by the operator under the target group's own configured KEK before the group is started | The target group's own configured KEK |
The physical join path for a newly added data group has one requirement that the seed group doesn't: the source group's KEK must be reachable and permissioned from the joining pod while the physical join is running, because the target's manager has to read the DEK out of the copied key.bin once before re-wrapping it under the target's KEK.
Recommendations
Register a customer key in every location that will host a data group, and select the appropriate key per data group under Security on the Data Groups tab. Each group's data at rest ends up encrypted under the key you selected for that group.
Per-region key sovereignty is supported. For a distributed high-availability cluster with data groups in different countries or regulatory zones, you can safely configure each group with its own KMS-managed key that has genuinely different underlying material from every other group. The operator re-wraps the encryption file under the target group's own KEK as part of the physical join, so each group's data at rest is protected by exactly the key you selected for that group. Each group's KEK can also live in a different KMS provider from the other groups', provided the same provider or on-premises family constraint that applies to the overall multi-DC deployment is respected.
Make the source group's KEK reachable from the joining group's location for the duration of the physical join. The joining group's Postgres pods use their own workload identity or credentials to run the one-time unwrap against the source's KEK. After the physical join completes, the source's KEK is no longer required by the joined group and you can remove that access if your security policy calls for it.
KEK rotation on one data group is local to that group. Rotating a data group's KEK rewrites that group's on-disk encryption file only. Other data groups continue with their own KEKs unchanged.
Scaling from single region to multi-region
Adding a replica cluster to an existing single-region cluster works the same way as creating a replica cluster alongside a primary at initial cluster creation. The new replica is bootstrapped from the primary using base-backup, so the primary's KEK becomes the effective KEK for the new replica. Register a compatible customer key in the target location before adding the replica.
Adding a data group to an existing distributed high-availability cluster finishes with the new group encrypted under its own configured KEK. The physical join uses bdr_init_physical — the source group's key.bin is copied to the new group and then re-wrapped by the operator under the new group's own configured KEK before the group starts. The new group's location must therefore have decrypt access on both the source group's KEK and the new group's KEK for the duration of the physical join. Once the physical join completes, only the new group's KEK is required. See Distributed high-availability (PGD) clusters across data centers for the full comparison.
Key rotation
Note
HM doesn't currently expose a built-in workflow to rotate the customer-managed key (KEK) of an existing cluster. Rotation is a customer-driven activity that you plan and execute against your own KMS provider, where the on-disk wrapped key must also be rewritten against the running Postgres instances.
Three distinct scenarios are commonly called "key rotation." They have very different requirements.
1. Automatic rotation inside your KMS (transparent)
Most cloud KMS providers can rotate the underlying cryptographic material of a customer-managed key on a schedule (for example, AWS KMS annual auto-rotation on customer-managed keys). In these cases the key's identifier (Amazon Resource Name (ARN), resource name, key URI) doesn't change, and prior versions of the material remain usable for decrypt. No action is required on the HM side. Your Postgres clusters continue to unwrap their on-disk encryption file transparently.
If you're only relying on your KMS provider's built-in rotation, there's nothing further to plan.
2. Replacing one customer key with a different customer key
If you want to move a cluster's encryption from one customer-managed key to a different one (for example, to migrate to a new HSM, a new region's key, or to retire an old key after a compromise), this isn't currently possible through HM without manual customer intervention. The Hybrid Manager console doesn't expose a rotate-key action.
If you need to perform this rotation, plan for it as a customer-side operation coordinated with your Postgres administrator. The high-level shape is:
- Register the new customer key in every HM location that hosts data for the cluster.
- On every Postgres pod that holds an on-disk encryption file for the cluster (primary and every replica for a replica-cluster topology; every data group's nodes for a distributed high-availability topology), run the Postgres-provided key rotation operation to rewrite the on-disk wrapped key under the new KEK. This is a per-pod operation and doesn't propagate over streaming replication.
- Verify each pod can restart cleanly using the new key before removing access to the old key.
- Keep the old KMS key available and accessible until every pod has been rotated and successfully restarted. Deleting the old key prematurely leaves any un-rotated pod unable to start.
For step 2, follow the guidance for your Postgres distribution (EDB Postgres Advanced Server or EDB Postgres Extended Server) for the exact command that rewrites the on-disk wrapped key. Contact EDB support if you require assistance planning or executing rotation across a multi-DC deployment.
3. Rotation across a multi-DC deployment per-topology
Replica clusters (PSR): the primary's KEK is the KEK of record for the entire cluster (see Replica clusters across data centers). If you rotate on the primary, you must also rotate on every replica. If you rotate a replica but not the primary, the replica's on-disk encryption file — copied from the primary at bootstrap — remains under the primary's KEK, so the replica's new KEK doesn't actually take effect until the file itself is rewritten by the rotation operation on that replica's pod. Plan the rotation as: primary first, then each replica in turn.
Distributed high-availability (PGD) clusters: each data group's KEK is independent. You can rotate one group's KEK without touching the others. Rotate each group's KEK on the pods of that group only.
What to prepare before rotating
- Register the new customer key in every affected location in HM before you begin.
- Verify the new key is reachable from every pod that will need to unwrap under it (the workload identity or credentials attached to the pod must have decrypt permission on the new key).
- Take a fresh backup of the cluster.
- Plan a maintenance window; each pod will need to restart under the new key.
- Don't delete the old key in your KMS provider until every pod has been rotated and has successfully restarted at least once under the new key.
Summary
| Scenario | Can each location use a different KEK material? |
|---|---|
| Primary cluster with replica cluster(s) in different locations | Not recommended. All locations should resolve to compatible key material. The primary's KEK is the effective KEK for every replica. |
| Distributed high-availability cluster with data groups in different locations, each configured with its own customer key | Yes. Each group is encrypted under its own configured KEK. For the seed group this is true from the start; for any group added later, the operator re-wraps key.bin under the target group's own KEK during the physical join. PGD replicates decrypted row images between groups over TLS — no key material crosses locations. |
| Adding a replica cluster to a single-region cluster later | Same rule as above, the primary's KEK protects the new replica's data at rest. |
| Adding a data group to a distributed high-availability cluster later | Yes. The new group's key.bin is re-wrapped under its own KEK during the physical join; no post-join rotation is needed. Both the source group's KEK and the new group's KEK must be reachable from the target during the physical join. |
| KEK rotation on the primary of a cluster with replicas | Must be run on the primary and repeated on each replica. |
| KEK rotation on a distributed high-availability data group | Local to that group only. |