Authenticating...
Skip to main content

0074: Kubernetes Target Architecture

STATUS​

Accepted

CONTEXT​

Today, Kubernetes at AdAction is two clusters. The data-tools cluster from ADR-0007 runs Datahub and Metabase in production: hand-built, managed node groups, IRSA, deployed via GitHub Actions running cdk deploy with prepackaged Helm charts. Alongside it, a recently built EKS Auto Mode cluster hosts Airbyte and already runs most of what this ADR proposes — Auto Mode, Pod Identity, External Secrets from Secrets Manager, ALB ingress — including a working answer to ALB-terminated SSO. Airbyte is not yet carrying live workloads.

That second cluster matters to this decision twice over. It is evidence: the core choices below are not paper designs, they are running in our accounts on our networking. And it is the reason several sections cite in-house findings rather than vendor documentation. Application workloads still run on a mix of Elastic Beanstalk (ADR-0036), EC2, and Lambda (Keystone, the postback processor — ADR-0033), each with its own bespoke deploy pipeline and runtime configuration.

This ADR defines the target architecture for consolidating application workloads onto Kubernetes (EKS). It is a companion to the Kubernetes Evolution Plan, which describes how we get there from the current state. (That plan is still in review; once it merges, this link becomes ../../engineering/k8s-evolution-plan.md.)

Drivers​

  1. Platform consistency — one way to build, deploy, observe, and operate every service, instead of one bespoke setup per app.
  2. Deploy velocity and safety — progressive delivery, instant rollbacks, preview environments, and a git audit trail for every change.
  3. Scalability and resilience — horizontal autoscaling and self-healing for spiky consumer traffic (offerwall, postback fan-out) beyond what Beanstalk provides.
  4. Ecosystem alignment — Kubernetes is where the industry and our vendors invest first. Moving with it unlocks a steady stream of k8s-first tooling (notably new Datadog features), better community support, and easier hiring and onboarding.

Cost is a consideration throughout, but not a driver.

Constraints​

  • AWS with EKS as the Kubernetes flavor (settled; alternatives not revisited).
  • Infrastructure provisioning stays in TypeScript CDK (ADR-0035).
  • CI stays on GitHub Actions (ADR-0036); the CD half is decided here.
  • A dedicated platform team owns the clusters and the paved road; app teams consume the platform through templates.

Scope​

This ADR is shared: it governs both AdAction and AdGem workloads, and the shared platform accounts that host them.

All long-running application workloads: the Laravel apps (AdGem API, offerwall), Player API, Prism, Offer API, and the existing data tools. Event-driven serverless workloads are governed by the exception policy in the DECISION section — Keystone stays on Lambda.

Considered Options​

Each major axis was evaluated independently:

  1. Cluster topology
    • Centralized: one shared cluster per environment, all apps as tenants
    • Decentralized: one cluster per product domain per environment
    • Hub + spokes: shared per-environment clusters by default, blueprint-stamped dedicated clusters by exception
  2. Networking
    • Deploy clusters into the existing RAM-shared VPCs (ADR-0062) with a dedicated pod CIDR
    • Dedicated VPC per cluster connected via Transit Gateway
    • Shared VPCs now, with pre-agreed triggers for escalating a cluster to a dedicated VPC behind a Transit Gateway
  3. Delivery (CD)
    • Push-based: GitHub Actions runs helm upgrade against clusters
    • Pull-based GitOps with ArgoCD
    • Pull-based GitOps with Flux
  4. Compute management
    • EKS Auto Mode (AWS-managed node lifecycle)
    • Managed node groups plus self-managed Karpenter
    • Fargate
  5. Service mesh
    • No mesh at adoption; written triggers for later adoption
    • Istio ambient mode from day one (mTLS-first posture)
    • Linkerd
    • AWS App Mesh — excluded outright: deprecated, EOL September 2026
  6. Workload identity
    • IRSA (the ADR-0007 choice)
    • EKS Pod Identity

DECISION​

1. Cluster topology: hub + spokes — shared by default, dedicated by exception​

Three shared clusters — one per environment (prod, SDLC, sandbox) — living in dedicated platform accounts host all standard stateless services as tenant namespaces. A reusable CDK cluster blueprint (cluster, addons, ArgoCD registration, observability, baseline policies) can stamp out a dedicated "spoke" cluster when a workload meets written exception criteria:

  • a compliance or regulatory isolation boundary,
  • a tenant requiring hard isolation the shared cluster cannot provide,
  • sustained scale that measurably degrades shared-cluster neighbors.

The data-tools workloads become the first spoke rather than tenants of a shared cluster: they keep their own cluster and account. In practice the Airbyte Auto Mode cluster is the closest thing we already have to that spoke, so the likely path is converging it onto the blueprint and moving Datahub and Metabase onto it, retiring the ADR-0007 cluster — not rebuilding from nothing.

Either way, "adopted, not migrated" describes the org chart, not the work. The ADR-0007 cluster predates Auto Mode, Pod Identity, and ArgoCD, and both existing clusters place pods on routable subnets, so conforming to this ADR means a real pod-network retrofit. Section 2 and section 4 record what that costs. One timing note the evolution plan acts on: that retrofit is node replacement, which is free on a cluster carrying no live traffic and a migration on one that is — so the Airbyte cluster is cheapest to converge before it goes live.

Trade-offs accepted: shared clusters concentrate blast radius (a bad upgrade affects every tenant) and demand real multi-tenancy discipline (see paved road below). We take that in exchange for the smallest possible operational surface — three or four clusters instead of six to nine — and the strongest consistency story. Because blast radius is the headline cost of this choice, shared-component changes ship tenant-by-tenant rather than cluster-wide: see the staged-rollout policy in section 12.

Full decentralization (cluster per domain) was rejected because operating and keeping consistent that many clusters directly fights the consistency driver; pure centralization with no exit was rejected because it has no honest answer for workloads that legitimately do not fit.

2. Networking: shared VPCs with a pod CIDR now; Transit Gateway as a documented escape hatch​

Clusters deploy into the existing RAM-shared VPCs (10.30/.31/.32, ADR-0062). Each shared VPC gains a secondary CIDR from CG-NAT space (100.64.0.0/16) used exclusively for pod IPs, so pods draw from a non-routable 65k pool and cannot exhaust the routable /16. Nodes, load balancers, and databases keep using routable space. Pod-to-database traffic stays intra-VPC: no gateway hops, no per-GB charges.

Pod placement in the secondary CIDR is explicit configuration, not a default. Auto Mode's default NodeClass puts pods on the same routable subnets as the nodes; CG-NAT placement requires a custom NodeClass with podSubnetSelectorTerms and podSecurityGroupSelectorTerms — Auto Mode requires the pair together to separate pod networking from node networking, so the blueprint owns a dedicated pod security group (and its ingress and egress rules) as part of the same contract, not as a follow-up. (Auto Mode does not support classic ENIConfig custom networking; on non-Auto-Mode clusters, VPC CNI custom networking achieves the same placement.) This is confirmed in-house: the Airbyte cluster build landed pods on routable 10.64.x under the default NodeClass. So the blueprint must set the custom NodeClass from the start, and the data-cluster adoption in section 1 includes re-homing its pod network, which is node replacement rather than a config flip.

Separate pod networking also costs pod density: the node's primary network interface is reserved for the node's own address, so pod IPs come only from secondary interfaces. That is a second reason the pod-scheduling trigger below is not redundant with routable IP pressure — density limits bite before either CIDR looks full.

The escape hatch, pre-agreed here so future escalation is a checklist rather than a debate — any one trigger justifies moving a cluster to its own VPC behind a Transit Gateway:

TriggerMeasure
Compliance boundaryA regulation or customer contract requires network isolation the shared VPC cannot demonstrate
Hard-isolation tenantA workload qualifies for a dedicated spoke for isolation reasons (topology criteria above)
Routable IP pressureRoutable IP utilization of a shared VPC sustained above 70 percent
Pod address or scheduling pressurePod-CIDR utilization sustained above 70 percent, or pods pending on IP/ENI exhaustion rather than CPU and memory

The second trigger exists because pods draw from a separate pool: the routable /16 can look healthy while the pod CIDR, per-ENI IP limits, or max-pods per node are what actually block scheduling. Both pools get their own alarm in the blueprint's monitor template.

Transit Gateway does not route overlapping CIDRs, so the escape hatch only stays cheap if the addresses are reserved up front. We therefore pre-allocate a non-overlapping node CIDR and pod CIDR per prospective cluster from a central IPAM registry (AWS IPAM pools) at blueprint level, whether or not the cluster ever escalates. The blueprint takes networking mode (shared subnets vs dedicated VPC) plus that reserved CIDR pair as parameters, so escalation is configuration rather than rework — and re-addressing, which would be rework, never enters the picture.

Trade-offs accepted: we knowingly keep ADR-0062's weak east-west isolation — mitigated by default-deny NetworkPolicy from day one (paved road, below). Day-one Transit Gateway was rejected because per-GB processing charges on chatty pod-to-RDS paths (offerwall, postback fan-out) recreate the cost problem 0062 avoided, while buying isolation we do not yet need. Known sharp edges of EKS on RAM-shared subnets (subnet tagging for the load balancer controller, cross-account ENI ownership) are accepted; the data cluster already proves the pattern in-house.

3. Delivery: pull-based GitOps with ArgoCD​

CI (GitHub Actions) ends at "build, test, push image to ECR, bump the image tag in the GitOps config repo." ArgoCD reconciles every cluster from git:

  • ApplicationSets stamp each service across the cluster-by-environment matrix and generate preview environments per pull request, under the constraints below.
  • Argo Rollouts provides progressive delivery: canary releases analyzed against Datadog metrics with automatic rollback.
  • Clusters are never reachable from GitHub runners; rollback is git revert; drift is detected and self-healed; git history is the deployment audit log.

Write access to the GitOps repo is production deploy authority. Pull-based delivery closes the cluster API but moves the blast radius to the repo ArgoCD reconciles from: anything that lands on its tracked branch reaches every cluster. The GitOps repo is therefore governed as production infrastructure, not as a config dump:

  • Protected tracked branch: no direct pushes, no force pushes, PR required, required status checks (manifest render, policy tests, schema validation).
  • Human review required on any change to cluster-scoped resources, the paved road's base chart, or ArgoCD's own configuration. Image-tag bumps from CI are the only automerge-eligible change class, and only within a path allowlist.
  • CI writes through a least-privilege scoped token limited to that path allowlist on that one repo. No org-wide PATs, no reuse of the app repo's credentials.
  • Commit signing required, and ArgoCD verifies signatures on the tracked branch, so a leaked token alone cannot produce a commit ArgoCD will apply.
  • CODEOWNERS on the platform directories. ArgoCD AppProject carries part of the tenant boundary but not all of it: spec.sourceRepos restricts which repositories a project may deploy from and spec.destinations restricts which clusters and namespaces it may deploy to, but neither restricts spec.source.path inside an allowed repo. So path isolation comes from app teams not authoring Application manifests at all — the platform-owned ApplicationSet generates them from a fixed path convention — backed by a CI policy check that rejects any Application whose path falls outside its team's directory. Destination restrictions then cap the damage if that check is ever bypassed.

This fulfills the intent ADR-0007 recorded ("If we later decide to deploy custom applications on Kubernetes, ArgoCD would be the preferred best practice"). Push-based GHA-plus-Helm was rejected: 0007 itself flagged the exposed-cluster-API problem, and push pipelines scale as N pipelines times M clusters with no drift detection. Flux was rejected on one decisive point: no built-in UI — self-serve deploy visibility for app teams is part of the velocity driver — and contradicting 0007's recorded preference needs a stronger reason than footprint.

Preview environments run untrusted input, and are constrained accordingly. A PR's manifests and image are proposed code, so previews get the standing of a tenant we do not trust, in the SDLC cluster only — never prod:

  • One preview namespace per PR, under the same paved-road guardrails as any tenant (quota, PSS restricted, default-deny NetworkPolicy) plus a tighter quota ceiling, since previews are the easiest way to accidentally exhaust a shared cluster.
  • No production credentials reachable. Previews get a dedicated Pod Identity role scoped to SDLC data, and their SecretStore may only read the preview-scoped secret path. A preview cannot name a prod secret.
  • Egress restricted to an allowlist; no ingress path from the public internet beyond an SSO-gated hostname.
  • Fork PRs do not auto-deploy. A preview from a fork requires a maintainer label, because otherwise an outside PR would be arbitrary code execution inside a shared cluster.
  • TTL-based cleanup: the ApplicationSet PR generator removes the namespace on merge or close, plus a hard expiry (days, not weeks) so abandoned PRs cannot accumulate.

Trade-offs accepted: ArgoCD is a new stateful platform component (requires SSO and RBAC at install, and its own upgrade care), and the two-repo mental model (app repo vs GitOps repo) is a real onboarding cost.

4. Compute: EKS Auto Mode​

Auto Mode delegates node provisioning (Karpenter-based), addon lifecycle, and node patching/rotation to AWS. The platform team's capacity goes to the paved road instead of AMI hygiene. Pods scale via HPA; spot capacity is enabled per node pool for tolerant workloads.

Cost, stated precisely. AWS does not charge a flat percentage. Auto Mode bills a separate per-hour management fee that varies by instance type, on top of the EC2 instance price. For the general-purpose and compute families we would run, that fee works out to roughly 10 to 12 percent of On-Demand list — which is where the "10 percent" shorthand comes from, and it should be read as an internal estimate for our instance mix, not an AWS-published rate. Two qualifiers matter for planning:

  • The fee is charged independently of purchase option. Against Spot or a Savings Plan, the same absolute fee is a materially higher percentage of what we actually pay for the instance, so the premium is worst exactly where we planned to save.
  • The premium is on compute we would buy anyway, so consolidation and right-sizing dominate it. In-house data point: the Airbyte cluster moved to Graviton and came out roughly 29 percent cheaper with more RAM, comfortably absorbing the Auto Mode fee. Bin-packing several bespoke Beanstalk and EC2 footprints into shared clusters is expected to behave the same way.

The Karpenter fallback is real but not free. Self-managed Karpenter on managed node groups stays the recorded exit (more control, custom AMIs, no management fee), and the AWS-supported migration direction is Karpenter to Auto Mode, so reversing is the less-trodden path. Stateless HTTP services and queue consumers move cleanly, which is most of our scope, and section 11 keeps datastores out of cluster. But "workloads cannot tell the difference" is too strong: Auto Mode owns storage, load balancing, and node APIs under its own identifiers, and three of those do not transfer in place.

ResourceFieldAuto ModeSelf-managedReversal cost
StorageClassprovisionerebs.csi.eks.amazonaws.comebs.csi.aws.comPVCs are not portable between provisioners; each volume needs PV/PVC recreate against a retained EBS volume
IngressClasscontrollereks.amazonaws.com/albingress.k8s.aws/albNo in-place migration; blue-green new load balancers plus a DNS traffic shift
ServiceloadBalancerClasseks.amazonaws.com/nlbservice.k8s.aws/nlbSame as above
IngressClassParams, TargetGroupBindingapiVersioneks.amazonaws.com/v1elbv2.k8s.aws/v1beta1Manifest rewrite; spec.targetType is required under Auto Mode
NodeClassapiVersioneks.amazonaws.com/v1karpenter.sh/v1Blueprint-level change, platform-owned

So the exit is cheap for stateless workloads behind a shared ingress, and it is a planned project for anything holding a PVC or owning its own load balancer. The blueprint keeps that bounded by treating StorageClass and IngressClass as platform-owned indirections that services reference by name, never by provisioner or controller identifier. Before invoking the fallback, the checks are: which namespaces hold PVCs, which own dedicated load balancers, and whether a DNS shift is acceptable for each public hostname.

Fargate was rejected: per-pod pricing at our density, no DaemonSets (complicates the Datadog agent), and slower cold starts for spiky consumer traffic.

5. Workload identity: EKS Pod Identity (updates ADR-0007)​

ADR-0007 chose IRSA for team familiarity. For all new clusters we adopt EKS Pod Identity instead: no per-cluster OIDC provider to manage, and simpler cross-account role chaining — which matters here, because pods in platform-account clusters routinely assume roles into workload accounts for RDS, SQS, and DynamoDB. This decision supersedes 0007's IRSA choice for new clusters; the data cluster migrates when it is adopted under the blueprint.

6. Multi-tenancy and the paved road​

Every service gets a namespace stamped from a template: ResourceQuota, LimitRange, default-deny NetworkPolicy with explicit allows, and Pod Security Standards restricted. PriorityClass is deliberately not in that list — it is cluster-scoped, so a namespace template is the wrong place for it. The blueprint creates a small fixed set of approved priority classes once per cluster (platform, production-critical, standard, batch) and services reference one by name from their values file. Preemption ordering across tenants is a platform decision, not a per-tenant one.

Creating the classes centrally is only half of it: Kubernetes lets any pod name any existing PriorityClass, and a higher-priority pod can preempt a lower-priority one. So a tenant that simply set priorityClassName: platform would be able to evict its neighbors. The namespace template therefore records each namespace's permitted classes, the base chart validates the value against that list, and an admission policy enforces it at the API server so the chart is not the only gate. platform stays reserved for platform namespaces, and production-critical requires a written case, the same way a dedicated spoke does.

Default-deny only counts once the controller is enabled. On Auto Mode, NetworkPolicy objects are accepted whether or not enforcement is on, so the guardrail can look present while traffic flows freely. Enabling it requires the amazon-vpc-cni ConfigMap in kube-system with enable-network-policy-controller: "true". The blueprint applies that ConfigMap and the cluster's baseline policies before any tenant namespace is admitted, and a smoke test asserts a denied connection is actually refused. Auto Mode also offers ClusterNetworkPolicy (Admin-tier rules that namespace policies cannot override) and DNS-based egress filtering via ApplicationNetworkPolicy; the Admin tier is the natural home for cross-tenant deny rules that must not be weakened by a tenant's own manifest.

One shared base Helm chart is consumed by all services with a small per-service values file — teams declare what they run; the chart encodes how (same philosophy as our _common.yml merge pattern elsewhere). ArgoCD Projects give per-team RBAC. Humans authenticate through Identity Center mapped to EKS access entries.

7. Ingress and traffic; service mesh deferred with triggers​

Shared clusters use Auto Mode's built-in load balancing: an IngressClass with controller: eks.amazonaws.com/alb (configured through IngressClassParams at eks.amazonaws.com/v1, since Auto Mode ignores annotations on the IngressClass itself) and loadBalancerClass: eks.amazonaws.com/nlb for L4. The separately installed AWS Load Balancer Controller is reserved for two cases: non-Auto-Mode fallback clusters, and the OIDC exception below. external-dns is deployed by us in the blueprint — it is compatible with Auto Mode and watches ordinary Ingress and Service objects, but it is not part of Auto Mode, so it is our component to run and upgrade. Cloudflare sits in front in Full (strict) mode (ADR-0066).

One known gap, confirmed in-house on the Airbyte build: Auto Mode's managed controller does not support alb.ingress.kubernetes.io/auth-type: oidc (AWS documents it as unsupported). Any endpoint that needs ALB-terminated SSO therefore runs a second, self-managed AWS Load Balancer Controller on its own IngressClass, which is supported — the two controllers coexist as long as each Ingress and Service is explicitly claimed by class. Existing load balancers cannot be moved between the two controllers, so the class is chosen when the service is onboarded, not later. Internal tooling endpoints are the expected users; application traffic authenticates in-app and stays on the Auto Mode class.

No service mesh at adoption. Datadog APM already covers tracing and golden metrics; our internal synchronous call graph is small; and mesh complexity would land during the migration itself. Adoption triggers, any one of which reopens the decision with Istio ambient mode as the pre-selected candidate (sidecarless, low retrofit cost, composes with Auto Mode):

TriggerMeasure
Zero-trust requirementCompliance or customer requirement for encryption-in-transit or workload-identity auth inside the VPC
Internal canary needA release class where ALB-level traffic splitting cannot canary an internal service-to-service dependency
L7 authorizationA concrete need to authorize specific routes between tenants, beyond NetworkPolicy's L3/L4

AWS App Mesh is excluded regardless: deprecated, EOL September 2026.

8. Configuration and secrets​

External Secrets Operator syncs from AWS Secrets Manager and SSM Parameter Store into namespaces. No secrets in git; the GitOps repo holds only per-environment Helm values.

"No secrets in git" is a statement about the repo, not about the cluster. ESO materializes provider values into real Kubernetes Secret objects, so the cluster becomes a secret store and needs its own boundaries:

  • At rest: clusters enable EKS envelope encryption of secrets with a customer-managed KMS key, so etcd-level access is not sufficient to read secret material. The key policy is per-environment.
  • Namespace-scoped, not cluster-scoped: each tenant gets a namespaced SecretStore, not a shared ClusterSecretStore. A ClusterSecretStore in a multi-tenant cluster is a cross-tenant read path by construction, so its use is restricted to platform namespaces.
  • Least-privilege by path: each service's Pod Identity role may read only its own prefix (/<env>/<service>/*) in Secrets Manager and Parameter Store. The prefix convention exists so the IAM policy can be generated from the namespace template rather than hand-written per service.
  • In-cluster read access: RBAC on Secret objects is namespace-scoped to the owning team; no tenant role carries cluster-wide get/list on secrets, and Datadog and other platform agents run without secret read permission outside their own namespace.
  • Rotation stays owned by the provider: ESO refreshes on an interval, so rotating in Secrets Manager propagates without a deploy.

9. Observability​

The Datadog agent ships in the cluster blueprint: APM, logs, Kubernetes state metrics, and control-plane logs, with service naming per ADR-0018 and cost tracking per the Datadog Cloud Cost Management runbook. A per-service dashboard and monitor template means every migrated service lands with monitoring on day one. Argo Rollouts canary analysis reads Datadog metrics.

10. Serverless exception policy​

Default: long-running HTTP services and queue consumers run on Kubernetes. A workload may remain on (or ship new to) Lambda when it is event-driven, highly spiky or infrequent, or integration glue — and does not need the paved road's deploy/observability machinery. Keystone, the postback processor (ADR-0033), stays on Lambda under this policy. Placement debates become a checklist against these criteria, not a philosophy discussion.

11. State stays on managed services​

RDS, ElastiCache, and DynamoDB remain the homes for data. No datastores run in-cluster — consistent with ADR-0007's approach for Datahub's backing services. Clusters must remain cattle.

12. Upgrade and failure posture​

The SDLC cluster always runs one Kubernetes minor version ahead of prod; upgrades soak in SDLC before promotion. Auto Mode handles node rotation.

Staged rollout of shared changes. Version skew across environments is not enough on its own: within the prod cluster, a shared-component change still reaches every tenant at once, which is the blast-radius cost recorded in section

  1. So changes to shared surfaces — the base Helm chart, cluster-scoped policy, addons, ArgoCD itself — roll out in waves rather than cluster-wide: platform namespaces first, then a designated low-risk tenant, then the rest, with a soak window and the tenant's own monitors as the gate between waves. ArgoCD sync waves and per-service chart version pinning are what make this expressible: a tenant can sit one base-chart version behind during a rollout without being drifted. Kubernetes minor upgrades stay all-at-once per cluster (the control plane is shared and cannot be waved), which is precisely why the SDLC-ahead soak policy carries that case. Making tenant-by-tenant rollout cheap is an explicit blueprint requirement, not a later optimization.

Disaster recovery. The default posture is rebuild-from-blueprint: cluster state is reproducible from CDK, desired workload state re-syncs from git via ArgoCD, and data lives in managed services. That covers the cluster and the workloads, and it is deliberately the easy half. The parts it does not cover are named here so they get owners rather than being assumed:

ComponentRecovery sourceOwner
Cluster and addonsCDK blueprint re-applyPlatform
Workload desired stateArgoCD sync from GitOps repoPlatform
ArgoCD itselfBootstrapped by the blueprint, not by ArgoCD — the bootstrap path is tested, since a circular dependency here blocks every other recoveryPlatform
Application dataRDS automated backups and PITR, DynamoDB PITR, ElastiCache snapshotsData-store owner per service
SecretsSecrets Manager replica secrets in the secondary region (native replication)Platform
ParametersSSM Parameter Store has no native cross-region replication, so parameters are defined in CDK and recreated by re-applying the stack in the secondary regionPlatform
Container imagesECR, replicated to the secondary regionPlatform
DNSRoute 53 and Cloudflare config in IaCPlatform

Targets are set per service rather than globally, because a cluster rebuild is minutes-to-hours while an RDS PITR restore is bounded by data volume, and it does no good to claim a cluster RTO the datastore cannot match. Each service records its RPO and RTO in its onboarding values as part of joining the paved road, and the platform publishes the rebuild time it can commit to. Restores are proven, not assumed: a scheduled drill rebuilds a cluster from blueprint in sandbox and restores at least one datastore into it, and an untested restore path is treated as a known gap rather than a recovery plan.

On regional scope, to be unambiguous: serving is single-region, and we do not claim a cross-region RTO. What crosses regions is only the cheap, stateless prerequisites for ever rebuilding there: container images, secrets, and parameters recoverable from IaC. That is replication we would regret not having, and it is a different claim from being able to serve from there. There is no warm cluster, no replicated database, and no tested regional failover, so a region loss is an extended outage, not a failover. Making that real means cross-region data replication and a drilled promotion path, which is deliberate future work and not implied by "rebuild from blueprint."

Target architecture overview​

CONSEQUENCES​

Positive​

  • One paved road: identical deploy, observability, and access patterns for every service; onboarding a service or an engineer converges to a template.
  • Progressive delivery with metric-gated automatic rollback directly serves deploy velocity and safety; preview environments per PR shorten feedback.
  • Drift-free, auditable delivery: git is the single source of deployed truth.
  • Ecosystem tailwind: k8s-first vendor features (Datadog and beyond) become usable the day they ship.
  • The existing data cluster stops being a special case and becomes the first conforming spoke.

Negative​

  • The platform team takes on ArgoCD and cluster-blueprint ownership — genuinely new operational surface, adopted deliberately.
  • Shared clusters concentrate blast radius; multi-tenancy guardrails are load-bearing from day one, not optional hardening.
  • Two-repo GitOps model and Kubernetes itself carry a real learning curve for app teams during the migration.
  • EKS Auto Mode carries a per-instance-type management fee (order of 10 to 12 percent of On-Demand list for our mix, and a higher effective share on Spot) plus less node customization; accepted, with self-managed Karpenter as the documented exit and section 4's compatibility table as its real price.
  • The GitOps repo becomes a production-critical access boundary: repo write access is deploy authority, so branch protection, scoped tokens, and commit signing are load-bearing controls rather than hygiene.
  • Auto Mode's managed ingress does not cover ALB-terminated OIDC, so SSO-gated endpoints mean running a second load balancer controller — a small permanent exception to "one way to do it."

Risks​

  • Shared-VPC isolation stays weak (inherited from ADR-0062): mitigated by default-deny NetworkPolicy, PSS restricted, and the pre-agreed Transit Gateway and service-mesh triggers.
  • Cluster upgrade coordination across all tenants: mitigated by the SDLC-ahead-of-prod soak policy, staged tenant-by-tenant rollout of shared components (section 12), and Auto Mode's managed rotation.
  • Guardrails that are present but not enforcing: NetworkPolicy on Auto Mode is the concrete case (objects accepted while the controller is off), and the general failure mode is a policy that reads as applied and is not. Mitigated by blueprint smoke tests that assert a denied path is actually denied, rather than asserting the object exists.
  • Untested recovery paths: rebuild-from-blueprint is only credible if exercised, and the ArgoCD bootstrap is the circular-dependency risk; mitigated by scheduled sandbox rebuild-plus-restore drills and per-service RPO/RTO ownership (section 12).
  • Blueprint quality risk: a second-class blueprint makes dedicated spokes drift; mitigated by CDK assertion tests and periodic ephemeral cluster builds in sandbox.
  • "Escape hatch never exercised" risk: triggers are written as measurable conditions in this ADR specifically so escalation cannot be indefinitely deferred by inertia.

NOTES​

This ADR records the target; sequencing, service order, and cutover mechanics live in the Kubernetes Evolution Plan.

References​

Original Author​

Ron White (ronco)

Approval date​

Approved by​

Appendix​

Appendix A — Decision matrix summary​

AxisChosenRejected (short reason)
TopologyHub + spokesCentralized-only (no exit for misfits); per-domain (ops burden times N, drift)
NetworkingShared VPCs + pod CIDR, TGW triggersDay-one TGW (per-GB cost on chatty paths, unneeded isolation)
DeliveryArgoCD GitOpsPush GHA+Helm (exposed API, no drift detection); Flux (no UI for self-service)
ComputeEKS Auto ModeSelf-managed Karpenter (fallback, more toil); Fargate (cost, no DaemonSets)
MeshDefer with triggersDay-one ambient mTLS (complexity during migration); App Mesh (deprecated)
IdentityEKS Pod IdentityIRSA (per-cluster OIDC, harder cross-account chaining)