Kubernetes Evolution Plan

How we get from today's mixed estate (Elastic Beanstalk, EC2, Lambda, and two EKS clusters) to the target described in the Kubernetes Target Architecture ADR. The ADR records what and why; this plan records in what order and how we know each step worked.

Where we actually start

Worth stating precisely, because it is better than the ADR's original framing suggests:

• The legacy data-tools cluster (ADR-0007): hand-built, managed node groups, IRSA, deployed by GitHub Actions running cdk deploy. Datahub and Metabase run here, and they are live.
• A new EKS Auto Mode cluster, built recently for Airbyte. It already runs the target stack — Auto Mode, Pod Identity, External Secrets from Secrets Manager, ALB — and the ALB-terminated SSO problem has been solved on it with a second self-managed load balancer controller. Airbyte is not yet serving live workloads.

That second cluster changes the plan's shape in two ways. Phase 0 is no longer greenfield: the blueprint should be extracted from a cluster that already works, not designed from documentation. And there is a closing window — while Airbyte carries no live workloads, that cluster can be reconciled onto the blueprint (or re-stamped outright) with no soak and no rollback plan. Once Airbyte goes live, the same work becomes a migration. Phase 0.5 exists to spend that window rather than let it close.

Each phase has entry criteria (do not start until true), exit criteria (the phase is done when true), and a rollback posture. Phases are sequential except where noted below, and work inside a phase parallelizes freely.

Guiding rules

1. The platform earns trust with boring workloads before it gets critical ones. Traffic order: internal and low-risk first, consumer-facing last.
2. No service migrates without its monitoring. The Datadog dashboard/monitor template lands with the service, before cutover.
3. Every cutover is reversible for its first two weeks. DNS/Cloudflare weighted routing back to the legacy target stays wired until the soak passes.
4. The paved road is finished before the third service uses it. Gaps found by services one and two are platform work, not app-team workarounds.
5. A guardrail counts as done when it's proven enforcing, not when it's applied. Test the denied case. Kubernetes and Auto Mode will both happily accept policy objects that do nothing, and a policy that reads as applied but isn't is worse than no policy, because it stops anyone looking.

Phase 0 — Platform foundation

Stand up what everything else rides on.

Start by harvesting the Airbyte cluster. Its Auto Mode, Pod Identity, ESO, and ALB configuration is a working reference in our own accounts, on our own networking, and the second-controller SSO pattern is already solved there. Reading that config into the blueprint retires a real chunk of this phase's risk and is strictly faster than rediscovering it. Two things to capture explicitly while doing so: which of its settings were deliberate versus defaulted, and where it diverges from the blueprint we want (its pods are on routable subnets, so at minimum the pod-network decisions differ). Treat its owner as a required reviewer of the blueprint, not a consumer of it.

Work:

• Add secondary CG-NAT CIDR (100.64.0.0/16) and pod subnets to the prod, SDLC, and sandbox shared VPCs (CDK change in the networking app).
• Stand up an IPAM registry and reserve a non-overlapping node CIDR and pod CIDR for every prospective cluster, including spokes we have not built yet. Transit Gateway cannot route overlapping ranges, so the ADR's escape hatch is only configuration-not-rework if the addresses are claimed before anything ships. Doing this after the hubs exist means re-addressing.
  • Write down the connectivity rule while doing it. Per ADR-0062 the prod, SDLC, and sandbox shared VPCs are not Transit Gateway attached, and this plan does not change that: adding 100.64.0.0/16 to them is safe precisely because they stay off the TGW. The reserved CIDRs exist for escalated clusters, which by definition leave the shared VPC for their own TGW-attached VPC. If anyone later proposes attaching a shared VPC to the TGW, that is a separate decision and this reservation scheme is its precondition, not its consequence.
  • Pod addresses are also not routed off-VPC: pod traffic to external destinations SNATs to the node address under the CNI's default behaviour, so the hard TGW uniqueness constraint lands on routable node/VPC CIDRs. Unique pod CIDRs are hygiene for the day we want pod-level routing or turn off external SNAT. Confirm that SNAT posture on the first cluster rather than assuming it, since it decides whether RDS and cross-account rules see pod IPs or node IPs.
• Build the cluster blueprint CDK construct: EKS Auto Mode cluster with its built-in pod identity and load-balancing capabilities enabled, installed addons — external-dns, External Secrets Operator, Datadog agent, ArgoCD (hub clusters) or ArgoCD registration (spokes) — and baseline policies (PSS restricted, default-deny NetworkPolicy, PriorityClasses). Blueprint details that are easy to miss and expensive to retrofit:
  • Pod placement on the secondary CIDR needs a custom NodeClass with both podSubnetSelectorTerms and podSecurityGroupSelectorTerms — Auto Mode requires the pair to separate pod networking from node networking. So the blueprint also owns a dedicated pod security group and its ingress/egress rules. The default NodeClass silently puts pods on routable subnets.
  • Enable the network-policy controller (amazon-vpc-cni ConfigMap in kube-system, enable-network-policy-controller: "true"). Auto Mode accepts NetworkPolicy objects whether or not enforcement is on, so default-deny ships inert without this.
  • Secrets encryption at cluster creation: envelope encryption with a per-environment customer-managed KMS key. This is a create-time cluster setting, not something to add later.
  • An admission policy restricting which PriorityClass each namespace may reference. Creating the classes centrally does not stop a tenant naming platform and preempting its neighbours, and chart-level validation only binds teams who use the chart, so this has to bite at the API server.
  • ArgoCD Project resource restrictions, not just destinations. Path and destination limits say where a chart may deploy; these say what it may create, and a chart that can mint a ClusterRoleBinding has escaped its namespace. Three specific controls on tenant projects:
    • clusterResourceWhitelist (default-deny for cluster-scoped kinds) kept empty or minimal, so ClusterRoleBinding and Namespace are refused;
    • namespaceResourceBlacklist denying argoproj.io/Application, so a tenant chart cannot generate its own Applications and escape the platform-owned ApplicationSet;
    • a negative test per kind, plus a review of the rendered AppProject, since these are easy to write and easy to get subtly wrong.
  • The default-deny egress allows that the paved road itself depends on: ALB health checks and target traffic inbound, plus egress to CoreDNS, the AWS APIs reached through Pod Identity, ECR, and the Datadog agent's endpoints. Without these baked into the baseline policy, the Phase 0 demo service cannot serve traffic or report telemetry, so the exit criteria below would fail for the right reason and look like the wrong one.
  • StorageClass and IngressClass exposed as platform-owned names that services reference indirectly, never by provisioner or controller identifier — this is what keeps the Karpenter fallback bounded.
  • A second, self-managed AWS Load Balancer Controller on its own IngressClass, for endpoints needing ALB-terminated SSO. Auto Mode's managed controller does not support auth-type: oidc, and load balancers cannot be moved between controllers later, so the class is picked at onboarding. This already exists on the Airbyte cluster — templatize that rather than solving it twice.
• Monitor template covers both address pools, using one capacity definition that Phase 0 and Phase 3 both reference so the two phases cannot drift:
  • Routable utilisation = assigned IPs ÷ assignable IPs across the VPC's routable subnets. Pod-CIDR utilisation = same ratio over the secondary-CIDR subnets. ENI/pod-slot utilisation = scheduled pods ÷ the cluster's max-pods capacity at current node count.
  • Aggregation is the 1-hour average per pool, per cluster; the gate is sustained breach over a 24-hour window, so a spike does not trip it.
  • Warn at 70 percent utilisation, fail at 85. Stated as utilisation throughout: higher is worse. (Phase 3's gate previously said "headroom above 70 percent," which inverts the comparison and would fail a healthy cluster at 40 percent utilisation.)
  • Pending pods: steady-state target is zero, and the load-test gate allows pending pods only during an active scale-out, cleared within 5 minutes of the triggering demand. Separate pod networking reserves the node's primary interface, so pod-slot utilisation can breach while both CIDRs still look healthy — which is the whole reason all three are tracked rather than just the routable pool.
• Give the thresholds a defined response, or they are just a dashboard: first action is expansion (another secondary CIDR, larger pod subnets, or raising node count/instance size). Only a compliance or hard-isolation trigger, or expansion genuinely running out, justifies escalating a cluster to its own VPC behind a Transit Gateway.
• Stamp the three hub clusters (prod, SDLC, sandbox) in platform accounts.
• Stand up the GitOps repo, governed as production infrastructure from the first commit, because write access to it is deploy authority: protected tracked branch, required status checks, CODEOWNERS on platform paths, a least-privilege scoped CI token, and commit signing with ArgoCD verifying signatures (cheap now, awkward to retrofit over existing history). Plus the ApplicationSet structure for the cluster-by-environment matrix and ArgoCD SSO via Identity Center.
• Per-team ArgoCD Projects for RBAC and destination limits. Note these do not constrain spec.source.path within an allowed repo, so path isolation comes from app teams never authoring Application manifests (the platform-owned ApplicationSet generates them) plus a CI check rejecting out-of-directory paths.
• CDK assertion tests for the blueprint; one ephemeral cluster build/teardown exercised in sandbox.

Entry criteria: ADR accepted. Platform team staffed.

Exit criteria: A demo "hello" service deploys to all three hub clusters via a git commit, receives traffic through Cloudflare → ALB, reports APM and logs to Datadog, and rolls back via git revert — with zero console access. Pods verifiably hold secondary-CIDR addresses, and per guiding rule 5 the guardrails pass negative tests rather than existence checks:

• a connection that should be blocked by default-deny actually is;
• a privileged or hostPath pod is rejected by Pod Security Standards restricted;
• a workload naming a PriorityClass outside its namespace's allowlist is rejected at admission;
• a tenant manifest rendering a ClusterRoleBinding is refused by its ArgoCD Project.

These four become the blueprint's regression suite, re-run on every blueprint change, since each one is a control that fails silently.

Rollback posture: Mostly none needed, with one exception worth naming: the secondary CIDR and pod subnets land in the live production shared VPC, so that step is a prod network change even though no workload uses it yet. Treat it like any other prod networking change (SDLC and sandbox first, off-peak window, route-table and NACL diffs reviewed). Rollback is removing the added subnets and CIDR association, which is clean only while nothing has launched into them, so it stops being reversible the moment the first cluster is stamped. Everything after that point is additive and disposable: unused clusters get deleted.

Phase 0.5 — Reconcile the Airbyte cluster while it's still free

Bring the existing Auto Mode cluster onto the blueprint before Airbyte carries live workloads. Today this is a maintenance task; after go-live it is a migration with a soak window, a rollback plan, and someone's data pipeline at risk. The work is the same either way, so the only variable is what it costs.

This is a gate on Airbyte's go-live, not a race against it. Two of these items cannot be done later without a migration: pod-network re-homing (node replacement) and secrets envelope encryption (a create-time cluster setting, so "later" means rebuilding the cluster). If Airbyte's go-live date arrives first, the decision is explicit — either hold go-live until this phase exits, or accept in writing that the cluster ships non-conforming and its convergence moves into Phase 4 with a full soak. What must not happen is go-live drifting past this phase by default, since that quietly converts a free change into an expensive one.

Work:

• Decide the mechanism honestly: re-stamp from the blueprint and move Airbyte onto the new cluster, or reconcile the existing cluster in place. Re-stamping is cleaner and exercises the blueprint for real; in-place is less work if the divergence turns out to be small. Pick after the Phase 0 harvest shows how far apart they actually are.
• Either way, do the pod-network re-homing now (custom NodeClass with podSubnetSelectorTerms and podSecurityGroupSelectorTerms, pods onto the secondary CIDR). This is node replacement, and node replacement is free when nothing is serving.
• Confirm secrets envelope encryption with a customer-managed KMS key, and if the cluster was created without it, that alone decides the mechanism above: it cannot be enabled after the fact, so the cluster has to be re-stamped. Cheaper to discover this now than to inherit an unencrypted-at-rest cluster that later holds real ESO-materialized secrets.
• Register it with the hub ArgoCD and move its deploys off cdk deploy / GitHub-Actions push onto GitOps.
• Apply the baseline policies and run the four negative tests against it. A real cluster with a real workload is a much better test of the guardrails than the Phase 0 demo service.

Entry criteria: Blueprint exists and has been stamped at least once (Phase 0 work substantially done). Does not require the full Phase 0 exit criteria — waiting for that risks the window closing.

Exit criteria: Airbyte runs on a blueprint-conforming Auto Mode cluster, reconciled or re-stamped, with pods on the secondary CIDR, secrets encrypted with a customer-managed KMS key, managed by hub ArgoCD, and the guardrail tests passing. No cdk deploy path remains for it.

Rollback posture: Trivial while Airbyte is not live, which is the entire point of doing it here. If the window has already closed by the time this starts, this phase stops being special and folds into Phase 4 under normal soak rules.

Phase 1 — Paved road proven end to end

Migrate one real, low-risk internal service (candidate criteria: internal consumers only, stateless, single queue or cron dependency, an engaged owning team). Build the missing paved-road pieces the migration exposes: the base Helm chart, the namespace template, the per-service Datadog template, the preview-environment ApplicationSet, the service onboarding doc.

Requirements the ADR pins on these pieces:

• Namespace template: a namespaced SecretStore per tenant (not a shared ClusterSecretStore, which is a cross-tenant read path by construction), and a Pod Identity role scoped to that service's /<env>/<service>/* prefix. The prefix convention exists so the IAM policy generates from the template instead of being hand-written per service.
• Base chart: per-service chart version pinning, so shared changes can roll out tenant-by-tenant later without a lagging tenant reading as drift. Cheaper to design in now than to add once every service consumes the chart.
• Onboarding doc and values: each service records its RPO and RTO as part of joining the paved road, and declares which PriorityClass it may use.
• Preview environments: SDLC only, never prod. Tighter quota than a normal tenant, a preview-scoped Pod Identity role and secret path (a preview must not be able to name a prod secret), egress allowlist, no auto-deploy for fork PRs without a maintainer label, and TTL cleanup on merge/close plus a hard expiry.

Entry criteria: Phase 0 exit criteria met.

Exit criteria: The pilot service runs in prod on the hub cluster for two weeks with error rate and latency at or better than its legacy baseline; its legacy infrastructure is decommissioned; a second team can onboard a service using only the written paved-road doc (validated by actually doing it).

Rollback posture: Legacy environment kept warm; cutover is a DNS weight.

Phase 2 — Internal APIs

Migrate Player API, Prism, and Offer API (order by ascending traffic). These prove Pod Identity cross-account access against RDS/SQS/DynamoDB at real volume and exercise HPA under production load. Laravel Horizon workers (per the Prism pattern) move as in-cluster Deployments alongside their services.

Entry criteria: Phase 1 exit criteria met, including the second-team onboarding validation.

This is also the first point with enough tenants to prove the staged rollout of shared changes: push one base-chart or cluster-policy change through the wave sequence (platform namespaces, then one low-risk tenant, then the rest) rather than cluster-wide. The mechanism needs exercising while a bad shared change is still survivable, not first during Phase 3.

Exit criteria: All three APIs serving 100 percent of production traffic from the hub cluster for two weeks; Beanstalk/EC2 environments for them decommissioned; at least one canary release with automated Datadog analysis has run in anger; at least one shared change delivered in waves with a tenant deliberately held a version behind.

Rollback posture: Per-service DNS weight back to legacy; two-week window.

Phase 3 — Consumer-facing surfaces

Migrate the offerwall and AdGem API — highest traffic, spiky, revenue-bearing. Precondition each cutover on a load test at 2x observed peak against the hub cluster, and release via Argo Rollouts canary (1 → 5 → 25 → 100 percent, gated on Datadog error-rate and latency analysis).

The load test has to validate pod scheduling headroom, not just request throughput. Separate pod networking reserves each node's primary interface for the node address, which lowers pods-per-node; at offerwall scale that constrains scale-out before CPU does. A test that reaches 2x peak on warm capacity proves less than one that forces Auto Mode to add nodes under load.

"Ran a load test" is not a gate, so the pass condition is explicit, using the Phase 0 capacity definitions:

• the service holds its existing p99 latency and error-rate SLOs at 2x peak;
• node scale-out from the pre-test node count to steady state completes within 5 minutes of the demand appearing;
• pending pods are zero outside an active scale-out, and clear within those 5 minutes;
• routable, pod-CIDR, and pod-slot utilisation all stay below 85 percent at peak.

Any one of those failing blocks the cutover rather than becoming a caveat in the retro.

If we are banking on consolidation to absorb the Auto Mode management fee, the lever is Graviton (an in-house cluster came out ~29 percent cheaper with more RAM after moving). That needs arm64 images, which is per-service build work and belongs in each service's migration rather than discovered at the end.

Entry criteria: Phase 2 exit criteria met; load-test harness exists.

Exit criteria: Offerwall and AdGem API at 100 percent on k8s through at least one observed traffic peak; autoscaling observed handling the peak without manual intervention. Legacy fleets are decommissioned after the two-week soak, not as part of reaching 100 percent — guiding rule 3 means the rollback target has to outlive the cutover, so "at 100 percent" and "legacy gone" are two weeks apart, not the same milestone.

Rollback posture: Canary gates auto-rollback in minutes; Cloudflare weighted routing to legacy stays wired for the soak window.

Phase 4 — Retire the legacy data cluster

Phase 0.5 already gave us a blueprint-conforming Auto Mode spoke. What remains is moving Datahub and Metabase off the hand-built ADR-0007 cluster and deleting it. The target is that existing spoke (or a sibling stamped from the same blueprint, if the data tools warrant their own), which means this phase is a workload migration rather than a cluster build — a meaningfully smaller job than the original plan assumed, and the main reason Phase 0.5 is worth doing early.

Their state lives in managed services (RDS, S3), so the move is a redeploy rather than a data migration. IRSA gives way to Pod Identity in the process, completing the ADR's supersession of 0007's identity choice.

These are live services, so normal cutover rules apply here even though Phase 0.5 got to skip them. The constraints that make this a migration rather than an upgrade:

• No in-place load balancer migration. Load balancers cannot transfer between the self-managed controller and Auto Mode, so cutover is blue-green behind a DNS shift. Already the plan's posture; now it is a constraint rather than a preference.
• Audit for PVCs before assuming redeploy. Auto Mode's EBS provisioner (ebs.csi.eks.amazonaws.com) cannot mount volumes created by the open-source driver (ebs.csi.aws.com). Anything holding a PVC needs PV/PVC recreation against a retained volume, so confirm per workload that state really is all in RDS/S3 rather than trusting the summary.
• SSO endpoints need the second ALB controller. Datahub and Metabase are the ALB-OIDC case Auto Mode's managed controller does not cover. The pattern is already proven on the Airbyte cluster, so this is applying a known solution rather than discovering one.

Discharge the ADR's DR obligations here, since this is the last phase with a disposable cluster lying around: restore at least one datastore into the spoke (not just re-sync manifests) and prove the ArgoCD bootstrap path, since ArgoCD cannot be the thing that recovers ArgoCD. Replaying git alone leaves the interesting half untested. If the Phase 0.5 re-stamp happened, that already exercised blueprint rebuild; what is still unproven is restore and bootstrap.

Entry criteria: Blueprint stable — no breaking blueprint changes for a full month.

Exit criteria: Datahub and Metabase running on a blueprint-stamped spoke managed by the hub ArgoCD; GitHub-Actions-push deploys to it retired.

Deleting the old cluster is the last step and is conditional, not simultaneous. It goes only after all of: the two-week soak passes, a datastore restore into the spoke succeeds with a data-integrity check (not just "the pod started"), the ArgoCD bootstrap path is exercised from nothing, and the measured restore time is recorded against the RPO/RTO the services declared. A drill that is merely documented does not count; the old cluster is the fallback that makes the drill safe to attempt, so it outlives the drill rather than the reverse.

Rollback posture: Old cluster kept until the new one passes a two-week soak, matching guiding rule 3. The earlier draft said one week; there's no reason for the data cluster to get a shorter reversibility window than a consumer service, and it is the cheapest cluster to leave running.

Phase 5 — Close out

• Decommission remaining Beanstalk applications and the deploy tooling that served them (ADR-0036's Beanstalk section becomes historical).
• Audit legacy VPCs (10.0/16, 10.20/16, 172.31/16) for stragglers; plan their retirement separately.
• Put the DR drill on a schedule rather than leaving it as a Phase 4 artifact: a periodic sandbox rebuild-plus-restore, so the recovery path stays proven instead of decaying into a document.
• Record the regional posture honestly in the runbook: serving is single-region, images and secrets replicate to a secondary region, and there is no tested regional failover. A region loss is an extended outage, not a failover. Cross-region data replication and a drilled promotion path are follow-on work, and should be filed as such rather than left implied by "rebuild from blueprint."
• Write the retrospective ADR amendments: mark superseded sections of ADR-0007 and ADR-0036, record any trigger-table hits (Transit Gateway, service mesh) that occurred during the migration, and swap this document's link to the target-architecture ADR from the PR URL to the merged doc path once its number is assigned.

Exit criteria: No production application traffic served outside k8s or the serverless exception policy; platform runbook and on-call rotation in steady state.

What does not migrate

Per the ADR's serverless exception policy: Keystone (the postback processor) stays on Lambda; DynamoDB/RDS/ElastiCache stay managed; future event-driven glue may ship straight to Lambda without debate. Anything matching the exception criteria lists itself in the service onboarding doc rather than migrating by default.

Measuring success

We already committed to DORA metrics (ADR-0003). The migration should move them, per migrated service: deployment frequency up, lead time down, change failure rate flat-or-down (canaries), MTTR down (git revert + auto-rollback). Baseline each service's DORA numbers before its migration phase; publish the before/after in the phase retro. Platform cost per service is tracked via Datadog Cloud Cost Management from Phase 1 onward.

On cost specifically, track the EKS Auto Mode management fee as its own line rather than folding it into compute. It is billed per second (one-minute minimum) at a rate that varies by instance type, separately from both the EC2 instance price and the EKS control-plane fee. It is not discounted by Spot, Reserved Instances, or Savings Plans, so the cheaper we make the underlying instance, the larger a share of that instance's cost the fee becomes. Reporting it separately is what makes the consolidation-and-Graviton argument checkable instead of a claim, and the per-instance-type rates should come off the EKS pricing page for our regions rather than from a remembered percentage.

Sequencing overview

Phase 0.5 branches off Phase 0 rather than queueing behind Phase 1, because its timing is set by Airbyte's go-live rather than by our readiness. It is the one phase in this plan with an external clock.

Phases 3 and 4 can still overlap: consumer cutovers and the data-tools migration touch different failure domains.