0069: Dedicated, auto-scaling worker tier for Offer API background jobs

STATUS

Accepted

CONTEXT

The Offer API web tier currently does two unrelated jobs on the same instances: it serves live v1/offers traffic, and it runs heavy background processing (queue workers and the scheduler). Because they share CPU, any sufficiently heavy background workload competes directly with request handling. There is no architectural separation between "serving the API" and "doing batch work," so background load and customer latency are coupled.

This is not hypothetical: the production web environment (offer-api-green-production) dips to Warning health every 15 minutes, and is expected to reach Degraded under heavier load. We localized it to the cache-warming workload. Cache warming (AGPI-1814) is a valuable feature, not a fault: it proactively recomputes v1/offers responses so real requests hit a warm cache, and it has materially improved latency for the apps it covers. The issue is purely where it runs. Because its queue workers live on the web instances, each cycle saturates their CPU (70-87% on both 2-vCPU t4g.medium instances) for roughly 90 seconds, and that briefly degrades the customer-facing path.

We localized the cause with four independent signals that all land on the :00 / :15 / :30 / :45 marks and nothing in between:

Mark (UTC)   EB health        Peak CPU (both instances)   warm-cache logs
13:30        Warning (15)     74 / 82 %                   job burst
13:45        Warning (15)     71 / 85 %                   job burst
14:00        Warning (15)     68 / 82 %                   job burst
14:15        Warning (15)     71 / 82 %                   job burst
14:30        Warning (15)     67 / 86 %                   dispatch + job burst
14:45        Warning (15)     67 / 87 %                   dispatch 14:45:02, jobs to 14:46:50
:05/:10/:20  OK (0)           10 to 30 % baseline         none

The every-5-minute stats job and every-minute event-sync consumer produce no spike, which rules out the other scheduled work. It is not errors: zero 5xx spans appeared in APM over the window. It is pure CPU contention from background work sharing the web instances.

For reference, how warming runs today (AGPI-1814): the scheduler dispatches offers:warm-cache every 15 minutes (onOneServer); the command pushes one WarmOfferCacheJob per tracked app onto the cache-warming Redis queue; each job recomputes every tracked filter combination for that app with forceRefresh (full DB query plus serialize); and the Horizon cache-warming supervisor that drains the queue runs on the web instances themselves.

The coupling will tighten, not loosen, which is why it matters now:

• Cache-warming load grows with the Prism ramp. Prism (formerly Targeted API) reads v1/offers heavily, and warming work scales with each app's per-country and per-sort filter diversity, not request volume. The largest single app already has ~432 legitimate filter combinations; totals climb as the ramp completes (~93 tracked apps today). This is healthy growth of a feature we want to keep.
• Event-sync is the next heavy workload. Offer API must continuously consume EventBridge/SQS events; that load grows with the same ramp and, like warming, should not have to contend with request handling.

The underlying need is a durable home for heavy background work, separate from the request path, that can absorb and scale with demand. Cache warming is simply the first workload to surface the limit; solving it generically also covers event-sync and whatever heavy job comes next, rather than fighting each one in place.

Considered Options

• 1. Dedicated worker tier under the existing offer-api EB application (same code artifact, same Redis/Postgres, no public traffic). - DECISION
• 2. Scale the web tier up or out to absorb the spike.
• 3. Throttle/chunk the warming dispatch only, leaving it on the web tier.
• 4. Trim the number of warmed filter combinations.
• 5. Move everything onto EB's formal SQS worker tier.
• 6. Extract the background jobs into a dedicated worker codebase/service (separate repo), rather than sharing the Offer API code.
• 7. Run event-sync on Lambda (PHP via Bref), leaving warming where it is.

DECISION

Stand up a dedicated Elastic Beanstalk worker tier under the existing offer-api application. It deploys the same code artifact and shares the same Redis and Postgres, but its job is background processing, not serving HTTP. The web tier goes back to only serving traffic. This is not a new application and requires no v1/offers behavior change.

Two things could in principle be "shared," and they are separate decisions: the codebase/artifact (one Laravel app deployed to both tiers) and the runtime/instances (the same boxes doing both jobs). The contention we are fixing comes entirely from sharing the runtime. This ADR breaks that sharing while deliberately keeping the codebase shared, which is the standard Laravel/Horizon pattern and the right fit here (see "Why the alternatives were not chosen" for why a separate worker codebase was rejected).

                      EB Application: offer-api
       +------------------------------+-----------------------------+
       |   WEB tier                   |   WORKER tier (new)         |
       |   offer-api-green-production  |   offer-api-worker-prod     |
       |                              |                             |
       |  * ALB, serves v1/offers     |  * no public traffic        |
       |  * queue worker for          |  * runs the scheduler       |
       |    user-facing queues only   |  * runs heavy job workers   |
       |  * runs NO heavy jobs        |  * AUTO-SCALES on load      |
       +--------------+---------------+--------------+--------------+
                      |                              |
                      +--------- same Redis ---------+
                      +--------- same Postgres ------+

What moves to the worker tier

• The cache-warming Redis queue (the Horizon cache-warming supervisor) is consumed only by the worker tier. This is the critical change: the recompute work leaves the web instances.
• The scheduler (artisan-scheduler.service, schedule:work) runs only on the worker tier.
• Over time, other heavy background jobs move here too (see Candidate workloads).

What the web tier keeps: serving v1/offers, plus a queue worker scoped to user-facing queues only (no longer cache-warming).

What stays the same: same application, repository, and build artifact (the deploy pipeline is extended to ship that one artifact to both environments, see Infrastructure and CI/CD); same Redis cluster (load-bearing, since the worker writes warmed entries and the web tier reads them); same Postgres; no change to forceRefresh or which apps are warmed. Prism keeps reading the same cache.

Candidate workloads

The worker tier is worth building once and reusing. Background jobs in the current schedule, by load shape:

Job	Cadence	Load shape	Move?
offers:warm-cache	every 15 min	bursty CPU	yes, first
eventsync:consume	every minute	sustained	yes, second
dd-stats:update-active_offer_count	every 5 min	light	optional
dd-stats:update-invalid_...percentage	every 3 hours	light	optional
offers:archive / offers:prune	twice daily	heavy, off-peak	optional

Warming and event-sync are the two that matter, and they have opposite load shapes, which drives the auto-scaling design.

Auto-scaling

Auto-scaling is a first-class requirement, but the right signal differs by workload:

• Reactive CPU auto-scaling does not catch the warming burst. The spike is ~90 seconds; CloudWatch breach + cooldown + instance boot takes several minutes, so a new instance arrives after the spike is already over. CPU-only scaling adds cost without helping.
• Event-sync (sustained): runs as a systemd-supervised daemon (eventsync-consumer.service, Restart=always) per PUB-158, not a Horizon queue worker and not a backlog-scaled pool. Because SQS delivers each message to exactly one consumer, scaling is simply how many worker instances run the unit, and correctness holds at any count. Default to Min=Max=1 (matching today's single-runner semantics); adding instances is a later scale-out lever if consume throughput becomes the bottleneck. Backlog-driven autoscaling (on SQS age-of-oldest-message) is possible but an optimization, not a prerequisite.
• Cache warming (bursty, scheduled): likely a combination of (a) flattening the burst via chunked/throttled dispatch so a steady fleet can track it, and (b) right-sizing plus higher worker concurrency with a scheduled minimum capacity around the known 15-minute cadence rather than purely reactive scaling.

Because the two workloads pull in opposite directions, a single shared scaling policy is hard to tune. Two options to weigh: one worker environment with multiple queues scaled on the dominant signal (simpler, harder to tune) vs. separate worker environments per workload class (cleaner scaling, slightly more infra).

How each tier knows its role

Both environments deploy the identical artifact. Nothing in application logic branches on "am I web or worker." Role is decided entirely by deployment configuration, driven by a single per-environment role flag (an env var, or simply a distinct Horizon environment name, which Horizon already uses to select its supervisor set). That flag controls four things:

• Which queues run. Job placement is already queue-based: config/horizon.php defines the supervisor-1 (default), archiving, and cache-warming supervisors under an environments key, and jobs are dispatched to named queues. Today one Horizon master on the web instances runs all of them. Under this ADR, we add per-role supervisor sets (e.g. a web set with only supervisor-1 and a worker set with cache-warming, then archiving) and each environment selects its set by Horizon environment name. Two gotchas the implementation must respect: (1) select the supervisor set via Horizon's own environment override (horizon.env), not by repointing APP_ENV, which stays production because it is load-bearing for logging, error handling, and config caching; (2) both Horizon masters share one Redis, so give them distinct horizon.prefix values or their dashboards, metrics, and tags collide. No job code changes; only which supervisor set each environment boots.
• Whether the scheduler runs. The scheduler is schedule:work under the artisan-scheduler.service systemd unit, installed by the predeploy hook 91-configure-artisan-scheduler-service.sh. That hook becomes role-gated so the scheduler is enabled only on the worker tier. (The schedule already uses onOneServer, so this is about which tier owns it, not double-firing.) Whichever environment owns the scheduler must keep a minimum capacity of at least one instance: the scheduler is what dispatches warm-cache and the other periodic jobs, so if its environment ever scales to zero, nothing dispatches the next cycle (see open questions).
• Whether the event-sync consumer runs. eventsync:consume is a long-running SQS poller, not a Horizon queue worker, so it is supervised as its own systemd unit (eventsync-consumer.service, Restart=always) modeled on the existing horizon.service, enabled only on the worker tier via a role-gated predeploy hook (PUB-158). This replaces today's scheduler-driven invocation (onOneServer + withoutOverlapping + the --max-time=540 band-aid from PR #1340), which was a daemon shoehorned into cron and caused host memory exhaustion on the 2026-05-11 rollout. Because SQS delivers each message to exactly one consumer, this is correctness-safe whether one instance or several run the unit.
• Whether it serves traffic. "No public traffic" is an infrastructure fact, not an app toggle: the worker environment is simply not in public DNS and is locked down at the security group (see AWS networking and security groups). nginx and php-fpm can stay up to answer the EB health check (which keeps enhanced health meaningful); it just receives no customer requests.

Infrastructure and CI/CD

• IaC. offer-api provisions EB through CDK in provision-v2 (lib/offer-api-stack.ts). The worker tier is a second EB environment under the same application, defined in that stack: its own Auto Scaling group and scaling policy, no public ALB listener. It is reviewed and deployed through the existing deploy-iac.yml pipeline, not click-ops.
• Deploy pipeline. deploy-eb.yml currently ships the built artifact to the web environment. It must fan the same artifact out to both environments (a matrix or a second deploy target). One build, two eb deploys, so web and worker never drift in code version.
• Migrations. Migrations run from the predeploy hook 07-run-migrations.sh, which today runs migrate --force on every instance at predeploy. With two environments deploying the same artifact, this must run on exactly one tier (gate it to web) to avoid two environments racing the same migration during a deploy. Deploy ordering also matters: deploy the migrating tier (web) first, so the worker never comes up against a schema the web code has not yet applied, or vice versa. This is the same deploy-order hazard that bit the EventBridge routing change (consumer/rules before producer). The cleanest end state decouples migrations from the per-tier deploy hook entirely.
• Queue transport. Redis-queue jobs (warming, archiving) do not need EB's formal SQS worker tier. Event-sync already consumes SQS, so its scaling signal is naturally available; warming scales on Redis queue depth or scheduled capacity.
• Metrics for scaling. Confirm the auto-scaling metric source: whether SQS backlog and Redis queue depth are already in CloudWatch/Datadog, or whether we publish a custom metric to scale on.

AWS networking and security groups

Both environments run in the same VPC, so giving the worker tier access to Postgres and Redis is an intra-VPC security-group change, not a networking project. The relevant facts, from the CDK in provision-v2 (LaravelBeanstalkApplication):

• VPC and subnets. The stack does not create a VPC; it looks up the existing one (vpc-0c3f02232ce195227 in production) and places instances in its private subnets. RDS Postgres and the ElastiCache Redis cluster already live in those private subnets. The worker environment launches into the same VPC and private subnets, so no peering, new subnet group, or cross-VPC routing is involved.
• How the web tier reaches the datastores today. The persistence layer owns the datastore security groups: offer-api-green-production-database-sg (Postgres, 5432) and offer-api-green-production-cache-sg (Redis, 6379). The web environment has its own instance security group (offer-api-green-production-application-sg) and, at synth time, opens its own ingress by looking the datastore SGs up by name and adding a rule from the application SG (allowDefaultPortFrom on 5432, addIngressRule on 6379). So the EB instance SG is the principal the datastores trust, and the consuming environment opens that ingress itself.
• What the worker tier needs. The worker environment gets its own instance security group, which the datastores do not trust yet. Two options:
  • Reuse the web instance SG for the worker environment. No new ingress rules, since the datastores already trust it. Simplest, but the two tiers share a network identity, so there is no SG-level isolation and no separate "worker datastore connections" visibility.
  • Give the worker its own SG and add two ingress rules (worker SG to Postgres 5432, worker SG to Redis 6379). Cleaner isolation. Following the construct's existing pattern, the worker environment looks the persistence SGs up by name and opens ingress from its own SG, so the web stack never has to reference the worker SG (which would create a circular stack dependency).
  • Proposed: a dedicated worker SG. Per-tier isolation is the point of this ADR, and we want to read worker datastore load on its own. Recorded as an open question below.
• This is not the EB SQS "Worker" environment tier. Because we keep nginx/php-fpm up for the health check and we want auto-scaling, the worker is a load-balanced WebServer-tier environment with an internal-scheme ALB and no public DNS record, not a true EB Worker tier and not a single-instance (no-ALB) environment. "No public traffic" means not in public DNS and locked at the SG, not "no load balancer."
• Credentials and instance role. Because it deploys the same artifact, the worker reads the same DB secret, DB_HOST, and REDIS_HOST; its environment must be fed the same persistence endpoints and secret name the web env uses. Its instance role also needs the same grants this stack attaches today (the Cognito secret read plus cross-account assume, the internal-user secret CMK, the S3 archive bucket). Either share the web instance role or replicate those managed policies onto the worker role.
• CDK structuring. The current construct builds one EB application with one environment and the datastores baked in; it has no second-environment concept, so this is new CDK, not a flag. The cleanest shape is to add the worker environment in the same stack, consuming the persistence layer's outputs (RDS endpoint, cache endpoint, db-secret name, the two datastore SG names) and opening its own ingress. Same-stack keeps the migration-gating decision (above) in one place and avoids cross-stack exports.

The shared-datastore ceiling in the Risks section is the flip side of this: opening a second ingress path adds no Postgres or Redis capacity, it only lets the worker load the same instances harder once it scales. The isolation this ADR buys is CPU isolation on the app tier, not datastore isolation.

Why the alternatives were not chosen

• Scale the web tier up or out. Pays continuously for headroom to absorb a 90-second spike on the customer-facing path, and reactive scaling cannot catch a sub-minute burst anyway. Rejected.
• Throttle/chunk warming dispatch only. Useful, and likely part of the warming scaling story, but on its own it keeps the load on the web tier. Complement, not a substitute.
• Trim warmed combinations. The combinations are legitimate Prism per-country and per-sort coverage, and Prism reads every one. Excluding apps would cause cache misses on real reads. Rejected.
• EB SQS worker tier for everything. Would mean rewiring the Redis-queue warming jobs onto SQS for no real benefit. Event-sync already uses SQS and keeps its model; warming stays on Redis. Rejected as a blanket approach.
• Dedicated worker codebase/service (separate repo). Rejected. The jobs are not standalone scripts; they are built on the Offer API domain layer. WarmOfferCacheJob reuses the same query scopes, serializers, and caching logic as the v1/offers endpoint, and the event-sync consumer patches the same Eloquent models. A separate codebase would force either duplicating that domain logic (constant drift and double-maintenance) or having the worker call back into Offer API over an internal API (re-adding a network hop and coupling, and putting load back on the web tier). Laravel jobs are also serialized by class name and rehydrated by the worker, so the worker must run matching code anyway, which is why both tiers deploy the same artifact together. A separate codebase would increase coupling-management burden to solve a compute-isolation problem that a separate environment already solves. Code separation is the wrong tool for this problem; runtime separation is the right one.
• Run event-sync on Lambda (PHP via Bref). Event-sync already consumes SQS, so an SQS-triggered Lambda is a natural fit, and Bref runs the existing Laravel app on Lambda without rewriting the domain logic. Rejected as the solution here for three reasons. First, it only addresses event-sync; the workload degrading the web tier today is cache warming, which is Redis-queue and scheduled, not SQS, and does not map onto an SQS-triggered Lambda, nor do the scheduler and the other Horizon supervisors (archiving, warming itself). Second, it adds a second runtime and deploy path (a Bref layer plus per-function packaging) alongside the EB artifact, cutting against this ADR's "one build, two deploys, same artifact" property and doubling operational surface. Third, moving the consumer to per-message Lambda invocations is a larger paradigm change than the backlog-scaled consumer pool already planned, with its own cold-start, 15-minute-limit, and concurrency concerns. Lambda/Bref stays a reasonable thing to revisit specifically for event-sync later, but it is not a home for the heavy background work this ADR is scoped to isolate.

Phasing

Each phase is shippable on its own:

1. Stand up the worker tier and move cache warming; confirm web health stays green across the 15-minute marks and the v1/offers cache hit rate is unchanged.
2. Add auto-scaling for warming (chunked dispatch + scaling policy).
3. Move event-sync to the worker tier as a systemd-supervised daemon (PUB-158): add eventsync-consumer.service, drop eventsync:consume from the scheduler, run Min=Max=1. SQS serialization keeps this correctness-safe; no consumer-pool coordination needed.

Phase 1 success criteria: web environment health stays green across the :00/:15/:30/:45 marks, web CPU loses the periodic spikes, worker-tier health spikes are expected and fine, and web-tier cache hit rate is unchanged.

CONSEQUENCES

• Positive outcomes
  • Web-tier health stops flapping; the customer-facing v1/offers path no longer competes with background CPU.
  • Background capacity scales with the Prism ramp on a per-workload signal.
  • Reusable tier: warming, event-sync, and other heavy jobs share one piece of infrastructure.
  • No API behavior change and no cache-correctness change.
• Negative outcomes
  • A new environment to provision, deploy, and operate (CDK under provision-v2).
  • Added compute cost for the worker tier, partly offset if the web tier can scale down once heavy jobs leave it.
  • This isolates app-tier CPU contention only. It relocates background CPU off the web instances; it does not reduce the work. The jobs' Postgres queries and Redis reads/writes are unchanged and still hit the shared instances, so this does not address datastore contention. If Postgres or Redis is or becomes the bottleneck, this ADR does not solve that, and scaling the worker tier up would not help (see Risks).

Risks

• Moving event-sync to a systemd-supervised daemon (PUB-158) is low-risk and well-templated (it mirrors horizon.service); the behavior change to call out is that dropping onOneServer lets more than one instance run the consumer, which SQS makes correctness-safe, and keeping it at Min=Max=1 initially avoids even that. It should still land in its own phase after warming is proven, but it is no longer the highest-design-surface item.
• A poorly chosen auto-scaling signal (e.g. reactive CPU for the warming burst) would add cost without fixing the spike. The per-workload signal choice above must be honored.
• If warming dispatch is not flattened, the worker tier still sees a 90-second saturation burst; acceptable on an isolated tier, but it constrains how small the steady fleet can be.
• Shared datastores set a ceiling. Postgres and Redis stay shared between tiers. Moving the work to a worker, and especially scaling that worker up (more concurrency or instances), increases peak load on those same datastores. If either becomes the bottleneck, the worker tier cannot solve it and could make it worse; that would need a separate effort (e.g. read replica, Redis right-sizing, query/serialization optimization, or TTL-aware warming to cut work at the source).
• Redis is the binding constraint today, so its sizing is a prerequisite, not a follow-on. The 2026-06-25 capacity and readiness assessment (see References) finds Redis is already on a trajectory to overflow on its own, independent of any further app onboarding, and flags it as the top-priority item. Because warming with forceRefresh rewrites cache entries, raising worker concurrency fills Redis faster. Phase 2 (warming auto-scaling) must therefore be sequenced after a Redis right-sizing, or it accelerates the exact failure it is meant to avoid. Phase 1 (relocating warming at today's concurrency) does not raise Redis load and can proceed first.

NOTES

References

• AGPI-1814 (cache warming) and the Offer API cache-warming strategy.
• ADR 0057: Real-time Offer API syncing (EventSync) - the second workload this tier absorbs.
• Targeted API (Prism) ADRs 0010-0012, 0040 - the driver of warming load growth.
• Source proposal: "A dedicated, auto-scaling worker tier for Offer API background jobs" (Luiz Bueno, 2026-06-16).
• Capacity and readiness assessment (2026-06-25): production telemetry on offers, cache hit rate, and Redis growth ahead of app onboarding. Independently recommends moving warming to dedicated workers and identifies Redis as the top-priority (P0) constraint.
• PUB-158: Supervise eventsync:consume via systemd instead of the scheduler. The mechanism for running the consumer on the worker tier as a daemon; supersedes the cron + --max-time band-aid (PR #1340).

• PR #167: docs(adr): dedicated, auto-scaling worker tier for Offer API background jobs

Original Author

Luiz Bueno

Approval date

Approved by

Appendix

Key facts:

• Environment: offer-api-green-production, account 010438502987, us-east-2, t4g.medium (2 vCPU), two instances.
• Background processing today: Laravel Horizon (horizon.service systemd unit) runs all supervisors on the web instances; the scheduler runs as schedule:work (artisan-scheduler.service). Horizon supervisors in production: supervisor-1 (default, 10 procs), cache-warming (2 procs), archiving (1 proc).
• Schedule: offers:warm-cache every 15 min; eventsync:consume every minute; dd-stats jobs; offers:archive and offers:prune twice daily. All onOneServer.
• Warming queue: cache-warming (Redis), one WarmOfferCacheJob per tracked app, forceRefresh.
• Event-sync: EventBridge to SQS to the eventsync:consume consumer (single runner today).
• Scale today: ~93 tracked apps; largest single app ~432 filter combinations (per-country times sort coverage); total climbing as Prism ramps.
• Networking: VPC vpc-0c3f02232ce195227 (production, looked up, not created); instances and both datastores in its private subnets. Security groups: app instances offer-api-green-production-application-sg; Postgres offer-api-green-production-database-sg (5432); Redis offer-api-green-production-cache-sg (6379). The web environment opens its own ingress to the Postgres and cache SGs from the application SG at synth time.

Open questions

1. One shared worker environment vs. separate environments per workload class.
2. Auto-scaling signal sources: are SQS backlog and Redis queue depth already available in CloudWatch/Datadog, or do we publish a custom metric?
3. Event-sync consumer fleet size: default Min=Max=1 (a single systemd-supervised daemon, matching today's single-runner semantics) vs. allowing more than one instance to run eventsync-consumer.service and relying on SQS serialization for scale-out (PUB-158).
4. Instance type, min/max capacity, and scaling thresholds per workload.
5. Role-flag mechanism: a dedicated env var vs. reusing a distinct Horizon environment name to select supervisors.
6. Which tier owns migrations on deploy (proposed: web only), and how the pipeline gates it.
7. Whether the web tier can scale down once heavy jobs leave it, to offset worker-tier cost.
8. Whether shared Postgres/Redis have headroom for warming to scale into, or whether datastore scaling needs to be sequenced alongside this work.
9. Worker security group: a dedicated worker SG with its own ingress rules to the Postgres (5432) and Redis (6379) SGs (proposed) vs. reusing the web instance SG (no new ingress, but no per-tier isolation).
10. Worker instance role: share the web instance role vs. a separate role replicating the Cognito, internal-user-secret, and archive-bucket grants.
11. Worker ALB scheme: internal-scheme ALB with no public DNS (proposed) vs. another approach to "no public traffic."
12. Scheduler placement once it leaves the web tier: which worker environment owns schedule:work, and how we guarantee that environment never scales to zero (otherwise no cycle dispatches warm-cache or eventsync:consume). Most acute under separate-env-per-workload, where a backlog-scaled event-sync env could otherwise reach zero between bursts.

Future: how this maps to Kubernetes

Moving Offer API off Elastic Beanstalk onto Kubernetes (EKS) is a separate, longer-term goal, but it is worth noting that this decision survives that move unchanged. The principle here is "share the codebase/artifact, separate the runtime," and Kubernetes expresses that split more naturally than EB does, so this is not throwaway work.

• Tiers become Deployments. The web tier and the worker tier become separate Deployments (or the worker a separate Deployment per workload class) running the same container image. The web Deployment runs php-fpm/nginx behind a Service/Ingress; the worker Deployment runs Horizon (php artisan horizon) and the scheduler, with no Ingress. The role flag this ADR describes becomes the pod's command/args (which supervisors to boot) rather than an EB Horizon environment name. No application code changes, same as on EB.
• Auto-scaling gets better, not just equivalent. The "reactive CPU cannot catch a 90-second burst" problem is exactly what queue-depth scaling solves. KEDA scales worker pods directly on SQS backlog (event-sync) and on Redis queue depth (warming), which is the per-workload signal the Auto-scaling section asks for, without the CloudWatch breach plus instance-boot lag of EB ASG scaling. CPU-based HPA remains available for the web tier.
• Networking is the same idea, different primitive. RDS and ElastiCache would almost certainly stay in the same VPC (EKS already coexists there; the stack already trusts a datahub EKS SG, sg-048401008a846a8a2). Datastore access then comes from the worker pods' or nodes' security group being allowed ingress on the Postgres (5432) and Redis (6379) SGs, the same model as today, optionally tightened further with NetworkPolicy. Pod-level SGs (security groups for pods) can give the worker its own identity, mirroring the dedicated-worker-SG option above.
• IAM moves to IRSA. The per-tier instance-role grants (Cognito secret, internal-user secret CMK, S3 archive bucket) become an IAM role bound to the worker's ServiceAccount via IRSA, instead of an EB instance profile. Same permissions, scoped per workload.
• What does not change. Shared Postgres and Redis, the shared-datastore ceiling in Risks, the migration-gating decision (a one-shot Job or init container gated to a single owner rather than a per-tier predeploy hook), and the fact that this isolates compute, not datastores.

Sequencing note: doing this on EB now is the faster path to relief, because it reuses the existing VPC, the already-provisioned RDS and ElastiCache, the build artifact, the deploy pipeline, and Horizon; the worker is a second EB environment plus a role flag, not a platform migration. The work also carries over rather than being thrown away: the role flag, the per-role Horizon supervisor split, the scheduler and migration gating, and the backlog-scaling design all transfer directly to a future containerized worker, which is the natural first workload to migrate given it has no public traffic and no latency SLO.