0073: Config-as-Data Reference Tables — Versioned CSV Loaded into Redshift on Merge

STATUS

Accepted

Documents a pattern that is already in production. The first instance — the conversion auto-approval rule engine — shipped between 2026-07-31 and 2026-08-05 and now drives production auto-approval decisions. This ADR is written as-built, and the observed behaviour in the Consequences section is measured rather than projected.

CONTEXT

The Problem

A growing class of our operational logic is not really logic at all — it is config: exclusion lists, allow lists, thresholds, rule tables. Small, human-edited, frequently changed, and read by a dbt model or a DAG. Today it lives one of three ways, none of them good:

Where it lives	How it changes	Problem
Inline in SQL	Edit the model, deploy dbt	No provenance for which value matched; unreadable by anything but that one model; every ops change is an engineering change
A dbt seed CSV	Edit the CSV, wait for the next DAG run	Reloads on every DAG run; load is not atomic; a partial load is silent
A hand-written Liquibase INSERT	New changeset per change	Too heavy for a list that changes weekly

Reference tables are proliferating rather than shrinking. dbt-adgem/seeds/ alone holds adgem_geo, apps_to_check, banned_exemptions, banned_model_rules, and data_incidents; dbt-adaction/seeds/ holds turbo_goals_raw. Each one has re-invented its own answers to the same four questions: who owns the table, who can read it, when does it load, and what happens when the data is wrong.

The immediate driver was the conversion auto-approval rule engine (Linear PEX-299). The auto-approval heuristic was a single WHERE clause carrying, at the time, 63 excluded values across six dimensions — offer IDs, AdGem app IDs, a publisher ID, device brands, device models, OS versions. Restructuring it into named, data-driven rules required those 63 values to become a table. The question of how that table gets its contents is the subject of this ADR, and the answer generalizes.

Exclusionary Config Fails Open

This is the load-bearing constraint, and it drives every part of the design below.

Most data problems fail loudly. A missing table throws; an empty fact table produces an obviously empty report. Config that expresses an exclusion inverts this. A table that lists what must not be auto-approved has the property that an empty table excludes nothing. A partial load doesn't error — it quietly widens approval and looks exactly like a clean run.

We have already paid for this, twice over.

Incident 203

Sub-goals gate high-payout turbo goals, so they must never be auto-approved — they have to go through manual fraud review. The original data-driven implementation excluded at the goal level by referencing the seeded turbo_goals table.

The seeded table sometimes came back empty or not complete. With an empty exclusion list, a batch of sub-goals was auto-approved. Contemporaneous theories were a race condition, or the seeder running against adaction_analysis rather than adgem_analysis (CAM-343). Row-count instability in that table was suspected across more than one incident (CAM-361).

The resolution (airflow-dags #1143) was to abandon goal-level exclusion and revert to offer-level exclusion — a blunter instrument that stops auto-approving every conversion on a turbo offer, not just the sub-goals. That over-blocking drove a large sustained volume increase for the Fraud Manual Review team (CAM-374).

The cost of an unreliable load, then, was not a one-off bad batch. It was the permanent loss of the precise exclusion we wanted, traded for a coarse one we could trust.

The Same Table, Operationally

Independently of correctness, reloading config on every DAG run is wasted work with a recurring risk window. dbt seed on turbo_goals_raw pushed the Turbo Conversion Creator DAG past its 12-minute duration monitor and paged the team repeatedly (CAM-472); a --debug flag had to be added to the seed command just to see what it was doing.

The Generalizable Lesson

For exclusionary config, silent under-load is worse than a hard failure. Any mechanism we adopt must therefore:

1. Load atomically — a failed load leaves the previous contents intact, never an empty table.
2. Validate before writing — a malformed value fails in CI, not at INSERT, and never silently.
3. Detect shrinkage — a load that drops most of the list turns red, rather than quietly widening approval.

Decision Drivers

• Fail closed. Per the above — the dominant constraint.
• Reviewable and auditable changes. A config change should be a git diff on a pull request, not a manual UPDATE against production.
• Load on merge, not per run. Config becomes live when it is merged. Reloading on every DAG run is both wasted work and a repeated opportunity to fail.
• Deterministic table ownership and grants. In Redshift, whoever creates a table owns it, and a table created by the wrong principal is unreadable downstream. This is the standing hazard described in ADR-0032.
• Swappable behind a stable contract. The consuming model's interface — for auto-approval, an autoapprove boolean plus the rules that blocked each conversion — must not encode where the rules are stored.

Considered Options

Option A: Status Quo — Config Inline in SQL

Keep the values in the model's WHERE clause.

Pros:

• No new infrastructure, no new credentials, no ownership question
• Changes ship through the existing dbt deploy path, fully reviewed
• Cannot fail open — there is no separate artifact to fail to load

Cons:

• No provenance: the model emits a single boolean, so there is no way to know which value matched
• The list is invisible to every other consumer and to reporting
• Every ops-driven change is an engineering change to a SQL file
• Does not scale past one consumer

Option B: dbt seed

Keep the CSV in a dbt project's seeds/ directory and load it with dbt seed, either from the consuming DAG or from CI.

Pros:

• Already the house pattern in both dbt projects — six seeds in use
• Grants come free from the project post-hooks plus the on-run-end schema USAGE hook
• Self-provisioning: local and sandbox dbt runs get the table with no extra step
• The seed CSV lives next to the model that reads it

Cons:

• Seeds have demonstrably loaded empty in production — this is Incident 203, and the affected data was exclusionary
• No pre-write validation: a bad value lands in the table and is discovered downstream, if at all
• Run from the DAG, it reloads on every run, with the duration and paging cost above
• Run from CI instead, the ci dbt target's user does not own the prod schemas, so it needs a new prod-capable dbt profile and new secrets — the largest lift of the four options

Option C: Liquibase runOnChange + loadData

Add a changeset to the database's changelog in data-warehouse that deletes and re-loads the table from the CSV, marked runOnChange so it re-runs whenever the file's checksum changes.

Pros:

• Content-addressed: any CSV edit re-runs the changeset, and a revert restores the prior checksum, so no spurious re-runs
• Changesets are transactional, so delete + loadData is all-or-nothing
• Inherits the entire existing migration apparatus for free — stage-then-prod promotion, the rollback workflow, per-database deploy concurrency, 1Password credentials, the JDBC driver
• Table DDL, grants, and data all live in one tool, with one owner
• One multi-row INSERT rather than per-row statements

Cons:

• No pre-write validation. Liquibase will happily insert an unrecognized risk tier or an over-long value. This is precisely the guarantee Incident 203 says matters most
• The shared migration workflow gates on the changelog path only, so a CSV-only edit does not trigger a deploy and the config never loads. Fixing that means adding an input to a workflow shared by all six databases
• Config changes are coupled to the migration pipeline, so a routine list edit inherits migration-grade ceremony and blast radius

Option D: Standalone Python Loader Run by a CI Job

Keep the CSV in the consuming repo. A dependency-light Python module parses and validates it, then performs an atomic replace against the target table. A reusable workflow runs it on merge to main. Table DDL and grants remain Liquibase's responsibility.

Pros:

• Validation runs before any database connection, so a malformed CSV fails in CI and in unit tests
• Full control of transaction semantics — an atomic replace that cannot leave the table empty
• The CSV lives beside the code that reads it, and a config change is a one-line reviewed diff
• Loads exactly once per merge; no per-run cost
• Runnable locally against a sandbox schema with no Airflow dependency

Cons:

• A second path from CI into Redshift, with its own credentials to manage
• The table is not dbt-managed, so local and sandbox dbt runs need the loader run once by hand
• DDL is expressed in two places — the authoritative Liquibase changeset and a sandbox-only CREATE TABLE IF NOT EXISTS — and can drift
• More code to own than the three alternatives

Comparison

Dimension	A: Inline SQL	B: dbt seed	C: Liquibase loadData	D: Python loader in CI
Reviewable diff	Yes	Yes	Yes	Yes
Provenance (which value matched)	No	Yes	Yes	Yes
Load timing	Deploy	Every DAG run (or CI)	Migration deploy	Every merge to main
Atomic load	N/A	No	Yes	Yes
Pre-write validation	N/A	No	No	Yes
Shrinkage detection	N/A	No	No	Yes (row-count floor in CI)
Ownership + grants	N/A	dbt hooks	Liquibase	Liquibase
Stage → prod promotion	Existing	Partial	Free	Manual
Rollback path	Existing	Re-run	Free	Revert + re-run, or manual dispatch
New credentials	None	Yes (CI variant)	None	None (reuses existing)
Self-provisions sandbox	N/A	Yes	No	Partially
Effort	None	High (CI variant)	Low	Medium

DECISION

Adopt Option D: config-as-data reference tables are stored as a versioned CSV in the consuming repository and loaded into a Liquibase-owned Redshift table by a reusable CI workflow on every merge to main.

The pattern has four parts. Each is stated as a general rule, followed by how the conversion auto-approval rules realize it — the first instance, now running in production.

1. The CSV in Git Is the Source of Truth

The file lives in the repository that consumes it, beside the code that reads it. A config change is a one-line diff on a reviewed pull request — the same review path as any code change, with the same history and the same blame.

Instance: lib/conversion_review/rules/conversion_autoapproval_rules.csv in airflow-dags, columns dimension, value, risk_tier, active, comment. Seeded with the 63 values migrated verbatim from the inline exclusion lists in conversions_to_autoapprove.sql; 68 rows as of 2026-08-05. The active column lets ops disable a rule without deleting it, preserving its history.

Keep the CSV dumb — pure value lists plus a named category the consuming SQL interprets. It is not a generic operator-driven DSL. Thresholds, patterns, and compound conditions stay in SQL, which is better at them and is unit-testable.

The division has held under change. When network publishers needed exclusion by app ID with any per-network suffix stripped, the fix was a new dimension (adgem_app_id_base, matching the whole segment before the first underscore rather than a prefix) — a new named attribute the SQL knows how to compute, not an operator or a pattern in the CSV. That is the seam working as intended: the CSV gained rows, the SQL gained a split_part, and neither learned anything about the other.

2. DDL and Grants Belong to Liquibase, Never to the Loader

The table is created, altered, and granted by a Liquibase changeset in data-warehouse. The loader writes rows and nothing else. This keeps ownership deterministic: the table is owned by the admin principal Liquibase runs as, not by whichever principal happened to load it first.

A consumer needs both USAGE on the schema and SELECT on the table. A table-level grant without schema USAGE silently grants nothing usable, and the consumer fails with permission denied for schema. This is the recurring trap described in ADR-0032; existing loaders in airflow-dags carry explicit tasks granting both because it has bitten before.

The two live at different lifecycles, and conflating them is what causes the trap:

• Schema USAGE is granted once, when the schema is created — not per table. Alongside it, an ALTER DEFAULT PRIVILEGES ... GRANT SELECT ON TABLES on the schema makes future tables readable by default. Because default privileges are scoped to objects created by the user who set them, they are a backstop rather than a guarantee.
• Table SELECT is granted per table, in the changeset that creates it.

So the rule for a new reference table is: if the schema is already granted, the table changeset only needs the table grant. If the table is the first thing in a brand-new schema, the changeset must do both, and a table-only grant will look correct and read as permission denied for schema.

Ownership is the third thing to hold separately from both, and it is easy to misread. The schema owner implicitly has USAGE and CREATE on it and needs no grant at all — so an identity can be absent from every grant in the changelog and still have full access, while another identity holding an explicit table grant is blocked because it lacks schema USAGE. Read ownership before concluding anything from the grant list.

Instance: schema USAGE on conversion_review for the reader, reporter, and transformer roles — plus ALTER DEFAULT PRIVILEGES for SELECT on tables — came from changeset 7 in migrations/changelog-adaction_analysis.yaml, which created the schema for dbt models long before this work. The rules table therefore needed table grants only. The schema itself is owned by airflow, which is why the pipeline principal appears in no USAGE grant yet reaches the schema freely; the table is owned by admin, the principal Liquibase runs as. Both confirmed against pg_namespace / pg_class.

Ordering rule: the Liquibase changeset merges and deploys first, always. If the loader runs against a schema where the table does not yet exist, it creates the table itself and the table gets the wrong owner — exactly the failure this split is designed to prevent.

The loader may carry a CREATE TABLE IF NOT EXISTS as a non-production convenience only, so it works against a sandbox schema. In production it is a no-op because the table already exists.

Instance: data-warehouse #517 added changeset 16 to migrations/changelog-adaction_analysis.yaml, creating conversion_review.conversion_autoapproval_rules and granting SELECT on it to the reader, reporter, and transformer roles plus SELECT, INSERT, DELETE to airflow — table grants only, since the schema was already granted. It merged at 16:33 UTC on 2026-07-31; the loader merged at 17:14 the same day — the required ordering, in that order. Grants persist across the loader's DELETE + INSERT, so they are a one-time concern rather than something the loader re-applies.

3. Load on Every Merge to main, Ungated, via a Reusable Workflow

The loader runs unconditionally on every push to main. It is deliberately not gated on the CSV path having changed.

The reason is a property of the merge strategy in force when this shipped: airflow-dags was rebase-only at the time. A rebase merge replays every commit of the branch onto main in a single push, so a path gate keyed on the last commit — or on a shallow diff — sees only that final commit. Any config pull request where the CSV edit is not the last commit, which is to say any pull request that took a round of review feedback, would skip the load and leave the config unchanged with a green build. That is Incident 203's failure mode reconstructed in CI, and re-run-on-failure does not recover it because nothing failed.

A correct gate was expressible even then — diffing the full push range with unshallowed history — but for a load that is idempotent and measured in dozens of rows, the gate bought nothing and could only introduce a way to silently not run. So it was dropped.

That constraint has since lifted. airflow-dags is now squash-only (allow_squash_merge: true, rebase and merge-commit both disabled). A squash merge lands the entire pull request as one commit, so a path gate keyed on the last commit sees the whole change and the silent-skip failure mode disappears. Gating the load on the CSV path is now straightforward, and is a reasonable future improvement — see the follow-ups.

That improvement would make the merge strategy load-bearing, which is worth anticipating rather than discovering. Today the load is unconditional, so it is safe under either strategy; the moment a path gate exists, switching the repository back to rebase-only silently reintroduces the skip, with no failing build to reveal it. A repository setting is not somewhere anyone thinks to look for downstream consequences, so whoever adds the gate should also leave a note in the repository README recording that the gate depends on squash-only merges — so that anyone proposing to change the setting sees the implication before making the change, rather than after.

The general rule to carry forward is not "never gate" but match the gate to the merge strategy, and prefer running unconditionally when you cannot make the gate provably correct. For an exclusionary table, a gate that can silently skip is worse than no gate at all.

The load lives in its own reusable workflow, not inlined in the deploy workflow. Three properties follow, and all three matter enough to make this part of the pattern rather than a stylistic choice:

• The deploy workflow calls it in two lines, matching how it delegates each of the six database migrations. The second table to adopt the pattern is a few lines of configuration, not a copy of the whole job.
• workflow_dispatch gives a real "Run workflow" button. Because the loader is idempotent, this is a legitimate manual force-reload and recovery path — after a transient failure, or to reload the table without a code change. That is a materially better recovery story than re-running a deploy.
• It is not in the deploy job's needs. The load runs in parallel with the database migrations and neither gates the MWAA deploy nor is gated by it. Verified in production: the deploy run at 17:24 UTC on 2026-08-05 failed overall while its rules load succeeded independently.

Instance: .github/workflows/load-autoapproval-rules.yml, invoked by deploy.yml via workflow_call with secrets: inherit, reusing the existing 1Password Redshift credentials and mirroring the established setup steps from the database-deploy and Python workflows.

4. The Write Is Atomic, and Validated Before It Happens

Four mechanisms. The first three answer the three requirements from the Context; the fourth catches what CI structurally cannot.

Atomic replace. Delete and re-insert inside a single transaction, then commit. Notably, not TRUNCATE: Redshift's TRUNCATE forces an implicit commit, so it cannot participate in a transaction — a TRUNCATE followed by a failed INSERT leaves the table empty, which for exclusionary config is the worst possible outcome. DELETE is transactional, so DELETE + INSERT + COMMIT is all-or-nothing and a failed load preserves the previous contents.

Any DDL the loader does run happens first and separately, since DDL also implicitly commits in Redshift.

Pre-write validation. Parsing and validation happen before any database connection is opened, so every check is exercised by unit tests in CI on the pull request:

• the column set matches exactly;
• categorical columns are members of their allowed sets;
• boolean columns parse as booleans;
• key columns are non-empty, and are rejected outright if they carry surrounding whitespace — a value that looks correct in a diff but never matches anything is worse than one that is obviously wrong;
• the natural key is unique across rows;
• field widths are checked in UTF-8 bytes against the declared VARCHAR(n) widths. Redshift measures VARCHAR limits in bytes, not characters, so a byte-length check catches an over-long value — including a short string of multi-byte characters — in CI rather than at INSERT.

Shrinkage detection. A test asserts a row-count floor, not an exact count. An exact count turns CI red every time someone legitimately adds an entry, which trains people to update the number without reading it. A floor is the direct Incident 203 countermeasure: it stays green when the list grows by one and fails when a bad merge drops most of it. An empty-file guard alone only catches the degenerate case; the failure that actually hurt us was a partial load. The floor has already absorbed five rule additions without a single spurious edit.

Post-write assertions on the table itself. The pre-write checks live in the loading repository and protect against a bad CSV. Declaring the table as a dbt source() with not_null and accepted_values tests on the categorical columns protects against everything else — a hand-edited row, a partially applied load, a Liquibase change that widened a column. Cheap, and it closes the gap between "the CSV was valid when CI ran" and "the table is valid now".

No swallowed exceptions in the load path. A caught-and-printed failure produces a green job and a broken table, which is the whole failure class this pattern exists to eliminate. Anything genuinely non-fatal should surface as a CI warning annotation rather than a log line in a job nobody reads.

Flow

Rationale

Config that decides what not to approve must fail closed. Of the four options, only the standalone loader places validation before the write while keeping table ownership and grants in the tool that already manages them.

Option C is the close runner-up and is genuinely attractive — it inherits promotion, rollback, and credentials for free, and if the data in question were not exclusionary it would likely win on effort alone. It loses here on the one axis this decision turns on: it has no way to reject bad data before it lands. Given that our concrete failure history is a silently wrong exclusion list rather than a botched deploy, pre-write validation is worth more to us than a free rollback path.

Option B is disqualified rather than merely outscored. It is the mechanism that produced Incident 203, on data of exactly this kind.

When to Use This Pattern

Use it when the data is small, human-edited, reviewed, read as a dbt source(), and — most of all — when a partial load would be silent. That covers exclusionary config, which fails open by approving what it should have held, and equally a positive list, whose missing rows mean work quietly not done. In both cases an incomplete load looks exactly like a clean one.

Do not use it when:

• the data is large or machine-generated — that is a pipeline, not config;
• rows carry system-versioned history, where validity is derived from when a load happened rather than declared in the file — see below;
• an application writes it at runtime — the CSV cannot be the source of truth for something the system itself mutates;
• it is genuinely a single value read by a single model — a dbt variable is simpler.

Effective Dating and History

"This needs history" usually means one of three things, and only the last is genuinely incompatible with a full-replace load. Worth separating them before reaching for a slowly-changing-dimension table.

Declared effective dates — supported, just add the columns. If a rule should apply from one date to another, valid_from / valid_to are ordinary data columns in the CSV, subject to the same pre-write validation as everything else, and the consuming SQL filters on them. Full replace does not destroy anything, because the dates are declared in the file rather than inferred from load history — replacing every row rewrites the same dates. Nothing about the pattern has to change. If you want scheduled exclusions, do this.

An audit trail of who changed what, when, and why — you already have it, in git. Every change is a reviewed commit with an author, a timestamp, a diff, and a pull request. That is a better audit record than a database table, which typically captures only the value and the load time. The gap is that it is not queryable from SQL. If you need point-in-time answers in the warehouse — "which rules were live when this conversion came in?" — append each load's rows to a separate history table (loaded_at plus the row) and leave the live table a clean full replace. That keeps the loader idempotent and adds one insert. Gate that append on the file actually having changed, or an unconditional load appends an identical snapshot on every unrelated deploy.

System-versioned history — this is the one that breaks it. If valid_from / valid_to must be derived from when the loader observed a change, the load stops being a replace and becomes a merge: diff against current state, close out rows that changed, insert new versions. That costs the properties this pattern is built on — the load is no longer idempotent, the atomic DELETE + INSERT no longer applies, the row-count floor stops meaning anything, and "just re-run the workflow" stops being a safe recovery. At that point the data has outgrown being config, and wants a real dimension table with its own model.

CONSEQUENCES

Observed in Production

Measured on the first instance, over the first five days of operation:

Property	Observed
Loader job outcome	10 of 10 successful across every deploy from 2026-08-04 to 2026-08-05
Load duration	45s–315s, median ≈95s
Effect on deploy time	None — runs in parallel with the six database migrations, and is not in the deploy job's needs
Independence	A deploy that failed overall still loaded the rules successfully
Config changes absorbed	5 rule additions plus 1 new dimension since adoption (63 → 68 rows)
Table contents vs CSV	Exact match — 68 rows, 68 distinct (dimension, value) keys, all active, 7 dimensions

Queried directly against the warehouse: the live table holds exactly the committed CSV, with no duplicate keys and no rows the file does not declare. That is the property the whole design exists to guarantee, and it holds after a week of unconditional reloads on every merge.

On permissions, reader and reporter hold USAGE on conversion_review and SELECT on the table, both confirmed in svv_schema_privileges / svv_relation_privileges. Grants held by other identities are not visible from a non-superuser connection, so they are taken from the changelog rather than observed.

Positive

• Config changes are reviewed diffs with full git history, rather than manual production UPDATEs.
• Bad config is rejected on the pull request, before it can reach Redshift.
• The failure mode for exclusionary config becomes closed rather than open: a failed load preserves the prior contents, and a shrunken list fails CI.
• Loads happen once per merge instead of once per DAG run, removing both the recurring risk window and the per-run duration cost — and, as measured above, without extending deploy time.
• The storage layer is swappable. Consumers depend on the table's shape, not on how it is filled.
• Consumers other than the original model — reporting, ML feature work, ad-hoc analysis — can read the config as data, which was impossible when it was a WHERE clause. The engine also emits which rules blocked each conversion, which the hardcoded clause never could.
• There is now a reusable workflow, not just a pattern, so the second adopter is nearly free.

Negative / Accepted Costs

• A second path from CI into Redshift exists, with credentials and error handling to own.
• The table is not dbt-managed, so local and sandbox dbt runs need the loader run once by hand against the dev schema. dbt seed provided this for free.
• DDL is expressed three times: the authoritative Liquibase changeset, the loader's sandbox-only CREATE TABLE IF NOT EXISTS, and the width constants used for validation. Three places that must agree, and nothing structurally binds them. This is a maintenance cost rather than a hazard, because every way they can diverge fails closed and says so: a widened column makes validation stricter than the table, so valid values are rejected in CI naming the exact constraint; a narrowed one fails at INSERT — Redshift raises rather than truncating — leaving the previous contents intact via the atomic replace; and the loader's CREATE TABLE is a no-op wherever the table already exists, so it can only affect a fresh sandbox schema.
• Ownership must be established deliberately by Liquibase rather than falling out of the tooling, which makes merge ordering between two repositories load-bearing.
• Every-push loading is cheap per table but scales linearly with adoption.

Risks

• Grant regression. A future Liquibase change that recreates the table takes ownership and drops its grants, leaving downstream consumers unable to read it. This is the standing ADR-0032 hazard; any changeset touching the table must re-grant explicitly.
• Cross-repo deploy ordering. The loader merging before its Liquibase changeset produces a wrong-owner table. The rule is documented above but is not enforced by tooling.
• Silent staleness — open. If the CI job stops running — a runner outage, a rotated credential, a workflow refactor that drops the job — the table simply keeps its last contents. A failing job is visible; a job that no longer runs is not. No freshness check exists on the table today, and adding one is the clearest remaining gap in the pattern.
• Out-of-band writes. Nothing prevents a manual UPDATE against the table. The CSV is the source of truth by convention only, until the next merge overwrites the drift — which means a manual production fix silently disappears on the next unrelated merge. The post-write source tests catch malformed manual edits, not well-formed wrong ones.
• Validation is only as good as its rules. Categorical and width checks catch malformed data, not wrong data. A valid entry excluding the wrong offer passes every check. Code review remains the only control on semantic correctness.
• Adoption gradient. The pattern is still more work than adding a dbt seed. The reusable workflow narrows the gap considerably, but until an existing seed is actually migrated onto it, the easy path and the right path are not yet the same path.

NOTES

References

• Linear PEX-299 — restructure auto-approval heuristics into a named rule engine; the first adopter
• CAM-343 — Incident 203, sub-goals auto-approved via an empty seeded table
• CAM-374 — Incident 203 resolution and the cost of reverting to offer-level exclusion
• CAM-361 — seeded table row-count instability across incidents
• CAM-472 — per-run dbt seed duration and paging
• airflow-dags #1143 — the goal-level to offer-level exclusion revert
• airflow-dags #2303 — the loader and CI workflow implementing this pattern (merged 2026-07-31)
• airflow-dags #2304 — the rules CSV and dbt source declaration
• airflow-dags #2345 — cutover; the rule engine drives production approvals
• data-warehouse #517 — the Liquibase changeset creating and granting the rules table (merged 2026-07-31, ahead of the loader)
• ADR-0032: Redshift Permissions Management Tooling
• Liquibase usage guide — which repository owns which migrations
• PR #198: docs(ADR): config-as-data reference tables loaded into Redshift on merge

Original Author

Emma Worthington

Approval date

Approved by

Appendix

Follow-Ups

None is required to adopt the pattern; all three reduce its long-run cost.

1. Add a freshness check on loaded reference tables. The one open risk with no mitigation. A job that silently stops running leaves stale config in place indefinitely, and nothing currently notices.
2. Re-introduce a path gate, now that merges are squash-only. The unconditional load was a response to rebase-only merges (Decision §3), and that constraint no longer applies. With one commit per merge, a path gate on the CSV is reliable, and would stop every unrelated deploy from reloading the table. Worth doing only with the squash-only setting treated as load-bearing — if the repository ever allows rebase merges again, the gate must be dropped or made push-range-correct in the same change. Ship it with a note in the repository README stating that dependency, so the implication reaches whoever proposes the settings change while they can still act on it.
3. Migrate turbo_goals_raw off dbt seed. It is the table Incident 203 broke on, and it still seeds on every DAG run. It is not exclusionary — it is a positive list of turbo goals and their sub-goals that drives conversion creation — so an incomplete load fails quietly in the other direction: conversions that should have been created simply are not, on that run and on every run until someone notices. It is also the table people read during incident response, where a list that is sometimes short is worse than no list, because it looks authoritative. Same silent-under-load problem, and the clearest second adopter: it would retire the paging noise from CAM-472 and prove that adopting the pattern is as cheap as the reusable workflow suggests.

Retired Follow-Up

Extracting the loader job into a reusable workflow_call file — proposed when this pattern was first written, and done in the initial implementation. deploy.yml now delegates in two lines, and workflow_dispatch provides the manual reload path. Recorded here because the reusability it bought is now part of the pattern (Decision §3) rather than an aspiration.