Datadog Cloud Cost Management (CCM) Runbook

Operational reference for AdAction/AdGem cloud cost data in Datadog: how it flows, how allocation tagging works, and how to troubleshoot it.

Owner: Platform team Last validated: 2026-06-10 (PLA-125)

Data flow

AWS payer account 172122050326 (consolidated billing, all AdAction/AdGem accounts)
  └─ Cost and Usage Report "DDCostReport"
       └─ s3://adaction-datadog-172122050326/cur-report-dir (us-east-2)
            └─ Datadog CCM ingestion (config id 2977, 15-month window)
                 └─ cost metrics: aws.cost.amortized, aws.cost.unblended, ...

• The CUR is generated in the payer account, so member accounts are covered automatically — no per-account setup is needed for cost data.
• Cost data lags ~24–48h behind spend (normal CUR behavior). The current day is always partial.
• Check ingestion health: pup api /api/v2/cost/aws_cur_config — status should be active.

Cost allocation tagging

Three layers have to line up for a tag to become a usable cost dimension:

1. Resource tags in IaC. Every CDK app applies the mandatory tag set at the app level: Environment, Product, Service, Repository, ManagedBy, Teams. Rolled out 2026-04 across: conversion-automation, singular-export, incident-bot, aws-governance-cdk, mcp-server, cpa_platform_aggregator, appsflyer-cost-api, aws-airflow, corebiz-infrastructure, adaction-docs-infra, partners_postback_endpoints.

2. Cost allocation tag activation in the payer account's billing settings. A resource tag only appears in the CUR (and therefore in Datadog) after its key is activated. Active keys as of 2026-06-10: the six mandatory keys above plus AdAction:CostCenter and Project. A CUR backfill from 2026-01-01 was requested 2026-06-10 so historical months gain the new columns.

   # Check current state
   aws ce list-cost-allocation-tags --status Active --profile adgem --region us-east-2
   # Activate a new key (takes ~24h to appear in the CUR)
   aws ce update-cost-allocation-tags-status --profile adgem --region us-east-2 \
     --cost-allocation-tags-status TagKey=<Key>,Status=Active

3. Datadog tag pipelines (Cloud Cost → Tag Pipelines, ruleset "Key Alignment") normalize CUR tag keys to the org-standard Datadog tags:

   Source keys (CUR)	Destination	Applied when
   teams, owner	team	destination missing
   servicename	service	destination missing
   kube_app_name, application	product	destination missing

   This is why dashboards filter cost by team: even though the AWS tag key is Teams.

Where to look at cost

• CCM Explorer: Datadog → Cloud Cost → Explorer (aws.cost.amortized, group by team, service, aws_product, aws_account).
• Team Overview dashboard (ubi-zkn-2c4): per-team cost section driven by the $team template variable.
• For finance-facing numbers, report UnblendedCost from AWS Cost Explorer in the payer account; CCM amortized figures include savings-plan amortization and will not match invoices.

Adding a new repo or account

• New repo/CDK app: apply the six mandatory tags at the app level (Tags.of(app).add(...)). No Datadog-side change needed — keys are already activated and piped.
• New AWS account: joins the CUR automatically via consolidated billing. Verify its resources carry the mandatory tags.
• New tag key: activate it in billing (step 2 above), then add a tag pipeline rule if the key needs aliasing to a standard Datadog tag.

Troubleshooting

Symptom	Likely cause / fix
No cost data at all	CUR config error: pup api /api/v2/cost/aws_cur_config → check status / error_messages.
Tag missing as a cost dimension	Key not activated in billing (aws ce list-cost-allocation-tags), or activated <48h ago, or resources untagged.
Tag active but dashboard empty	Dashboard filters the Datadog-standard key (e.g. team:) — check the tag pipeline rule for the source key exists and is enabled: pup api /api/v2/tags/enrichment.
Historical months missing tag columns	Request a backfill: aws ce start-cost-allocation-tag-backfill --backfill-from <ISO date> (one request at a time; takes days).
Costs don't match the AWS invoice	Amortized vs unblended. Use UnblendedCost for invoice reconciliation.

Quick freshness check:

pup api -X POST /api/v2/query/timeseries --input - << 'EOF'
{"data":{"type":"timeseries_request","attributes":{
  "formulas":[{"formula":"query1"}],
  "queries":[{"data_source":"cloud_cost","name":"query1","query":"sum:aws.cost.amortized{*} by {team}"}],
  "from":<epoch_ms_7d_ago>,"to":<epoch_ms_now>,"interval":86400000}}}
EOF

Known gaps / follow-ups

• No Datadog cost monitors (anomaly alerts / budgets) exist yet — tracked separately.
• Stray tag keys in billing data need cleanup at the source: Team (vs Teams), Environment (trailing space), environment (lowercase), AcAction:CostCenter (typo).
• Container-level (EKS workload) cost allocation has not been evaluated.