Authenticating...
Skip to main content

0070: Team-Scoped APM Tagging for Laravel Services

STATUS​

Accepted

Scope: This ADR sits under shared/ intentionally. Team tagging is one convention applied across services owned by teams spanning both AdAction and AdGem (offerwall/player, dashboard, offer, and postback services), not a single-product decision.

CONTEXT​

The Team Overview dashboard is wired to query APM spans by a team: tag so each engineering team can see the latency, error rate, and request volume of the work they own. Today it can't: the tag is effectively missing from our span data.

A group_by: team aggregation across the last four hours returns spans for only one team, and even those are incidental:

aggregate_spans  *  group_by: team  (last 4h)
→  team=platform : 135,239 spans
   (no other teams)

All of those team:platform spans come from the postgresql integration's queries, inherited from the Postgres service config. None of our Laravel services are deliberately stamping a team tag. The five other product teams (player-experience, publisher-integrations, advertiser-integrations, campaign-delivery, cosmic-rewards) have zero APM spans carrying team:.

Why this gap exists​

Datadog's Service Catalog stores team association as metadata, but that metadata does not propagate to span tags. The Datadog Unified Service Tagging spec covers env, service, and version, but not team. To get team: on a span we have to apply it at the tracer level, either through the DD_TAGS environment variable or via runtime code on the root span.

Why this matters​

Without team-scoped APM data, the APM section of the Team Overview dashboard stays permanently empty for five of six teams. A few related capabilities are unaffected and worth distinguishing:

  • DORA per-team metrics (lead time, change failure rate, MTTR) work because they query the dedicated dora data source, which honors the team field on deployment events sent by CI.
  • SLOs work because their team association comes from the SLO definition itself.
  • Anything that runs through APM (resource latency, error rate by team, request rate by team, trace search filtered by team) requires the team: tag on spans. That's the gap this ADR addresses.

Considered Options​

  • [Chosen] Stamp team at the tracer level using two patterns picked per service: DD_TAGS env var for single-team services, and a Laravel route-group middleware for multi-team services
  • PHP 8 #[Team('x')] attribute on controllers, read by middleware
  • Set the tag manually in each controller's __construct
  • Use Datadog Agent tag-processing rules to derive team from resource_name
  • Status quo: no team tagging
  • Split each multi-team DD_SERVICE into per-team services (see Appendix A)

DECISION​

We will tag spans with team:<team> at the tracer level, using one of two mechanisms depending on whether the service serves a single team or several.

Service shapeMechanismLives in
Single-team (one repo, one team)DD_TAGS=team:<team> env varDeployment config (CDK / Compose / .env)
Multi-team (one repo serves several teams via routes)Laravel middleware that stamps team on the request's root span, applied via route groupsPHP source: middleware + routes/*.php

Multi-team services also set DD_TAGS=team:untagged as a fallback, so routes that miss the middleware surface as an explicit gap rather than being silently attributed to a real team. See Fallback for unwrapped routes for the trade-off.

Pattern A, Single-Team Service​

For services where one repo serves one team, set the tag in deployment config alongside the other Unified Service Tagging variables:

// CDK / ECS task definition
environment: {
  DD_SERVICE: 'player-api',
  DD_ENV: 'prod',
  DD_VERSION: process.env.GIT_SHA ?? 'dev',
  DD_TAGS: 'team:player-experience',  // ← the change
}

No PHP changes. Every span the tracer produces inherits team:player-experience.

Pattern B, Multi-Team Service​

For services where one repo serves several teams via different route groups, install a tiny middleware that writes team onto the root span based on a parameter:

// app/Http/Middleware/TagTeam.php
class TagTeam
{
    public function handle(Request $request, Closure $next, string $team)
    {
        if (function_exists('\\DDTrace\\root_span')) {
            $span = \DDTrace\root_span();
            if ($span !== null) {
                $span->meta['team'] = $team;
            }
        }
        return $next($request);
    }
}
// routes/api.php
Route::middleware('team:player-experience')->group(function () {
    Route::get('/offers', [OfferController::class, 'index']);
});

Route::middleware('team:campaign-delivery')->group(function () {
    Route::resource('campaigns', CampaignController::class);
});

The function_exists guard makes the middleware a no-op when the dd-trace PHP extension isn't loaded (local dev, CI), so tests don't need to fake out the tracer.

Why root-span tagging is sufficient​

The middleware tags the request's root (entry) span — not the child DB, HTTP, and cache spans beneath it. That's deliberate. dd-trace-php doesn't propagate custom meta tags from parent to child, and forcing it to (a per-span start hook reading a request-scoped team) is fragile across tracer upgrades and uneven across integrations. It also isn't necessary:

  • The Team Overview dashboard's APM views — Top Endpoints, request/error/latency (RED) metrics, trace search — read the service entry span, which is exactly what the middleware tags.
  • Team-scoped queries should count entry spans, not every span. One request emits one entry span plus many child spans, so a naive group_by: team over * conflates requests with internal operations. Scope the query to the entry span — filter on the request operation (confirm the exact name against your spans, typically laravel.request or web.request) — and the count reflects requests.
  • Tagging shared-infra child spans (a Postgres query, a cache read) with the requesting team would muddy infra-level views without adding signal the dashboard uses.

For the rare case that genuinely needs per-child attribution across a whole trace — "how much DB time does team X consume" — Datadog Trace Queries can filter on a tag present on any span in the trace, so the root-span tag is enough to scope the trace without duplicating it onto every child. If that stops being enough, the DD_SERVICE split in Appendix A is the escalation path.

Fallback for unwrapped routes​

Multi-team services set DD_TAGS=team:untagged as a process-wide fallback. Any route that misses a team: middleware group surfaces as team:untagged rather than being silently folded into a real team. We chose a sentinel over a real default team deliberately:

FallbackWhat happens to an unwrapped route
Real team (e.g. team:campaign-delivery)Silently miscategorized. The dashboard looks correct but isn't, and the only way to find the gap is a manual weekly audit of which teams are missing.
Sentinel team:untagged (chosen)The gap becomes first-class data. A monitor — spans with team:untagged exceeds N/min — gives continuous rollout health, and a reader who sees team:untagged in a breakdown knows immediately what it means. The CI lint becomes a backstop rather than the only line of defense.

Service Triage​

ServiceOwner(s)Pattern
player-apiplayer-experienceA
offer-apipublisher-integrationsA
targeted-apipublisher-integrationsA
partners-postback-endpointsadvertiser-integrationsA
offer-interceptorcampaign-deliveryA
apimixed (player-experience, campaign-delivery, publisher-integrations)B
new_dashboardmixedB

Alternatives Considered​

OptionVerdict
PHP 8 #[Team('x')] attribute on controllers, read by middlewareNicer for greenfield code; more boilerplate to retrofit. Reconsider for new services.
Controller __construct sets the tag manuallyEasy to forget on new controllers, clutters every class. Rejected.
Datadog Agent tag-processing rules (regex on resource_name)Zero code change, but brittle and hard to keep in sync with route refactors. No compile-time check. Rejected as the primary mechanism.
Status quo (no tagging)Rejected. Leaves the Team Overview dashboard permanently empty and blocks team-scoped APM observability indefinitely.
Split into separate DD_SERVICE values per team (a.k.a. service split)Heavier refactor; deferred. See Appendix A for the full rationale and the signals that would trigger a revisit.

Rollout​

  1. Land this ADR.

  2. Sample PRs, one Pattern A and one Pattern B, tracked in the team-tag-propagation implementation plan.

  3. Each team owns rollout for the services they own.

  4. Weekly progress check via aggregate_spans group_by team, scoped to entry spans (see Why root-span tagging is sufficient). The goal is all six teams represented.

  5. After two weeks of green, add a CI lint that fails when a route in a multi-team repo lacks a team: middleware.

    This is a natural fit for a shared GitHub Action rather than a copy-pasted per-repo script. A small composite (or JavaScript) action published in a central repo — e.g. AdAction/github-actions/team-tag-lint — would load the route table (php artisan route:list --json), assert every route resolves to a team: middleware alias, and check each team value against the canonical team list. Multi-team repos consume it in one workflow step:

    - uses: AdAction/github-actions/team-tag-lint@v1

    The rule then lives in one place and updates propagate by bumping the version tag, rather than drifting across repos. Tracked as a follow-on, not in scope here.

CONSEQUENCES​

Positive​

  • The Team Overview dashboard's APM section populates for all teams.
  • Team-scoped APM queries become available org-wide: resource latency by team, error rate by team, request rate by team, trace search filtered by team.
  • Foundation for future per-team SLOs on APM-derived signals.
  • No PHP change for the roughly five single-team services. The two multi-team services get a small, reviewable middleware diff.

Negative​

  • Coordination overhead: roughly five PRs across team-owned repos for Pattern A, plus two PRs for the Pattern B middleware install.
  • Ongoing discipline is required to wrap new multi-team routes in the team middleware. Mitigated by the proposed CI lint, which fails when a route in a multi-team repo lacks a team: middleware.
  • Pattern B adds a tiny per-request overhead, one method call plus one meta assignment. Negligible.

Neutral​

  • The team tag is layered on top of the existing DD_SERVICE. No existing dashboards, monitors, or SLOs scoped to service: break.
  • DORA aggregation "across teams within a shared service" stays unchanged. Per-team DORA at the service level remains a future concern, covered in Appendix A.

Risks​

  • New multi-team routes ship without the middleware, leaving a slice of traffic unattributed. Mitigated by the team:untagged sentinel fallback, which makes the gap explicit and alertable rather than silently misattributing it to a real team, and by the proposed CI lint.
  • Child spans aren't team-tagged. Pattern B tags the entry span only, so team-scoped queries must target entry spans (see Why root-span tagging is sufficient); aggregations run naively over * will undercount a team's child DB/HTTP spans. Accepted: the dashboard reads entry spans, and trace-level attribution is available via Datadog Trace Queries.
  • Middleware parameter typos (e.g. team:plyer-experience) create silent miscategorization. Mitigated by keeping the canonical team list in one place and proposing the same CI lint check the value against it.
  • Local dev divergence: the function_exists guard means local environments never set the tag, which is intentional but means tag-related bugs only show up in deployed environments. Acceptable for now.

NOTES​

References​

Original Author​

Ron White

Approval date​

Approved by​

Appendix​

Appendix A, Future Direction: DD_SERVICE Split​

The team: tag answers "who owns this request" as a span attribute. A heavier alternative would be to make multi-team services report as several distinct APM services, one per team. api would become api-player-experience, api-campaign-delivery, and so on.

What it gains​

  • Each team gets a first-class APM service entry: its own Service Catalog tile, its own SLOs, its own DORA deployment events, its own service map, its own error-tracking grouping.
  • Cleaner oncall routing.
  • Team-scoped dashboards become trivial. Filter by service: instead of doing template-variable gymnastics on a team: tag.
  • DORA deployment frequency works correctly per-team because the underlying deploy events are keyed on service.

What it costs​

  • The "all of api together" operational view fragments. Cross-app incidents (DB exhaustion, FPM saturation) require summing across services.
  • Cross-team request tracing inside a single HTTP request becomes cross-service noise.
  • Service Catalog inflates from roughly 40 to 50+ services.
  • Span-creation-time service binding in some PHP integrations may not pick up per-request overrides. A clean split may require N deployments behind path-based load balancer rules.
  • Existing dashboards, monitors, and SLOs scoped to service:api need rewriting.

Implementation paths if we revisit​

  1. Per-request service override in middleware: \DDTrace\root_span()->service = 'api-player-experience'. Lowest infra change. Child spans created before middleware runs may carry the original service.
  2. Multiple deployments with distinct DD_SERVICE env vars, requests routed via path-based ALB rules. Cleanest separation. More deployments to manage.

Why we're deferring​

Team tagging is reversible and stackable. Once teams are tagged via route groups, the same route-group mechanism can later set service instead of (or in addition to) team. We get the dashboard working immediately, prove out the operational value, and only take on the heavier refactor if we hit concrete limitations.

Signals that would trigger a revisit​

  • Teams consistently report that "filter by team tag in APM" doesn't give them what they need.
  • We want per-team SLOs, per-team deployment tracking, or per-team oncall routing that the tag-based approach can't cleanly support.
  • DORA aggregated-by-repo numbers become misleading enough to leadership that per-team service-level DORA matters.