Authenticating...
Skip to main content

0056: Batch Resend Failed Webhooks API

STATUS​

Accepted

CONTEXT​

When offer-converted webhooks fail to be delivered to publishers (due to network issues, publisher endpoint downtime, etc.), these failed webhooks are captured in the failed_webhooks table for later retry. Currently, resending failed webhooks requires running an Artisan command (webhooks:resend) via CLI access to the API servers.

This approach has several limitations:

  1. Only engineers with server access can trigger resends
  2. No visibility into failed webhooks without database queries
  3. No preview capability to see what would be resent
  4. Difficult to track progress of batch operations

We need a dashboard tool that allows admins to view, preview, and batch resend failed webhooks without requiring server access.

Considered Options​

  • Dashboard calls API endpoint (Queued) - Dashboard sends requests to API, which queues jobs for processing. Dashboard polls for status.
    • DECISION
  • Dashboard calls API endpoint (Synchronous) - Dashboard sends requests to API, which processes webhooks synchronously and returns results.
  • Dashboard executes command directly - Dashboard uses SSH or similar to execute the existing Artisan command.

DECISION​

We will create a new REST API in the api repository that exposes four endpoints for the dashboard to consume:

  1. List failed webhooks (GET /v1/admin/failed-webhooks) - Paginated, filterable list of failed webhooks
  2. Preview resend (POST /v1/admin/failed-webhooks/preview) - Shows what would be resent without actually sending (dry-run)
  3. Batch resend (POST /v1/admin/failed-webhooks/resend) - Queues webhooks for resending, returns a batch ID
  4. Get batch status (GET /v1/admin/failed-webhooks/resend/{batchId}/status) - Returns progress and results of a batch operation

Architecture​

Queued Processing​

We chose queued (asynchronous) processing over synchronous for the following reasons:

  1. No timeout constraints - Synchronous processing would be limited by HTTP request timeouts. With webhooks taking up to 20 seconds each, even 100 webhooks could timeout.
  2. Scalability - Large batches (thousands of webhooks) can be processed without blocking.
  3. Resilience - If a worker fails, unprocessed jobs remain in the queue for retry.
  4. Progress tracking - Laravel's Job Batching provides built-in progress visibility.

Batch Tracking with Laravel Job Batching​

We use Laravel's built-in Job Batching feature (Bus::batch()) for tracking batch operations. This requires adding the job_batches table migration to the common package.

Laravel's job_batches table provides:

  • total_jobs - Total number of jobs in the batch
  • pending_jobs - Jobs still waiting to be processed
  • failed_jobs - Number of failed jobs
  • failed_job_ids - IDs of failed jobs for inspection
  • finished_at - Timestamp when batch completed

This eliminates the need for a custom batch tracking table.

Authentication​

The API endpoints use internal API key authentication via the X-Internal-Api-Key header, consistent with other internal APIs (e.g., OfferApiClient). When Cognito is integrated, we will migrate to using that for authentication.

Admin Audit Trail​

Admin identity is primarily logged in the dashboard. Additionally, the resend endpoint accepts an optional initiated_by parameter:

{
  "app_id": 456,
  "initiated_by": "admin@example.com"
}

When provided, this value is stored in the batch metadata (in the job_batches.options column), making API logs self-contained for debugging without requiring cross-reference to dashboard logs.

The API does not validate or enforce this parameter - the dashboard is responsible for passing the authenticated user's email. This keeps the API simple while providing useful audit context in batch records.

Idempotency​

To prevent duplicate resends (e.g., admin triggers resend twice, or different admins resend the same webhook concurrently):

  1. Status-based filtering - The resend endpoint only selects webhooks with resend_pending status. When queued, status changes to resend_in_progress.
  2. Atomic status update - Use database transactions to atomically select and update status, preventing race conditions.
  3. Post-processing status - After processing, webhooks are marked manually_resolved or returned to resend_pending for retry.

This ensures each webhook can only be in one active batch at a time. Webhooks already in progress are excluded from the list endpoint and cannot be selected for a new batch.

Webhook Lifecycle​

Records in failed_webhooks are not deleted after resend - they are updated with a new status to preserve an audit trail:

OutcomeNew StatusBehavior
Successmanually_resolvedRecord retained for audit history
Failureresend_pendingCan be retried in a future batch

This matches the existing behavior in the webhooks:resend Artisan command and allows tracking of resend attempts, success rates, and failure patterns.

Other Options​

Synchronous processing was rejected because:

  • HTTP timeouts would limit batch sizes
  • Large batches would block and eventually timeout
  • No graceful handling of partial failures

Direct command execution was rejected because:

  • Requires SSH access or similar infrastructure
  • Difficult to track progress
  • No preview capability
  • Poor user experience

CONSEQUENCES​

Positive​

  1. Admins can resend failed webhooks without engineering involvement
  2. Full visibility into failed webhooks with filtering and search
  3. Preview capability prevents accidental resends
  4. Progress tracking for large batch operations via Laravel's Job Batching
  5. Reusable ResendWebhookService extracted from command
  6. No custom batch tracking table needed - uses Laravel's built-in job_batches

Risks​

  1. Queue backlog - Large batch resends could create queue pressure. Mitigated by using a dedicated queue or rate limiting if needed.
  2. Database load - Frequent polling of batch status could create load. Mitigated by reasonable polling intervals in the dashboard.
  3. Batch pruning - Laravel's batch records can be pruned. Ensure appropriate retention period is configured if historical data is needed.

NOTES​

Prerequisites​

  • Add job_batches table migration to common package (required for Laravel Job Batching)

Future Improvements​

  • Consider migrating from Redis to SQS for the job queue

References​

  • Parent ticket: AGPI-1284
  • Existing command: api/app/Console/Commands/ResendFailedWebhooks.php
  • FailedWebhook model: common/src/Models/FailedWebhook.php
  • Laravel Job Batching Documentation

Original Author​

  • Micah Wierenga

Approval date​

Approved by​

Appendix​

API Endpoint Details​

List Failed Webhooks​

GET /v1/admin/failed-webhooks

Query parameters: app_id, webhook_type, status, player_id, campaign_id, per_page, page, sort_by, sort_dir

Preview Resend​

POST /v1/admin/failed-webhooks/preview
Content-Type: application/json

{
  "webhook_ids": [123, 124],  // or
  "app_id": 456
}

Returns regenerated URLs/signatures without sending.

Batch Resend​

POST /v1/admin/failed-webhooks/resend
Content-Type: application/json

{
  "webhook_ids": [123, 124],  // or
  "app_id": 456
}

Returns batch_id for tracking.

Get Batch Status​

GET /v1/admin/failed-webhooks/resend/{batch_id}/status

Returns progress: total, processed, succeeded, failed, progress_percent, status.