Skip to content

0036: Optimal CI/CD Pipeline

STATUS: Approved

CONTEXT

To ensure high-quality, secure, and reliable software delivery, we require a robust CI/CD pipeline that automates code validation, testing, release, and deployment. The pipeline must support both manual and automated (Dependabot) contributions, enforce best practices (e.g., secrets scanning, PR title linting), and enable safe, automated deployments to both application and infrastructure environments. The goal is to minimize manual intervention, reduce risk, and accelerate feedback for engineers.

Considered Options

Option 1: Jenkins

Pros:

  • Highly customizable and extensible with a vast plugin ecosystem.
  • Can be self-hosted, providing full control over the environment and security.
  • Supports complex workflows and integrations with almost any tool.
  • Large community and extensive documentation.

Cons:

  • Requires ongoing maintenance, upgrades, and security patching.
  • UI/UX is less modern compared to managed solutions.
  • Scaling and high availability require additional infrastructure effort.
  • Plugin compatibility issues can arise during upgrades.

Option 2: AWS CodePipeline

Pros:

  • Fully managed service with seamless integration into AWS ecosystem.
  • Scales automatically with minimal operational overhead.
  • Pay-as-you-go pricing model.
  • Native integrations with AWS services (e.g., CodeBuild, CodeDeploy, Lambda).

Cons:

  • Limited flexibility compared to Jenkins or GitHub Actions.
  • Less support for non-AWS tools and on-premises resources.
  • UI and pipeline configuration can be less intuitive.
  • Vendor lock-in to AWS.

Option 3: GitHub Actions

Pros:

  • Native integration with GitHub repositories and workflows.
  • Easy to set up and use with YAML-based configuration.
  • Large marketplace of reusable actions.
  • Supports matrix builds and parallel jobs.
  • Managed infrastructure with minimal maintenance.

Cons:

  • Limited to GitHub-hosted repositories.
  • Resource limits and concurrency restrictions on free/standard plans.
  • Debugging can be more challenging compared to local runners.
  • Some advanced enterprise features require paid plans.

NOTES

  • The pipeline integrates GitHub Actions for automation.
  • Pre-commit hooks and secrets scanning are enforced before code is committed.
  • All pull requests trigger builds and tests, with PR title linting for consistency.
  • Semantic release automates versioning and changelog generation.
  • Releases trigger builds/tests for both application and infrastructure code.
  • Application artifacts are uploaded to an S3 bucket and deployed to AWS Elastic Beanstalk.
  • Infrastructure changes are deployed via AWS CDK.
  • Dependabot PRs for minor/patch versions can be auto-merged.

Slack Notifications for Failed Workflows

To ensure rapid response to CI/CD issues and keep teams informed, Slack notifications should be configured for failed workflows and deployments. This implementation supports the pipeline's goal of minimizing manual intervention while maintaining high visibility into system health.

Setup Options:

  1. GitHub App for Slack Integration:
  2. Install the GitHub app for Slack in your workspace
  3. Subscribe to repository workflows using: /github subscribe owner/repo workflows

  4. Configure filters to reduce notification noise: /github subscribe owner/repo workflows:{name:"workflow_name" event:"workflow_event" branch:"branch_name"}

  5. GitHub Actions with Slack Notifications:

  6. Add notification steps to workflows using actions like ravsamhq/notify-slack-action@v2.5.0
  7. Configure notifications to trigger only on failures using if: failure() conditions
  8. Store Slack webhook URLs as repository secrets for security

Implementation Example:

notify-on-failure:
  runs-on: ubuntu-latest
  needs: [build, test, deploy]
  if: failure()
  steps:
    - name: Send Slack Notification
      uses: ravsamhq/notify-slack-action@v2.5.0
      with:
        status: ${{ job.status }}
        notification_title: "{workflow} has failed"
        message_format: "{emoji} *{workflow}* failed in <{repo_url}|{repo}>"
        footer: "Linked Repo <{repo_url}|{repo}> | <{workflow_url}|View Workflow>"
      env:
        SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Benefits:

  • Immediate notification of deployment failures enables rapid response
  • Reduces time to detection and resolution of CI/CD issues
  • Improves team awareness of system health and deployment status
  • Supports the pipeline's reliability and automation goals

DECISION

  • GitHub Actions

GitHub Actions was chosen because it integrates directly with our repositories and developer workflows, reducing context switching and maintenance overhead. Jenkins and AWS CodePipeline require separate dashboards and more upkeep, while GitHub Actions keeps everything in one place for developers.

CONSEQUENCES

  • Removal of all Jenkins Pipelines.
  • Removal of all AWS CodePipelines.
  • Migration of any old Pipelines to GitHub Actions pipelines in the repositories that they deal with.

Risks

  • There's risk of broken/missing functionality when migrating old pipelines.

References

Original Author

  • Dakota Washok

Approval date

Approved by

Appendix