TechJuiceBlog
#Tech#Web Development#Programming#DevOps

CI/CD Pipelines: Complete Guide

A comprehensive guide to CI/CD pipelines, covering tools, patterns, and best practices for automating deployments.

CI/CD Pipelines: The Complete Guide

Continuous Integration and Continuous Deployment (CI/CD) has become essential for modern software development. By automating the build, test, and deployment process, teams can ship code faster, more frequently, and with higher confidence.

This comprehensive guide covers everything you need to know about CI/CD pipelines, from basic concepts to advanced patterns.

What are CI and CD?

Continuous Integration (CI)

CI is the practice of merging all developer copies to a shared mainline several times a day. Each merge triggers an automated build and test sequence to detect integration errors as early as possible.

Key Principles:

  • Automate builds - No manual compilation steps
  • Run tests automatically - Every commit triggers test suite
  • Fast feedback - Developers know within minutes if build fails
  • Maintain clean repository - Always keep mainline deployable

Continuous Deployment (CD)

CD is the practice of having code deployed to production at all times, typically through automated pipelines.

Key Principles:

  • Automate deployment - Zero manual steps
  • One-click deploy - Deploy on approval or automatic
  • Rollback capability - Instant revert if issues occur
  • Environment parity - Dev/staging/production environments match

CI/CD Workflow

┌─────────────────────────────────────────┐
│          Developer Workflow          │
├─────────────────────────────────────────┤
│                                     │
│  1. Create Branch                  │
│     git checkout -b feature/new     │
│                                     │
│  2. Make Changes                   │
│     Edit code                      │
│     Write tests                    │
│                                     │
│  3. Commit Changes                  │
│     git add .                      │
│     git commit -m "Add feature"    │
│                                     │
│  4. Push to Remote                 │
│     git push origin feature/new      │
│                                     │
├───────────────┬─────────────────────┤
│              ▼                     │
│    ┌─────────────────┐              │
│    │  CI Pipeline    │              │
│    ├─────────────────┤              │
│    │                 │              │
│    │  ┌───────────┐  │              │
│    │  │ Build       │  │              │
│    │  │ - Compile   │  │              │
│    │  │ - Lint     │  │              │
│    │  │ - Test     │  │              │
│    │  │ - Package  │  │              │
│    │  └───────────┘  │              │
│    │                 │              │
│    │  ┌───────────┐  │              │
│    │  │ Deploy     │  │              │
│    │  │ - Staging  │  │              │
│    │  │ - Production│  │              │
│    │  └───────────┘  │              │
│    └─────────────────┘              │
│                                     │
├───────────────┴─────────────────────┤
│              ▼                     │
│    ┌─────────────────┐              │
│    │  Production     │              │
│    │  Application    │              │
│    └─────────────────┘              │
└─────────────────────────────────────────┘

Popular CI/CD Tools

GitHub Actions

Basic Workflow

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run linter
        run: npm run lint

      - name: Run tests
        run: npm test -- --coverage

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info

Multi-Stage Pipeline

# .github/workflows/ci-cd.yml
name: CI/CD

on:
  push:
    branches: [main]
  workflow_dispatch:  # Manual trigger

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test -- --coverage

      - name: Upload coverage
        uses: actions/upload-artifact@v3
        with:
          name: coverage
          path: coverage/

  build:
    name: Build
    needs: test  # Runs after test job succeeds
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Build
        run: npm run build

      - name: Upload build
        uses: actions/upload-artifact@v3
        with:
          name: build
          path: dist/

  deploy-staging:
    name: Deploy to Staging
    needs: build
    runs-on: ubuntu-latest
    environment: staging  # Requires approval

    steps:
      - uses: actions/checkout@v3

      - name: Download build
        uses: actions/download-artifact@v3
        with:
          name: build

      - name: Deploy to Vercel
        uses: amondnet/vercel-action@v20
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          working-directory: ./vercel

  deploy-production:
    name: Deploy to Production
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production

    steps:
      - uses: actions/checkout@v3

      - name: Download build
        uses: actions/download-artifact@v3
        with:
          name: build

      - name: Deploy to Vercel
        uses: amondnet/vercel-action@v20
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          vercel-args: '--prod'

GitLab CI/CD

# .gitlab-ci.yml
stages:
  - test
  - build
  - deploy

variables:
  NODE_VERSION: "18"
  CACHE_KEY: "$CI_COMMIT_REF_SLUG"

test:
  stage: test
  image: node:$NODE_VERSION
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  script:
    - npm ci
    - npm test -- --coverage
  coverage: '/Coverage: \d+\.\d+\%/'

build:
  stage: build
  image: node:$NODE_VERSION
  dependencies:
    - test  # Only runs if test passes
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

deploy_staging:
  stage: deploy
  image: node:$NODE_VERSION
  dependencies:
    - build
  only:
    - main
  when: manual  # Requires manual trigger
  script:
    - npm install -g vercel
    - vercel --token=$VERCEL_TOKEN --yes --prod=false

deploy_production:
  stage: deploy
  image: node:$NODE_VERSION
  dependencies:
    - build
    - deploy_staging
  only:
    - main
  when: manual
  script:
    - npm install -g vercel
    - vercel --token=$VERCEL_TOKEN --yes --prod

CircleCI

# .circleci/config.yml
version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:18.18
    steps:
      - checkout
      - restore_cache:
          keys:
            - node-deps-{{ checksum "package-lock.json" }}
      - run: npm ci
      - save_cache:
          keys:
            - node-deps-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - run: npm test -- --coverage
      - store_artifacts:
          path: coverage/

  build:
    docker:
      - image: cimg/node:18.18
    steps:
      - checkout
      - restore_cache:
          keys:
            - node-deps-{{ checksum "package-lock.json" }}
      - run: npm ci
      - run: npm run build
      - store_artifacts:
          path: dist/

  deploy:
    docker:
      - image: cimg/node:18.18
    steps:
      - checkout
      - attach_workspace:
          at: dist
      - run:
          name: Deploy to Heroku
          command: |
            git remote add heroku https://git.heroku.com/myapp.git
            git push heroku HEAD:main --force
workflows:
  version: 2
  test-build-deploy:
    jobs:
      - test
      - build
      - deploy

Build Strategies

Build Caching

# GitHub Actions with caching
steps:
  - name: Setup Node.js
    uses: actions/setup-node@v3
    with:
      node-version: '18'
      cache: 'npm'  # Caches node_modules

  - name: Get npm cache directory
    id: npm-cache-dir-path
    run: echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT

  - name: Cache node modules
    uses: actions/cache@v3
    id: npm-cache
    with:
      path: ${{ steps.npm-cache-dir-path.outputs.dir }}
      key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
      restore-keys: |
        ${{ runner.os }}-node-

  - name: Install dependencies
    run: npm ci

Parallel Jobs

# Run jobs in parallel
jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run linter
        run: npm run lint

  test-unit:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run unit tests
        run: npm test --unit

  test-integration:
    name: Integration Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run integration tests
        run: npm test --integration

  build:
    name: Build
    needs: [lint, test-unit, test-integration]  # Runs after all three jobs complete
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build
        run: npm run build

Matrix Builds

# Build and test across multiple environments
jobs:
  test:
    name: Test (${{ matrix.os }} - Node ${{ matrix.node-version }})
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node-version: [16, 18, 20]
      fail-fast: false  # Continue testing even if one fails

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Upload results
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: test-results-${{ matrix.os }}-node-${{ matrix.node-version }}
          path: test-results/

Deployment Strategies

Blue-Green Deployment

# Deploy to blue and green environments
deploy_blue:
  runs-on: ubuntu-latest
  environment:
    name: blue
    url: https://blue.example.com
  steps:
    - uses: actions/checkout@v3
    - name: Deploy to Blue
      run: |
        npm run build
        aws s3 sync dist/ s3://blue-bucket/
        aws cloudfront create-invalidation --distribution-id BLUE_DISTRIBUTION_ID --paths /*

deploy_green:
  runs-on: ubuntu-latest
  needs: deploy_blue
  environment:
    name: green
    url: https://green.example.com
  steps:
    - uses: actions/checkout@v3
    - name: Deploy to Green
      run: |
        npm run build
        aws s3 sync dist/ s3://green-bucket/
        aws cloudfront create-invalidation --distribution-id GREEN_DISTRIBUTION_ID --paths /*

switch_traffic:
  runs-on: ubuntu-latest
  needs: deploy_green
  steps:
    - name: Switch traffic to Green
      run: |
        aws route53 change-resource-record-sets \
          --hosted-zone-id example.com \
          --change-batch file://traffic-switch.json

Canary Deployment

# Gradual rollout to production
deploy_canary:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - name: Deploy canary (10%)
      run: |
        npm run build
        aws s3 sync dist/ s3://production-bucket/canary/
        kubectl patch deployment myapp -p '{"spec":{"replicas":2}}'  # 2 pods for 10% traffic

monitor_canary:
  runs-on: ubuntu-latest
  needs: deploy_canary
  steps:
    - name: Monitor canary metrics
      run: |
        node scripts/monitor-canary.js --duration=300 # Monitor for 5 minutes

deploy_full:
  runs-on: ubuntu-latest
  needs: monitor_canary
  if: success()  # Only runs if monitoring succeeds
  steps:
    - uses: actions/checkout@v3
    - name: Deploy to production (100%)
      run: |
        npm run build
        aws s3 sync dist/ s3://production-bucket/
        kubectl patch deployment myapp -p '{"spec":{"replicas":20}}'  # 20 pods for 100% traffic

Rolling Update

# Kubernetes rolling update
deploy:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - name: Build Docker image
      run: |
        docker build -t myapp:${{ github.sha }} .
        docker push myapp:${{ github.sha }}

    - name: Deploy with rolling update
      run: |
        kubectl set image deployment/myapp myapp=myapp:${{ github.sha }}
        kubectl rollout status deployment/myapp --watch=true

Quality Gates

Code Quality Checks

steps:
  - name: Lint code
    run: npm run lint

  - name: Check code style
    run: npm run format:check

  - name: Type check
    run: npm run type-check

  - name: Security audit
    run: npm audit --audit-level=moderate

  - name: Dependency check
    run: npm outdated

Test Coverage

steps:
  - name: Run tests with coverage
    run: npm test -- --coverage

  - name: Check coverage threshold
    run: |
      if [ $(cat coverage/coverage-summary.json | jq '.total.lines.pct') -lt 80 ]; then
        echo "Coverage below 80%"
        exit 1
      fi

  - name: Upload coverage to Codecov
    uses: codecov/codecov-action@v3
    with:
      files: ./coverage/lcov.info
      flags: unittests
      fail_ci_if_error: true

Performance Budgets

steps:
  - name: Build application
    run: npm run build

  - name: Check bundle size
    run: |
      npm run build:analyze

  - name: Compare with budget
    run: |
      BUDGET_SIZE=244000  # 244KB

      ACTUAL_SIZE=$(wc -c dist/bundle.js | awk '{print $1}')

      if [ $ACTUAL_SIZE -gt $BUDGET_SIZE ]; then
        echo "Bundle size exceeds budget"
        exit 1
      fi

Best Practices

1. Keep Pipelines Fast

# ✅ Good: Parallel jobs
jobs:
  lint-test-build:
    strategy:
      matrix:
        task: [lint, test, build]
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Run ${{ matrix.task }}
        run: npm run ${{ matrix.task }}

# ❌ Bad: Sequential jobs
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Lint
        run: npm run lint

  test:
    needs: lint  # Waits for lint
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Test
        run: npm run test

  build:
    needs: test  # Waits for test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build
        run: npm run build

2. Use Caching Effectively

steps:
  - name: Cache node modules
    uses: actions/cache@v3
    id: cache-node
    with:
      path: |
        node_modules
        ~/.npm
      key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
      restore-keys: |
        ${{ runner.os }}-node-

  - name: Install dependencies
    run: npm ci

3. Fail Fast

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest]
        node-version: [16, 18]
      fail-fast: true  # Stop all jobs if one fails
    steps:
      - name: Test
        run: npm test

4. Environment Secrets

# ✅ Good: Use secrets
steps:
  - name: Deploy
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    run: |
      aws s3 sync dist/ s3://my-bucket/

# ❌ Bad: Hardcoded secrets
steps:
  - name: Deploy
    env:
      AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
      AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    run: |
      aws s3 sync dist/ s3://my-bucket/

5. Artifact Management

steps:
  - name: Build
    run: npm run build

  - name: Upload build artifacts
    uses: actions/upload-artifact@v3
    with:
      name: build-output
      path: |
        dist/
        build-logs/
      retention-days: 30  # Keep for 30 days

  - name: Upload test results
    if: always()  # Upload even if tests fail
    uses: actions/upload-artifact@v3
    with:
      name: test-results
      path: test-results/
      if-no-files-found: ignore

Monitoring and Notifications

Status Checks

# .github/workflows/status.yml
name: Status Check

on:
  schedule:
    # Run every 15 minutes
    - cron: '*/15 * * * *'

jobs:
  status:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Check website status
        run: |
          STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://example.com)

          if [ $STATUS -ne 200 ]; then
            echo "Website is down"
            exit 1
          fi

Notifications

# Send Slack notifications on deployment
steps:
  - name: Notify Slack
    uses: 8398a7/action-slack@v3
    with:
      status: ${{ job.status }}
      text: |
        Deployment ${{ job.status }} for ${{ github.sha }}
        Branch: ${{ github.ref }}
        Author: ${{ github.actor }}
      webhook_url: ${{ secrets.SLACK_WEBHOOK_URL }}
    if: always()  # Send on both success and failure

Conclusion

CI/CD pipelines are the backbone of modern software development. By implementing comprehensive automation, you can ship code faster, with higher confidence, and reduce manual errors.

Key Takeaways

  1. Automate Everything - Build, test, deploy should all be automated
  2. Parallelize Where Possible - Run jobs concurrently to speed up pipelines
  3. Use Caching - Cache dependencies and build artifacts
  4. Implement Quality Gates - Lint, test coverage, security checks
  5. Choose Right Deployment Strategy - Blue-green, canary, rolling updates
  6. Monitor Pipelines - Get notified of failures quickly
  7. Secure Secrets - Never hardcode credentials

Next Steps

  1. Assess your current build and deploy process
  2. Choose a CI/CD platform (GitHub Actions, GitLab CI, CircleCI)
  3. Create your first pipeline
  4. Add stages for build, test, and deploy
  5. Implement caching and parallel jobs
  6. Add quality gates and notifications
  7. Iterate and improve over time

The journey to continuous delivery starts now. Build your pipeline and start shipping faster.

Ready to automate your deployments? Start by creating a simple CI pipeline and iterate from there. Your team will thank you.