Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Chapter 29: Deployment Guide

Overview

This chapter provides complete deployment procedures for the POS Platform, including Docker containerization, environment configuration, deployment strategies, and rollback procedures.


Deployment Architecture

                                    PRODUCTION ENVIRONMENT
+-----------------------------------------------------------------------------------+
|                              Load Balancer (Nginx/HAProxy)                        |
|                                    Port 443 (HTTPS)                               |
+-----------------------------------------------------------------------------------+
           |                              |                              |
           v                              v                              v
+-------------------+        +-------------------+        +-------------------+
|   POS-API-01      |        |   POS-API-02      |        |   POS-API-03      |
|   (Container)     |        |   (Container)     |        |   (Container)     |
|   Port 8080       |        |   Port 8080       |        |   Port 8080       |
+-------------------+        +-------------------+        +-------------------+
           |                              |                              |
           +------------------------------+------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
|                              PostgreSQL Cluster                                   |
|                         Primary (Write) + Replica (Read)                          |
|                                    Port 5432                                      |
+-----------------------------------------------------------------------------------+
           |                              |                              |
           v                              v                              v
+-------------------+        +-------------------+        +-------------------+
|    Redis          |        |    RabbitMQ       |        |   Prometheus      |
|   (Cache/Session) |        |   (Event Bus)     |        |   (Metrics)       |
|    Port 6379      |        |    Port 5672      |        |    Port 9090      |
+-------------------+        +-------------------+        +-------------------+

Container Images

Complete Dockerfile for API

# File: /pos-platform/docker/api/Dockerfile
# Multi-stage build for ASP.NET Core POS API

#=============================================
# Stage 1: Build Environment
#=============================================
FROM mcr.microsoft.com/dotnet/sdk:8.0-alpine AS build

WORKDIR /src

# Copy solution and project files for layer caching
COPY ["POS.sln", "./"]
COPY ["src/POS.Api/POS.Api.csproj", "src/POS.Api/"]
COPY ["src/POS.Core/POS.Core.csproj", "src/POS.Core/"]
COPY ["src/POS.Infrastructure/POS.Infrastructure.csproj", "src/POS.Infrastructure/"]
COPY ["src/POS.Application/POS.Application.csproj", "src/POS.Application/"]

# Restore dependencies (cached unless .csproj changes)
RUN dotnet restore "POS.sln"

# Copy remaining source code
COPY . .

# Build release version
WORKDIR "/src/src/POS.Api"
RUN dotnet build "POS.Api.csproj" -c Release -o /app/build

#=============================================
# Stage 2: Publish
#=============================================
FROM build AS publish

RUN dotnet publish "POS.Api.csproj" \
    -c Release \
    -o /app/publish \
    --no-restore \
    /p:UseAppHost=false \
    /p:PublishTrimmed=false

#=============================================
# Stage 3: Runtime Environment
#=============================================
FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine AS runtime

# Security: Run as non-root user
RUN addgroup -S posgroup && adduser -S posuser -G posgroup

WORKDIR /app

# Install health check dependencies
RUN apk add --no-cache curl

# Copy published application
COPY --from=publish /app/publish .

# Set ownership
RUN chown -R posuser:posgroup /app

# Switch to non-root user
USER posuser

# Expose port
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# Set environment
ENV ASPNETCORE_URLS=http://+:8080
ENV ASPNETCORE_ENVIRONMENT=Production
ENV DOTNET_RUNNING_IN_CONTAINER=true

# Entry point
ENTRYPOINT ["dotnet", "POS.Api.dll"]

Dockerfile for Frontend (Blazor WASM)

# File: /pos-platform/docker/frontend/Dockerfile
# Multi-stage build for Blazor WebAssembly

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY ["src/POS.Client/POS.Client.csproj", "src/POS.Client/"]
RUN dotnet restore "src/POS.Client/POS.Client.csproj"
COPY . .
WORKDIR "/src/src/POS.Client"
RUN dotnet publish "POS.Client.csproj" -c Release -o /app/publish

FROM nginx:alpine AS runtime
COPY --from=build /app/publish/wwwroot /usr/share/nginx/html
COPY docker/frontend/nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Docker Compose Configuration

Complete Production docker-compose.yml

# File: /pos-platform/docker/docker-compose.yml
# Production deployment configuration

version: '3.8'

services:
  #=========================================
  # POS API Service (Scalable)
  #=========================================
  pos-api:
    build:
      context: ..
      dockerfile: docker/api/Dockerfile
    image: pos-api:${TAG:-latest}
    container_name: pos-api-${INSTANCE:-1}
    restart: unless-stopped
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 512M
      update_config:
        parallelism: 1
        delay: 30s
        failure_action: rollback
        order: start-first
      rollback_config:
        parallelism: 1
        delay: 10s
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ConnectionStrings__DefaultConnection=${DB_CONNECTION_STRING}
      - ConnectionStrings__ReadReplicaConnection=${DB_READ_CONNECTION_STRING}
      - Redis__ConnectionString=${REDIS_CONNECTION_STRING}
      - RabbitMQ__Host=${RABBITMQ_HOST}
      - RabbitMQ__Username=${RABBITMQ_USER}
      - RabbitMQ__Password=${RABBITMQ_PASSWORD}
      - Jwt__Secret=${JWT_SECRET}
      - Jwt__Issuer=${JWT_ISSUER}
      - Jwt__Audience=${JWT_AUDIENCE}
      - Payment__StripeApiKey=${STRIPE_API_KEY}
      - Payment__StripeWebhookSecret=${STRIPE_WEBHOOK_SECRET}
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://prometheus:9090
    ports:
      - "${API_PORT:-8080}:8080"
    networks:
      - pos-network
    depends_on:
      postgres-primary:
        condition: service_healthy
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"
    volumes:
      - pos-data-protection:/app/keys
      - pos-logs:/app/logs

  #=========================================
  # PostgreSQL Primary (Write)
  #=========================================
  postgres-primary:
    image: postgres:16-alpine
    container_name: pos-postgres-primary
    restart: unless-stopped
    environment:
      - POSTGRES_DB=pos_db
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - PGDATA=/var/lib/postgresql/data/pgdata
    ports:
      - "${DB_PORT:-5432}:5432"
    networks:
      - pos-network
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./postgres/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
      - ./postgres/postgresql.conf:/etc/postgresql/postgresql.conf:ro
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d pos_db"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 8G
        reservations:
          cpus: '1'
          memory: 2G

  #=========================================
  # PostgreSQL Replica (Read)
  #=========================================
  postgres-replica:
    image: postgres:16-alpine
    container_name: pos-postgres-replica
    restart: unless-stopped
    environment:
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - PGDATA=/var/lib/postgresql/data/pgdata
    networks:
      - pos-network
    volumes:
      - postgres-replica-data:/var/lib/postgresql/data
    command: postgres -c hot_standby=on
    depends_on:
      postgres-primary:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G

  #=========================================
  # Redis Cache
  #=========================================
  redis:
    image: redis:7-alpine
    container_name: pos-redis
    restart: unless-stopped
    command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
    ports:
      - "${REDIS_PORT:-6379}:6379"
    networks:
      - pos-network
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G

  #=========================================
  # RabbitMQ Message Broker
  #=========================================
  rabbitmq:
    image: rabbitmq:3-management-alpine
    container_name: pos-rabbitmq
    restart: unless-stopped
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_PASSWORD}
      - RABBITMQ_DEFAULT_VHOST=pos
    ports:
      - "${RABBITMQ_PORT:-5672}:5672"
      - "${RABBITMQ_MGMT_PORT:-15672}:15672"
    networks:
      - pos-network
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G

  #=========================================
  # Nginx Load Balancer
  #=========================================
  nginx:
    image: nginx:alpine
    container_name: pos-nginx
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    networks:
      - pos-network
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
      - nginx-cache:/var/cache/nginx
    depends_on:
      - pos-api
    healthcheck:
      test: ["CMD", "nginx", "-t"]
      interval: 30s
      timeout: 10s
      retries: 3

#=========================================
# Networks
#=========================================
networks:
  pos-network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16

#=========================================
# Volumes
#=========================================
volumes:
  postgres-data:
    driver: local
  postgres-replica-data:
    driver: local
  redis-data:
    driver: local
  rabbitmq-data:
    driver: local
  nginx-cache:
    driver: local
  pos-data-protection:
    driver: local
  pos-logs:
    driver: local

Environment Variables Reference

Complete .env Template

# File: /pos-platform/docker/.env.template

#=============================================
# ENVIRONMENT
#=============================================
ENVIRONMENT=Production
TAG=latest

#=============================================
# DATABASE - PostgreSQL
#=============================================
DB_HOST=postgres-primary
DB_PORT=5432
DB_NAME=pos_db
DB_USER=pos_admin
DB_PASSWORD=<GENERATE_STRONG_PASSWORD>
DB_CONNECTION_STRING=Host=postgres-primary;Port=5432;Database=pos_db;Username=pos_admin;Password=${DB_PASSWORD};Pooling=true;MinPoolSize=5;MaxPoolSize=100
DB_READ_CONNECTION_STRING=Host=postgres-replica;Port=5432;Database=pos_db;Username=pos_admin;Password=${DB_PASSWORD};Pooling=true

#=============================================
# CACHE - Redis
#=============================================
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=<GENERATE_STRONG_PASSWORD>
REDIS_CONNECTION_STRING=redis:6379,password=${REDIS_PASSWORD},abortConnect=false

#=============================================
# MESSAGE BROKER - RabbitMQ
#=============================================
RABBITMQ_HOST=rabbitmq
RABBITMQ_PORT=5672
RABBITMQ_USER=pos_admin
RABBITMQ_PASSWORD=<GENERATE_STRONG_PASSWORD>
RABBITMQ_VHOST=pos

#=============================================
# SECURITY - JWT
#=============================================
JWT_SECRET=<GENERATE_256_BIT_SECRET>
JWT_ISSUER=pos-platform
JWT_AUDIENCE=pos-clients
JWT_EXPIRY_MINUTES=60
JWT_REFRESH_EXPIRY_DAYS=7

#=============================================
# PAYMENT PROCESSING
#=============================================
STRIPE_API_KEY=sk_live_<YOUR_KEY>
STRIPE_WEBHOOK_SECRET=whsec_<YOUR_SECRET>
STRIPE_PUBLIC_KEY=pk_live_<YOUR_KEY>

#=============================================
# EXTERNAL SERVICES
#=============================================
SHOPIFY_API_KEY=<YOUR_KEY>
SHOPIFY_API_SECRET=<YOUR_SECRET>
QUICKBOOKS_CLIENT_ID=<YOUR_ID>
QUICKBOOKS_CLIENT_SECRET=<YOUR_SECRET>

#=============================================
# MONITORING
#=============================================
PROMETHEUS_PORT=9090
GRAFANA_PORT=3000
GRAFANA_ADMIN_PASSWORD=<GENERATE_STRONG_PASSWORD>

#=============================================
# LOGGING
#=============================================
LOG_LEVEL=Information
LOG_PATH=/app/logs
SERILOG_SEQ_URL=http://seq:5341

#=============================================
# API CONFIGURATION
#=============================================
API_PORT=8080
API_RATE_LIMIT_PER_MINUTE=100
API_CORS_ORIGINS=https://pos.yourcompany.com

Deployment Checklist

Pre-Deployment Checklist

## Pre-Deployment Verification

### 1. Code Readiness
- [ ] All tests passing (unit, integration, e2e)
- [ ] Code review approved
- [ ] Security scan completed (no critical/high vulnerabilities)
- [ ] Version number updated in csproj
- [ ] CHANGELOG.md updated
- [ ] Database migrations tested

### 2. Infrastructure Readiness
- [ ] Target environment accessible
- [ ] SSL certificates valid (> 30 days)
- [ ] Database backup completed (< 1 hour old)
- [ ] Sufficient disk space (> 20% free)
- [ ] Load balancer health checks configured
- [ ] DNS pointing to correct servers

### 3. Configuration Verification
- [ ] .env file populated with production values
- [ ] Secrets stored in secure vault
- [ ] Connection strings validated
- [ ] External API keys verified

### 4. Rollback Preparation
- [ ] Previous version image tagged and available
- [ ] Rollback script tested
- [ ] Database rollback script prepared (if schema changes)
- [ ] Rollback communication template ready

### 5. Team Readiness
- [ ] Deployment window communicated
- [ ] On-call engineer identified
- [ ] Customer support notified
- [ ] Monitoring dashboard accessible

Deployment Script

#!/bin/bash
# File: /pos-platform/scripts/deploy.sh
# Production deployment script

set -e  # Exit on error

#=============================================
# CONFIGURATION
#=============================================
DEPLOY_DIR="/opt/pos-platform"
DOCKER_COMPOSE="docker compose"
TAG=${1:-latest}
BACKUP_DIR="/backups/pos"
LOG_FILE="/var/log/pos-deploy.log"

#=============================================
# FUNCTIONS
#=============================================
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

check_prerequisites() {
    log "Checking prerequisites..."

    # Check Docker
    if ! command -v docker &> /dev/null; then
        log "ERROR: Docker not installed"
        exit 1
    fi

    # Check disk space (require 20% free)
    FREE_SPACE=$(df / | tail -1 | awk '{print $5}' | sed 's/%//')
    if [ "$FREE_SPACE" -gt 80 ]; then
        log "ERROR: Insufficient disk space (${FREE_SPACE}% used)"
        exit 1
    fi

    log "Prerequisites check passed"
}

backup_database() {
    log "Creating database backup..."
    BACKUP_FILE="${BACKUP_DIR}/pos_db_$(date +%Y%m%d_%H%M%S).sql.gz"

    $DOCKER_COMPOSE exec -T postgres-primary pg_dump -U pos_admin pos_db | gzip > "$BACKUP_FILE"

    if [ $? -eq 0 ]; then
        log "Database backup created: $BACKUP_FILE"
    else
        log "ERROR: Database backup failed"
        exit 1
    fi
}

pull_images() {
    log "Pulling new images (tag: $TAG)..."

    $DOCKER_COMPOSE pull

    log "Images pulled successfully"
}

deploy_with_zero_downtime() {
    log "Starting zero-downtime deployment..."

    # Scale up new containers first
    $DOCKER_COMPOSE up -d --scale pos-api=4 --no-recreate

    # Wait for new containers to be healthy
    log "Waiting for health checks..."
    sleep 60

    # Verify new containers are healthy
    HEALTHY_COUNT=$($DOCKER_COMPOSE ps | grep "healthy" | wc -l)
    if [ "$HEALTHY_COUNT" -lt 3 ]; then
        log "ERROR: Not enough healthy containers"
        rollback
        exit 1
    fi

    # Rolling update
    $DOCKER_COMPOSE up -d --force-recreate

    # Scale back to normal
    $DOCKER_COMPOSE up -d --scale pos-api=3

    log "Deployment completed successfully"
}

verify_deployment() {
    log "Verifying deployment..."

    # Check health endpoint
    HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)

    if [ "$HTTP_CODE" -eq 200 ]; then
        log "Health check passed (HTTP $HTTP_CODE)"
    else
        log "ERROR: Health check failed (HTTP $HTTP_CODE)"
        rollback
        exit 1
    fi

    # Check version
    VERSION=$(curl -s http://localhost:8080/health | jq -r '.version')
    log "Deployed version: $VERSION"
}

rollback() {
    log "ROLLBACK: Initiating rollback..."

    # Get previous image tag
    PREVIOUS_TAG=$(docker images pos-api --format "{{.Tag}}" | sed -n '2p')

    if [ -z "$PREVIOUS_TAG" ]; then
        log "ERROR: No previous version found for rollback"
        exit 1
    fi

    log "Rolling back to version: $PREVIOUS_TAG"

    TAG=$PREVIOUS_TAG $DOCKER_COMPOSE up -d --force-recreate

    log "Rollback completed"
}

#=============================================
# MAIN EXECUTION
#=============================================
main() {
    log "=========================================="
    log "POS Platform Deployment - Started"
    log "Tag: $TAG"
    log "=========================================="

    cd "$DEPLOY_DIR"

    check_prerequisites
    backup_database
    pull_images
    deploy_with_zero_downtime
    verify_deployment

    log "=========================================="
    log "Deployment completed successfully!"
    log "=========================================="
}

# Run main function
main "$@"

Zero-Downtime Deployment Strategy

Rolling Update Process

┌─────────────────────────────────────────────────────────────────────────────┐
│                         ZERO-DOWNTIME DEPLOYMENT                            │
└─────────────────────────────────────────────────────────────────────────────┘

STEP 1: Initial State
┌─────────────────────────────────────────────────────────────────────────────┐
│  Load Balancer                                                              │
│       │                                                                     │
│       ├──────► API-1 (v1.0) [HEALTHY] ◄── Receiving traffic               │
│       ├──────► API-2 (v1.0) [HEALTHY] ◄── Receiving traffic               │
│       └──────► API-3 (v1.0) [HEALTHY] ◄── Receiving traffic               │
└─────────────────────────────────────────────────────────────────────────────┘

STEP 2: Add New Container
┌─────────────────────────────────────────────────────────────────────────────┐
│  Load Balancer                                                              │
│       │                                                                     │
│       ├──────► API-1 (v1.0) [HEALTHY]                                      │
│       ├──────► API-2 (v1.0) [HEALTHY]                                      │
│       ├──────► API-3 (v1.0) [HEALTHY]                                      │
│       └─ - - ► API-4 (v2.0) [STARTING] ◄── Not yet in rotation            │
└─────────────────────────────────────────────────────────────────────────────┘

STEP 3: New Container Healthy
┌─────────────────────────────────────────────────────────────────────────────┐
│  Load Balancer                                                              │
│       │                                                                     │
│       ├──────► API-1 (v1.0) [HEALTHY]                                      │
│       ├──────► API-2 (v1.0) [HEALTHY]                                      │
│       ├──────► API-3 (v1.0) [HEALTHY]                                      │
│       └──────► API-4 (v2.0) [HEALTHY] ◄── Now receiving traffic           │
└─────────────────────────────────────────────────────────────────────────────┘

STEP 4: Drain Old Container
┌─────────────────────────────────────────────────────────────────────────────┐
│  Load Balancer                                                              │
│       │                                                                     │
│       ├─ X ─► API-1 (v1.0) [DRAINING] ◄── Finishing existing requests     │
│       ├──────► API-2 (v1.0) [HEALTHY]                                      │
│       ├──────► API-3 (v1.0) [HEALTHY]                                      │
│       └──────► API-4 (v2.0) [HEALTHY]                                      │
└─────────────────────────────────────────────────────────────────────────────┘

STEP 5: Replace Old Container
┌─────────────────────────────────────────────────────────────────────────────┐
│  Load Balancer                                                              │
│       │                                                                     │
│       ├──────► API-1 (v2.0) [HEALTHY] ◄── Replaced                         │
│       ├──────► API-2 (v1.0) [DRAINING]                                     │
│       ├──────► API-3 (v1.0) [HEALTHY]                                      │
│       └──────► API-4 (v2.0) [HEALTHY]                                      │
└─────────────────────────────────────────────────────────────────────────────┘

STEP 6: Complete (Scale back to 3)
┌─────────────────────────────────────────────────────────────────────────────┐
│  Load Balancer                                                              │
│       │                                                                     │
│       ├──────► API-1 (v2.0) [HEALTHY]                                      │
│       ├──────► API-2 (v2.0) [HEALTHY]                                      │
│       └──────► API-3 (v2.0) [HEALTHY]                                      │
│                                                                             │
│  Result: Zero downtime, all traffic served continuously                     │
└─────────────────────────────────────────────────────────────────────────────┘

Rollback Procedures

Automated Rollback Script

#!/bin/bash
# File: /pos-platform/scripts/rollback.sh
# Emergency rollback script

set -e

DEPLOY_DIR="/opt/pos-platform"
DOCKER_COMPOSE="docker compose"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] ROLLBACK: $1"
}

#=============================================
# ROLLBACK TO PREVIOUS VERSION
#=============================================
rollback_containers() {
    log "Starting container rollback..."

    # Get previous image
    PREVIOUS_TAG=$(docker images pos-api --format "table {{.Tag}}\t{{.CreatedAt}}" | \
                   grep -v latest | head -2 | tail -1 | awk '{print $1}')

    if [ -z "$PREVIOUS_TAG" ]; then
        log "ERROR: No previous version available"
        exit 1
    fi

    log "Rolling back to: $PREVIOUS_TAG"

    cd "$DEPLOY_DIR"
    export TAG=$PREVIOUS_TAG

    # Force recreate with previous version
    $DOCKER_COMPOSE up -d --force-recreate pos-api

    log "Containers rolled back to $PREVIOUS_TAG"
}

#=============================================
# ROLLBACK DATABASE (IF NEEDED)
#=============================================
rollback_database() {
    BACKUP_FILE=$1

    if [ -z "$BACKUP_FILE" ]; then
        log "No database backup specified, skipping DB rollback"
        return
    fi

    log "Rolling back database from: $BACKUP_FILE"

    # Restore from backup
    zcat "$BACKUP_FILE" | $DOCKER_COMPOSE exec -T postgres-primary psql -U pos_admin pos_db

    log "Database rolled back"
}

#=============================================
# VERIFY ROLLBACK
#=============================================
verify_rollback() {
    log "Verifying rollback..."

    sleep 30  # Wait for containers to stabilize

    HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)

    if [ "$HTTP_CODE" -eq 200 ]; then
        log "Rollback verified successfully (HTTP $HTTP_CODE)"
    else
        log "ERROR: Rollback verification failed (HTTP $HTTP_CODE)"
        log "CRITICAL: Manual intervention required!"
        exit 1
    fi
}

#=============================================
# MAIN
#=============================================
main() {
    log "=========================================="
    log "EMERGENCY ROLLBACK INITIATED"
    log "=========================================="

    rollback_containers
    rollback_database "$1"
    verify_rollback

    log "=========================================="
    log "ROLLBACK COMPLETED"
    log "=========================================="
}

main "$@"

Health Check Endpoints

Health Check Implementation

// File: /src/POS.Api/Health/HealthCheckEndpoints.cs

public static class HealthCheckEndpoints
{
    public static void MapHealthChecks(this WebApplication app)
    {
        // Basic liveness probe (is the app running?)
        app.MapHealthChecks("/health/live", new HealthCheckOptions
        {
            Predicate = _ => false,  // No checks, just confirms app is running
            ResponseWriter = WriteResponse
        });

        // Readiness probe (is the app ready to serve traffic?)
        app.MapHealthChecks("/health/ready", new HealthCheckOptions
        {
            Predicate = check => check.Tags.Contains("ready"),
            ResponseWriter = WriteResponse
        });

        // Full health check (all dependencies)
        app.MapHealthChecks("/health", new HealthCheckOptions
        {
            ResponseWriter = WriteResponse
        });
    }

    private static async Task WriteResponse(
        HttpContext context,
        HealthReport report)
    {
        context.Response.ContentType = "application/json";

        var response = new
        {
            status = report.Status.ToString(),
            version = Assembly.GetExecutingAssembly()
                .GetCustomAttribute<AssemblyInformationalVersionAttribute>()
                ?.InformationalVersion ?? "unknown",
            timestamp = DateTime.UtcNow,
            checks = report.Entries.Select(e => new
            {
                name = e.Key,
                status = e.Value.Status.ToString(),
                duration = e.Value.Duration.TotalMilliseconds,
                description = e.Value.Description,
                data = e.Value.Data
            })
        };

        await context.Response.WriteAsJsonAsync(response);
    }
}

Health Check Response Example

{
  "status": "Healthy",
  "version": "2.1.0",
  "timestamp": "2025-12-29T10:30:00Z",
  "checks": [
    {
      "name": "database",
      "status": "Healthy",
      "duration": 12.5,
      "description": "PostgreSQL connection is healthy"
    },
    {
      "name": "redis",
      "status": "Healthy",
      "duration": 3.2,
      "description": "Redis cache is accessible"
    },
    {
      "name": "rabbitmq",
      "status": "Healthy",
      "duration": 8.1,
      "description": "RabbitMQ broker is connected"
    },
    {
      "name": "disk",
      "status": "Healthy",
      "duration": 1.0,
      "description": "Disk space: 45% used"
    }
  ]
}

Summary

This chapter provides complete deployment procedures including:

  1. Docker Configuration: Multi-stage Dockerfile and production docker-compose.yml
  2. Environment Variables: Complete reference for all configuration
  3. Deployment Checklist: Pre-deployment verification steps
  4. Zero-Downtime Strategy: Rolling update process diagram
  5. Rollback Procedures: Automated rollback scripts
  6. Health Checks: Implementation and response format

Next Chapter: Chapter 30: Monitoring and Alerting


“Deploy with confidence. Rollback without fear.”