> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling

> Scale Chunkr for high-throughput production workloads

Chunkr is designed to scale horizontally by increasing the number of worker replicas. This guide covers scaling strategies for different workload patterns.

## Understanding Chunkr's Architecture

Chunkr uses a distributed architecture with specialized workers:

```
┌─────────────┐
│   Client    │
└──────┬──────┘
       │
       ▼
┌─────────────┐      ┌─────────────┐
│   Server    │◄────►│    Redis    │
│  (1 replica)│      │   (Queue)   │
└──────┬──────┘      └─────────────┘
       │
       │ Enqueues tasks
       │
       ▼
┌─────────────────────────────────┐
│      Task Workers (30)          │
│  - Orchestrate processing       │
│  - Call ML services             │
└────┬──────────────────────┬─────┘
     │                      │
     ▼                      ▼
┌────────────┐      ┌──────────────┐
│Segmentation│      │     OCR      │
│   Workers  │      │   Workers    │
│ (6 replicas)│      │ (3 replicas) │
└────────────┘      └──────────────┘
```

## Default Replica Configuration

The default `compose.yaml` is optimized for medium workloads:

| Service                  | Default Replicas | Purpose                           |
| ------------------------ | ---------------- | --------------------------------- |
| **server**               | 1                | API server (stateless, can scale) |
| **task**                 | 30               | Background task orchestration     |
| **segmentation-backend** | 6                | Document layout analysis          |
| **ocr-backend**          | 3                | Text recognition                  |
| **web**                  | 1                | Frontend UI                       |
| **postgres**             | 1                | Database (single instance)        |
| **redis**                | 1                | Queue/cache (single instance)     |

## Scaling Strategies

### Vertical Scaling

Increase resources for individual containers:

```yaml theme={null}
services:
  task:
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 8G
        reservations:
          cpus: '2.0'
          memory: 4G
```

### Horizontal Scaling

Increase the number of worker replicas:

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 50  # Scale up from 30
  
  segmentation-backend:
    deploy:
      replicas: 12  # Scale up from 6
  
  ocr-backend:
    deploy:
      replicas: 6   # Scale up from 3
```

<Info>
  Horizontal scaling is more effective for Chunkr as it allows processing multiple documents in parallel.
</Info>

## Scaling for Different Workloads

### High Volume, Small Documents

**Characteristics:** Many PDF pages, mostly text

**Recommended configuration:**

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 50
  
  segmentation-backend:
    deploy:
      replicas: 8
  
  ocr-backend:
    deploy:
      replicas: 12  # OCR is bottleneck for text-heavy docs
```

### Large Documents, Complex Layouts

**Characteristics:** Multi-page documents with tables, images, complex formatting

**Recommended configuration:**

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 30
  
  segmentation-backend:
    deploy:
      replicas: 12  # Segmentation is bottleneck
  
  ocr-backend:
    deploy:
      replicas: 6
```

### Mixed Workload

**Characteristics:** Variety of document types and sizes

**Recommended configuration:**

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 40
  
  segmentation-backend:
    deploy:
      replicas: 10
  
  ocr-backend:
    deploy:
      replicas: 8
```

## GPU Scaling Considerations

### Single GPU

With one GPU (8GB+), balance workers to avoid memory contention:

```yaml theme={null}
segmentation-backend:
  deploy:
    replicas: 4
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  environment:
    - MAX_BATCH_SIZE=4

ocr-backend:
  deploy:
    replicas: 2
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
```

<Warning>
  Too many replicas on a single GPU can cause out-of-memory errors. Start conservative and scale up while monitoring GPU memory.
</Warning>

### Multiple GPUs

With multiple GPUs, scale workers proportionally:

**2 GPUs:**

```yaml theme={null}
segmentation-backend:
  deploy:
    replicas: 8  # 4 per GPU

ocr-backend:
  deploy:
    replicas: 4  # 2 per GPU
```

**4 GPUs:**

```yaml theme={null}
segmentation-backend:
  deploy:
    replicas: 16  # 4 per GPU

ocr-backend:
  deploy:
    replicas: 8   # 2 per GPU
```

### GPU Pinning

For optimal performance, pin specific workers to specific GPUs:

```yaml theme={null}
segmentation-backend-gpu0:
  build:
    context: .
    dockerfile: docker/segmentation/Dockerfile
  deploy:
    replicas: 4
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]

segmentation-backend-gpu1:
  build:
    context: .
    dockerfile: docker/segmentation/Dockerfile
  deploy:
    replicas: 4
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ['1']
            capabilities: [gpu]
```

## CPU Scaling

For CPU-only deployments, scale more conservatively:

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 10  # Reduced from GPU default
  
  segmentation-backend:
    deploy:
      replicas: 6
      resources: {}
    environment:
      - MAX_BATCH_SIZE=64
      - OMP_NUM_THREADS=12
      - MKL_NUM_THREADS=12
  
  ocr-backend:
    deploy:
      replicas: 3
      resources: {}
```

**CPU Threading Configuration:**

* **OMP\_NUM\_THREADS**: OpenMP threads per worker
* **MKL\_NUM\_THREADS**: Intel MKL threads per worker
* **NUMEXPR\_NUM\_THREADS**: NumExpr threads per worker

<Info>
  Set thread counts to: `(total_cpu_cores / replicas)` to avoid over-subscription.
</Info>

## Load Balancing

Chunkr uses nginx for load balancing ML workers:

### Segmentation Load Balancer

```yaml theme={null}
segmentation:
  image: nginx:latest
  ports:
    - "8001:8000"
  volumes:
    - ./nginx/segmentation.conf:/etc/nginx/nginx.conf:ro
  depends_on:
    - segmentation-backend
```

The nginx configuration distributes requests across all `segmentation-backend` replicas.

### OCR Load Balancer

```yaml theme={null}
ocr:
  image: nginx:latest
  ports:
    - "8002:8000"
  volumes:
    - ./nginx/ocr.conf:/etc/nginx/nginx.conf:ro
  depends_on:
    - ocr-backend
```

<Info>
  The load balancers automatically detect all replicas using Docker's DNS service discovery.
</Info>

## Monitoring and Optimization

### Monitor Queue Depth

Check Redis queue length to identify bottlenecks:

```bash theme={null}
# Connect to Redis
docker compose exec redis redis-cli

# Check queue length
LLEN task_queue_name

# Monitor in real-time
watch -n 1 'docker compose exec redis redis-cli LLEN task_queue_name'
```

**Interpretation:**

* **Queue growing**: Workers can't keep up, scale up
* **Queue near zero**: Workers idle, may be over-provisioned
* **Queue stable**: System balanced

### Monitor Worker Utilization

```bash theme={null}
# CPU and memory usage
docker stats

# GPU usage
watch -n 1 nvidia-smi

# Task worker logs
docker compose logs -f task | grep "Processing"
```

### Identify Bottlenecks

<Steps>
  <Step title="Monitor queue depth">
    If queues are growing, workers are the bottleneck.
  </Step>

  <Step title="Check resource usage">
    * CPU near 100%: Scale horizontally or upgrade CPU
    * GPU near 100%: Add more GPUs or increase batch size
    * Memory high: Reduce replicas or batch sizes
  </Step>

  <Step title="Review processing times">
    Check logs for average processing time per document:

    ```bash theme={null}
    docker compose logs task | grep "completed in"
    ```
  </Step>

  <Step title="Scale the bottleneck">
    Increase replicas for the slowest component first.
  </Step>
</Steps>

## Scaling Commands

### Scale Specific Service

```bash theme={null}
# Scale task workers to 50
docker compose up -d --scale task=50

# Scale segmentation workers to 12
docker compose up -d --scale segmentation-backend=12

# Scale multiple services
docker compose up -d --scale task=50 --scale segmentation-backend=12
```

### Update docker-compose.yaml

For persistent scaling, update the compose file:

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 50
```

Then apply:

```bash theme={null}
docker compose up -d
```

### Zero-Downtime Scaling

```bash theme={null}
# Scale up gradually
docker compose up -d --scale task=40 --no-recreate
sleep 10
docker compose up -d --scale task=50 --no-recreate
```

## Database Scaling

PostgreSQL is a single instance by default. For production:

### Connection Pooling

Add PgBouncer for connection pooling:

```yaml theme={null}
pgbouncer:
  image: pgbouncer/pgbouncer:latest
  environment:
    - DATABASES_HOST=postgres
    - DATABASES_PORT=5432
    - DATABASES_USER=postgres
    - DATABASES_PASSWORD=postgres
    - DATABASES_DBNAME=chunkr
    - POOL_MODE=transaction
    - MAX_CLIENT_CONN=1000
    - DEFAULT_POOL_SIZE=25
  ports:
    - "6432:6432"
```

Update connection string:

```bash theme={null}
PG__URL=postgresql://postgres:postgres@pgbouncer:6432/chunkr
```

### Read Replicas

For read-heavy workloads, add PostgreSQL read replicas and route read queries accordingly.

## Redis Scaling

### Redis Cluster

For high availability and better performance:

```yaml theme={null}
redis-cluster:
  image: redis:latest
  command: redis-server --cluster-enabled yes --cluster-config-file nodes.conf
  volumes:
    - redis_cluster_data:/data
```

### Redis Sentinel

For automatic failover:

```yaml theme={null}
redis-sentinel:
  image: redis:latest
  command: redis-sentinel /etc/redis/sentinel.conf
  volumes:
    - ./redis/sentinel.conf:/etc/redis/sentinel.conf
```

## Cost Optimization

### Auto-scaling Strategy

1. **Monitor queue depth** every minute
2. **Scale up** when queue > 100 tasks for 2 minutes
3. **Scale down** when queue \< 10 tasks for 5 minutes
4. **Minimum replicas**: Keep baseline capacity
5. **Maximum replicas**: Set budget limits

### Resource Limits

Prevent runaway resource usage:

```yaml theme={null}
services:
  task:
    deploy:
      replicas: 30
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
        reservations:
          cpus: '1.0'
          memory: 2G
```

## Production Scaling Checklist

<Steps>
  <Step title="Baseline testing">
    * Test with expected document types
    * Measure processing times
    * Identify resource bottlenecks
  </Step>

  <Step title="Configure monitoring">
    * Set up metrics collection
    * Configure alerts for queue depth
    * Monitor GPU/CPU utilization
  </Step>

  <Step title="Scale incrementally">
    * Start with default configuration
    * Increase replicas by 25-50% at a time
    * Monitor impact before further scaling
  </Step>

  <Step title="Optimize bottlenecks">
    * Scale the slowest component first
    * Tune batch sizes and threading
    * Consider adding GPUs if needed
  </Step>

  <Step title="Set resource limits">
    * Prevent memory exhaustion
    * Ensure predictable performance
    * Enable graceful degradation
  </Step>
</Steps>

## Next Steps

* Review [GPU Setup](/deployment/gpu-setup) for GPU scaling
* Configure [Environment Variables](/deployment/environment-variables)
* Return to [Docker Compose Deployment](/deployment/docker-compose)