> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Setup

> Configure GPU acceleration for Chunkr's ML models

Chunkr uses GPU acceleration to significantly improve document processing performance. This guide covers GPU configuration for NVIDIA GPUs.

## Prerequisites

### Hardware Requirements

* NVIDIA GPU with CUDA support (compute capability 6.0+)
* At least 6GB GPU memory (12GB+ recommended for production)
* Multiple GPUs supported for increased throughput

### Software Requirements

* NVIDIA GPU drivers (version 470.x or later)
* NVIDIA Container Toolkit
* Docker Engine 19.03 or later
* Docker Compose V2

## Installing NVIDIA Container Toolkit

<Steps>
  <Step title="Add NVIDIA package repository">
    ```bash theme={null}
    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
    curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
    curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
      sudo tee /etc/apt/sources.list.d/nvidia-docker.list
    ```
  </Step>

  <Step title="Install nvidia-container-toolkit">
    ```bash theme={null}
    sudo apt-get update
    sudo apt-get install -y nvidia-container-toolkit
    ```
  </Step>

  <Step title="Configure Docker daemon">
    ```bash theme={null}
    sudo nvidia-ctk runtime configure --runtime=docker
    sudo systemctl restart docker
    ```
  </Step>

  <Step title="Verify installation">
    Test GPU access from Docker:

    ```bash theme={null}
    docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
    ```

    You should see your GPU(s) listed in the output.
  </Step>
</Steps>

## GPU Configuration in Docker Compose

Chunkr's default `compose.yaml` configures GPU access for ML services:

### Segmentation Backend

```yaml theme={null}
segmentation-backend:
  build:
    context: .
    dockerfile: docker/segmentation/Dockerfile
  deploy:
    replicas: 6
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
  volumes:
    - /dev/shm:/dev/shm
  environment:
    - MAX_BATCH_SIZE=4
    - BATCH_WAIT_TIME=0.2
    - OVERLAP_THRESHOLD=0.025
    - SCORE_THRESHOLD=0.2
```

**Configuration details:**

* **replicas: 6** - Six worker processes share available GPUs
* **count: all** - All GPUs are available to workers
* **capabilities: \[gpu]** - Enables GPU support
* **/dev/shm** - Shared memory for faster data transfer

### OCR Backend

```yaml theme={null}
ocr-backend:
  build:
    context: .
    dockerfile: docker/doctr/Dockerfile
  deploy:
    replicas: 3
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
  volumes:
    - /dev/shm:/dev/shm
```

**Configuration details:**

* **replicas: 3** - Three OCR workers for parallel processing
* Full GPU access for text recognition

<Info>
  The `/dev/shm` volume mount enables faster GPU memory transfers and is critical for performance.
</Info>

## Performance Tuning

### Batch Size Configuration

Adjust batch sizes based on your GPU memory:

```yaml theme={null}
environment:
  - MAX_BATCH_SIZE=4        # Increase for GPUs with >8GB memory
  - BATCH_WAIT_TIME=0.2     # Time to wait for batch to fill
```

**Recommended batch sizes:**

* 6GB GPU: `MAX_BATCH_SIZE=2`
* 8GB GPU: `MAX_BATCH_SIZE=4` (default)
* 12GB+ GPU: `MAX_BATCH_SIZE=8`

### Replica Count Optimization

Adjust worker replicas based on GPU count and memory:

**Single GPU (8GB+):**

```yaml theme={null}
segmentation-backend:
  deploy:
    replicas: 4

ocr-backend:
  deploy:
    replicas: 2
```

**Multiple GPUs:**

```yaml theme={null}
segmentation-backend:
  deploy:
    replicas: 6  # 3 per GPU for 2 GPUs

ocr-backend:
  deploy:
    replicas: 4  # 2 per GPU for 2 GPUs
```

### Model Parameters

Fine-tune model inference parameters:

```yaml theme={null}
environment:
  - OVERLAP_THRESHOLD=0.025  # Lower = stricter duplicate detection
  - SCORE_THRESHOLD=0.2      # Lower = more detections, higher recall
```

## Multi-GPU Configuration

To specify exact GPU allocation:

```yaml theme={null}
segmentation-backend:
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ['0', '1']  # Use first two GPUs
            capabilities: [gpu]
```

Or limit to a specific count:

```yaml theme={null}
devices:
  - driver: nvidia
    count: 2  # Use exactly 2 GPUs
    capabilities: [gpu]
```

## Monitoring GPU Usage

### Real-time Monitoring

Monitor GPU utilization in real-time:

```bash theme={null}
watch -n 1 nvidia-smi
```

### Per-Container GPU Stats

```bash theme={null}
# List containers using GPU
docker ps --format "table {{.Names}}\t{{.Status}}"

# Monitor specific container
docker stats <container_name>
```

### GPU Memory Usage

```bash theme={null}
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
```

## Switching to CPU-Only Mode

If GPUs are unavailable or for testing, use CPU mode:

```bash theme={null}
docker compose -f compose.yaml -f compose.cpu.yaml up -d
```

The CPU configuration removes GPU requirements and adjusts settings:

```yaml theme={null}
segmentation-backend:
  deploy:
    replicas: 6
    resources: {}  # No GPU reservation
  environment:
    - MAX_BATCH_SIZE=64
    - OMP_NUM_THREADS=12
    - MKL_NUM_THREADS=12
    - NUMEXPR_NUM_THREADS=12
```

<Warning>
  CPU mode is significantly slower. Expect 5-10x longer processing times compared to GPU acceleration.
</Warning>

## Troubleshooting

### GPU not detected

**Check NVIDIA driver:**

```bash theme={null}
nvidia-smi
```

If this fails, reinstall NVIDIA drivers.

**Verify Docker GPU access:**

```bash theme={null}
docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
```

**Check container toolkit:**

```bash theme={null}
nvidia-ctk --version
```

### Out of memory errors

1. **Reduce batch size:**
   ```yaml theme={null}
   environment:
     - MAX_BATCH_SIZE=2
   ```

2. **Decrease replica count:**
   ```yaml theme={null}
   deploy:
     replicas: 2  # Fewer workers per GPU
   ```

3. **Monitor GPU memory:**
   ```bash theme={null}
   nvidia-smi dmon -s mu
   ```

### Performance issues

1. **Check GPU utilization** - Should be >70% during processing
2. **Verify shared memory** - Ensure `/dev/shm` is mounted
3. **Review batch settings** - Optimize `MAX_BATCH_SIZE` and `BATCH_WAIT_TIME`
4. **Check for GPU throttling** - Monitor temperature with `nvidia-smi`

### Docker Compose GPU errors

**Error: "could not select device driver"**

```bash theme={null}
# Reconfigure Docker runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

**Error: "failed to initialize NVML"**

```bash theme={null}
# Restart NVIDIA persistence daemon
sudo systemctl restart nvidia-persistenced
```

## Best Practices

1. **Monitor GPU temperature** - Keep below 80°C for optimal performance
2. **Use appropriate batch sizes** - Balance throughput vs. memory usage
3. **Scale replicas carefully** - More replicas isn't always faster
4. **Regular driver updates** - Keep NVIDIA drivers current
5. **Shared memory mounting** - Always include `/dev/shm` volume

## Next Steps

* Configure [Environment Variables](/deployment/environment-variables)
* Learn about [Scaling](/deployment/scaling) GPU workers
* Return to [Docker Compose Deployment](/deployment/docker-compose)