> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Models

> Configure LLM models for document processing with OpenAI, Google AI, OpenRouter, or self-hosted solutions

## Overview

Chunkr supports any OpenAI-compatible API for LLM processing. You can configure multiple models with different providers, set rate limits, and specify default and fallback models.

## Configuration File

LLM models are configured in a `models.yaml` file. Copy `models.example.yaml` to get started:

```bash theme={null}
cp models.example.yaml models.yaml
```

Set the path to your models configuration file using the environment variable:

```bash theme={null}
LLM__MODELS_PATH=./models.yaml
```

## Model Configuration Structure

Each model in the configuration requires:

<ParamField path="id" type="string" required>
  Unique identifier for the model. Use this ID in POST and PATCH requests to reference the model.
</ParamField>

<ParamField path="model" type="string" required>
  The model name/identifier used by the provider (e.g., `gpt-4o`, `gemini-2.0-flash-lite`).
</ParamField>

<ParamField path="provider_url" type="string" required>
  The API endpoint URL for the provider's chat completions service.
</ParamField>

<ParamField path="api_key" type="string" required>
  Your API key for authentication with the provider.
</ParamField>

<ParamField path="default" type="boolean">
  Mark one model as the default. This model is used when no specific model is requested.
</ParamField>

<ParamField path="fallback" type="boolean">
  Mark one model as the fallback. Used when `FallbackStrategy::Default` is configured.
</ParamField>

<ParamField path="rate-limit" type="integer">
  Optional rate limit in requests per minute for this model.
</ParamField>

<Note>
  You must configure exactly **one default model** and **one fallback model** (they can be the same model).
</Note>

## Provider Examples

<Tabs>
  <Tab title="OpenAI">
    ```yaml models.yaml theme={null}
    models:
      - id: gpt-4o
        model: gpt-4o
        provider_url: https://api.openai.com/v1/chat/completions
        api_key: "your_openai_api_key_here"
        default: true
        rate-limit: 200
    ```

    <Note>
      Rate limits help prevent API quota exhaustion. OpenAI has different rate limits based on your tier.
    </Note>
  </Tab>

  <Tab title="Google AI Studio">
    ```yaml models.yaml theme={null}
    models:
      - id: gemini-2.0-flash-lite
        model: gemini-2.0-flash-lite
        provider_url: https://generativelanguage.googleapis.com/v1beta/openai/chat/completions
        api_key: "your_google_ai_studio_api_key_here"
        fallback: true
    ```

    Google AI Studio provides an OpenAI-compatible endpoint for Gemini models.
  </Tab>

  <Tab title="OpenRouter">
    ```yaml models.yaml theme={null}
    models:
      - id: gemini-pro-1.5
        model: google/gemini-pro-1.5
        provider_url: https://openrouter.ai/api/v1/chat/completions
        api_key: "your_openrouter_api_key_here"
    ```

    OpenRouter provides access to multiple LLM providers through a unified API. Use the provider-prefixed model names (e.g., `google/gemini-pro-1.5`).
  </Tab>

  <Tab title="Self-Hosted">
    ```yaml models.yaml theme={null}
    models:
      - id: local-llm
        model: mistral-7b
        provider_url: http://localhost:8000/v1/chat/completions
        api_key: "your_local_api_key_or_leave_empty_if_not_required"
    ```

    Any OpenAI-compatible server works, including:

    * vLLM
    * LocalAI
    * Ollama (with OpenAI compatibility layer)
    * Text Generation Inference

    <Warning>
      Leave the `api_key` empty or use a placeholder if your self-hosted server doesn't require authentication.
    </Warning>
  </Tab>
</Tabs>

## Complete Example

Here's a complete `models.yaml` with multiple providers:

```yaml models.yaml theme={null}
models:
  # Primary model (OpenAI)
  - id: gpt-4o
    model: gpt-4o
    provider_url: https://api.openai.com/v1/chat/completions
    api_key: "sk-..."
    default: true
    rate-limit: 200

  # Fallback model (Google AI)
  - id: gemini-2.0-flash-lite
    model: gemini-2.0-flash-lite
    provider_url: https://generativelanguage.googleapis.com/v1beta/openai/chat/completions
    api_key: "AIza..."
    fallback: true

  # Additional model via OpenRouter
  - id: gemini-pro-1.5
    model: google/gemini-pro-1.5
    provider_url: https://openrouter.ai/api/v1/chat/completions
    api_key: "sk-or-..."

  # Self-hosted model
  - id: local-llm
    model: mistral-7b
    provider_url: http://localhost:8000/v1/chat/completions
    api_key: ""
```

## Using Models in Requests

### Specifying a Model

Reference your configured model by its `id` in the `llm_processing` configuration:

<CodeGroup>
  ```json POST Request theme={null}
  {
    "llm_processing": {
      "model_id": "gpt-4o",
      "temperature": 0.0,
      "max_completion_tokens": 4096
    }
  }
  ```

  ```python Python SDK theme={null}
  from chunkr_ai import Chunkr

  chunkr = Chunkr()

  chunkr.upload(
      file_path="document.pdf",
      llm_processing={
          "model_id": "gpt-4o",
          "temperature": 0.0,
          "max_completion_tokens": 4096
      }
  )
  ```
</CodeGroup>

### Default Model

If you don't specify a `model_id`, the model marked with `default: true` is used automatically:

```json theme={null}
{
  "llm_processing": {
    "temperature": 0.0
  }
}
```

## Fallback Strategy

Configure how Chunkr handles LLM failures:

<ParamField path="fallback_strategy" type="enum" default="Default">
  * `None`: No fallback, task fails on LLM error
  * `Default`: Use the model marked with `fallback: true`
  * `Model("model-id")`: Use a specific model as fallback
</ParamField>

<CodeGroup>
  ```json No Fallback theme={null}
  {
    "llm_processing": {
      "model_id": "gpt-4o",
      "fallback_strategy": "None"
    }
  }
  ```

  ```json Default Fallback theme={null}
  {
    "llm_processing": {
      "model_id": "gpt-4o",
      "fallback_strategy": "Default"
    }
  }
  ```

  ```json Specific Fallback theme={null}
  {
    "llm_processing": {
      "model_id": "gpt-4o",
      "fallback_strategy": {
        "Model": "gemini-2.0-flash-lite"
      }
    }
  }
  ```
</CodeGroup>

## Rate Limiting

Rate limits prevent exceeding provider quotas:

```yaml theme={null}
models:
  - id: gpt-4o
    model: gpt-4o
    provider_url: https://api.openai.com/v1/chat/completions
    api_key: "sk-..."
    rate-limit: 200  # Max 200 requests per minute
```

Chunkr automatically queues requests when approaching the limit.

## Best Practices

<Warning>
  **Never commit API keys to version control.** Use environment variables or secure secret management.
</Warning>

1. **Use different models for different purposes**
   * Fast model (e.g., GPT-4o-mini, Gemini Flash) for simple segments
   * High-quality model (e.g., GPT-4o) for complex tables and formulas

2. **Configure appropriate rate limits**
   * Check your provider's rate limits
   * Set conservative limits to avoid throttling

3. **Always configure a fallback**
   * Ensures processing continues if primary model fails
   * Use a reliable, fast model as fallback

4. **Test your configuration**
   ```bash theme={null}
   # Verify your models.yaml is valid
   chunkr validate-models
   ```

## Troubleshooting

### Model Not Found

If you get a "model not found" error:

1. Verify the model `id` exists in your `models.yaml`
2. Check that `LLM__MODELS_PATH` points to the correct file
3. Restart the Chunkr service after updating `models.yaml`

### Authentication Errors

1. Verify your API key is correct and not expired
2. Check that the API key has the necessary permissions
3. For self-hosted models, verify the endpoint is accessible

### Rate Limit Errors

1. Lower the `rate-limit` value in your configuration
2. Upgrade your provider tier for higher limits
3. Configure a fallback model with higher limits