> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision-Language Model Processing

> Use VLMs to enhance document understanding with AI-powered content generation and analysis

Chunkr integrates Vision-Language Models (VLMs) to provide enhanced content generation, intelligent table processing, formula extraction, and custom segment analysis using visual understanding.

## Overview

VLM processing in Chunkr allows you to:

* Generate enhanced content using fine-tuned models (`strategy: LLM`)
* Add custom LLM-powered analysis to any segment type
* Process complex tables with better structure understanding
* Extract mathematical formulas as LaTeX
* Generate custom descriptions for images and diagrams

## Generation Strategies

### Auto Strategy

Uses heuristic-based generation - fast and efficient:

```json theme={null}
{
  "segment_processing": {
    "Text": {
      "strategy": "Auto"
    }
  }
}
```

**Best for:** Standard text, lists, simple formatting

### LLM Strategy

Uses Chunkr's fine-tuned Vision-Language Models:

```json theme={null}
{
  "segment_processing": {
    "Table": {
      "strategy": "LLM"
    }
  }
}
```

**Best for:** Tables, formulas, pictures, complex layouts

<Note>
  LLM strategy uses the page image as context to generate more accurate structured content
</Note>

## Default VLM Configuration

Some segment types use LLM strategy by default:

| Segment Type | Default Strategy | Reason                         |
| ------------ | ---------------- | ------------------------------ |
| `Table`      | LLM              | Better structure understanding |
| `Formula`    | LLM              | LaTeX extraction from images   |
| `Picture`    | LLM              | Visual content description     |
| `Page`       | LLM              | Full page understanding        |

## Custom LLM Prompts

Add custom LLM-powered analysis to any segment using the `llm` field:

```python Python theme={null}
import requests

response = requests.post(
    "https://api.chunkr.ai/api/v1/task/parse",
    headers={"Authorization": "YOUR_API_KEY"},
    json={
        "file": "https://example.com/document.pdf",
        "segment_processing": {
            "Picture": {
                "format": "Markdown",
                "strategy": "Auto",
                "llm": "Describe this image in detail, focusing on key visual elements and any text present."
            },
            "Table": {
                "format": "Html",
                "strategy": "LLM",
                "llm": "Extract the data and create a summary of key insights from this table."
            },
            "Text": {
                "format": "Markdown",
                "strategy": "Auto",
                "llm": "Summarize this section in 2-3 sentences."
            }
        }
    }
)
```

The LLM output is stored in `segment.llm` and can be included in chunks via `embed_sources`.

## Extended Context

Use the full page image as context for LLM generation:

```json theme={null}
{
  "segment_processing": {
    "Table": {
      "format": "Html",
      "strategy": "LLM",
      "extended_context": true
    }
  }
}
```

**Default:** `false`

<Warning>
  Extended context provides better results but increases processing time and token usage
</Warning>

## Configuring LLM Models

Chunkr supports global LLM configuration to control which models are used for VLM processing.

### LLM Processing Configuration

```python Python theme={null}
{
    "llm_processing": {
        "model_id": "gpt-4o",
        "fallback_strategy": "Default",
        "max_completion_tokens": 2048,
        "temperature": 0.0
    }
}
```

<Tabs>
  <Tab title="Model ID">
    ```json theme={null}
    {
      "llm_processing": {
        "model_id": "gpt-4o"
      }
    }
    ```

    Specify which model to use. Check documentation for available models.
  </Tab>

  <Tab title="Fallback Strategy">
    ```json theme={null}
    {
      "llm_processing": {
        "fallback_strategy": "Default"
      }
    }
    ```

    Options:

    * `None`: No fallback
    * `Default`: Use system default fallback
    * `Model("model-id")`: Use specific model as fallback
  </Tab>

  <Tab title="Max Completion Tokens">
    ```json theme={null}
    {
      "llm_processing": {
        "max_completion_tokens": 2048
      }
    }
    ```

    Control the maximum length of LLM-generated content.
  </Tab>

  <Tab title="Temperature">
    ```json theme={null}
    {
      "llm_processing": {
        "temperature": 0.0
      }
    }
    ```

    Control randomness (0.0 = deterministic, 1.0 = creative).
  </Tab>
</Tabs>

## Image Cropping

Control when segment images are cropped and stored:

<Tabs>
  <Tab title="Auto">
    ```json theme={null}
    {
      "crop_image": "Auto"
    }
    ```

    **Default** for most segments - Only crop when needed for post-processing
  </Tab>

  <Tab title="All">
    ```json theme={null}
    {
      "crop_image": "All"
    }
    ```

    **Default** for Pictures - Always crop and store segment images
  </Tab>
</Tabs>

Cropped images are available in `segment.image` as presigned URLs.

## Complete VLM Example

```python Python theme={null}
import requests
import time

# Advanced VLM configuration
config = {
    "file": "https://example.com/research-paper.pdf",
    "segmentation_strategy": "LayoutAnalysis",
    "ocr_strategy": "All",
    "high_resolution": true,  # Better image quality for VLM
    "llm_processing": {
        "model_id": "gpt-4o",
        "fallback_strategy": "Default",
        "temperature": 0.0
    },
    "segment_processing": {
        "Table": {
            "format": "Html",
            "strategy": "LLM",
            "crop_image": "All",
            "extended_context": true,
            "llm": "Convert this table to structured HTML and provide a brief summary of the data.",
            "embed_sources": ["Content", "LLM"]
        },
        "Picture": {
            "format": "Markdown",
            "strategy": "Auto",
            "crop_image": "All",
            "llm": "Describe this figure, including any charts, graphs, or diagrams. Explain what it illustrates.",
            "embed_sources": ["Content", "LLM"]
        },
        "Formula": {
            "format": "Markdown",
            "strategy": "LLM",
            "crop_image": "All",
            "llm": "Extract this formula as LaTeX and explain what it represents.",
            "embed_sources": ["Content", "LLM"]
        },
        "Text": {
            "format": "Markdown",
            "strategy": "Auto",
            "llm": "Provide a concise summary of this section.",
            "embed_sources": ["Content"]
        }
    }
}

# Create task
response = requests.post(
    "https://api.chunkr.ai/api/v1/task/parse",
    headers={"Authorization": "YOUR_API_KEY"},
    json=config
)
task = response.json()
task_id = task["task_id"]

# Poll for completion
while True:
    response = requests.get(
        f"https://api.chunkr.ai/api/v1/task/{task_id}",
        headers={"Authorization": "YOUR_API_KEY"}
    )
    task = response.json()
    
    if task["status"] == "Succeeded":
        break
    elif task["status"] == "Failed":
        raise Exception("Task failed")
    
    time.sleep(2)

# Access VLM-processed content
for chunk in task["output"]["chunks"]:
    for segment in chunk["segments"]:
        print(f"\nSegment Type: {segment['segment_type']}")
        print(f"Content: {segment['content'][:200]}...")
        
        # Access custom LLM output if configured
        if segment.get('llm'):
            print(f"LLM Analysis: {segment['llm'][:200]}...")
        
        # Access cropped image if available
        if segment.get('image'):
            print(f"Image URL: {segment['image']}")
```

## Accessing VLM Output

VLM-generated content is available in multiple fields:

```python Python theme={null}
for segment in chunk["segments"]:
    # Primary content (based on format and strategy)
    content = segment["content"]  # HTML or Markdown
    
    # Original OCR text
    text = segment["text"]
    
    # Custom LLM output (if configured)
    if "llm" in segment and segment["llm"]:
        llm_analysis = segment["llm"]
    
    # Specific format outputs (backward compatibility)
    html = segment["html"]
    markdown = segment["markdown"]
    
    # Cropped image URL (if available)
    if "image" in segment and segment["image"]:
        image_url = segment["image"]
```

## Embed Sources with VLM

Control which VLM outputs are included in chunk embeddings:

```python Python theme={null}
{
    "segment_processing": {
        "Table": {
            "format": "Html",
            "strategy": "LLM",
            "llm": "Summarize key data points from this table.",
            "embed_sources": ["Content", "LLM"]
        }
    }
}
```

**Result:** The chunk's `embed` field will contain:

1. The HTML table structure (`Content`)
2. The LLM summary (`LLM`)

<Note>
  Order matters! Sources appear in the embed field in the order specified.
</Note>

## Use Cases

<Accordion title="Enhanced Table Extraction">
  ```python theme={null}
  {
      "Table": {
          "format": "Html",
          "strategy": "LLM",
          "extended_context": true,
          "embed_sources": ["Content"]
      }
  }
  ```

  VLM provides better table structure understanding, especially for complex or merged cells.
</Accordion>

<Accordion title="Mathematical Formula Extraction">
  ```python theme={null}
  {
      "Formula": {
          "format": "Markdown",
          "strategy": "LLM",
          "crop_image": "All",
          "llm": "Extract this formula as LaTeX.",
          "embed_sources": ["Content", "LLM"]
      }
  }
  ```

  Convert formula images to LaTeX for better searchability and rendering.
</Accordion>

<Accordion title="Image Description for RAG">
  ```python theme={null}
  {
      "Picture": {
          "format": "Markdown",
          "strategy": "Auto",
          "crop_image": "All",
          "llm": "Describe this image in detail for a text-based search system.",
          "embed_sources": ["LLM"]
      }
  }
  ```

  Make images searchable by generating detailed text descriptions.
</Accordion>

<Accordion title="Document Summarization">
  ```python theme={null}
  {
      "Text": {
          "format": "Markdown",
          "strategy": "Auto",
          "llm": "Summarize this section in 2-3 sentences, highlighting key points.",
          "embed_sources": ["LLM", "Content"]
      }
  }
  ```

  Provide summaries alongside full text for multi-level retrieval.
</Accordion>

## Performance Considerations

<Warning>
  VLM processing increases latency and cost. Use strategically for best results.
</Warning>

**Tips:**

* Use `strategy: LLM` only for complex content (tables, formulas, pictures)
* Use `strategy: Auto` for simple text segments
* Set `extended_context: false` unless you need full page context
* Configure `max_completion_tokens` appropriately to control costs
* Use `temperature: 0.0` for consistent, deterministic output

## Error Handling with VLM

Control how VLM errors are handled:

```json theme={null}
{
  "error_handling": "Continue"
}
```

**Options:**

* `Fail`: Stop processing on any error (default)
* `Continue`: Continue processing despite LLM refusals or failures

<Note>
  Use `Continue` for fault-tolerant processing when some VLM failures are acceptable
</Note>

## Best Practices

1. **Use VLM strategically** - Only apply to segments that benefit from visual understanding
2. **Write clear prompts** - Be specific about what you want in custom `llm` prompts
3. **Enable high resolution** - Set `high_resolution: true` for better VLM input quality
4. **Test and iterate** - Experiment with different prompts and configurations
5. **Monitor costs** - VLM processing can be expensive at scale
6. **Choose appropriate models** - Different models have different strengths and costs

## Next Steps

* Learn about [custom chunking](/guides/custom-chunking) to optimize VLM output usage
* See [processing documents](/guides/processing-documents) for core API usage
* Review the [migration guide](/guides/migration-guide) for content generation changes