> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Chunking Strategies

> Configure semantic chunking and segment processing to optimize your documents for RAG and embedding workflows

Chunkr provides flexible chunking strategies to combine document segments into optimal chunks for retrieval-augmented generation (RAG) and embedding systems.

## Understanding Chunks vs Segments

<Note>
  **Segments** are individual layout elements (paragraphs, tables, images) detected during analysis. **Chunks** are groups of segments combined for embedding.
</Note>

* **Segment**: A single structural element (e.g., one paragraph, one table)
* **Chunk**: One or more segments grouped together based on your `target_length`

## Chunk Processing Configuration

Configure chunking behavior through the `chunk_processing` parameter:

```python Python theme={null}
import requests

response = requests.post(
    "https://api.chunkr.ai/api/v1/task/parse",
    headers={"Authorization": "YOUR_API_KEY"},
    json={
        "file": "https://example.com/document.pdf",
        "chunk_processing": {
            "target_length": 512,
            "ignore_headers_and_footers": true,
            "tokenizer": "Word"
        }
    }
)
```

### Target Length

Controls the approximate size of each chunk:

<Tabs>
  <Tab title="512 tokens (Default)">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 512
      }
    }
    ```

    **Best for:** Standard RAG applications and most embedding models
  </Tab>

  <Tab title="1024 tokens">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 1024
      }
    }
    ```

    **Best for:** Models with larger context windows, more comprehensive chunks
  </Tab>

  <Tab title="256 tokens">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 256
      }
    }
    ```

    **Best for:** Fine-grained retrieval, smaller embedding models
  </Tab>

  <Tab title="0 (Single Segment)">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 0
      }
    }
    ```

    Each chunk contains exactly one segment. **Best for:** When you need segment-level granularity
  </Tab>
</Tabs>

<Note>
  Chunkr never breaks segments apart - they remain intact within chunks. The `target_length` is the maximum size, and Chunkr will fit as many complete segments as possible without exceeding it.
</Note>

### Tokenizer Selection

Choose how text length is measured:

<Tabs>
  <Tab title="Word (Default)">
    ```json theme={null}
    {
      "chunk_processing": {
        "tokenizer": "Word"
      }
    }
    ```

    Splits text by word boundaries. Simple and fast.
  </Tab>

  <Tab title="Cl100kBase">
    ```json theme={null}
    {
      "chunk_processing": {
        "tokenizer": "Cl100kBase"
      }
    }
    ```

    OpenAI's tokenizer for GPT-3.5, GPT-4, and text-embedding-ada-002.
  </Tab>

  <Tab title="XlmRobertaBase">
    ```json theme={null}
    {
      "chunk_processing": {
        "tokenizer": "xlm-roberta-base"
      }
    }
    ```

    For RoBERTa-based multilingual models.
  </Tab>

  <Tab title="BertBaseUncased">
    ```json theme={null}
    {
      "chunk_processing": {
        "tokenizer": "bert-base-uncased"
      }
    }
    ```

    For BERT-based models.
  </Tab>

  <Tab title="Custom HuggingFace Tokenizer">
    ```json theme={null}
    {
      "chunk_processing": {
        "tokenizer": "Qwen/Qwen-tokenizer"
      }
    }
    ```

    Any valid HuggingFace tokenizer ID can be specified.
  </Tab>
</Tabs>

### Ignore Headers and Footers

<Warning>
  Headers and footers can break reading order across pages. It's recommended to ignore them.
</Warning>

```json theme={null}
{
  "chunk_processing": {
    "ignore_headers_and_footers": true
  }
}
```

**Default:** `true`

When enabled, page headers and footers are excluded from chunking but still available in the output if needed.

## Segment Processing

Control how individual segments are processed and what content is included in chunks.

### Content Format Selection

Each segment type can be configured with a preferred output format:

```python Python theme={null}
response = requests.post(
    "https://api.chunkr.ai/api/v1/task/parse",
    headers={"Authorization": "YOUR_API_KEY"},
    json={
        "file": "https://example.com/document.pdf",
        "segment_processing": {
            "Text": {
                "format": "Markdown",
                "strategy": "Auto"
            },
            "Table": {
                "format": "Html",
                "strategy": "LLM"
            },
            "Picture": {
                "format": "Markdown",
                "strategy": "Auto"
            }
        }
    }
)
```

### Format Options

<Tabs>
  <Tab title="Markdown">
    ```json theme={null}
    {
      "format": "Markdown"
    }
    ```

    Text-based format, good for most content. Default for most segment types.
  </Tab>

  <Tab title="Html">
    ```json theme={null}
    {
      "format": "Html"
    }
    ```

    Preserves complex structure. Default for tables.
  </Tab>
</Tabs>

### Generation Strategy

<Tabs>
  <Tab title="Auto">
    ```json theme={null}
    {
      "strategy": "Auto"
    }
    ```

    Uses heuristic-based generation. Fast and efficient.
  </Tab>

  <Tab title="LLM">
    ```json theme={null}
    {
      "strategy": "LLM"
    }
    ```

    Uses Chunkr's fine-tuned models for enhanced quality. See [VLM Processing](/guides/using-vlm) for details.
  </Tab>
</Tabs>

### Embed Sources

Control which content is included in the chunk's `embed` field:

```python Python theme={null}
{
    "segment_processing": {
        "Text": {
            "format": "Markdown",
            "embed_sources": ["Content", "LLM"]
        },
        "Table": {
            "format": "Html",
            "embed_sources": ["Content"]
        }
    }
}
```

**Available sources:**

* `Content`: The generated content (HTML or Markdown based on format)
* `LLM`: Custom LLM-generated output (when configured)
* `HTML`: ⚠️ Deprecated - use `Content` with `format: Html`
* `Markdown`: ⚠️ Deprecated - use `Content` with `format: Markdown`

<Note>
  The order of sources in the array determines the sequence in the embed field. For example, `["Content", "LLM"]` means content appears first, followed by LLM output.
</Note>

## Complete Example

```python Python theme={null}
import requests
import time

config = {
    "file": "https://example.com/research-paper.pdf",
    "segmentation_strategy": "LayoutAnalysis",
    "ocr_strategy": "All",
    "chunk_processing": {
        "target_length": 512,
        "ignore_headers_and_footers": true,
        "tokenizer": "Cl100kBase"  # For OpenAI embeddings
    },
    "segment_processing": {
        "Title": {
            "format": "Markdown",
            "strategy": "Auto",
            "embed_sources": ["Content"]
        },
        "Text": {
            "format": "Markdown",
            "strategy": "Auto",
            "embed_sources": ["Content"]
        },
        "Table": {
            "format": "Html",
            "strategy": "LLM",  # Use VLM for better table processing
            "embed_sources": ["Content"]
        },
        "Picture": {
            "format": "Markdown",
            "strategy": "Auto",
            "crop_image": "All",
            "embed_sources": ["Content"]
        },
        "Formula": {
            "format": "Markdown",
            "strategy": "LLM",
            "embed_sources": ["Content"]
        }
    }
}

# Create task
response = requests.post(
    "https://api.chunkr.ai/api/v1/task/parse",
    headers={"Authorization": "YOUR_API_KEY"},
    json=config
)
task = response.json()
task_id = task["task_id"]

# Poll for completion
while True:
    response = requests.get(
        f"https://api.chunkr.ai/api/v1/task/{task_id}",
        headers={"Authorization": "YOUR_API_KEY"}
    )
    task = response.json()
    
    if task["status"] == "Succeeded":
        break
    elif task["status"] == "Failed":
        raise Exception("Task failed")
    
    time.sleep(2)

# Process chunks
for chunk in task["output"]["chunks"]:
    print(f"\nChunk {chunk['chunk_id']}")
    print(f"Length: {chunk['chunk_length']} tokens")
    print(f"Segments: {len(chunk['segments'])}")
    print(f"\nEmbed content:\n{chunk['embed'][:200]}...")
```

## Segment Types

All available segment types you can configure:

| Segment Type    | Default Format | Default Strategy | Description                          |
| --------------- | -------------- | ---------------- | ------------------------------------ |
| `Title`         | Markdown       | Auto             | Document titles                      |
| `SectionHeader` | Markdown       | Auto             | Section headings                     |
| `Text`          | Markdown       | Auto             | Body paragraphs                      |
| `ListItem`      | Markdown       | Auto             | Bullet/numbered lists                |
| `Table`         | Html           | LLM              | Tables and grids                     |
| `Picture`       | Markdown       | LLM              | Images and figures                   |
| `Caption`       | Markdown       | Auto             | Image/table captions                 |
| `Formula`       | Markdown       | LLM              | Mathematical formulas                |
| `Footnote`      | Markdown       | Auto             | Footnotes                            |
| `PageHeader`    | Markdown       | Auto             | Page headers                         |
| `PageFooter`    | Markdown       | Auto             | Page footers                         |
| `Page`          | Markdown       | LLM              | Full page (when using Page strategy) |

## Chunking Strategies by Use Case

<Accordion title="RAG for Q&A Systems">
  ```json theme={null}
  {
    "chunk_processing": {
      "target_length": 512,
      "tokenizer": "Cl100kBase",
      "ignore_headers_and_footers": true
    }
  }
  ```

  Balanced chunks for good context and retrieval precision.
</Accordion>

<Accordion title="Long-Form Document Analysis">
  ```json theme={null}
  {
    "chunk_processing": {
      "target_length": 1024,
      "tokenizer": "Cl100kBase",
      "ignore_headers_and_footers": true
    }
  }
  ```

  Larger chunks preserve more context.
</Accordion>

<Accordion title="Fine-Grained Semantic Search">
  ```json theme={null}
  {
    "chunk_processing": {
      "target_length": 256,
      "tokenizer": "Word",
      "ignore_headers_and_footers": true
    }
  }
  ```

  Smaller chunks for precise retrieval.
</Accordion>

<Accordion title="Segment-Level Processing">
  ```json theme={null}
  {
    "chunk_processing": {
      "target_length": 0,
      "tokenizer": "Word",
      "ignore_headers_and_footers": false
    }
  }
  ```

  One segment per chunk for maximum granularity.
</Accordion>

## Best Practices

1. **Match tokenizer to your embedding model** - Use `Cl100kBase` for OpenAI models, the corresponding tokenizer for others
2. **Start with defaults** - The default 512 tokens works well for most cases
3. **Consider your retrieval strategy** - Smaller chunks for precise retrieval, larger for context-rich answers
4. **Use segment types strategically** - Configure different formats for different content (HTML for tables, Markdown for text)
5. **Test with your data** - Optimal settings vary by document type and use case

## Next Steps

* Learn about [VLM processing](/guides/using-vlm) for enhanced content generation
* See [processing documents](/guides/processing-documents) for core API usage
* Review the [migration guide](/guides/migration-guide) for recent API changes