> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# Segmentation

> Layout analysis and document segmentation strategies in Chunkr

Segmentation is the process of detecting and classifying layout elements in documents. Chunkr's segmentation engine identifies regions like tables, images, text blocks, and other document components.

## Segmentation Strategies

Chunkr supports two segmentation strategies:

<CodeGroup>
  ```json Configuration theme={null}
  {
    "segmentation_strategy": "LayoutAnalysis"
  }
  ```
</CodeGroup>

<Tabs>
  <Tab title="LayoutAnalysis">
    **Detect and classify layout elements**

    The `LayoutAnalysis` strategy uses computer vision models to detect document layout elements and classify them into specific types.

    **Features**:

    * Detects 11 different segment types
    * Provides bounding boxes for each element
    * Assigns confidence scores
    * Enables fine-grained chunking
    * Supports complex document layouts

    **Use cases**:

    * Academic papers with tables and formulas
    * Reports with mixed content
    * Documents requiring precise element extraction
    * When you need to process tables and images separately

    **Performance**: Adds minimal latency (batched processing)
  </Tab>

  <Tab title="Page">
    **Treat each page as a single segment**

    The `Page` strategy processes each page as one segment without detecting individual layout elements.

    **Features**:

    * Fastest processing
    * Simple output structure
    * Full-page OCR results
    * Basic chunking only

    **Use cases**:

    * Simple text documents
    * When speed is critical
    * Documents without complex layouts
    * When element-level processing isn't needed

    **Performance**: Fastest option, no layout analysis overhead
  </Tab>
</Tabs>

## Segment Types

When using `LayoutAnalysis`, Chunkr detects these segment types:

<AccordionGroup>
  <Accordion title="Title" icon="heading">
    Document or section titles

    **Characteristics**:

    * Large, prominent text
    * Usually at document/section start
    * High hierarchy level (3)
    * Often triggers new chunks

    **Example**: Main document title, chapter headings
  </Accordion>

  <Accordion title="SectionHeader" icon="section">
    Section and subsection headers

    **Characteristics**:

    * Medium hierarchy level (2)
    * Introduces new sections
    * Triggers chunk boundaries
    * Smaller than Title, larger than body text

    **Example**: Section headings, subsection titles
  </Accordion>

  <Accordion title="Text" icon="align-left">
    Regular body text paragraphs

    **Characteristics**:

    * Most common segment type
    * Hierarchy level 1
    * Combines with adjacent text in chunks

    **Example**: Paragraphs, body content
  </Accordion>

  <Accordion title="ListItem" icon="list">
    Bullet points and numbered list items

    **Characteristics**:

    * Individual list entries
    * Can be chunked together
    * Preserves list structure

    **Example**: Bullet points, enumerated items
  </Accordion>

  <Accordion title="Table" icon="table">
    Tabular data and structured content

    **Characteristics**:

    * Complex structure
    * Usually processed with LLM strategy
    * Can be paired with Caption
    * Cropped image available

    **Example**: Data tables, comparison charts

    **Default processing**: LLM-based HTML generation
  </Accordion>

  <Accordion title="Picture" icon="image">
    Images, figures, and diagrams

    **Characteristics**:

    * Visual content
    * Always cropped
    * Can be paired with Caption
    * May contain OCR results if text present

    **Example**: Photos, diagrams, charts

    **Default processing**: Image URL with optional description
  </Accordion>

  <Accordion title="Caption" icon="closed-captioning">
    Image and table captions

    **Characteristics**:

    * Describes associated visual element
    * Kept with paired Picture/Table in chunks
    * Usually smaller text below/above element

    **Example**: "Figure 1: Architecture diagram"
  </Accordion>

  <Accordion title="Formula" icon="function">
    Mathematical formulas and equations

    **Characteristics**:

    * Mathematical notation
    * Processed with LLM for LaTeX generation
    * May be inline or block-level

    **Example**: Equations, mathematical expressions

    **Default processing**: LLM-based LaTeX extraction
  </Accordion>

  <Accordion title="Footnote" icon="note">
    Footnotes and references

    **Characteristics**:

    * Small text at page bottom
    * References to main content
    * Usually numbered or marked

    **Example**: Citations, additional notes
  </Accordion>

  <Accordion title="PageHeader" icon="header">
    Page headers

    **Characteristics**:

    * Appears at top of pages
    * Often repetitive across pages
    * Can be excluded from chunks

    **Example**: Document title, chapter name

    **Note**: Excluded by default when `ignore_headers_and_footers: true`
  </Accordion>

  <Accordion title="PageFooter" icon="footer">
    Page footers

    **Characteristics**:

    * Appears at bottom of pages
    * Often contains page numbers
    * Can be excluded from chunks

    **Example**: Page numbers, copyright text

    **Note**: Excluded by default when `ignore_headers_and_footers: true`
  </Accordion>

  <Accordion title="Page" icon="file">
    Full page segment (only with Page strategy)

    **Characteristics**:

    * Entire page as one segment
    * Used when no layout analysis performed
    * Contains all page OCR results

    **When used**: Only with `segmentation_strategy: "Page"`
  </Accordion>
</AccordionGroup>

## Segment Structure

Each segment contains rich metadata:

```typescript theme={null}
interface Segment {
  // Identification
  segment_id: string;
  segment_type: SegmentType;
  
  // Position
  bbox: {
    left: number;
    top: number;
    width: number;
    height: number;
  };
  
  // Quality
  confidence?: number;  // 0.0 to 1.0
  
  // Content (generated in segment processing step)
  text: string;         // OCR-extracted text
  content: string;      // Formatted (HTML or Markdown)
  html: string;         // HTML representation
  markdown: string;     // Markdown representation
  llm?: string;         // LLM-generated content
  
  // Page context
  page_number: number;
  page_width: number;
  page_height: number;
  
  // Additional data
  image?: string;       // URL to cropped image
  ocr?: OCRResult[];    // Detailed OCR results
}
```

## How Segmentation Works

### 1. Object Detection

Chunkr uses object detection models to identify layout elements:

```mermaid theme={null}
graph LR
    A[Page Image] --> B[Object Detection Model]
    B --> C[Bounding Boxes]
    B --> D[Class Predictions]
    B --> E[Confidence Scores]
    C --> F[Segments]
    D --> F
    E --> F
```

The model outputs:

* **Bounding boxes**: `[left, top, width, height]` coordinates
* **Class predictions**: Integer class IDs (0-10)
* **Confidence scores**: Detection confidence (0.0-1.0)

### 2. Class Mapping

Class IDs are mapped to segment types:

```rust theme={null}
0 => Caption
1 => Footnote
2 => Formula
3 => ListItem
4 => PageFooter
5 => PageHeader
6 => Picture
7 => SectionHeader
8 => Table
9 => Text
10 => Title
```

### 3. OCR Assignment

OCR results are assigned to segments based on spatial overlap:

1. Add padding to segment bounding boxes
2. Calculate intersection area with each OCR result
3. Assign OCR result to segment with maximum overlap
4. Adjust OCR coordinates relative to segment

<Note>
  Segmentation padding is configurable via `segmentation_padding` in worker config. Default padding ensures OCR results near segment edges are captured.
</Note>

### 4. Fallback Handling

If no segments are detected:

```typescript theme={null}
// Creates a full-page segment
{
  segment_type: "Page",
  bbox: { left: 0, top: 0, width: page_width, height: page_height },
  confidence: 1.0,
  // ... all page OCR results
}
```

## Segmentation Quality

### Confidence Scores

Each segment includes a confidence score from the detection model:

* **> 0.9**: High confidence, very reliable
* **0.7 - 0.9**: Good confidence, usually accurate
* **0.5 - 0.7**: Medium confidence, may need review
* **\< 0.5**: Low confidence, likely false positive

<Tip>
  Filter segments by confidence threshold if you need high-precision extraction:

  ```typescript theme={null}
  const highConfidenceSegments = segments.filter(s => s.confidence > 0.8);
  ```
</Tip>

### Accuracy Factors

**Improves accuracy**:

* ✅ High-resolution images (`high_resolution: true`)
* ✅ Clear, well-formatted documents
* ✅ Standard layouts (papers, reports)
* ✅ Good contrast and quality scans

**May reduce accuracy**:

* ❌ Low-resolution or blurry images
* ❌ Unusual layouts or designs
* ❌ Heavily stylized documents
* ❌ Poor scan quality

## Batched Processing

Segmentation uses batched processing for efficiency:

```typescript theme={null}
// Pages are processed in batches
const batchSize = config.segmentation_batch_size;  // e.g., 10 pages

// Reduces API calls and improves throughput
batches.forEach(async (batch) => {
  const segments = await performSegmentationBatch(batch);
});
```

**Benefits**:

* Faster processing for multi-page documents
* Efficient resource utilization
* Reduced network overhead

## Configuration Examples

### Academic Papers

```json theme={null}
{
  "segmentation_strategy": "LayoutAnalysis",
  "high_resolution": true,
  "segment_processing": {
    "Table": { "strategy": "LLM", "format": "Html" },
    "Formula": { "strategy": "LLM", "format": "Markdown" },
    "Picture": { "crop_image": "All" }
  }
}
```

### Simple Documents

```json theme={null}
{
  "segmentation_strategy": "Page",
  "high_resolution": false,
  "segment_processing": {
    "Page": { "strategy": "Auto", "format": "Markdown" }
  }
}
```

### Reports with Tables

```json theme={null}
{
  "segmentation_strategy": "LayoutAnalysis",
  "segment_processing": {
    "Table": { 
      "strategy": "LLM", 
      "format": "Html",
      "crop_image": "Auto"
    },
    "Text": { "strategy": "Auto", "format": "Markdown" }
  },
  "chunk_processing": {
    "ignore_headers_and_footers": true
  }
}
```

## Error Handling

With `error_handling: "Continue"`, segmentation failures fall back gracefully:

```typescript theme={null}
// If layout analysis fails
try {
  segments = await layoutAnalysis(page);
} catch (error) {
  console.log('Layout analysis failed, using page segmentation');
  // Falls back to full-page segment
  segments = [createPageSegment(page, ocrResults)];
}
```

## Next Steps

<CardGroup cols={2}>
  <Card title="OCR Strategies" icon="eye" href="/concepts/ocr">
    Learn about text extraction methods
  </Card>

  <Card title="Segment Processing" icon="wand-magic-sparkles" href="/concepts/pipelines#segment-processing-configuration">
    Configure content generation per segment type
  </Card>

  <Card title="Chunking" icon="scissors" href="/concepts/chunking">
    Understand how segments are combined
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference">
    See complete API documentation
  </Card>
</CardGroup>