> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipelines

> Document processing pipeline architecture and configuration

Chunkr's pipeline orchestrates the entire document processing workflow, from upload to final output. Understanding the pipeline helps you optimize processing for your use case.

## Pipeline Architecture

The Chunkr pipeline consists of sequential steps executed in order:

```mermaid theme={null}
flowchart TD
    A[Initialize Pipeline] --> B{Convert to Images}
    B --> C{Chunkr Analysis}
    C --> D[Crop Segments]
    D --> E[Segment Processing]
    E --> F[Chunking]
    F --> G[Complete]
    
    style A fill:#e1f5ff
    style G fill:#d4edda
```

## Pipeline Steps

Each step in the pipeline is represented by the `PipelineStep` enum:

### Available Steps

<CodeGroup>
  ```rust Pipeline Steps theme={null}
  pub enum PipelineStep {
      ConvertToImages,
      ChunkrAnalysis,
      Crop,
      SegmentProcessing,
      Chunking,
  }
  ```
</CodeGroup>

<AccordionGroup>
  <Accordion title="ConvertToImages" icon="image">
    Converts PDF pages into high-quality JPEG images for processing.

    **When it runs**: After task initialization

    **What it does**:

    * Converts each PDF page to a JPEG image
    * Applies scaling factor based on `high_resolution` setting
    * Stores images in memory for subsequent steps

    **Configuration**:

    * `high_resolution: true` → Higher quality, slower (\~7s per page)
    * `high_resolution: false` → Standard quality, faster
  </Accordion>

  <Accordion title="ChunkrAnalysis" icon="magnifying-glass">
    Performs OCR and layout analysis on document pages.

    **When it runs**: After image conversion

    **What it does**:

    * Extracts text using OCR based on `ocr_strategy`
    * Detects layout elements based on `segmentation_strategy`
    * Creates initial segments with bounding boxes and OCR results
    * Scales coordinates according to resolution settings

    **Output**: Array of segments with type classifications and OCR data
  </Accordion>

  <Accordion title="Crop" icon="crop">
    Crops segment images from page images for visual elements.

    **When it runs**: After analysis, before segment processing

    **What it does**:

    * Extracts image regions for each segment based on bounding boxes
    * Stores cropped images for pictures, tables, and other visual elements
    * Applies padding configured in `segmentation_padding`

    **Configuration**: Controlled per segment type via `crop_image` setting
  </Accordion>

  <Accordion title="SegmentProcessing" icon="wand-magic-sparkles">
    Post-processes segments to generate structured content.

    **When it runs**: After cropping

    **What it does**:

    * Generates HTML/Markdown from segments using configured strategy
    * Applies LLM models for complex elements (tables, formulas)
    * Creates the `content`, `html`, `markdown`, and optionally `llm` fields
    * Processes each segment type according to its configuration

    **See**: [Segment Processing Configuration](#segment-processing-configuration)
  </Accordion>

  <Accordion title="Chunking" icon="scissors">
    Combines segments into semantically meaningful chunks.

    **When it runs**: Final step

    **What it does**:

    * Groups segments based on hierarchy and target length
    * Generates embed text for each chunk
    * Calculates chunk lengths using specified tokenizer
    * Optionally filters out headers and footers

    **See**: [Chunking](/concepts/chunking)
  </Accordion>
</AccordionGroup>

## Pipeline Configuration

The pipeline behavior is controlled through the `Configuration` object:

### Core Settings

```json theme={null}
{
  "segmentation_strategy": "LayoutAnalysis",
  "ocr_strategy": "Auto",
  "high_resolution": true,
  "error_handling": "Fail",
  "expires_in": 3600
}
```

<ParamField path="segmentation_strategy" type="SegmentationStrategy" default="LayoutAnalysis">
  Controls layout analysis approach:

  * `LayoutAnalysis`: Detect layout elements (tables, images, etc.)
  * `Page`: Treat each page as a single segment

  See [Segmentation](/concepts/segmentation) for details.
</ParamField>

<ParamField path="ocr_strategy" type="OcrStrategy" default="All">
  Controls OCR behavior:

  * `All`: Process all pages with OCR (\~0.5s per page)
  * `Auto`: Use existing text layer when available

  See [OCR](/concepts/ocr) for details.
</ParamField>

<ParamField path="high_resolution" type="boolean" default="true">
  Use high-resolution images for processing.

  * `true`: Better accuracy, \~7s latency per page
  * `false`: Faster processing, standard quality
</ParamField>

<ParamField path="error_handling" type="ErrorHandlingStrategy" default="Fail">
  How to handle processing errors:

  * `Fail`: Stop processing on any error
  * `Continue`: Attempt to continue despite non-critical errors
</ParamField>

<ParamField path="expires_in" type="number" default="null">
  Seconds until task is automatically deleted. Expired tasks cannot be accessed or updated.
</ParamField>

### Segment Processing Configuration

Each segment type can be configured independently:

```json theme={null}
{
  "segment_processing": {
    "Table": {
      "strategy": "LLM",
      "format": "Html",
      "crop_image": "Auto",
      "llm": "Optional custom prompt",
      "embed_sources": ["Content"],
      "extended_context": false
    },
    "Text": {
      "strategy": "Auto",
      "format": "Markdown",
      "crop_image": "Auto",
      "embed_sources": ["Content"]
    },
    "Picture": {
      "strategy": "Auto",
      "format": "Markdown",
      "crop_image": "All",
      "embed_sources": ["Content"]
    }
  }
}
```

**Available segment types**:

* `Title`
* `SectionHeader`
* `Text`
* `ListItem`
* `Table`
* `Picture`
* `Caption`
* `Formula`
* `Footnote`
* `PageHeader`
* `PageFooter`
* `Page`

**Configuration options per segment type**:

<ResponseField name="strategy" type="GenerationStrategy">
  How content is generated:

  * `Auto`: Heuristic-based conversion (default for most types)
  * `LLM`: Use fine-tuned Chunkr models (default for Table, Formula)
</ResponseField>

<ResponseField name="format" type="SegmentFormat">
  Output format:

  * `Markdown`: Markdown representation (default for most types)
  * `Html`: HTML representation (default for Table)
</ResponseField>

<ResponseField name="crop_image" type="CroppingStrategy">
  When to crop segment images:

  * `Auto`: Crop only when needed for processing
  * `All`: Always crop (for Picture type)
</ResponseField>

<ResponseField name="llm" type="string">
  Optional custom prompt for LLM processing. Use this to generate a custom `llm` field in the segment output using off-the-shelf models.
</ResponseField>

<ResponseField name="embed_sources" type="EmbedSource[]">
  Which content sources to include in chunk embed field:

  * `Content`: The main content field (HTML or Markdown based on format)
  * `LLM`: LLM-generated custom content
  * `HTML`: (Deprecated) HTML representation
  * `Markdown`: (Deprecated) Markdown representation

  Order matters - sources are concatenated in array order.
</ResponseField>

<ResponseField name="extended_context" type="boolean" default="false">
  Use full page image as context for LLM generation. Provides more context but increases processing time.
</ResponseField>

### Chunk Processing Configuration

```json theme={null}
{
  "chunk_processing": {
    "target_length": 512,
    "tokenizer": "Word",
    "ignore_headers_and_footers": true
  }
}
```

<ResponseField name="target_length" type="number" default="512">
  Target number of tokens/words per chunk. Set to `0` to keep one segment per chunk.
</ResponseField>

<ResponseField name="tokenizer" type="TokenizerType" default="Word">
  Tokenizer for counting chunk length:

  * `Word`: Simple whitespace tokenization
  * `Cl100kBase`: OpenAI tokenizer (GPT-3.5, GPT-4)
  * `xlm-roberta-base`: Multilingual RoBERTa
  * `bert-base-uncased`: BERT base
  * Any Hugging Face tokenizer (e.g., `"Qwen/Qwen-tokenizer"`)
</ResponseField>

<ResponseField name="ignore_headers_and_footers" type="boolean" default="true">
  Whether to exclude page headers and footers from chunks. Recommended as they break reading order across pages.
</ResponseField>

## Pipeline State

The pipeline maintains state throughout processing:

```rust theme={null}
pub struct Pipeline {
    pub input_file: Option<Arc<NamedTempFile>>,
    pub pdf_file: Option<Arc<NamedTempFile>>,
    pub page_images: Option<Vec<Arc<NamedTempFile>>>,
    pub segment_images: DashMap<String, Arc<NamedTempFile>>,
    pub chunks: Vec<Chunk>,
    pub task: Option<Task>,
    pub task_payload: Option<TaskPayload>,
}
```

The pipeline:

1. Downloads the input file from storage
2. Converts to PDF if needed
3. Generates page images
4. Performs OCR and segmentation
5. Crops segment images
6. Processes segments
7. Creates chunks
8. Uploads artifacts to storage

## Error Handling

The pipeline includes retry logic and error handling:

```json theme={null}
{
  "error_handling": "Continue"
}
```

**Strategies**:

<Tabs>
  <Tab title="Fail">
    **Stop processing on any error**

    * Task fails immediately when an error occurs
    * Provides fastest feedback on issues
    * Recommended for production when quality is critical

    Use when you need guaranteed quality and can handle failures.
  </Tab>

  <Tab title="Continue">
    **Attempt to continue despite errors**

    * Falls back to alternatives when steps fail
    * Uses PDF text layer if OCR fails
    * Treats pages as single segments if layout analysis fails
    * Skips LLM processing if models refuse/error

    Use when you want best-effort processing and can tolerate imperfect results.
  </Tab>
</Tabs>

Each pipeline step has configurable retry logic with exponential backoff (default: 3 retries).

## Monitoring Pipeline Progress

Track pipeline progress through task status:

```json theme={null}
{
  "status": "Processing",
  "message": "Running Chunkr analysis",
  "started_at": "2024-03-02T10:30:00Z"
}
```

**Pipeline messages**:

* `"Task initialized"` → Ready to start
* `"Converting pages to images"` → Step 1
* `"Running Chunkr analysis"` → Step 2 (OCR + Segmentation)
* `"Cropping segments"` → Step 3
* `"Processing segments"` → Step 4 (LLM processing)
* `"Chunking"` → Step 5
* `"Finishing up"` → Uploading artifacts

## Performance Considerations

<Warning>
  Processing time scales with:

  * **Number of pages**: Linear scaling
  * **High resolution**: \~7s per page additional latency
  * **OCR strategy**: `All` adds \~0.5s per page
  * **LLM processing**: Depends on number of tables/formulas
</Warning>

**Optimization tips**:

1. **Use `ocr_strategy: "Auto"`** for PDFs with text layers
2. **Disable high resolution** for simple documents
3. **Limit LLM processing** to only segment types that need it
4. **Set appropriate `target_length`** to control chunk count
5. **Use `error_handling: "Continue"`** for best-effort processing

## Next Steps

<CardGroup cols={2}>
  <Card title="Segmentation Strategies" icon="grid" href="/concepts/segmentation">
    Learn about layout analysis options
  </Card>

  <Card title="OCR Configuration" icon="eye" href="/concepts/ocr">
    Understand text extraction strategies
  </Card>

  <Card title="Chunking" icon="scissors" href="/concepts/chunking">
    Learn how segments are combined
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference">
    See complete API documentation
  </Card>
</CardGroup>