> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lumina-ai-inc/chunkr/llms.txt
> Use this file to discover all available pages before exploring further.

# Task Settings

> Configure chunking, tokenization, and task-level settings for document processing

## Overview

Task settings control how documents are processed, chunked, and managed throughout their lifecycle. These settings include chunk processing, expiration, resolution, and error handling strategies.

## Configuration Structure

Task configuration is provided when creating a new processing task:

<CodeGroup>
  ```json POST /api/v1/task theme={null}
  {
    "file": "base64_encoded_file_or_url",
    "chunk_processing": {
      "target_length": 512,
      "tokenizer": "Word",
      "ignore_headers_and_footers": true
    },
    "high_resolution": true,
    "expires_in": 3600,
    "error_handling": "Continue",
    "llm_processing": {
      "model_id": "gpt-4o",
      "temperature": 0.0
    }
  }
  ```

  ```python Python SDK theme={null}
  from chunkr_ai import Chunkr

  chunkr = Chunkr()

  task = chunkr.upload(
      file_path="document.pdf",
      chunk_processing={
          "target_length": 512,
          "tokenizer": "Word",
          "ignore_headers_and_footers": True
      },
      high_resolution=True,
      expires_in=3600
  )
  ```
</CodeGroup>

## Chunk Processing

Controls how document segments are grouped into chunks for retrieval and embedding.

### Parameters

<ParamField path="chunk_processing.target_length" type="integer" default={512}>
  Target number of tokens per chunk. If set to `0`, each chunk contains exactly one segment.

  **How it works:**

  * Segments are combined until reaching the target length
  * Individual segments are never split (they remain intact)
  * Final chunks may exceed the target slightly to include complete segments
</ParamField>

<ParamField path="chunk_processing.tokenizer" type="string | enum" default="Word">
  Tokenizer for measuring chunk length. Supports:

  **Predefined tokenizers:**

  * `Word` - Simple whitespace-based tokenization
  * `Cl100kBase` - OpenAI tokenizer (GPT-3.5, GPT-4, text-embedding-ada-002)
  * `xlm-roberta-base` - Multilingual RoBERTa tokenizer
  * `bert-base-uncased` - BERT base uncased tokenizer

  **Custom HuggingFace tokenizers:**

  * Any valid HuggingFace model ID (e.g., `"Qwen/Qwen-tokenizer"`, `"facebook/bart-large"`)
</ParamField>

<ParamField path="chunk_processing.ignore_headers_and_footers" type="boolean" default={true}>
  Whether to exclude page headers and footers from chunks.

  **Recommended:** Keep this `true` as headers/footers often break reading order across pages.
</ParamField>

### Examples

<Tabs>
  <Tab title="Single Segment Chunks">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 0
      }
    }
    ```

    Each chunk contains exactly one segment. Useful for:

    * Fine-grained retrieval
    * Segment-level processing
    * Maximum precision in search
  </Tab>

  <Tab title="Word-Based Chunking">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 512,
        "tokenizer": "Word"
      }
    }
    ```

    Default configuration. Fast and simple tokenization based on whitespace.
  </Tab>

  <Tab title="OpenAI Tokenizer">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 512,
        "tokenizer": "Cl100kBase"
      }
    }
    ```

    Use when working with OpenAI embeddings (e.g., `text-embedding-ada-002`).
  </Tab>

  <Tab title="Custom HuggingFace">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 512,
        "tokenizer": "Qwen/Qwen-tokenizer"
      }
    }
    ```

    Use any HuggingFace tokenizer by providing its model ID.
  </Tab>

  <Tab title="Include Headers/Footers">
    ```json theme={null}
    {
      "chunk_processing": {
        "target_length": 512,
        "tokenizer": "Word",
        "ignore_headers_and_footers": false
      }
    }
    ```

    Include page headers and footers in chunks. Use when headers/footers contain important content.
  </Tab>
</Tabs>

## Processing Strategies

### OCR Strategy

<ParamField path="ocr_strategy" type="enum" default="All">
  Controls Optical Character Recognition behavior:

  * `All` - Process all pages with OCR (Latency: \~0.5s per page)
  * `Auto` - Selectively apply OCR only to pages with missing or low-quality text
</ParamField>

<CodeGroup>
  ```json All Pages theme={null}
  {
    "ocr_strategy": "All"
  }
  ```

  ```json Auto Detection theme={null}
  {
    "ocr_strategy": "Auto"
  }
  ```
</CodeGroup>

<Note>
  `Auto` mode uses existing text layers when available, falling back to OCR only when needed.
</Note>

### Segmentation Strategy

<ParamField path="segmentation_strategy" type="enum" default="LayoutAnalysis">
  Controls document segmentation:

  * `LayoutAnalysis` - Detect layout elements (tables, pictures, formulas) with bounding boxes. Provides fine-grained segmentation.
  * `Page` - Treat each page as a single segment. Faster but without layout element detection.
</ParamField>

<CodeGroup>
  ```json Layout Analysis theme={null}
  {
    "segmentation_strategy": "LayoutAnalysis"
  }
  ```

  ```json Page-Level theme={null}
  {
    "segmentation_strategy": "Page"
  }
  ```
</CodeGroup>

## Error Handling

<ParamField path="error_handling" type="enum" default="Fail">
  Controls how errors are handled during processing:

  * `Fail` - Stop processing and fail the task on any error
  * `Continue` - Attempt to continue despite non-critical errors (e.g., LLM refusals, rate limits)
</ParamField>

<Tabs>
  <Tab title="Fail on Error">
    ```json theme={null}
    {
      "error_handling": "Fail"
    }
    ```

    **Use when:**

    * Complete accuracy is critical
    * You want to manually review and fix errors
    * Processing can be safely retried
  </Tab>

  <Tab title="Continue on Error">
    ```json theme={null}
    {
      "error_handling": "Continue"
    }
    ```

    **Use when:**

    * Partial results are acceptable
    * Processing large batches where individual failures shouldn't block the entire task
    * You have fallback models configured
  </Tab>
</Tabs>

## Resolution Settings

<ParamField path="high_resolution" type="boolean" default={true}>
  Whether to use high-resolution images for cropping and post-processing.

  **Trade-offs:**

  * `true` - Better quality for image segments, tables, and formulas (Latency: \~7s per page)
  * `false` - Faster processing with standard resolution
</ParamField>

<CodeGroup>
  ```json High Resolution theme={null}
  {
    "high_resolution": true
  }
  ```

  ```json Standard Resolution theme={null}
  {
    "high_resolution": false
  }
  ```
</CodeGroup>

## Task Expiration

<ParamField path="expires_in" type="integer">
  Number of seconds until the task is deleted. Expired tasks cannot be:

  * Updated
  * Polled
  * Accessed via web interface

  If not specified, uses the system default from `JOB__EXPIRATION_TIME` environment variable.
</ParamField>

<Tabs>
  <Tab title="1 Hour">
    ```json theme={null}
    {
      "expires_in": 3600
    }
    ```
  </Tab>

  <Tab title="24 Hours">
    ```json theme={null}
    {
      "expires_in": 86400
    }
    ```
  </Tab>

  <Tab title="7 Days">
    ```json theme={null}
    {
      "expires_in": 604800
    }
    ```
  </Tab>

  <Tab title="No Expiration">
    ```json theme={null}
    {
      "expires_in": null
    }
    ```

    <Warning>
      Tasks without expiration consume storage indefinitely. Set appropriate cleanup policies.
    </Warning>
  </Tab>
</Tabs>

## LLM Processing

See the [LLM Models](/configuration/llm-models) page for detailed LLM configuration.

<ParamField path="llm_processing.model_id" type="string">
  ID of the model to use (from your `models.yaml`). If not provided, the default model is used.
</ParamField>

<ParamField path="llm_processing.temperature" type="float" default={0.0}>
  Temperature for LLM generation. Range: 0.0 (deterministic) to 2.0 (creative).
</ParamField>

<ParamField path="llm_processing.max_completion_tokens" type="integer">
  Maximum tokens in LLM responses. Limits output length and cost.
</ParamField>

<ParamField path="llm_processing.fallback_strategy" type="enum" default="Default">
  Fallback behavior when primary LLM fails:

  * `None` - No fallback
  * `Default` - Use configured fallback model
  * `Model("id")` - Use specific model
</ParamField>

## Complete Example

```json theme={null}
{
  "file": "base64_encoded_file_or_url",
  "file_name": "research_paper.pdf",
  "chunk_processing": {
    "target_length": 512,
    "tokenizer": "Cl100kBase",
    "ignore_headers_and_footers": true
  },
  "segment_processing": {
    "table": {
      "format": "Html",
      "strategy": "LLM"
    }
  },
  "ocr_strategy": "All",
  "segmentation_strategy": "LayoutAnalysis",
  "high_resolution": true,
  "error_handling": "Continue",
  "expires_in": 86400,
  "llm_processing": {
    "model_id": "gpt-4o",
    "temperature": 0.0,
    "max_completion_tokens": 4096,
    "fallback_strategy": "Default"
  }
}
```

## Task Status

Tasks progress through these states:

* `Starting` - Task queued and initializing
* `Processing` - Active processing
* `Succeeded` - Completed successfully
* `Failed` - Encountered an error
* `Cancelled` - Manually cancelled

## Best Practices

1. **Match tokenizer to your embedding model**
   * Use `Cl100kBase` for OpenAI embeddings
   * Use model-specific tokenizers for other embeddings

2. **Set appropriate chunk sizes**
   * Smaller chunks (256-512) for precise retrieval
   * Larger chunks (1024+) for more context

3. **Use `Continue` error handling for batch processing**
   * Prevents individual failures from blocking entire batches
   * Review logs for partial failures

4. **Configure expiration based on usage**
   * Short expiration (1 hour) for temporary processing
   * Long expiration (7+ days) for production results
   * Monitor storage usage

5. **Enable high resolution selectively**
   * Use for documents with important visual elements
   * Disable for text-heavy documents to improve speed