Skip to main content

Agentic Document Intelligence

This document describes the current agentic metadata extraction pipeline.

Entry Point​

The current extraction entry point is HybridMetadataExtractor.

Behavior:

  • if no supported LLM provider is configured, it uses deterministic extraction
  • if at least one of OpenAI, Gemini, or Mistral is configured, it delegates to AgenticDocumentIntelligenceService

Framework Choice​

The agentic pipeline is implemented with LangGraph.

Current internal pipeline marker:

  • framework: langgraph-ready
  • version: v1

The design goal is orchestrated, bounded intelligence rather than free-form autonomous behavior.

Supported LLM Providers​

The current provider order inside the agentic pipeline is:

  1. mistral
  2. gemini
  3. openai

LlmService supports explicit fallback order resolution and returns provider/model metadata together with completion results.

Workflow Shape​

The AgenticDocumentIntelligenceService builds a LangGraph state graph with nodes for:

  • routing
  • title and summary generation
  • typed metadata extraction
  • correspondent resolution
  • tagging
  • normalization and validation

The final result is converted into the shared MetadataExtractionResult shape expected by the rest of the processing pipeline.

Structured Outputs​

All four agent LLM calls (routing, title/summary, per-type extraction, tagging) send a JSON schema from apps/api/src/processing/agent-schemas.ts:

  • Mistral and OpenAI enforce it via response_format: { type: "json_schema", strict: true }; the schemas are strict-compatible (every property required, nullable where optional in spirit, additionalProperties: false)
  • Gemini's schema dialect rejects JSON-Schema type unions, so Gemini falls back to plain JSON mode; caller-side validation covers the gap
  • the per-type extraction schema is generated from the registry's relevant fields; its fieldConfidence entries are required but nullable, and null placeholders are accepted by the response validator and ignored during normalization
  • every response is validated with Zod (parseWithSchema) before use; a schema-violating payload falls back to the deterministic extraction path, and the brace-slice JSON heuristic remains only as the last-ditch parse fallback
  • merged fields are confidence-aware: an LLM value only replaces a deterministic regex hit when the deterministic slot is empty or the LLM reported at least the deterministic confidence, and field provenance follows the winner

Parse-Provider Annotation Hints​

When MISTRAL_OCR_DOCUMENT_ANNOTATIONS=true (default off), the Mistral OCR request additionally asks for a document annotation: a vision-capable model extracts a generic metadata schema (document type from the registry enum, title, summary, the union of relevant fields, each with confidence) inside the same OCR call. The validated result lands on parsed.preExtracted — a capability of the parse output, not a new interface, so other providers are unaffected.

The graph shape is unchanged; nodes consume the hint conditionally:

  • routing skips its LLM call when the annotated type is valid and its confidence is at least 0.7 (provider recorded as mistral-annotation)
  • title/summary skips its LLM call when the annotation carries a title AND the parse provider reported a document language that matches the configured processing language. Mistral OCR reports no language, so in practice titles are regenerated through the language-aware LLM path — the annotation request carries no language preference, so a mismatch would persist titles in the document's language instead of the one selected in Settings
  • typed extraction seeds from the annotation fields via the same confidence-aware merge used for LLM values (provenance provider_annotation). The seed is filtered to the routed type's relevant fields, so the generic annotation schema cannot inject values that type could never produce. The LLM call is skipped only when the annotation itself supplied every required field with sufficient annotation confidence (a value scored below 0.5, or with no score at all, does not count as coverage) — deterministic parsing filling the gaps does not count. Type-specific refiners still run afterwards, and confidence/provenance are rebuilt for the values they replace
  • correspondent resolution, tagging, and validation are unchanged — they need archive context (candidate lists, deterministic seeds) the annotation cannot provide

Cost is roughly neutral (annotations ~+$1/1000 pages vs. up to three saved chat calls per document); latency drops because up to three sequential round-trips disappear. Documents longer than ~8 pages get the warning annotation_hint_partial_document because annotations only consider the leading pages. Every node keeps its LLM and deterministic fallback, so the flag can be toggled per environment and compared.

Routing Stage​

The routing stage determines the likely document type and stores:

  • selected document type
  • subtype when present
  • confidence
  • reasoning hints
  • provider and model metadata

This data is later exposed to the frontend through metadata.intelligence.routing.

Supported Document Types​

Current document type registry entries:

  • invoice
  • receipt
  • contract
  • tax_document
  • utility_bill
  • bank_statement
  • payslip
  • insurance_document
  • generic_letter

The registry defines:

  • canonical names
  • aliases
  • summaries
  • required fields
  • relevant fields
  • label hints for dates, references, and similar extraction targets

Type-Specific Extraction​

The typed extraction stage uses dedicated modules under:

  • apps/api/src/processing/type-specific-extractors/

This keeps extraction logic structured per document family instead of relying on one generic prompt path.

Correspondent Resolution​

Correspondent resolution is not a separate ad-hoc step in the UI. It is part of the extraction workflow.

The pipeline feeds extracted or candidate correspondent data into CorrespondentResolutionService, which returns:

  • resolved correspondent name
  • confidence
  • match strategy metadata

That metadata is preserved under metadata.correspondentExtraction and metadata.intelligence.correspondentResolution.

Tagging​

The tagging stage produces normalized tag suggestions and confidence metadata.

These are surfaced through metadata.intelligence.tagging.

Validation and Normalization​

The final validation stage normalizes extracted fields and produces:

  • normalized field values
  • warnings
  • errors
  • duplicate signals

This stage is also responsible for shaping data that will later affect:

  • confidence
  • review reasons
  • review evidence

Final Output Shape​

The agentic pipeline returns the standard extraction fields expected elsewhere in the backend, including:

  • title
  • summary
  • issue date
  • due date
  • expiry date
  • amount and currency
  • reference number
  • holder name
  • issuing authority
  • correspondent name
  • document type name
  • tags
  • confidence
  • review reasons

It also writes detailed intelligence metadata into metadata.intelligence.

Frontend Exposure​

The web app currently exposes agentic output in several places:

  • explorer row badges and summaries
  • review queue badges and summaries
  • document detail intelligence tab

The document detail page currently exposes:

  • routing
  • generated summary
  • type-specific fields
  • field confidence and provenance
  • tagging and correspondent resolution
  • validation warnings and errors
  • pipeline metadata and durations

Design Constraints​

Current design constraints intentionally keep the system predictable:

  • agent stages are fixed and orchestrated
  • extracted data is normalized before it is trusted
  • review remains explicit and separate from processing
  • manual overrides still outrank automated output