feat(agentic idp): First version of agentic IDP using strands

[kazmer97]

Summary

• Add agentic extraction module using Strands agents for structured data output without JSON parsing errors • Enhance
extraction service with dynamic Pydantic model generation from configuration attributes • Maintain full backward
compatibility - traditional extraction remains default behavior

Changes Made

• New: lib/idp_common_pkg/idp_common/extraction/agentic_idp.py - Core agentic functionality with tool-based
extraction
• Modified: lib/idp_common_pkg/idp_common/extraction/service.py - Added dynamic model generation and agentic
integration
• Updated: lib/idp_common_pkg/idp_common/extraction/README.md - Comprehensive documentation with usage examples

Benefits

• Eliminates JSON parsing errors through validated Pydantic models
• Improves extraction accuracy with self-correcting agent tools
• Provides dual extraction paths (traditional vs agentic) based on configuration
• Zero breaking changes - existing workflows continue unchanged

This enhancement significantly improves extraction reliability while preserving all existing functionality and
configuration patterns.

[rstrahan]

How do prompts work.. Does the agent still use the existing System prompt and Task prompt? (I had imagined the agent would need different prompt variants, but if not, that's great!)

[kazmer97]

How Prompts Work with Agentic Extraction
The agentic extraction uses the existing System and Task prompts, but in a different way than traditional extraction:

Traditional Extraction
System prompt: Sent directly to Bedrock as system message
Task prompt: Sent as user message
Model responds with JSON text that needs parsing
Agentic Extraction
System prompt → Passed via custom_instruction parameter and appended to agentic system prompt
Task prompt → Sent as user message (content blocks with text/images)
Uses Strands agent with tools for structured output
No JSON parsing needed - returns validated Pydantic model
Key Difference
The agentic system prompt (in agentic_idp.py) provides extraction guidelines and tool usage instructions. Your existing system/task prompts are incorporated as custom instructions to guide the extraction without changing the core agent behavior.

[kazmer97]

How Prompts Work with Agentic Extraction

The agentic extraction uses the existing System and Task prompts, but in a different way than traditional extraction:

Traditional Extraction

• System prompt: Sent directly to Bedrock as system message
• Task prompt: Sent as user message
• Model responds with JSON text that needs parsing

Agentic Extraction

• System prompt → Passed via custom_instruction parameter and appended to agentic system prompt
• Task prompt → Sent as user message (content blocks with text/images)
• Uses Strands agent with tools for structured output
• No JSON parsing needed - returns validated Pydantic model

Key Difference

The agentic system prompt (in agentic_idp.py) provides extraction guidelines and tool usage instructions. Your existing system/task prompts are incorporated as custom instructions to guide the extraction without changing the core agent behavior.

Result: Same prompts work for both methods, just applied differently under the hood.

[rstrahan]

We should be able to automate this.. already in idp_common bedrock client we have a list of models that support caching, so use that to auto detect and cache if model supports it.

[rstrahan]

Is it too soon to say this.. Better to say (Preview) or (Experimental) for now, IMHO, till we have more testing and ironed out bugs..

[rstrahan]

Can we say something about this enabling future extensions involving MCP servers to integrate validation or enrichment into the extraction process... (eg validate names, addresses, etc from external services)

[rstrahan]

Can you explain how the system_prompt and task_prompts apply to the agentic extraction.. Does the agent use the same prompt as 'traditional' method?

[rstrahan]

See slack comment on throttle catching..

it failed with
Execution Failed
Error: ValueError: Agent invocation failed: An error occurred (ThrottlingException) when calling the ConverseStream operation (reached max retries: 4): Too many tokens, please wait before trying again.
We need to keep retrying... for much much longer.. @kaznb can you check
idp_common bedrock client for exponential backoff/retry logic
state machine for retries at lambda invocation level..
We should never fail due to throttling.. at least not for several hours..
Note - I seem to get this error every time with Sonnet 4.5 in my isengard account when processing just 1 lending_package.pdf (Isengard has really low quota), but, whatever the customer quota is, we still need to defend against throttling when volumes are large and demand exceeds quota