Skip to content

Optional Classification and Extraction Steps #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Optional Classification and Extraction Steps #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a24400e
Corrected git | branch created from exsisting feature branch
Sep 23, 2025
7575f07
Moved gating logic from patterns to services
Sep 24, 2025
4fcb327
Decoupled Services from Main Processing Flow | OCR | CLASSIFICATION |…
Sep 25, 2025
1d09259
Surfacing errors to Processing Flow Ui for Extraction Step
Sep 25, 2025
ef94dab
Decoupled Assessment
Sep 27, 2025
0d74b98
Merge remote-tracking branch 'upstream/develop' into feature/optional…
Sep 29, 2025
2e91f28
reverted to original Flow with simple flag check
Sep 29, 2025
41a4dad
Decoupled Services from Main Processing Flow | OCR | CLASSIFICATION |…
Sep 25, 2025
f20862a
Surfacing errors to Processing Flow Ui for Extraction Step
Sep 25, 2025
f3bd5cb
Decoupled Assessment
Sep 27, 2025
37ceba3
reverted to original Flow with simple flag check
Sep 29, 2025
c1281a1
request changes made
Sep 29, 2025
4982fbb
removed interactive rebase leftovers
Sep 29, 2025
22ace72
added in configs
Sep 29, 2025
1b2ffc1
Removed excess Error collection in Summ Func | Updated docs | correct…
Sep 30, 2025
9238d97
Merge branch 'main' into feature/optional-classification-extraction
HatmanStack Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/pattern-2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ Each step includes comprehensive retry logic for handling transient errors:
```

#### Extraction Function
- **Purpose**: Extracts fields using Claude via Bedrock
- **Purpose**: Extracts fields using Claude via Bedrock.
- **Key Features**:
- Document class-specific attribute extraction
- Configurable extraction attributes
Expand Down Expand Up @@ -225,6 +225,9 @@ The pattern exports these outputs to the parent stack:

**Stack Deployment Parameters:**
- `ClassificationMethod`: Classification methodology to use (options: 'multimodalPageLevelClassification' or 'textbasedHolisticClassification')
- **OCR**: Control the ability to tune ocr via configuration file `ocr.tuning` property.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid I am confused now by the scope of this PR, including the introduction of this 'tuning' concept. I had thought this would be a simple PR, a quick win, to introduce only an 'enable' property for classification and extraction, with the simple logic in the associated lambdas to skip the associated processing if enabled is False. (Like we have already for Summarization).
And so I am confused by why this would require introduction of new stack deployment parameters and these .tuning properties.

- **Classification**: Control the ability to tune classification via configuration file `classification.tuning` property.
- **Extraction**: Control the ability to tune extraction via configuration file `extraction.tuning` property.
- **Summarization**: Control summarization via configuration file `summarization.enabled` property (replaces `IsSummarizationEnabled` parameter)
- `ConfigurationDefaultS3Uri`: Optional S3 URI to custom configuration (uses default configuration if not specified)
- `MaxConcurrentWorkflows`: Workflow concurrency limit
Expand Down
5 changes: 4 additions & 1 deletion docs/pattern-3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ Each step includes comprehensive retry logic for handling transient errors:
```

#### Classification Function
- **Purpose**: Classifies pages using UDOP model on SageMaker, and segments into sections using class boundaries
- **Purpose**: Classifies pages using UDOP model on SageMaker, and segments into sections using class boundaries.
- **Input**: Output from OCR function plus output bucket
- **Output**:
```json
Expand Down Expand Up @@ -209,6 +209,9 @@ The pattern exports these outputs to the parent stack:

**Stack Deployment Parameters:**
- `UDOPModelArtifactPath`: S3 path to UDOP model artifacts (see [Fine tuning a UDOP model](#fine-tuning-a-udop-model-for-classification))
- **OCR**: Control the ability to tune ocr via configuration file `ocr.tuning` property.
- **Classification**: Control the ability to tune classification via configuration file `classification.tuning` property.
- **Extraction**: Control the ability to tune extraction via configuration file `extraction.tuning` property.
- **Summarization**: Control summarization via configuration file `summarization.enabled` property (replaces `IsSummarizationEnabled` parameter)
- `ConfigurationDefaultS3Uri`: Optional S3 URI to custom configuration (uses default configuration if not specified)
- `MaxConcurrentWorkflows`: Workflow concurrency limit
Expand Down
2 changes: 1 addition & 1 deletion lib/idp_common_pkg/idp_common/assessment/granular_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2018,4 +2018,4 @@ def _convert_error_list_to_string(self, errors) -> str:
return f"{first_errors} and {len(errors) - 1} more errors"
else:
# Few errors - show all
return "; ".join(errors)
return "; ".join(errors)
Loading