Llm integration POC #1028

timfdev · 2025-08-12T22:23:17Z

pydantic-ai and ag-ui-protocol

need pydantic >= 2.10 and >=2.11.2 respectively, this breaks some of the unit tests

…s allowed token count. Make conflicting libraries pydantic-ai and ag-ui optional; disabling agent route if not installed. Make search routes async and fix small bugs in query building.

codspeed-hq · 2025-08-16T23:26:13Z

CodSpeed Performance Report

Merging #1028 will not alter performance

_{Comparing llm-integration (c35a073) with main (8a2b890)}

Summary

✅ 13 untouched

…hestrator-core into llm-integration

codecov · 2025-08-18T15:53:58Z

Codecov Report

❌ Patch coverage is 43.57218% with 1071 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.75%. Comparing base (8a2b890) to head (c35a073).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
orchestrator/search/indexing/indexer.py	22.94%	131 Missing ⚠️
orchestrator/search/retrieval/retriever.py	33.11%	101 Missing ⚠️
orchestrator/search/indexing/traverse.py	32.00%	85 Missing ⚠️
orchestrator/api/api_v1/endpoints/search.py	32.03%	70 Missing ⚠️
orchestrator/cli/speedtest.py	26.74%	62 Missing and 1 partial ⚠️
orchestrator/search/filters/base.py	42.05%	62 Missing ⚠️
orchestrator/cli/resize_embedding.py	21.21%	51 Missing and 1 partial ⚠️
orchestrator/search/retrieval/utils.py	22.72%	51 Missing ⚠️
orchestrator/search/core/types.py	65.03%	50 Missing ⚠️
orchestrator/search/retrieval/validation.py	25.00%	45 Missing ⚠️
... and 18 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1028      +/-   ##
==========================================
- Coverage   85.14%   78.75%   -6.40%     
==========================================
  Files         217      251      +34     
  Lines       10495    12387    +1892     
  Branches     1004     1214     +210     
==========================================
+ Hits         8936     9755     +819     
- Misses       1305     2372    +1067     
- Partials      254      260       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

orchestrator/api/api_v1/endpoints/search.py

orchestrator/api/api_v1/api.py

luc-tielen · 2025-08-19T09:41:05Z

orchestrator/cli/search_explore.py

+        dotenv run python main.py search semantic "Shop for an alligator store"
+        ...
+        {
+            "path": "subscription.shop.shop_description",
+            "value": "Kingswood reptiles shop"
+        },


We should make the examples more generic (also the ones below), since this is specific for the WFO instance where we built the initial POC.

...migrations/versions/schema/2025-08-12_52b37b5b2714_search_index_model_for_llm_integration.py

orchestrator/search/core/embedding.py

orchestrator/search/docs/running_local_text_embedding_inference.md

luc-tielen · 2025-08-19T20:36:37Z

orchestrator/search/filters/base.py

+FilterCondition = (
+    DateFilter  # DATETIME
+    | NumericFilter  # INT/FLOAT
+    | StringFilter  # STRING TODO: convert to hybrid search


do we need to make a ticket for this TODO?

Im thinking that maybe this stringfilter should be removed altogether, its already possible to do a hybrid search by passing a user query, passing something like the top 5 results back to the agent will probably yield better results.

For things like booleans/product blocks , we already have the equality filter. Matching on exact text by letting the agent fill in a string will probably not work well.

orchestrator/search/retrieval/builder.py

…ndpoints for autocompleting paths and UI compatible operators per field type for frontend rendering.

… settings and instructions.

…ption records in response, improve highlighting

…hestrator-core into llm-integration

…d substring highlighting

… data)

…ith just a field name and value type. Support component contains/not contains filters.

tjeerddie · 2025-09-15T07:39:04Z

orchestrator/search/agent/prompts.py

+    except Exception as e:
+        logger.warning(f"Failed to load schema for prompt: {e}")
+        schema_info = "    Schema temporarily unavailable"
+    logger.error(f"Generated schema for agent prompt:\n{schema_info}")


I don't think this is suppose to be an error log?

tjeerddie · 2025-09-15T09:16:49Z

orchestrator/search/indexing/traverse.py

+    # We are explicitly excluding 'traceback' and 'steps'
+    # to avoid overloading the index with too much data.
+    _process_fields_to_exclude: set[str] = {
+        "traceback",


only excluding traceback? update exclude list or the above comment

tjeerddie · 2025-09-15T09:29:48Z

orchestrator/settings.py

@@ -92,6 +92,34 @@ class AppSettings(BaseSettings):
    EXPOSE_SETTINGS: bool = False
    EXPOSE_OAUTH_SETTINGS: bool = False

+    # Pydantic-ai Agent settings
+    AGENT_MODEL: str = "openai:gpt-4o-mini"  # See pydantic-ai docs for supported models.


It might be nice to create a different settings class for LLM settings

tjeerddie · 2025-09-15T09:31:41Z

orchestrator/search/retrieval/engine.py

+def _extract_matching_field_from_filters(filters: FilterTree) -> MatchingField | None:
+    """Extract the first path filter to use as matching field for structured searches.
+
+    TODO: Should we allow a list of matched fields in the MatchingField model?


what to do with this? new issue?

Mark90

Nice, that's a lot of work 🔥

Overall structure of the code is good, that's why I was able to leave a lot of questions and small remarks. I mean this as a good thing :)

Mark90 · 2025-09-15T06:47:22Z

.github/workflows/run-codspeed-tests.yml

@@ -18,7 +18,7 @@ jobs:
      options: --privileged
    services:
      postgres:
-        image: postgres:15-alpine
+        image: pgvector/pgvector:pg15


What does this add on top of the normal postgres image? Is there documentation for the extra postgres configuration/extensions required?

Mark90 · 2025-09-15T07:14:05Z

orchestrator/api/api_v1/endpoints/search.py

+        search_params=search_params,
+        db_session=db.session,
+        pagination_params=pagination_params,
+    )


Have there been any thoughts about implementing access control?

This is planned for later.

Mark90 · 2025-09-15T07:20:48Z

orchestrator/api/api_v1/endpoints/agent.py

+
+
+def build_agent_app() -> ASGIApp:
+    if not app_settings.AGENT_MODEL or not app_settings.OPENAI_API_KEY:


These settings are strings that can't be None so by default it will be enabled. Since users need to configure the LLM setup, by default it should IMO be disabled with a bool variable like AGENT_ENABLED

Mark90 · 2025-09-15T07:24:38Z

orchestrator/api/api_v1/endpoints/agent.py

+
+        return agent.to_ag_ui(deps=StateDeps(SearchState()))
+    except Exception as e:
+        logger.error("Agent init failed; serving disabled stub.", error=str(e))


What kind of failures has this shown?

Mark90 · 2025-09-15T07:33:22Z

...migrations/versions/schema/2025-08-12_52b37b5b2714_search_index_model_for_llm_integration.py

+        sa.Column("entity_id", postgresql.UUID, nullable=False),
+        sa.Column("path", LtreeType, nullable=False),
+        sa.Column("value", sa.Text, nullable=False),
+        sa.Column("embedding", Vector(TARGET_DIM), nullable=True),


Does this require the database to have pgvector installed?

Mark90 · 2025-09-16T13:08:11Z

orchestrator/search/retrieval/retriever.py

+                entity_scores.join(entity_highlights, entity_scores.c.entity_id == entity_highlights.c.entity_id)
+            )
+        ).cte("ranked_results")
+


Could we split this function up in one for the DB interaction part which produces an output, and another function that performs the below computations based on the former's output? And preferably also some unittests for the latter

Mark90 · 2025-09-16T13:10:56Z

orchestrator/search/retrieval/retriever.py

@@ -0,0 +1,447 @@
+from abc import ABC, abstractmethod


Maybe split up into a package with a module for each retriever type, it's a lot of scrolling now :)

Mark90 · 2025-09-16T13:27:55Z

orchestrator/search/retrieval/retriever.py

+
+    def _quantize_score_for_pagination(self, score_value: float) -> BindParameter[Decimal]:
+        """Convert score value to properly quantized Decimal parameter for pagination."""
+        pas_dec = Decimal(str(score_value)).quantize(Decimal("0.000000000001"))


Should this change along with the SCORE_PRECISION if that ever changes?

If so maybe do something like f'{1 / 10**precision:.{precision}f}

Mark90 · 2025-09-16T13:38:49Z

orchestrator/search/retrieval/utils.py

+
+        if not matches:
+            substring_pattern = re.escape(word)
+            matches = list(re.finditer(substring_pattern, text, re.IGNORECASE))


If a resulting text has both word and substring matches, wouldn't we want to highlight the substring matches as well?

Mark90 · 2025-09-16T13:43:26Z

orchestrator/search/schemas/results.py

+
+class TypeDefinition(BaseModel):
+    operators: list[FilterOp]
+    valueSchema: dict[FilterOp, ValueSchema]


Is camelCase needed here?

timfdev added 8 commits August 13, 2025 00:20

Vector search and agent mode POC

99da746

l

960ce80

add ag-ui package

d34d467

fix linting

cede6f5

last lint fix

65e963d

Streaming pipeline for indexing, using litellm to track token count v…

90a5d1a

…s allowed token count. Make conflicting libraries pydantic-ai and ag-ui optional; disabling agent route if not installed. Make search routes async and fix small bugs in query building.

fix mypy issues & use pgvector image

d0a23ec

use pgvector for codspeed tests

4fce33d

timfdev and others added 3 commits August 17, 2025 01:30

Merge branch 'main' into llm-integration

b8b4eb8

update docs and cleanup

1ec3625

Merge branch 'llm-integration' of github.com:workfloworchestrator/orc…

5d4c316

…hestrator-core into llm-integration

mrijk reviewed Aug 19, 2025

View reviewed changes

orchestrator/api/api_v1/endpoints/search.py Outdated Show resolved Hide resolved

use python 3.10+ style type hinting

f837226

luc-tielen reviewed Aug 19, 2025

View reviewed changes

timfdev and others added 10 commits August 23, 2025 19:47

refactor from a list of filter conditions to a filter tree; Include e…

9a70722

…ndpoints for autocompleting paths and UI compatible operators per field type for frontend rendering.

small bugfixes

816ead4

Bump pydantic to 2.11

811ebea

CLI command to reshape vector embeddings column, improved local setup…

1348d0a

… settings and instructions.

Update mask_value for masking exposed settings and fix unit tests

ca2a4ff

Bump version to 5.0.0a1

5c8025f

Add keyset pagination, include search metadata, load detailed subscri…

29ae887

…ption records in response, improve highlighting

Merge branch 'llm-integration' of github.com:workfloworchestrator/orc…

c6635dc

…hestrator-core into llm-integration

Speedtest, improved retrieval speed by limiting search space, improve…

6a4cada

…d substring highlighting

Merge main into llm-integration branch (clean merge without sensitive…

e9597e1

… data)

timfdev force-pushed the llm-integration branch from 3840906 to e9597e1 Compare September 3, 2025 12:37

timfdev added 3 commits September 3, 2025 14:54

Normalize all retriever scores to 0-1 range and other small fixes

8830633

fix linting issues

3a37786

Add matchedfields for all endpoints and for structured searches

0e54272

timfdev added 5 commits September 10, 2025 01:58

refactor path endpoint and filters to simplify structured filtering w…

7634d24

…ith just a field name and value type. Support component contains/not contains filters.

negation on group level, not record level

3fdab01

Merge main branch

7a8db9e

Merge remote-tracking branch 'origin/pydantic-2.11' into llm-integration

0a39261

Make agent packages required and remove import safequards

c35a073

tjeerddie reviewed Sep 15, 2025

View reviewed changes

tjeerddie approved these changes Sep 15, 2025

View reviewed changes

Mark90 self-requested a review September 16, 2025 12:52

Mark90 reviewed Sep 16, 2025

View reviewed changes



		def build_agent_app() -> ASGIApp:
		if not app_settings.AGENT_MODEL or not app_settings.OPENAI_API_KEY:

Llm integration POC #1028

Are you sure you want to change the base?

Llm integration POC #1028

Uh oh!

Conversation

timfdev commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #1028 will not alter performance

Summary

Uh oh!

codecov bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mark90 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

timfdev commented Aug 12, 2025 •

edited

Loading

codspeed-hq bot commented Aug 16, 2025 •

edited

Loading

codecov bot commented Aug 18, 2025 •

edited

Loading