Temporal annual distribution functionality and temporal FeatureExtraction capability enablement

[alex-odysseus]

Addressing #2331 (temporal annual distribution requirement)

Cohort Characterization Controller has been extended with a new endpoint
GET /generation/{generationId}/temporalresult

Temporal annual distribution and temporal Cohort Characterization analysis execution results are selected from newly introduced tables in the result schema - cc_temporal_results and cc_temporal_annual_results

Each Feature Analysis has been extended with two attributes indicating their temporal and temporal annual distribution properties depending on their actual support by FeatureExtraction

[alex-odysseus]

Similar notation as for the table above should be used

IF OBJECT_ID('@results_schema.table', 'U') IS NULL

[alex-odysseus]

Is the list final?

[alex-odysseus]

Added items from https://github.com/OHDSI/FeatureExtraction/blob/main/inst/csv/PrespecTemporalAnalyses.csv

[alex-odysseus]

Chris, please pull the code @chrisknoll

[chrisknoll]

Error when generating:

2025-03-27 15:21:52.331 INFO taskExecutor-1 org.springframework.batch.core.job.SimpleStepHandler - [] - Executing step: [generateCohortCharacterization.generate]
2025-03-27 15:21:52.854 ERROR taskExecutor-1 org.ohdsi.webapi.common.generation.TransactionalTasklet - [] - JSONObject["sqlQueryFeatures"] not found.
org.json.JSONException: JSONObject["sqlQueryFeatures"] not found.
	at org.json.JSONObject.get(JSONObject.java:570) ~[json-20230227.jar:?]
	at org.json.JSONObject.getString(JSONObject.java:857) ~[json-20230227.jar:?]
	at org.ohdsi.cohortcharacterization.CCQueryBuilder.getQueriesForPresetAnalyses(CCQueryBuilder.java:251) ~[SkeletonCohortCharacterization-2.0.0.jar:?]

This came from a copy of an existing CC, where I switched on the new options in the design.

@alex-odysseus @oleg-odysseus

Attached is sample CC example: CC_TestCase.txt

[chrisknoll]

Will approve once errors are resolved.

[oleg-odysseus]

Hi @chrisknoll , @anthonysena . I was able to reproduce the problem, a small fix on the SkeletonCohortCharacterization was enough for the test CC you've provided to get generated locally (Report opens, the temporal distributions are in place). I assume we`ll need to release the skeleton lib once again.. Fix PR: SkeletonCohortCharacterization PR 47 , Thank you.

[chrisknoll]

Yes, please release SkeletonCC, and as part of this release if we could add this failure as a test case for the package.

[oleg-odysseus]

Hi @chrisknoll , @anthonysena , thank you for releasing SkeletonCohortCharacterization v2.0.1. Looks like the CI for this PR now succeeds and there are no merge conflicts so far. Please kindly merge if you have no other remarks. Thank you.

[chrisknoll]

While the functionality seems to work, my concern is on execution time. Here's a screen-shot of 2 executions, the 5 minute one is without the new options, the 13 hour one is with the options enabled. This was running on a large sample CDM on my local box:

So I'm concerned about releasing this feature and leading to 100-fold increase in execution time. Maybe we can spend some time at the meeting evaluating the queries and understanding if there's a performance optimization to be made.

@anthonysena @alex-odysseus

[chrisknoll]

@oleg-odysseus : per our conversation on WG call, you'll investigate the comparison flag that is leading to an empty STDDiff column in the UI.

For context: it seems that the new atlas branch shows the correct STDDiff values when the WebAPI branch is 'master'...only switching to the WebAPI temporal prevalence branch leads to the error in the UI where the StdDiff column is empty.

[oleg-odysseus]

@chrisknoll , I've reproduced the problem locally and decided to roll back the change that I've introduced some time ago that led to this behavior (that change forced disabling the comparative results). Now as per my local checks, the reports behave as usual for the analysis with multiple cohorts. If the analysis was generated with temporal distribution then the links to observe those distributions would be visible for 1, 3 or more cohorts. For the comparative mode (2 cohorts) the link is not enabled, but if a user needs to observe temporal/annual distribution for the cohort then he/she could exit the comparative mode by selecting, for example, only that particular cohort and observing temporal distributions for it by clicking the link.