Repository navigation
[bot] Fast-forward for 26.3.19 - #1014
Merged
Merged
Conversation
## Rationale Removes this module's last references to MergeSync, which is being retired. The labwork request form they belonged to is not registered, so none of this code was reachable at runtime. ## Changes - Removes the Merge integration from the labwork request form, along with its instructions panel. - Drops the prompt that offered to open Merge after a request was submitted.
…985) ## Rationale Make it possible to tell from the ETL job log which event data rows ended up holding no attribute values. The _SND Event Data step clears attribute values as a side effect of its merge and only the _SND Attribute Data step restores them; because the two steps decide independently which rows to process, a mismatch leaves event data silently stripped. Newly inserted rows carry the same exposure, arriving with no attribute values and depending on that same second step to supply them. Until now nothing in the log identified the rows either step handled, so the problem could only be found by querying the database after the fact. ## Changes - Both ETL steps log the EventDataIds they handled, so the two logs can be diffed against each other. - The event data step logs on every path that touches exp.Object: merge, insert, and delete. - The merge also logs which rows are about to have their attribute values cleared. - The attribute data step logs the rows it received, the rows it wrote, and the difference between them. - The attribute data step reports when its source rows arrive ungrouped, which is one way the values fail to be restored. - Failures that abort the attribute data step now name the row that failed and note that the batch was left with its attribute values cleared. - Both steps also log the source rowversion span of each batch, so the rows a step actually handled can be placed against the incremental window the ETL logs for the run. - The attribute data step pairs each EventDataId it left unwritten with its rowversion; one falling inside the window points at the source view dropping the row rather than at a window mismatch. - Id lists are sorted and chunked for diffing, and omitted above 2000 ids so the full batches of an initial load don't flood the log. - Everything logs at debug, which the ETL job logger runs at by default; the diagnostics that cost queries or memory are skipped when debug is off. - No change to what the merge writes.
## Rationale Stop the SND ETL losing attribute values when the event data and attribute data steps' incremental windows drift apart, and make a blanked source attribute clear its stored value instead of leaving it stale. The event data step deleted and recreated each merged row's exp object, which discarded its attribute values, and only the attribute data step restores them; because that step computes its own incremental window, a row whose change landed in the seconds between the two steps computing their windows was cleared by the first and never seen by the second, and its values were gone permanently. The delete dates from the days before the attribute data step existed and has been redundant since that step began replacing values property by property. The attribute data source view also filtered blank values out, so a cleared attribute never reached the ETL and its old value survived. That was masked by the same delete, which is why it surfaces alongside this fix. The view change is a source-database ALTER VIEW and deploys separately from the module. The module change is safe to deploy first, since no null-valued rows arrive until the view changes. ## Related Pull Requests - #985 added the per-step EventDataId logging that pinpointed the lost rows. ## Changes - The event data merge no longer deletes and recreates each row's exp object, so attribute values survive a merge that the attribute data step's window does not also cover. - The attribute data source view passes blank values through as null-valued rows instead of filtering them out, so a cleared attribute reaches the ETL. - The attribute data step removes the stored value for a null-valued source row; a lookup value that fails to resolve still leaves the old value in place. - The diagnostic that reported which values a merge was about to clear is removed, since nothing is cleared now.
This PR has code changes related to ASI Observation datasets: - Updated column title name - Renamed dataset to BiocontainmentObservations - Removed LocationComments and carry-over columns - Merged latest release26.3-SNAPSHOT changes
cnathe
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated automatically.
Approve all matching PRs simultaneously.
Approval will trigger automatic merge.
View all PRs: https://internal.labkey.com/Scrumtime/Backlog/harvest-gitOpenPullRequests.view?branch=26.3_ff_bot_26.3.19