CADPH possible connection discrepancies

[OHoward2023]

Problem statement

3 facilities reporting data to LA County/CADPH have experienced data transfer issues since November of 2024. Confirmation from CalREDIE states that all data received should be also received by LA county.

What you need to know

LA county inquired about receiving data submission from CEDARS-SINAI MEDICAL CARE FOUNDATION, ALPHA CLINICAL LABORATORY, INC, and Advanced Molecular Diagnostics. There has been confirmation of data sent and received via SimpleReport. Confirmation of no interruption in connection to CA via ReportStream.

Daily Data submission reports show multiple one result file upload.

-

To do

• Confirm SimpleReport connection is running as intended for ReportStream.
• Please confirm if there are CA connection issues to ReportStream and update Support team so we can reach out to the users.

[MikaelahD13]

Hi @victor-chaparro, may you please take a look at this ticket and see what is causing this discrepancies?

[victor-chaparro]

@TaneaY Starting November we moved California's connection from SFTP saphiresftp1.cdph.ca.gov to REST https://saphireapi.cdph.ca.gov/hl7 . If reports are not being routed to LA county that would be on CA-DPH to look into. If SimpleReport can send us some report ids from those facilities we can make sure the reports are getting routed to CA.

[OHoward2023]

@victor-chaparro
Latest Report IDs from SimpleReport for the following facilities:

Cedars-Sinai Medical Care Foundation

ALPHA CLINICAL LABORATORY

Advanced Molecular Diagnostics

We confirmed successful CSV upload via SimpleReport and will confirm with CADPH that they are receiving results from the correct file path.

[victor-chaparro]

We have not been getting any of these reports through the Universal PIpeline, this seems like a larger issue. I will get the Simple Report team involved.

[Jcavallo7]

Hi All, user came back with following response:

[victor-chaparro]

Hi, no we identified the error was because SimpleReport was not routing these reports to ReportStream and were being dropped. We have identified a solution to resend some of the dropped messages, but not all. @chris-kuryak Will have more direction on how we are going to communicate these to our users next week.

[chris-kuryak]

I am filling out an incident report now. I will schedule time with the SimpleReport team to discuss early next week. I will communicate any necessary updates to the support team ASAP.

[OHoward2023]

Thank you both for the updates. Should we alert CA of the interruption and anticipation of resubmissions, or should we wait until you all collaborate for resolution? Support is still awaiting confirmation of prior file path update from CADPH.

[chris-kuryak]

@OHoward2023 The issue was fixed on Jan 23, and you can message as such to CA.
Regarding resubmissions, we are still developing a plan and timeline for resubmitting missed messages. Once we begin resubmitting messages, we can let you know so that you can inform CA. I am estimating next week, but it is still TBD.

[OHoward2023]

@chris-kuryak We have drafted the following. Please let us know if this is okay to send.

Hi all,

Thank you for your patience.

The engineering team has confirmed that there was a communication error between SimpleReport and ReportStream, and has notated that the fix was complete on January 23rd.

However, we are still developing a plan and timeline for resubmitting missed messages. We anticipate that this will be done relatively quickly and will reach out with more information as we receive.

Thank you,

[chris-kuryak]

@OHoward2023 Looks good to me! Thanks for sending.

[TaneaY]

Response from user.

I wanted to follow up on the reporting issue that impacted COVID-19, Influenza, and RSV results transmitted via SimpleReport due to a communication error between SimpleReport and ReportStream, which affected data flow to CalREDIE ELR.

Latest Update

The ReportStream engineering team confirmed that the issue was resolved as of January 23rd. However, we are still awaiting a plan and timeline for resubmitting the missed messages predating that date. Could you provide an update on any progress regarding the resubmission process?

Impact Overview

This issue resulted in a backlog of results dating back to November 1, 2024, affecting multiple facilities. A significant portion of the backlog originates from Ananda Analytical Laboratories LLC (05D2217697) [RSV, Influenza, and COVID-19] due in part to their onboarding completion coinciding with the transmission error.

Affected Laboratories:

Laboratory Name

Ananda Analytical Laboratories LLC (05D2217697)

Advanced Molecular Diagnostics, LLC (31D2053667)

Cedars-Sinai Medical Care Foundation (05D2138287)

Cedars-Sinai Medical Care Foundation (05D2098707)

Personalized Pediatrics Lab (05D2281427)

Alpha Clinical Laboratory, Inc (05D0999149)

We appreciate any updates you can provide on the timeline and process for addressing this backlog. Please let us know if there’s anything we can do to support or facilitate resolution efforts.

Response from Support:

Thank you for your patience.

We are still developing a plan and timeline for resubmitting missed messages. We anticipate that this will be done by the end of February and will reach out with more information as we receive updates.

Thank you,

[chris-kuryak]

@TaneaY
Thanks for the update.

An email was sent out from SimpleReport to impacted organizations this morning (Fri 2/14) informing them to resubmit affected results through SimpleReport by the end of February. I can confirm that every organization listed in the email from CA (your comment) was included in the email outreach. So I would expect to see those results come in over the next two weeks.

You can feel free to update CA with that information, and you can indicate that we can reconnect with them at the end of February to identify if any results are still outstanding.

How does that sound?

[TaneaY]

@chris-kuryak -Follow-up requested from LA County:
**I wanted to follow up on the reporting issue that impacted COVID-19, Influenza, and RSV results transmitted via SimpleReport due to a communication error between SimpleReport and ReportStream, which affected data flow to CalREDIE ELR.

Latest Update

The ReportStream engineering team confirmed that the issue was resolved as of January 23rd. However, we are still awaiting a plan and timeline for resubmitting the missed messages predating that date. Could you provide an update on any progress regarding the resubmission process?

Impact Overview

This issue resulted in a backlog of results dating back to November 1, 2024, affecting multiple facilities. A significant portion of the backlog originates from Ananda Analytical Laboratories LLC (05D2217697) [RSV, Influenza, and COVID-19] due in part to their onboarding completion coinciding with the transmission error.

Affected Laboratories:

Laboratory Name

Ananda Analytical Laboratories LLC (05D2217697)

Advanced Molecular Diagnostics, LLC (31D2053667)

Cedars-Sinai Medical Care Foundation (05D2138287)

Cedars-Sinai Medical Care Foundation (05D2098707)

Personalized Pediatrics Lab (05D2281427)

Alpha Clinical Laboratory, Inc (05D0999149)

We appreciate any updates you can provide on the timeline and process for addressing this backlog. Please let us know if there’s anything we can do to support or facilitate resolution efforts.

Thank you for your time, and I look forward to your response.**

I do not see Advanced Molecular Diagnostics, LLC (31D2053667) on the resubmission contact list from SimpleReport all others above have not responded yet.

Suggested Support Response:
**Hi Mario,

We appreciate your patience.

We have reached out to the affected organizations and are still awaiting resubmission for those included. We are working diligently to have more updates later this week and will reach out with more information as we receive.

Thank you,**

[chris-kuryak]

@TaneaY

It appears that all of the senders in the list they responded with are in the list that SR said they reached out to. The only weird part is that "Advanced Molecular Diagnostics" was marked as being in NJ, not in CA. So you may want to check with SR to confirm they are the same sender.
Otherwise, your response looks accurate. We will be reviewing the list tomorrow with next steps. So you can also add to your response that you will be able to provide an update on the status of resubmissions next week.

[TaneaY]

3/5 confirmed Advanced Molecular in NJ is also reporting to CA email sent to user
"We appreciate your patience and apologize for any inconvenience.
We have reached out to the affected organizations and requested resubmission by 2/28/25. but are still awaiting resubmissions for those included. We are working diligently to have more updates within the week and will reach out with more information as we receive."

Support will reach out with updates after 3/17 as SR sent follow-up to resubmission confirmation on 3/13.

[chris-kuryak]

@TaneaY Any updates on this regarding support reaching out?

[TaneaY]

@chris-kuryak the above comment was the reply sent to the user with mention of updating after 3/17. This needs to be added to Incident report agenda today to discuss resubmission updates that can be sent to user. Confirmation of resubmissions has not been received via SR as of today @ 12:30pm est.

[chris-kuryak]

Just checking, but did you get what you needed out of today's discussion @TaneaY ? Happy to chat more if we need to.

[TaneaY]

@chris-kuryak I can follow up today just to keep CA updated with the following:
"We appreciate your patience and apologize for any inconvenience.
We have reached out to the affected organizations and requested resubmission again. However, we are still awaiting resubmissions for the organizations included in your confirmation request. We are working to develop a plan to decrease potential data loss if organizations fail to respond and will reach out with more information in early April."

[TaneaY]

@chris-kuryak Above sent to CA on 3/25. User replied 3/26 that Advanced Molecular Diagnostics confirmed resubmission to LA county on 3/19. Impacted org list not updated to reflect. Confirmation email received Support escalated to SR Product team for confirmation and update to incident report documentation.