Embed Size (px)
Citation preview
HL7 FHIR DevDays 2020, Virtual Edition, November 17–20, 2020 | @FirelyTeam | #fhirdevdays | www.devdays.com/november-2020
HL7®, FHIR® and the flame Design mark are the registered trademarks of Health Level Seven International and are used with permission.
Reconciliation of FHIR Data With Incoming Data
Roman Polyanovsky - Google Cloud
1
Who am I?
• Roman Polyanovsky
• Software Developer
2
What is this talk about?
- This talk is intended as a high level discussion
- Primarily of challenges of merging streams of data into FHIR at scale
- We'll also discuss ideas for solutions to these challenges as well as their
gaps and pitfalls
3
The world in FHIR
- FHIR represents a complete state of the world
- Other formats (HL7v2 for example) are not a complete state
- They are transactional/event based
- They represent a continuous stream of updates
4
For example: HL7v2 → FHIR
ORU_R01MSH|...SFT|...PID|...PD1|...OBS|...ORC|......
Input Message
5
For example: HL7v2 → FHIR
ORU_R01MSH|...SFT|...PID|...PD1|...OBS|...ORC|......
MessageHeader
Patient
Observation
Encounter
DiagnosticReport
Input Message
Mapping
Output FHIR Resources
github.com/GoogleCloudPlatform/healthcare-data-harmonization
6
For example: HL7v2 → FHIR
ORU_R01MSH|...SFT|...PID|...PD1|...OBS|...ORC|......
MessageHeader
Patient
Observation
Encounter
DiagnosticReport
Input Message
Mapping
Output FHIR Resources
DestinationFHIR Store
MessageHeader
Patient
Observation
Encounter
DiagnosticReport
Reconciliation
Merging continuous
updates into the current
state of FHIR
7
- Concurrency - many streams (maybe different formats), no synchronicity
Challenges
MessageHeader
Patient
Observation
Encounter
DiagnosticReport
ORU_R01MSH|...SFT|...PID|...PD1|...OBS|...ORC|......
MessageHeader
Patient
Observation
Encounter
DiagnosticReport
Reconcile
ADT_A08MSH|...SFT|...PID|...PD1|...AL1|...AL1|......
MessageHeader
Patient
AllergyIntolerance
Encounter
PractitionerRole
Reconcile
8
Challenges
- Concurrency - many streams (maybe different formats), no synchronicity
- Out of Order Updates - (maybe severely) delayed updates or backfills
Message 0 Message 4 Message 1 Message 2 Message 3
9
Challenges
- Concurrency - many streams (maybe different formats), no synchronicity
- Out of Order Updates - (maybe severely) delayed updates or backfills
- Maintaining accurate history - must avoid polluting FHIR store
10
Challenges
- Concurrency - many streams (maybe different formats), no synchronicity
- Out of Order Updates - (maybe severely) delayed updates or backfills
- Maintaining accurate history - must avoid polluting FHIR store
- Arbitrary merge logic - Merges spanning many resources with bespoke
merging requirements
11
Using an intermediate store or "staging" area
Concept Architecture
HL7v2 Message
HL7v2 Message
HL7v2 Message
12
Using an intermediate store or "staging" area
Concept Architecture
HL7v2 Message FHIR Resources
HL7v2 Message FHIR Resources
HL7v2 Message FHIR Resources
13
Using an intermediate store or "staging" area
Concept Architecture
HL7v2 Message FHIR Resources FHIR Resources
HL7v2 Message FHIR Resources
HL7v2 Message FHIR ResourcesStaging
14
Using an intermediate store or "staging" area
Concept Architecture
HL7v2 Message FHIR Resources FHIR Resources
HL7v2 Message FHIR Resources
Some Other Message FHIR Resources
Staging
DestinationFHIR Store
Patient
Observation
Encounter
DiagnosticReport
Reconcile
MessageHeader
15
Using an intermediate store or "staging" area
Concept Architecture
FHIR Resources
Staging
DestinationFHIR Store
Patient
Observation
Encounter
DiagnosticReport
Reconcile
MessageHeader- Concurrency- Staging area is just full of
immutable resources- Mapped messages are just
loosely dropped in- Gap: Need to execute
searches/several fetches to find groups of resources for reconciliation
- Reverse linking can help with this, as well as maybe clever labelling
- Search by extension would also help
16
Using an intermediate store or "staging" area
Concept Architecture
FHIR Resources
Staging
DestinationFHIR Store
Patient
Observation
Encounter
DiagnosticReport
Reconcile
MessageHeader- Out of Order Updates- Resources can be
(re)sorted upon reconciliation and (re)merged
- Pitfall: Need to be careful about picking the ordinal to sort by
- Links to a timestamped MessageHeader (can even duplicate the timestamp across all resources during mapping)
17
Using an intermediate store or "staging" area
Concept Architecture
FHIR Resources
Staging
DestinationFHIR Store
Patient
Observation
Encounter
DiagnosticReport
Reconcile
MessageHeader- Maintaining History- Sorted + grouped
resources from staging can paint an accurate history
- Gap: FHIR history is immutable in the Destination Store*
- Make it mutable or use Staging/Other store for historical queries.
18
Using an intermediate store or "staging" area
Concept Architecture
FHIR Resources
Staging
DestinationFHIR Store
Patient
Observation
Encounter
DiagnosticReport
Reconcile
MessageHeader- Arbitrary merge logic- The Reconciliation service
can pull all necessary resources from the intermediate store, and implement merge logic bespoke to each group
- Gap: May need to fetch (too) many resources from staging
- Remedy with "snapshots" and Container resources
19
Conclusions
- Reconciling FHIR resources is complicated by
- Concurrent streams/race conditions
- Out of order messages
- Merging instead of overwriting (e.x. HL7v2 omitted fields: ...|||...)
- Searching/querying limitations
- An intermediate staging area/store simplifies some of these problems by
- Introducing immutability
- Allowing sorting and resorting of messages
- Maintaining a complete history of individual events
20
The world in FHIR
-
21
Contact
• During DevDays, you can find / reach me here:• Via Whova App – Speaker’s Gallery
• If I can figure out what this is or how it works
• Email: [email protected]
22
Q&A
