CycleStack: Inferring Periodic Behavior via Temporal Sequence Visualization in Ultrasound Video

Teng-Yok Lee1, Abon Chaudhuri1

Fatih Porikli2, Han-Wei Shen1

1The Ohio State

University

2Mitsubishi Electric Research

Laboratories

Respiration-gated Therapy (1/2) Treatment procedures need

accurate identification of the tumor’s location

Reducing the damage to the surrounding normal tissues

Increasing the efficiency of the treatment

The patient’s respiration can cause the periodic movement of the organs/tissues/tumors

Respiration-gated Therapy (2/2)

Utilizes the patient’s respiration signal to control the gate of the beam

Requires the therapist to verify the synchronization between the tissue’s motion and the respiration signal

Using ultrasound to sense the tissue’s motion

Requirements & Challenges

Detect the periodic cycles inside the (ultrasound) video

Burden in the short term visual memory

Visually verify the synchronization between the motion (of the tumor or tissues) and signal (respiration)

Different visual representation of the video and the signal

Post-therapy Analysis

For the purpose of quality assurance

Challenge

Watching video and signal can be time-consuming and tedious

Video visualization

Courtesy: Daniel and Chen, Video Visualization. In IEEE Vis ‘03

Our Approaches

Movement trace image Visualization of the motion in a video

Side-by-side visualization between the video and the signal

CycleStack Overview of the entire video and the signal

Movement Trace Image

Reveal the periodic cycles inside the video

Side-by-side comparison of the motion and the signal

time

Movement Trace Image

Movement Trace Image: Result

Synchronous time steps Synchronous time steps

Asynchronous case Opposite case

Movement Trace Image: Issue

Comparison in a sliding window

Too many cycles in the sliding window is not effective

CycleStack (1/2)

Preserves all time steps on the screen Overview of the video and signal

Organizes the movement trace image and signal into small multiples Efficient utilization of the screen space

CycleStack (2/2)

Input Respiratory Signal

2. Stack the cycles

Movement Trace Image

1. Segment the signal into periodic cycles

2. Stack the cycles

= CycleStack

Periodic Cycle Segmentation

1-threshold segmentation1-threshold segmentation of false cycles

CycleStack: Layout

Movement Trace Image

Segmented Cycles

CycleStack

Inhale Phases

Exhale Phases

Time

CycleStack: Asynchronous Cases

CycleStack: Opposite Cases

Synchronous cycles

Oppositecycles

Online CycleStack

Issue

Growing CycleStack can exceed the display window

Solution

CycleStack can be automatically rescaled to fit the display window

User Study: Setting (1/2)

Purpose Evaluate the response time and accuracy to detect non-

synchronous cases

Testing videos Asynchronous cases (1 synthesized & 1 ultrasound video)

Opposite cases (1 synthesized & 1 ultrasound video)

User Study: Setting (2/2)

18 participants in 2 groups One group watched videos and signals in separate

windows

The other group watched the CycleStack

Measurement Record the response time when the participants identified

non-synchronous cases

Compute the detection rate and average response time

User Study: Asynchronous Cases Observation about CycleStack

Higher detection rate in the ultrasound video

Slightly slower response time and larger standard deviation

Explanation

CycleStack keeps all cycles on the screen, allowing the viewer to detect the asynchronous cycle even after it has passed

Video Method Detection Rate Mean ±Std. Dev.

Synthesized Video/Signal 100% 1370.67±13.46

CycleStack 100% 1388.11±25.66

Ultrasound Video/Signal 44.4% 1887.00±32.72

CycleStack 88.9% 1966.38±141.83

User Study: Opposite Cases

Observation about CycleStack

Slightly slower response time but smaller standard deviation in the ultrasound video

Explanation

Opposite cases last longer than asynchronous cases

Some viewers of video and signal can detect it, even in late

Video Method Detection Rate Mean ±Std. Dev.

Synthesized Video/Signal 100% 1155.86±55.84

CycleStack 100% 1171.00±51.40

Ultrasound Video/Signal 88.9% 2488.88±206.06

CycleStack 100% 2557.00±102.36

Conclusion

Contribution An effective visualization that allows the viewer to compare

and to verify the synchronization between a video and a signal

Visual detection of non-synchronous cases during the online therapy and offline post-therapy analysis

Questions?