The IllusionHole for Medical Applications

Yoshifumi Kitamura*1 Takashi Nakashima*1 Keisuke Tanaka*2 Takeshi Johkoh*2 *1 Graduate School of Information Science and Technology

Osaka University *2 Graduate School of Medicine

Osaka University ABSTRACT In this study, we discuss a display table suitable for collaborative work environments for medical use. Using an interactive stereoscopic display system allows simultaneous observation of accurate stereoscopic images generated from volume data. We further investigate all requirements for design guidelines of the display system, including hardware configuration, rendering software to generate the stereoscopic images, and the interface system to operate the displayed images. Index Terms: B.4.2 [Input/Output and Data Communications]: Input/Output Devices, I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism Keywords: stereoscopic display, 3D user interface, 3D interaction, multiple users, collaborative work, volume visualization.

1 INTRODUCTION Recently, noninvasive tests have been intensively conducted

due to the ability to visualize human internal structures using 3D data collected by CT and MRI scans, and this allows us to study research simulations for medical diagnoses, surgery plans, and education using volume data. For example, there is a method to observe the state of the affected region in cross-sections mapped to the images and other arbitrary forms [1]. Another is to examine each layer of images that have been advance-sorted into distinct groups such as skin and bone [2]. In addition, automated systems have been investigated to analyze the location and size of tumors and to determine the distance between vasculatures or tissues and tumors in medical images [3]. It is, however, still common for devices exhibiting volume data to show individual cross-sections on 2D displays. Although there have been some attempts to simplify spatial analysis for stereoscopically displayed volume data [4], it has been challenging for multiple users to observe volume data simultaneously and interactively.

Until now, multiple users needed to share stereoscopic images in situations such as surgical procedures. This has been cumbersome due to the significant difficulty in identifying the precise locations of affected regions on display because the face of the image for one user creates distortion and lack of depth on the display for another when multiple users interactively change their points of view. It is also essential for multiple users to have indirect communication systems to interact with each other, such as eye contact and facial expressions, during the procedure.

Therefore, we discuss our concept concerning an interactive stereoscopic display system conducive for multiple users for medical use. We show volume data for medical applications with the appropriate stereoscopic displays for multiple users and demonstrated multi-user coordinate manipulability of images. 2 DESIGN OF A MEDICAL-USE INTERACTIVE STEREOSCOPIC

DISPLAY SYSTEM FOR MULTIPLE USERS Volume data are frequently utilized during surgery, medical

treatments, diagnoses, and education. More effective medical

decisions, surgical plans or simulations can be made if the appropriate stereoscopic images are provided to several surgeons and their supporters simultaneously. In fact, many imaging instruments, such as MRIs, have already been installed in operating rooms. Volumetric scanning by CT or MRI makes it possible to monitor the conditions of the affected regions in real time during surgical procedures. It is even more effective and meaningful for medical doctors and patients if the surgeons and their supporters are able to watch these stereoscopic images as they observe during the real surgical procedures.

One of the reasonable medical-use multi-user interactive stereoscopic display systems is to be installed into a table in which a horizontal monitor screen has been placed [5]. This is the most effective way to view the stereoscopic images from the vantage point of individuals standing around a table, as is the case during surgical procedures. It is also essential for multiple users to maintain cooperative work environments to communicate effectively through eye contact and facial expressions during the procedure. For this purpose, using optical equipment which includes a parallax barrier [6], mirror [7][8] and revolving screen [9], allows multiple users to observe the stereoscopic images with motion parallax in any direction. It is, however, impossible to naturally interact with the stereoscopic images since image formation is located in the optical device. For example, users are unable to directly point to a particular part of the image. Therefore, we propose utilization of our concept of a multi-user interactive stereoscopic display system, the IllusionHole [10]. For this concept, volume data for medical use with proper stereoscopic display should be open to multiple users. In addition, a system in which multiple users can mutually manipulate the images needs to be developed.

In the previous prototype IllusionHole, two liquid crystal projectors were utilized to form the left and right images [11]. Those images were divided using circular polarizing filters. However, these filters generate some degree of crosstalk depending on the variation of hue with the viewing direction, although the advantage of this system configuration is its relative low cost. Therefore, we need to design the IllusionHole system to include projectors with high brightness and resolution in an active time-sequential stereoscopic manner in order to properly show high quality translucent images for medical use. Although active time-sequential stereovision was used in the prototype system shown in [10], there were few choices for projectors and screens for the general display.

IllusionHole needs to display twice the images (left and right) as the number of users according to their viewpoints in real time. Furthermore, handling volume rendering remains a burden to the system. Thus, it is necessary to investigate the configuration of effective software in order to construct the drawing systems that can display the outcome of volume rendering to IllusionHole in real time. We must further discuss suitable interactive environments to manipulate the images on display. It is possible that multiple users around IllusionHole cooperatively work on a given surgical simulation using personal devices. We also need to consider sterility issues for the medical doctors using general interactive devices in an operating room. Collectively, multiple users surrounding IllusionHole should be restricted to performing

*1 2-1 Yamadaoka, Suita, Osaka, 565-0871 Japan. kitamura@ist.osaka-u.ac.jp

231

IEEE Virtual Reality Conference 2007March 10 - 14, Charlotte, North Carolina, USA1-4244-0906-3/07/$20.00 ©2007 IEEE

tasks for the images on display, whereas support staff can take care of the rest of the work processing. 3 SYSTEM CONFIGURATION 3.1 Hardware

The IllusionHole multi-user stereoscopic display consists of a display device and a display mask with a hole formed in its center [10]. For the display in this prototype, Barco DLP projector GALAXY (5,000 ANSI lumens, resolution SXGA 1,280 × 1,024) was placed horizontally and was reflected vertically upward by a planar mirror onto a horizontal display screen. The size of the display surface is 70 inches (1,388 mm × 1,110 mm). The mask hole was designed to be circular in shape with a radius of 200 mm in order to display various organs to scale.

The system configuration is shown in Figure 1. The viewing position of each user is detected using an ultrasonic 3D motion tracker (ZPS-VK, by Furukawa Co., Ltd.) [12], which subsequently calculates the center and radii of the drawing area of each user. Images for left and right eyes according to each drawing area were created by a PC (CPU: Xenon 2.8 GHz, memory: 1GB, Graphics card: NVIDIA Quadro FX 4000), and were submerged into each buffer. Output of each buffer can be processed in turn using quad buffer of the graphics board. Each user wears a pair of liquid crystal shutter glasses (CrystalEyes3, by Real D), and observes each corresponding stereoscopic image through the mask hole. 3.2 Drawing software

Effective drawing software needs to be developed for IllusionHole in order to display the outcome of volume rendering in real time. For drawings, two different software programs have been developed; a volume rendering software to visualize volume data, and IllusionHole software to display adequate stereoscopic images for all users. Volume Rendering Software: For the use of volume data in the generation of stereoscopic images, VirtualPlace (by AZE) [13] is utilized as volume rendering software to create images of perspective projection in real time. When data concerning the viewpoints of all users’ left and right eyes and input operations using interactive devices are entered, volume rendering images based on all input data are generated and stored in memory. IllusionHole Software: This software acquires data concerning viewpoints of all users’ left and right eyes and input operations using interactive devices. Volume rendering images generated by the volume rendering software based on these information are obtained from the memory. Subsequently, these images are adequately allocated and displayed. 4. INTERFACE

We designed the interface to reflect a model where separate functions in cooperative work environments in which multiple users surrounding the IllusionHole are restricted to performing tasks for the image on display, and in which support operators are able to take care of remaining operations. Concretely speaking, the IllusionHole display device and a PC monitor are used in this prototype. We have designed adequate interfaces for each display system because we assume a situation in which several medical doctors share the stereoscopic images interactively and cooperatively over the IllusionHole during surgery, and operators perform assistance to the doctors through the monitor. 4.1 Interface over the IllusionHole

In IllusionHole, there are no obstacles which block handling of stereoscopic images over the stereoscopic display region. One of the distinct features of this system is that the absolute location of the stereoscopic image is consistent with all directions from all users. Thus, we use a direct pointing device that can recognize the

3D input and user’s directions without any conflict. In the prototype system we introduced a stick controller with a 6 DOF tracker, which makes it possible to have detailed 3D entry and operations. The interface to operate the controller over the IllusionHole is described below. Clipping: This is a method used to display a vertical or horizontal cross section of volume data. When a user selects clipping among several interactions, a wire-frame cube on the volume data is displayed as shown in Figure 2. The user selects a clipping face among the six faces of the cube by operating the key in the controller. The user then moves the cross section in the volume data that corresponds to the clipping face, inward or outward within the volume, by operating the key. Thus, the user is able to handle clipping through the controller. Cutting: This is a different form of clipping. Arbitrary cross sections can be displayed using the 3D position and pose in the controller. As Figure 3 indicates, the user can display arbitrary cross sections intuitively by way of creating a vertical cross section facing towards the controller at a position located slightly distal. It is also possible to adjust the slope of the cross section with fine movements using the cross key in the controller even after a decision is made. Transparency control: A specific part of the images can be visualized by adjusting the degree of transparency corresponding to the concentration value in the volume data. For example, bones or vasculatures are exclusively visualized from volume data of the entire body by changing the threshold of the concentration value to visualize. The user can use up/down keys in the controller to control the degree of transparency. Figure 4 illustrates the extracted image of the skull from volume data of the head region. Slope adjustment: The user is able to observe the stereoscopic image from different angles by adjusting the slope of the stereoscopic image in the volume data. The slope of the stereoscopic image can be adjusted by assigning keys into the x, y and z axes on the coordinates system in IllusionHole. Control-ring: It is important for the user to operate an axis rotation on the absolute coordinates system easily and intuitively in the interactive work environment. In order to achieve this, the user needs to rotate the stereoscopic image on IllusionHole about the vertical (z) axis on the absolute coordinates system using the control-ring placed along the mask hole, as illustrated in Figure 5 [14]. The user can only rotate the stereoscopic image to the angle calculated by the tracker installed in the control-ring based on the positions of x and y on the coordinates. This enables each user to observe the stereoscopic image considering the rotation angle intuitively, and to manipulate the system simultaneously with other users. Magnification and reduction control: The user can also alter the size of the stereoscopic image by changing its magnification rate in the volume data. The magnification rate of the stereoscopic image can be adjusted by the key in the controller.

Figure 1: System configuration.

232

Image switches: Images which have been previously created and stored in volume data can be easily displayed by manipulating buttons in the controller. This is useful in the case that the user wants to change the same angle and magnification rate in the same region of the image, or modifies parameters such as transparency or clipping in order to observe and compare the differences between original and modified images. Modification of parameters, including transparency and clipping, can be done by keys in the controller as described earlier. It remains time consuming, however, to perform a single key manipulation, such as modifying parameters, re-volume rendering, and image updating, therefore, multiple images need to be switched, displayed and compared promptly. For example, the two images shown in Figure 6 (a) and (b) require approximately 15 manipulations when switched without using this function, but can be switched with only one operation using this function when two images have already been generated. It is even possible to switch more than three images. Pointing and annotation: The position of the stereoscopic image is set so that it appears to be in the same place from each user’s perspective, and the users are able to directly reach out to and point at a particular position of the stereoscopic image. The other users also recognize the position at exactly the same position of the image, unless the finger or the devise occludes the 3D image. In addition to this natural interface, a specific point of the stereoscopic image can be indicated by putting a visual marker in the 3D common workspace as shown in Figure 7. The marker can be observed from all the users at exactly the same position; therefore, it can be used as a useful annotation. Menu selection: When operating over the IllusionHole, each user can recognize the currently selected operation by viewing the content of operation as a text format displayed at the bottom of volume data as described in Figures 2 and 4. This text display is shown when the content of operation is selected, which the user can manipulate by using the keys in the controller. In addition, it is clear which operation is currently in use from the text color. As shown in Figure 3, the user can also hide the text if necessary. 4.2 Interface on the PC monitor

We anticipate that several medical doctors will share the stereoscopic image on the IllusionHole, and operators can then edit the images based on the doctor’s directions on the PC monitor. They can also operate the system using a mouse and key board as for regular PC operations. In addition to the fact that the content of interaction can be elaborately manipulated in IllusionHole using the mouse and key board on the PC monitor as mentioned earlier, fundamental settings including patient data and selection for the rendering mode can also be achieved in this system. 5 DISPLAY EXAMPLES

Figure 8 shows a situation in which four users observe volume data for a human abdominal image from the fourth user’s viewpoint as displayed on the IllusionHole. The images in (b), (c), and (d), represent those taken from viewpoints of the users A, B and C, respectively, around the IllusionHole. As the figure shows, each user is able to observe the image from each of the different view points, and all users have an interactive work environment in which to share the same stereoscopic image. An operator sitting behind the users (or doctors) around the IllusionHole can edit the image precisely based on the doctor’s directions.

In the case of pulmonary cancer, if a malignant tumor is found, excision of the pulmonary lobe, which has wide-ranging effects, can be modified to the elimination of a single pulmonary region in order to retain lung function and to maintain cardiopulmonary activity after surgery. The image displayed in the prototype system shows the boundaries of veins spread in sub-regions located between the pulmonary areas, which represent putative

pulmonary sub-segments. We speculate that using these types of images would enable the evaluation of respiratory function before and after surgery, and allow the performance of simulations before surgery. Figure 9(a) shows an example of the image separated into pulmonary regions. Surgical support is more effective if images of vasculatures and bones are superimposed with the previous image shown in the same figure (b), because such images help doctors to determine the locations of organs and tissues when surgery is performed. Figure 10(a) illustrates a situation in which four users observe volume data for an image separated into pulmonary region. The images in figures (b)-(d), represent each image as observed by each user, respectively, from their own viewpoint in (a). 6 DISCUSSION

In this prototype system, we demonstrated the combination of volume rendering software, VirtualPlace [13], and IllusionHole software, for the drawing software. VirtualPlace is capable of creating volume rendering images corresponding to perspective projection on a PC monitor in real time using general PC configuration settings. However, the frequency of updating images provided for each user is not actually high enough, although IllusionHole software can update viewing positions and image drawing areas for each user in real time. The update speed of the image drawing areas is 20 frames per second but the frequency of image updates of volume rendering is approximately 1 frame per second, with data of 220 × 220 × 311. We consider that using more efficient volume rendering software or hardware-based rendering will allow volume rendering at a higher speed.

The prototype system is composed of an interface device which can easily recognize the user’s directions and actions over the IllusionHole, and comes with functions including direct handling of clipping and transparency controls by users. This is particularly beneficial for investigating medical cases, surgical plans, and education. Deformation of the volume data displayed on the IllusionHole is one of the most expected future work. We speculate that in the near future, users will enact improvements of the interfaces by utilizing various devices. Examples include devices with obstacle sense feedback systems for surgical simulations, and devices to display specific information concerning stereoscopic images. Furthermore, we consider that the device would need to be disinfected or sterilized in the case of doctor use in an operating room. Sterilization is not needed if manipulation of the system is performed by assistants, rather than doctors themselves.

We believe that the prototype system is beneficial in collaborative situations; for example, a medical conference in which multiple medical doctors need to mutually understand and confer. In particular, the prototype system is effective for creating images for various organs which do not change their shape during surgery, such as bones, brain, breast, and liver. The images for these organs can be stored in volume data, synthesized as stereoscopic images and displayed in IllusionHole. These stereoscopic images can be prepared in advance of surgery, and/or utilized by doctors during surgery. We further anticipate that the system will have many applications if the imaging devices are co-installed with MRIs in operating rooms. The condition of the affected region will be monitored during surgery in real time by volume scanning using CT or MRI analysis. In addition, incision lines can be drawn on the patient body surface by mapping the coordinate systems on the IllusionHole and the operation table. 7 CONCLUSION

In this study, we discussed the concept of the IllusionHole, an interactive stereoscopic display system designed for multiple users. We tested if multiple users could simultaneously observe adequate stereoscopic images of data volume for medical use, and

233

manipulate the images interactively. We will further investigate and construct useful systems such as image displays for other organs and further develop specialized image interfaces during actual medical practice and clinical trials. ACKNOWLEDGEMENTS We would like to thank Prof. Fumio Kishino and Ms. Yuko Tanaka of Osaka University for useful discussions. This study was partially funded by the Strategic Information and Communications R&D Promotion Program of the Ministry of Internal Affairs and Communications. REFERENCES [1] M. J. McGuffin, L. Tancau, and R. Balakrishnan: Using

deformations for browsing volumetric data, Proc.of IEEE Visualization pp. 401-408 (2003).

[2] D. Weiskopf, K. Engel, and T. Ertl: Volume clipping via per-fragment operations in texture-based volume visualization, Proc. of IEEE Visualization, pp. 93-100 (2002).

[3] B. Preim, W. Spindler, K. J. Oldhafer, and H. Peitgen: 3D-interaction techniques for planning of oncologic soft tissue operations, Proc. of Graphics Interface, pp. 183-190 (2001).

[4] B. Reitinger, D. Schmalstieg, A. Bornik, R. Beichel: Spatial analysis tools for virtual reality-based surgical planning, Proc. of IEEE 3DUI, pp. 37-44 (2006).

[5] M. Agrawala, A. C. Beers, B. Frohlich, and Pat Hanrahan: The two-user responsive workbench: support for collaboration through individual views of a shared space, Proc. of SIGGRAPH, pp. 327-332 (1997).

[6] T. Endo, Y. Kajiki, T. Honda, and M. Sato: Cylindrical 3D display observable from all directions, SIGGRAPH Emerging Technologies (2001).

[7] Bimber, B. Frohlich, D. Schmalstieg, L. M. Encarnacao: The virtual showcase, IEEE Computer Graphics and Applications, Vol. 21, No. 6, pp. 48-55 (2001).

[8] R. Otsuka, T. Hoshino, and Y. Horry: Transpost: all-around display system for 3D solid image, Proc of ACM VRST, pp.187-194 (2004).

[9] G. Favalora, R. Dorval, D. Hall, M. Giovinco, and J. Napoli: Volumetric three-dimensional display system with rasterization hardware, Proc. of SPIE Photonics West #4297, pp. 227-235 (2001).

[10] Y. Kitamura, T. Konishi, S. Yamamoto, and Fumio Kishino: Interactive stereoscopic display for three or more users, Proc. of SIGGRAPH, pp.231-239 (2001).

[11] Y. Kitamura, T. Nakayama, T. Nakashima, and S. Yamamoto: The IllusionHole with polarizarion filters, Proc of ACM VRST, pp. 244-251 (2006)

[12] Furukawa Co.,Ltd. http://www.furukawakk.co.jp/ [13] AZE Ltd. http://www.aze.co.jp/ [14] M. Hachet, R. Watanabe, and Y. Kitamura: A collaborative interface

for the IllusionHole using a control-ring and a set of mice, Proc. of IEEE 3DUI, pp. 67-70 (2006).

Figure 2: Clipping. Figure 3: Cutting.

Figure 4: Transparency control. Figure 5: Control-ring.

(a)

(b)

Figure 6: Image switches.

Figure 7: Pointing and annotation.

(a)

(b)

(c) (d) Figure 8: Example of the IllusionHole shared by four users (from the fourth user’s viewpoint). Volume data of a human body is displayed on the IllusionHole.

(a)

(b)

Figure 9: Display of segment-divided images of lugs on the IllusionHole.

(a)

(b)

(c)

(d) Figure 10: Volume data of segment-divided images of lugs is displayed on the IllusionHole.

234