Design of functional simulation of renal cancer in virtual reality environments

  • Published on

  • View

  • Download


  • D ULY

    BO KED

    ACOb ctoman realCo n dTh owinrooM withpo nalGr stemen d thwit ittenRe s, anco entCo irst,wa n incomputer in the operating roomwas created. This is the first step in bringing VR technology to the operating roomto assist the surgeon directly. UROLOGY 66: 732735, 2005. 2005 Elsevier Inc.






    73penpartial nephrectomyhasbeen shown topro-vide effective, long-term tumor control, with

    eservation of renal function in patients with local-d renal tumors.1 More recently, minimally inva-e laparoscopic techniques to perform partial ne-rectomy have been developed in an attempt toplicate the open surgical experience.2 Comparedth open surgery, laparoscopic partial nephrectomyults in lower analgesic requirements postopera-ely, earlier hospital discharge, and more rapid re-n to normal activities. Currently used techniqueslaparoscopic partial nephrectomy may, however,ult in a greater incidence of positive marginsd an increase in intraoperative complications.

    Furthermore, the warm ischemic time has beenshown to be longer during laparoscopic than dur-ing open partial nephrectomy.3 In addition, tactilehaptic feedback is lost during laparoscopic proce-dures and may make localizing the tumor moredifficult than during an open procedure. There-fore, additional refinements are required so thatthe results of laparoscopic partial nephrectomywill meet or exceed the results of the equivalentopen surgical procedure.The ability to view a kidney and an associated renal

    mass in a virtual reality (VR) environment may facil-itate preoperative planning and successful surgicalremoval of a renal tumor. Our goal was to design asystem that can seamlessly convert a patients cross-sectional imaging data (computed tomography [CT]or magnetic resonance imaging) into a three-dimen-sional (3D) VR image that can be viewed andmanip-ulated by the surgeon preoperatively in a 3D VR en-vironment and in the operating room on a personalcomputer. This report focused on the initial stepstaken in achieving this goal.

    m the Division of Urology, The Ohio State University, Colum-, Ohio; Integrated Manufacturing Techologies Institute, Na-al Research Council of Canada, London, Ontario; Division ofology, University of Western Ontario, London, Ontario; De-tment of Urology, Queens University, Kingston, Ontario; andision of Urology, University of Ottawa, Ottawa, Ontario,nada.eprint requests: Bodo E. Knudsen, M.D., 4835 UniversityESIGN OF FUNCTIONAL SIMIN VIRTUAL REALIT


    ABSTRjectives. The preoperative planning of partial nephred surrounding tissue in three-dimensional (3D) virtualmmunications in Medicine computed tomography scaemodel can be transferred to a personal computer, allm.ethods. Computed tomography data from a patientlygonal mesh using Amira running on a desktop persoaphics Monster Onyx2 running the Linux operating syvironments: either the CAVE or a specialized desk calleh the model on a desktop personal computer was wrsults. A 3D model of the kidney, the multiple tumoruld be viewed and manipulated in a true VR environmnclusions. This project completed two major goals. Fs created and viewed in a VR environment. Second, aAsco

    spitals Clinic, 456 West 10th Avenue, Columbus, OH, 43210-8. E-mail: knudsen-1@medctr.osu.eduubmitted: October 14, 2004, accepted (with revisions): April2005


    Ty can be facilitated by the ability to view the tumor

    ity (VR). A technique to convert Digital Imaging andata into a fully 3D VR environment was developed.g the surgeon to view the 3Dmodel in the operating

    multifocal renal masses was converted into a 3Dcomputer with Windows XP Professional. A Siliconwas used to view the 3D stereo model in the VR

    e Immersadesk. An application to view and interactin C.d the associated systems was created. The modeland on a desktop personal computer.a 3D model of a kidney containing multiple massesterface to display the model on a desktop personalMATERIAL AND METHODS

    79-year-old man initially presented for a workup of micro-pic hematuria. Ultrasonography suggested the presence of bi-


  • lateresfronegopttha













    URral renalmasses. Subsequent cross-sectionalCTandmagneticonance imaging revealed bilateral renal tumors ranging in sizem approximately 1 to 2.2 cm. The metastatic workup wasative. The patient was presented with his possible treatmentions but elected not to proceed with surgical management att time, instead opting for follow-up imaging.he patients CT data was encoded and stored in Digitalaging and Communications in Medicine (DICOM) is a standardized format for encoding and transmittingdical data. The format includes a header, which containsevant patient data and the imaging modality used, and theual image data. The segmentation process follows and in-ves extracting the target data from the raw DICOM data.r this project, two methods of segmentation were used andpared.he first method of segmentation was performed (Robarts Research Institute, London, On-io, Canada). This program uses an algorithm termed mathe-tical morphology. Once segmented, the data are output indical Image NetCDF (MINC) format, which is based on thetCDF (University Corporation for Atmospheric Research,lder, Colo) generic data format. The segmented data are con-ted to a 3Dpolygonalmeshusing a Python/VTK script createdAtamai Incorporated (London, Ontario, Canada). The result-3D polygonal mesh is displayed using OpenGL.he second method for segmentation was performed usingira (TGS, SanDiego, Calif). Amira is a commercially availabletware package that performs advanced 3D visualization andume modeling. Although Amira offers an automated optionsegmentation, amanualmethodwasused for this project. Thea to be segmented were marked on the individual slices, al-ing for more precise segmentation. A generated surface wasn applied to the model using Amira. In addition, Amiraoothes the segmentation data and removes artifacts to betterect the original data.oth segmentation processes create a standard 3D objectBJ) file. Regardless of the segmentation process used, theJ files are handled identically at this point. The files areported into 3D Studio Max (Discreet, Montreal, Quebec,nada) using a plug-in called OBJ2MAX, version 3.1. Addi-nal smoothing and optimization of the models is then done.e final model is exported to the viewer.Silicon Graphics Onyx2 Reality Monster (Mountain View,

    lif) running the Linux operating system was used to displayfinal models in a VR environment. The models could be

    wed and interacted within one of two environments, either anlosed room called the CAVE (FakeSpace Systems, Marshall-n, Iowa) or using a specialized desk called ImmersadeskkeSpace Systems).o view themodel on aWindows-based personal computer, acialized programwas written in C usingMicrosoft Visualdio.NET (Microsoft, Redmond, Wash), and OpenGL wasd to display the model. An interface was created to allownipulation of the model in the first and third person perspec-s. The controls use the keyboard andmouse and allow for fullnipulation of the model in three dimensions.he University of Western Ontario Research Ethics Boardroved this study.


    he two segmentation processes were able to pro-ce models successfully from the DICOM CT datad were imported into 3D Studio Max for finaloothing and optimization. However, we consid-d themodel created tounsatisfactory. The model produced using Amiras superior owing to the additional precision thatOLOGY 66 (4), 2005software provided. 3D Studio Max was used toooth and optimize the segmented data to create arking 3D model (Fig. 1). The completed modelig. 2) was then exported and viewed in two differ-t 3D VR environments, the CAVE and the Immer-esk. In the virtual environments, themodel couldmanipulated to allow for viewing from any per-ective, including from inside the kidney (eg, col-ting system). Specialized eyeglasses are required tow the model in this environment. Video clips ofmodel were recorded in Amira. The clip demon-

    ates the model, but, because it is only a recordedeo, it lacks the 3D VR features of the actual modelideo clip 5).he model was transferred to a desktop Win-ws-based personal computer. An interface wasated that allowed the model to be viewed usinge first and third person perspectives. The camerawpoint is controlled using a combination of theuse and keyboard. With this system, the model

    URE 1. Mesh 3D kidney model as viewed in 3Ddio Max.

    URE 2. Completed model before export to 3D VRwer.733

  • coinfi




    nifia nelestathaTh

    patheficthetinscaa rancenT






    73uld be fully manipulated, allowing viewing fromnite angles.


    onventional radical nephrectomy has been theimary treatment for the removal and cure of local-d renal tumors. The preservation of renal functions become an increasingly important considerationrecent years, resulting in growing prominence forrtial nephrectomy as a treatment option.1 The in-ations for partial nephrectomy are widening andw include patients with normal contralateral renalction. The goal is to achieve the same local cancerntrol as that achieved with open radical nephrec-y while preserving overall renal function.4 Al-ugh no randomized trial has been published tote comparing the oncologic control with radicalrsus partial nephrectomy, results have shown thatrtial nephrectomy does provide durable long-termccess.1,2 Furthermore, for small, unilateral tumorss than 4 cm in diameter, the cancer-specific sur-al rate at 10 years approaches 100%.1aparoscopic partial nephrectomy is an emergingphron-sparing treatment option for the treatmentrenal tumors, and the results approach thoseieved with open surgery.2 The advantages of thearoscopic approach include shorter hospital stays,re rapid convalescence, and decreased use of nar-tics postoperatively.3 Potential disadvantages existth the laparoscopic approach, including longer in-operative warm ischemic times and increased ma-intraoperative and postoperative urologic compli-ions. Therefore, improvements in the techniquest be continually sought to enhance the advan-es further and eliminate the disadvantages of lapa-scopic partial challenge during partial nephrectomy is ade-

    ately localizing the neoplasm and determining thepth of tumor involvement. Small renal tumorsmaycompletely intraparenchymal and not seen at all atsurface of the kidney. Furthermore, some pa-

    nts have underlying disease processes, such as vonppel-Lindau disease, that can predispose to multi-al renal tumors.5 Real-time color Doppler ultra-nography may be used intraoperatively to aid inating a renal tumor and to gain additional infor-tion with regard to the tumor size, depth of inva-n, presence of prominent blood vessels in the vi-ity of the tumor, and the existenceof other satelliteors in the kidney.6lthough real-time ultrasonography can be of sig-cant assistance during partial nephrectomy, it hasumber of limitations. The operating room is anctronically noisy environment because of the sub-ntial quantity of electric medical equipment usedt can degrade the quality of the ultrasound image.e ultrasound probemay not be able to visualize all4rts of the kidney owing to limitations imparted bywound size and shape. Small lesions may be dif-

    ult to see, especially if excess pressure is applied toultrasound transducer. It may be difficult to dis-guish small, hypoechoic renal tumors from renalrring. Finally, a practical limitation is the need foradiologist to be present during the procedure forextendedperiod.Thismaynotbepractical at someters.7hree-dimensional VR imaging of the kidney may

    entually obviate or reduce the need for intraopera-e ultrasonography. By being able to view a 3D VRdel of the kidney during the procedure, the sur-on may be better able to perform the resection ofrenal mass accurately. The entire kidney will beily viewed, and small lesions that were previouslyn on cross-sectional imaging should be readilyntifiable.oll et al.8 used volume-rendered CT to createh-quality 3D images in 60 patients with renalsses to facilitate the preoperative and intraopera-e evaluations. In addition, a 3 to 5-minute longeotapewasprepared andviewedbefore thepatientderwent open partial nephrectomy. This videomonstrated thepositionof thekidney; locationandpth of the tumor extension, renal arteries, andins; and the relationship of the tumor to the collect-system. The investigators demonstrated that thisaging techniquemay avert the need formore inva-e preoperative imaging such as renal angiography.e important difference in our study was that thedels we used can be fully manipulated in the VRvironment, which provides infinite viewing per-ectives and, therefore, a theoretical benefit overrrent 3D volume-rendered CT images.ertain challenges must be overcome before thishnology can become a core component in the op-ting room. The process of converting the cross-tional imaging data into the 3D VR model is cur-tly both labor and time intensive. Identifying thisitation, we have been able to automate several keyps of the process. Still, our present method re-ins quite labor intensive andwould not be feasibleould multiple models be required within a shortriod.ltimately, to streamline the process of creating 3Dimages, it would be ideal to incorporate the seg-ntation and rendering software directly into theworkstation. This would eliminate the cumber-

    me stepsof exporting theDICOMdata to a separaterkstation. Furthermore, if future automation ofvarious steps in the segmentation and rendering

    ocess can be done within the CT workstation, ituld provide an efficient system that could readilyincorporated into the standard preoperativerkup of patients with renal masses.urrently, our models can be viewed in the VR

    vironments (CAVE and Immersadesk, FakespaceUROLOGY 66 (4), 2005

  • SypuOunipwhhutiopemolaphelapmoitoredvisT


    surgeons to control the camera system during theprocedure.11,12
















    URstems), as well as on a desktop personal com-ter using a keyboard and mouse combination.r goal is to allow the surgeon to view and ma-ulate the model during the surgical procedureile maintaining surgical sterility. An efficientman/machine interface is required. Viewing op-ns include using a flat-screen LCD monitor sus-nded above the operative field or using a head-unted LCD worn by the surgeon. For aaroscopic procedure, the surgeon could use thead-mounted LCD display to visualize both thearoscopic camera image and the VR data. Head-unted displays are equivalent to standard mon-rs in terms of surgical efficacy and may help touce neck strain by providing alignment of theual and motor axis.9,10he current mouse/keyboard interface is not suit-

    le for the operating room environment. It is vitalrgeons not be distracted from their focus on theocedure, and the human/machine interactionsst be natural and nonintrusive. In the VR viewingvironments, PINCH Gloves (Fakespace Systems)mbined with the Flock of Birds Electromagneticacker (Ascension Technology, Burlington, Vt; Fig.are used to manipulate the models by a series ofnd gestures and eye movements. An advantage ofs technique is that it allows the user tomanipulatemodel in the VR environment in a fashion similarreality. Miniaturization of the technology wouldow it to be adopted in the operating room environ-nt. Ideally, the surgeonwould have sensors placedhis hands underneath the surgical gloves in a non-rusive fashion. Alternatively, a voice-actuated in-face could be developed. The model could be ma-ulated using a series of commands spoken by thergeon. This interface would be similar to the ro-tic system, AESOP, used by many laparoscopic

    URE 3. CAVE VR viewing environment with PINCHves and Flock of Birds Electromagenetic Tracker.OLOGY 66 (4), 2005CONCLUSIONS

    3D VR model was created using the CT cross-tional imaging data obtained from a patient withltifocal bilateral renalmasses. Themodel was suc-sfully viewed and manipulated in a VR environ-nt (CAVEand Immersadesk, FakeSpace Systems).e VR model was also successfully viewed and ma-ulated on a desktop Windows-based personalmputer after a custom interface was created.his project successfully demonstrated the firstps necessary to bring VR technology into the op-ting room to facilitate performance of partial ne-rectomy. This technology could be applied to tra-ional open, aswell as laparoscopic, surgery. Futurerk is needed to validate this approach to demon-ate a tangible benefit to operative outcomes.

    REFERENCES1. Fergany AF, Hafez KS, and Novick AC: Long-term re-ts of nephron sparing surgery for localized renal cell carci-ma: 10-year followup. J Urol 163: 442445, 2000.2. Gill IS, Desai MM, Kaouk JH, et al: Laparoscopic partialhrectomy for renal tumor: duplicating open surgical tech-ues. J Urol 167(2 Pt 1): 469470, 2002.3. Gill IS, Matin SF, Desai MM, et al:Comparative analysislaparoscopic versus open partial nephrectomy for renal tu-rs in 200 patients. J Urol 170: 6468, 2003.4. Van PH: Partial nephrectomy: the standard approachsmall renal cell carcinoma? Curr Opin Urol 13: 431432,3.5. Choyke PL, Pavlovich CP, Daryanani KD, et al: Intra-erative ultrasound during renal parenchymal sparing sur-y for hereditary renal cancers: a 10-year experience. J Urol5: 397400, 2001.6. Walther MM, Choyke PL, Hayes W, et al: Evaluation ofor Doppler intraoperative ultrasound in parenchymal spar-renal surgery. J Urol 152(6 Pt 1): 19841987, 1994.7. Choyke PL, and Daryanani K: Intraoperative ultra-nd of the kidney. Ultrasound Q 17: 245253, 2001.8. Coll DM, Uzzo RG, Herts BR, et al: 3-dimensional vol-e rendered computerized tomography for preoperativeluation and intraoperative treatment of patients undergo-nephron sparing surgery. J Urol 161: 10971102, 1999.9. Hanna G, and Cuschieri A: Image display technologyimage processing. World J Surg 25: 14191427, 2001.0. Herron DM, Lantis JC, Maykel J, et al: The 3-Dmonitorhead-mounted display: a quantitative evaluation of ad-ced laparoscopic viewing technologies. Surg Endosc 13:755, 1999.1. Krapichler C, Haubner M, Engelbrecht R, et al: VR in-action techniques for medical imaging applications. Com-t Methods Programs Biomed 56: 6574, 1998.2. Unger SW, Unger HM, and Bass RT: AESOP robotic. Surg Endosc 8: 1131, 1994.

    Video Clips cited in this article can be foundon the internet at: http://www.goldjournal.net735



View more >