J Oral Maxillofac Surg 55:456-462, 1997

Virtual Reality for Orthogna thic Surgery: The Augmented Reality Environment

Concept ARNE WAGNER, MD* MICHAEL RASSE, MD, DDS,t

WERNER MILLESI, MD, DDS,$ AND ROLF EWERS, MD, DD, DDS, PhD,§

Purpose: The objective of this study was to apply virtual reality technology to osteotomies of the facial skeleton.

Materials and Methods: Augmented reality can be considered a hybrid of virtual and real environment spaces, which are coregistered and simultaneously visualized. Using a see-through HMD (head-mounted display) and Interven- tional Video Tomography intraoperatively, partial visual immersion into a pa- tient-related virtual data space augments the surgeon’s perception as shown in an experimental study and clinical applications.

Results: Without limiting the surgical judgment, offering continuous observa- tion of the operating field, the presented technology additionally provides visual access to invisible data of anatomy, physiology, and function and thus guaran- tees unencumbered and fluent surgery.

Conclusion: Despite current shortcomings, augmented reality technology proved to be particularly well suited for use in osteotomies of the facial skeleton.

Virtual Reality (VR) has captured the imagination of the media and the public, yet its applications in medicine are only beginning to be explored. A search of the Medline Database of the National Library of Medicine performed in January 1995 showed that re- ports of applying virtual reality tools in medicinelm6 have increased by 500% since their introduction into the medical literature, a fact that gives evidence of the growing popularity and importance of VR in medi- cine.7

In 1987 Watanabe et a18,9 introduced a new device for computed intraoperative navigational assistance. Guided by the ‘ ‘Neuronavigator,” they fixed the pa- tient’s skull with a head clamp to assure intraoperative

Received from the Clinic of Maxillofacial Surgery, University of Vienna, Waehringer Guertel 18-20, 1090 Vienna, Austria.

* Resident. t Senior Consultant. $ Consultant. Q Professor and Head of Clinic. Address correspondence and reprint requests to Dr Wagner: Uni-

versitaetsklinik fur Kiefer-und Gesichtschirurgie, AKH Wien, Waeh- ringer Guertel 18-20, 1090 Wien, Austria.

0 1997 American Association of Oral and Maxillofacial Surgeons

0278-2391/97/5505-0005$3.00/O

calibration. Watanabe et al used a pointing device in connection with a mechanical tracking arm that al- lowed real-time visualization of the tip of his instru- ment as an overlay graphic point on computed tomog- raphy (CT) images. Advantages of this and similar systems”-13 were appreciated, and their application in neurosurgical procedures were found to be useful.

In an attempt to overcome the disadvantages inher- ent in current systems, the Clinic of Maxillofacial Sur- gery at the University of Vienna adapted the VPS sys- tem by ARTMA Biomedical Inc (Vienna, Austria), for use in the maxillofacial region. The ultimate objective was to provide support to the surgeon during the opera- tion by visualizing invisible topography and instru- ments in a nonobtrusive way. Our goal was to explore this concept and to promote the development of a sys- tem in which the surgeon was partially immersed within the virtual data space. Partial immersion is a hybrid of virtual and real environment spaces and thus can be termed composite reality14 or augmented reality. To accomplish this, the surgeon wears a see-through head-mounted display (HMD)15-1s to see virtual medi- cal structures orthotopically superimposed on the real patient. Thus, the technology should enhance rather than replace the real environment, and thereby aug-

WAGNER ET AL 457

FIGURE 1. Computer videophotograph as seen through the HMD. Sterolithographic skull model with virtual overlay graphic indicating the preplanned Le Fort I osteotomy line in blue, both palatal arteries and the tip of the saw in red, handpiece of the saw in blue.

mentation of the surgeon’s perception should improve rather than limit surgical judgment. When applying VR technology to osteotomies of the facial skeleton, the objective was to transfer points, lines, and planes from cephalometric drawings, stereolithographic skull models, splints, and imaging data to the patient. Virtual visualization should be accomplished intraoperatively by superimposition of these data.

Materials and Methods

Superimposition of virtual points, lines, and planes on the patient was made possible by defining a spatial relationship between these structures and fiducial points. Fiducials are defined points on a patient that are also discernible in images used for three-dimen- sional (3D) data acquisition. Preoperatively, anatomic structures, their positional changes caused by surgical procedures, and the osteotomy are simulated in virtual space. During surgery, the position of the bone seg- ment or soft tissue to be obtained is visualized as a real-time overlay. The osteotomy is visualized and su- perimposed on the patient. This superimposition is brought to the surgeon’s eye by use of a partially im- mersive HMD (Fig 1). The surgeon watches the op- erating field and, at the same time, recognizes com- puter-generated navigational data in a pseudo-natural and unencumbering way. The proprietary principle of IVT (ARTMA), is the dynamic correlation of video stereophotogrametry with the patient’s real-time posi- tion data. These data are provided by 6-degrees-of- freedom sensors, tracking the subject’s anatomy, and additionally, medical instruments and imaging devices.

Advancements in VR technology have led to the de- velopment of various digitizing technologies that match

virtual environments to the real world. In the presented configuration, the ARTMA Virtual Patient System is equipped with an electromagnetic tracking device. A rigidly attached sensor continuously registers spatially defined data during the real-time movement of any structure and provides permanent updating in real time. A video camera is used for acquisition of live real-time single-image frames. For every individual video frame, the position and orientation of the charged couplet de- vice (CCD) imaging plane is unambiguously defined and the optical characteristics and distortion of the video image are simulated in a mathematical system of equa- tions. It is then possible to project sensor data on the imaging plane for simultaneous orthotopic matching of virtual (imaged) data with real world (patient) data. The virtual data then become any structure identified and marked in the cephalograms, CT scans, or magnetic resonance (MR) images that can be backprojected in the correct spatial position on the IVT image set. The unique advantage of this interactive backprojection tech- nology is manipulation of the virtual computer-gener- ated graphic overlay production.

Anatomic structures are reconstructed in three di- mensions from planar cephalograms by stereophoto- metric analysis. 1g*20 A stereophotometric equational al- gorithm involving at least eight intrinsic fiducial markers is used. They are not fixed but are recon- structed and matched by digitizing their position using a digitizing stylus. The position of the fiducial markers can be confirmed at any time during the clinical study or surgical procedure and updated so that the aug- mented reality field can be continuously monitored for accuracy by the surgeon.

In this system, the extension of the IVT fusion of the virtual imaged data with digitally processed live video sequences can be considered augmented reality of live image fusion. The partial immersive visor display deliv-

FIGURE 2. Hardware setup. The operating microscope provides video input. Note the computer workstation and an HMD prototype.

458 AUGMENTED REALITY ENVIRONMENT FOR ORTHOGNATHIC SURGERY

FIG1 JRE! light weigl with an video 3 can

Surgeon usi ng a see-through I -IMD

dditional mini ature -a intraoperativ ely.

ers spatial localizing and navigational information and virtual imaging of nonvisual structures to the operating surgeon in an unencumbered, less restricting way, with- out interrupting the flow of the surgery.

Clinical Experience

Before actual surgical intervention, the planned pro- cedure was simulated in a preclinical experimental study using a stereolithographic skull model.21 Figure 2 shows the intraoperative visualization of a pre-

planned Le Fort I osteotomy line as a blue virtual overlay graphic. The handpiece of the saw is visualized in blue, whereas the blade of the saw and the palatal arteries on both sides are discernible as red lines. When planning the model operation, we assumed that this could reduce the risk of damage of the palatal vessels intraoperatively.

During the actual Le Fort I osteotomy, we found it appropriate to visualize the positioning of the upper jaw in combination with the simultaneous soft tissue

FIGURE 4. Sterile sheathed 3D sensor rigidly attached to the fore- head tracks every movement of the head.

FIGURE 5. Image coordinate transformation The surgeon uses a 3D sensor stylus for digitizing fiducial points, thereby matching the coordinate systems of virtual preoperative planning and the real patient.

WAGNER ET AL

FIGURE 6. Computer video- photograph as seen through the HMD. Virtual overlay graphic of the preplanned osteotomy line superimposed on the maxillary bone guides the tip of the oscil- lating saw (arrow).

changes. The predicted and intended soft tissue changes could be compared with the actual acquired profile in real time. If not determined by the occlusion, intraoperative decisions concerning the amount of ad- vancement could be made with respect to the cephalo- metric analysis. The cephalometric analysis can be vis- ualized by virtual overlay graphics intraoperatively. Conversely, the patient’s actual profile line can be real time overlayed on the cephalograms. This applies to any point, line, plane, or structure of the patient’s face.

Figure 3 shows the hardware setup. In this particular case, an operating microscope was used for video in- put. When investigating the benefits of various HMDs

(Fig 3), this see-through HMD in combination with a miniature camera provided reasonable visualization quality. Preoperatively, a sterile sheathed 3D tracker was rigidly attached to the patient’s forehead and draped completely (Fig 4), defining a patient-related coordinate system. In the next step, fiducial markers were digitized for image coordinate transformation. Repeated intraoperatively, this controls accuracy (Fig 5). Figures 6 and 7 show the operating field from the surgeon’s perspective. The preplanned osteotomy is superimposed on the maxilla, serving as intraoperative navigation information for the oscillating saw. The re- sulting Le Fort I osteotomy matches the virtual super-

FIGURE 7. Surgeon’s per- spective. When using a see- through HMD the physician views the operating field en- chanced by fusion of cyber- space-graphics with the real world. The yellow line indicates the preoperatively planned oste- otomy.

460 AUGMENTED REALITY ENVIRONMENT FOR ORTHOGNATHIC SURGERY

FIGURE! 8. Computer video- photograph of four split-screen views showing virtual control of the positioning procedure. A virtual interincisional point (arrows) provides numeric in- formation about the upper jaw’s spatial position in real time.

imposition. Virtual control of positioning of the max- illa (Fig S), by use of a virtual interincisal points providing numeric information on the spatial position of the upper jaw, was used in addition to the prefabri- cated occlusal splint.

When simulating soft tissue changes of the upper lip, as shown in Figures 9 and 10, the preoperative and preplanned postoperative overlay graphics on the drawing as well as on the postoperative live video

show accurate matching of the virtually planned proce- dures and the surgical result. The blue lines correlate with the upper lip contour; the two colored points indi- cate the preoperative and postoperative incisor position (Figs 9, 11).

Discussion

Since 1989, we have investigated various applica- tions of augmented reality environment technology in

FIGURE 9. Computer video- photograph of four split-screen views of the subject. The two colored points (short arrows) correspond to the preoperative and postoperative incisor posi- tion. The blue lines (long arrows) indicate the upper lip contour and thus show planned soft tissue changes intraopera- tively.

WAGNER ET AL 461

FIGURE 10. Computer video- photograph as seen through the HMD. Virtual superimposition of these graphics on the patient show correct soft tissue ad- vancement.

maxillofacial surgery.22-26 Subsequently, we used VR technology in orthognathic surgery. In 1995, we intro- duced the virtual visualization of preplanned osteot- omy lines and the virtual visualization of positioning of osteotomized bone segments intraoperatively.

Accurate matching and simultaneous visualization of virtual data and the real world can be termed “aug- mented reality.” To obtain this, the user wears a see- through HMD to see virtual medical structures super- imposed on the real patient. Partially immersive meth- ods allow complete interaction with the real world and simultaneously make accessible the virtual data envi-

ronment, thereby providing the safety needed when performing surgical procedures.

Virtual image-guided surgery either reduced or did not add operative time, but it did increase preoperative preparation time. Preoperative data collection, image processing, preparations for image coordinate transfor- mation, and preoperative planning are time consuming.

In general, surgery using virtual image guidance was believed to provide additional safety compared with surgery performed without it. Being able to view an unrestricted surgical field and at the same time recog- nize virtual anatomic and functional data orthotopi-

FIGURE 11. Computer video- photograph as seen through the HMD. Correct position of the maxilla postoperatively indi- cated by the two colored interin- cision points.

462 DISCUSSION

tally superimposed on the real patient allows surgery to become less invasive. In orthognathic surgery and in osteotomies of the facial skeleton, augmented reality concepts have, for the first time, allowed transparent transfer of preoperative planning to intraoperative vi- sualization.

Whenever bone segments are transferred, virtual structures serve as guidelines intraoperatively and, in addition show every motion of the osteotomized bone in relation to imaging data and cephalometric drawing (Figs 8, 9). For instance, in correction of posttraumatic enophthalmos, augmented reality can be used for intra- operative visualization of a symmetrical position of the globe.

System accuracy and consistancy are over-riding concerns. Because the surgeon relies on other cues and skills during surgery, he or she can always check the accuracy of the system using fiducial markers and noting what is encountered compared with what the virtual images show. The system showed good, but still insufficient, accuracy for extremely accurate pro- cedures (ie, less than 1 mm) in the operating room environment, because the system’s coregistration error cannot be expected to decrease the tracking system’s error. For instance, for determination of occlusal rela- tions, a splint is still better. In this sense, it can be stated that it is not always possible to be highly accu- rate; therefore, it is of paramount importance that the surgeon can always quantify system inaccuracy by direct visualization. Despite little shortcomings, how- ever, our system has already proved to be a valuable addition to our surgical armamentarium and is there- fore used in selected cases.

References

1. Dunkley P: Virtual reality in medical training. Lancet 343: 1218, 1994

2. Frisbie AG: Advances in educational technology and journeys into virtual reality. J-Allied-IVD, CD-I, Health 22:131, 1993

3. Kaltenbom KF, Rienhoff 0: Virtual reality in medicine. Meth- ods Inf Med 32:407, 1993

4. Noar MD: Endoscopy simulation: A brave new world. Endos- copy 23:147, 1991

5. Noar MD: Endoscopy simulation training devices. Endoscopy 24: 159. 1992

6. Vannier i’&W: Computer applications in radiology. Curr Opin Radio1 3:258. 1991

7. Gupta SC, Klein SA, Barker JH, et al: Introduction of new technology to clinical practice: A guide for assessment of new VR applications. J Med Virtual Reality 1: 16, 1995

8. Watanabe E. Watanabe S. Manaka S. et al: Three-dimensional digitizer ;Neuronavigaior): New equipment for CT-guided stereotaxis surgerv. Surg Neurol 27:543. 1987

9. Watanabe E, Mayaiagi Y: Kosugi Y, et al: Open surgery as- sisted by the neuronavigator, a stereotactic, articulated, sensi- tive arm. Neurosurgery 28:792, 1991

10. Fialkov JA, Phillips JH, Gruss JS, et al: stereotactic system for guiding complex craniofacial reconstruction. Plast Reconstr Surg 89:340, 1992

11. Giorgi C, Luzzara M: A computer controlled stereotactic arm: Virtual reality in neurosurgical procedures. Acta Neurochir Sum11 Wien 58:75, 1993

12. Has&id S, Miihling J. ZGller J: Intraoperative navigation in oral and maxillofacial surgery. Int J Oral Maxillofac Surg

I .

24:111, 1995 13. Reinhardt H, Meyer H, Amrein E: A computer-assisted device

for the intraooerative CT-correlated localization of brain tu- mors. Eur Su;g Res 20:51, 1988

14. Weinberg R: Neurosurgerv for third millenium: Neurosurgical topics: Vol 2. Ame&& Association of Neurological %ur- geons, 1992, pp 47-63

15. Stix G: See-through view: Virtual reality may guide physicians hands. Sci Am 267:166, 1992

16. Stix G: Reach out: Touch is added to virtual reality simulations. Sci Am 264:134, 1991

17. Wickham JEA: Future developments: Minimally invasive sur- -zerv. Br Med J 308:193. 1994

18. Wi&ms M, Wann JP: Binocular vision in a virtual world. Ouhthalmic Phvsiol Out 13:387. 1993

19. Lo&et-Higgins fiC: A computer ‘dgorithm for reconstructing a scene from two projections. Nature 293:133, 1981

20. Metz CE, Fencil LE: Determination of three-dimensional structure in biplane radiography without prior howledge of the relation- ship between the two views. Theor Med Phys 16:45, 1989

21. Fencil LE, Metz CE: Propagation and reduction of error in three-dimensional structure determined from biolane views of unknown orientation. Med Phys 17:951, 1996

22. Ploder 0, Wagner A, Enislidis G. et al: Comuuter assisted intra- opera&e v&ualization of den&l implants: Augmented reality in medicine. Der Radiologe 35:569, 1995

23. Wagner A, Ploder 0, Enislidis G, et al: Virtual image guided navigation in tumor surgery: Technical innovation. J Cranio- maxillofac Surg 23:271, 1995

24. Wagner A, Ploder 0, Enislidis G, et al: NAVIGOS: Navigation assistance by a virtual image guiding operation system. Initial description and model operation on a stereolithographic skull model. Stomatol 93:87-90, 1996

25. Wagner A, Ploder 0, Truppe M, et al: Image guided surgery. Int J Oral Maxillofac Surg 25:147, 1996

26. Wagner A, Ploder 0, Zuniga J, et al: Augmented reality environ- ment for temporomandibular joint motion analysis. Int J Adult Orthod Orthognath Surg 11:127, 1996

J Oral Maxillofac Surg 55:462-463. 1997

Discussion Virtual Reality in Orthognathic Surgery: The

Augmented Reality Environment Concept

Herman F. Sailer, MD, DDS and Ulrich Longerich, MD, DDS University Hospital Zurich, Zurich, Switzerland

The Virtual Patient System (VPS) described in this article and the further development, the Interventional Videotomog- raphy (IVT), offer the possibilities of so-called composite or augmented reality. VPS and IVT are based on the same software, with the difference being that IVT transmits the image data added to a special conference system via Internet, ISDN, or TCP/IP.