PAPER

DIGITAL &MULTIMEDIA SCIENCES

Minhua Ma,1 Ph.D.; Huiru Zheng,2 Ph.D.; and Harjinder Lallie,1 M.Sc.

Virtual Reality and 3D Animationin Forensic Visualization*

ABSTRACT: Computer-generated three-dimensional (3D) animation is an ideal media to accurately visualize crime or accident scenes to theviewers and in the courtrooms. Based upon factual data, forensic animations can reproduce the scene and demonstrate the activity at various pointsin time. The use of computer animation techniques to reconstruct crime scenes is beginning to replace the traditional illustrations, photographs, andverbal descriptions, and is becoming popular in today’s forensics. This article integrates work in the areas of 3D graphics, computer vision, motiontracking, natural language processing, and forensic computing, to investigate the state-of-the-art in forensic visualization. It identifies and reviewsareas where new applications of 3D digital technologies and artificial intelligence could be used to enhance particular phases of forensic visualizationto create 3D models and animations automatically and quickly. Having discussed the relationships between major crime types and level-of-detail incorresponding forensic animations, we recognized that high level-of-detail animation involving human characters, which is appropriate for manymajor crime types but has had limited use in courtrooms, could be useful for crime investigation.

KEYWORDS: forensic science, virtual reality, augmented reality, 3D animation, forensic computing, forensic visualization, scene recon-struction, natural language processing, computer vision, motion tracking

Computer-generated animation is an ideal media to accuratelyvisualize crime or accident scenes to the viewer to help understandthe situation and retain complex spatial information. Computer-related crime does not involve physical motion in the same way asfor instance a homicide; it would, therefore, serve little purpose forsuch a crime to be reconstructed visually. Computer-generated ani-mation has a limited role to play in the reconstruction of computer-related crime and computer forensics; at most it may be restrictedto the demonstration of technical data for the education of juries.For instance, the operation of hard disks and the manner in whichthey store data can be visualized for judge and jury rather thanpresented them with a technical description. The problem with suchan approach, however, is that the computer-based animations mustbe acceptable in a courtroom situation.

Based upon factual data, forensic animations can reproduce thescene and demonstrate the activity and location of vehicles, objects,and involved persons at various points in time. Once the animationhas been produced, it is easy and cost-effective to observe thescene from various viewpoints such as a driver’s view, a victim’sview, and a witness’ view. Using computer animation techniques toreconstruct crime scenes is replacing the traditional illustrations,photographs, and verbal descriptions, and it is becoming popular intoday’s forensics.

Computer graphics technology has been successfully applied inforensic visualization, ranging from traffic accident reconstruction

to major crime scenes (1–5). Computer-generated (CG) animationhas been used to investigate crimes and shown to the jury in thecourtroom for evidence presentation (1).

This article integrates 3D graphics, computer vision, motiontracking, natural language visualization, and forensic computing, toinvestigate the state-of-the-art in forensic 3D animation. The rest ofarticle is organized as the follows: First, section CG Animation forForensic Visualization discusses approaches of applying CG tech-nologies to forensic reconstruction in mixed reality and virtual real-ity; Second, Technologies for Forensic Visualization, followed by adescription of Crime Types and Level-of-Detail (LOD), and finallythe article is concluded by Summary and Discussion.

CG Animation for Forensic Visualization

There are four approaches of adopting computer-generated ani-mation in forensic reconstruction: virtual reality (VR), augmentedreality (AR), CG 3D animation, and combining real and syntheticimagery.

Virtual reality simulation in forensic process starts from model-ing 3D virtual objects and humans based on measurements andphotos and animates the models to recreate the crime scene or inci-dents concerned. Applications of VR in forensic animation includepathological visualization, murder reconstruction, and shooting casebriefing tool.

Augmented reality is the combination of VR and real-world con-tent where CG virtual objects or humans are superimposed overreal objects or into video footage in real time. An AR user maywear translucent goggles, through which he could see the realworld as well as CG images projected on top of that world.Applications of AR in forensics include simulating road trafficaccidents where virtual vehicles are blended into the real scene or

1School of Computing, University of Derby, Derby, U.K.2School of Computing & Mathematics, University of Ulster, Co. Antrim,

U.K.*Work supported by the Research Inspired Curriculum Fund 2007 ⁄ 8 from

University of Derby, U.K.Received 14 April 2009; and in revised form 14 July 2009; accepted 23

July 2009.

J Forensic Sci, September 2010, Vol. 55, No. 5doi: 10.1111/j.1556-4029.2010.01453.x

Available online at: interscience.wiley.com

� 2010 American Academy of Forensic Sciences 1227

footage and augmented crime scenes where 3D models of victim,suspect, or missing weapon are blended into the real crime scenefor evidence presentation. The advantage of using AR in forensicvisualization is saving time and costs by reducing the amount of3D modeling required and higher immersion because of use ofreal-world elements.

Figure 1 shows the techniques used for forensic visualization.The horizontal axis indicates the proportion of virtual content, andthe vertical axis expresses the degree of interactivity. Use of video,photos, and illustrations provides no interaction and all the contentsare real; therefore, it locates at the bottom left. CG 3D animationcontains all virtual content and has no interactivity involved whenwatching the animation, hence it is at the bottom right. Combiningreal and synthetic imagery is somewhere in the middle, anddepends on how much synthetic content is used. It has become apopular special effects technique in the movie industry.

The difference between VR ⁄ AR and the other three techniquesin Fig. 1 is interactivity and real time. VR and AR provide aninteractive real-time 3D graphical environment that responds to useractions such as moving around the virtual world or maneuveringvirtual objects. VR ⁄ AR can put the user in the driver’s seat foraccident reconstructions and allow the user to observe a crimescene from a desired vantage point, which is impossible to film orobserve. A VR ⁄AR user can also, for example, play the variousroles of victim, perpetrator, or witness to experience the reconstruc-tion of the incident.

In 2003, Linden Lab released Second Life, a virtual world inwhich users can create and animate their own objects and charac-ters. Object creation is facilitated by the availability of basic prims(i.e., primitives), which can be modified, textured, and scripted.The behaviors of objects and nonplayer characters (bots) are pro-grammed using the Linden scripting language. Second Life hasbeen used for crime scene reconstruction and role-playing for crimeinvestigation, mainly for educational purposes. There are placesbuilt in Second Life for role-play in forensic science teaching toprovide live forensic science experience, with roles of victim, per-petrator, witness, crime scene investigator, police, prosecutor, anddefender (e.g., Midian City - A Dark RP Community) and forensicpathology laboratories (e.g., Medical Examiner’s office). In a pro-ject reported in (6), a group of forensic students have constructed avirtual crime scene complete with a fully furnished house, shore-line, and backyard deck with barbeque, which is a part of the storyin Second Life. They then planted clues around the crime scene ofa stolen sword. Another group of students examines the crime

scene, collects clues, runs forensic tests using functioning equip-ment, and tries to solve the crime.

Though forensic visualization and digital entertainment industryoften use the same sets of software and hardware for creatinganimation or interactive systems, it is important to differentiatebetween forensic reconstruction and entertainment. The majordifferences between them are similar to the differences betweenanimation and simulation: accuracy and purpose. What is aestheti-cally and perceptively real is unnecessarily physically accurate.Take character motion as an example: professional animators usu-ally tweak the timing, joint ranges, and motion shape of charactersfor aesthetic edits in digital games and 3D movies industry;whereas forensic animators aim to generate kinematic and dynamicmotion as close as possible to the reality.

Technologies for Forensic Visualization

Recent developments in computer technology, especially graph-ics, have created a climate where novel computing forensic applica-tions have emerged. In this section, we identify the areas wherenew applications of 3D digital technologies and artificial intelli-gence (AI) could be used to enhance particular phases of forensicvisualization to create 3D models and animations automatically andquickly or to provide real-time interactivity of the reconstructedscenes.

Firstly, we identified three types of data involved in the forensicvisualization processes: original, conclusive, and user data. Originaldata obtained through observation and measurement of the scene;conclusive data is calculation results based on analysis of the origi-nal data; and user data is solely for evidence presentation purposes.

Motion Tracking

VR and AR use motion tracking technologies to capture users’movement to provide interaction in real time. Zhou and Hu (7)provided a survey of motion tracking technologies, which consis-tently update spatiotemporal information with regard to humanmovement. Figure 2 shows the main categories of motion trackingtechniques, including sensor-based, camera-based, glove-basedtracking, and haptic interface. A VR ⁄ AR user usually wears eithersensors or markers on his limbs ⁄ joints ⁄ head to enable the systemto capture his movement in the virtual environment.

Sensor-based systems use electromagnet, electromechanical,inertial, or ultrasound technologies to collect precise data of three-dimensional location and orientation by transmitting and sensingsignals from the sensors. Camera-based tracking uses videocameras to detect active (LEDs) or passive (coated with a retrore-flective material) markers attached at specific body locations totracking the movements. Some of the camera-based systems evendo not need to place markers on user. Data glove systems usuallyuse tilt and flex sensors to capture the bending of user’s fingers.Haptic interface is a special motion tracking device that interactswith a user via touch.

Computer Vision Technologies

Besides being a part of motion tracking technologies in camera-based systems, computer vision technologies have been applied forautomatic generation of photo-realistic 3D calibrated models froma sequence of images, e.g., instant Scene Modeler (8), an automatic3D modeling system that creates calibrated photo-realistic 3D mod-els by using a handheld stereo camera for recording images and alaptop for acquisition and processing.

FIG. 1—Forensic visualization techniques.

1228 JOURNAL OF FORENSIC SCIENCES

Dynamic Simulation

Use of CG visualization in forensic evidence presentation hasbecome admissible in the United Kingdom and globally. In theUnited Kingdom, computer records are acceptable as evidence ifthey can be proved to be correct and accurate (1). This leads to thesignificance of introducing dynamic simulation (a.k.a. physicallybased simulation) to forensic animation, because dynamic simula-tion is capable of accurate modeling of objects and human motionsbased on Newton’s laws of motion.

All virtual objects, human kinematics, and motion must obeyNewton’s laws of motion in a precise simulation. For example, inthe case of traffic accident reconstruction, high physical fidelity ofthe creation of car and victim models is crucial to answer questionsabout how the accident happened, the sequence of events, and howa specific vehicle moved in the situation. This is the reason thatmost forensic reconstructionists recognized by courts have beenmechanical engineers, and the accepted laws of motion andmechanics form the basis of this expertize.

Currently, a forensic reconstructionist calculates the placement ofvehicles before, during, and after an accident based on the data col-lected from the accident scene and comes up with data representingthe movement of all the participants in the accident. An animatorthen creates 3D models of virtual objects and humans and con-structs the scene according to the measurement of the scene, pho-tos, or scale drawings, and data calculated by the forensicreconstructionist.

Dynamic simulation provides realistic motion of virtual objectsby modeling the behavior of virtual objects and their responses toexternal force (e.g., gravity) and torque in a physically realisticmanner. Dynamic simulation models objects with their physicalproperties such as mass, inertia, barycentre, joint limitations, restitu-tion, and surface friction—it can make objects in virtual worlds notonly look real but act real. Typical dynamic simulation includescollision detection and the simulation of gravity, friction force,

torque, and kinematics in motor actions. Physics can be applied torigid bodies or deformable bodies such as human tissues.

Real-time dynamic simulation has been used quite heavily indigital games, mechanical simulations, medical visualization andtraining, and engineering (3). The advantage of applying physicallybased simulation techniques in forensic visualization is that byvirtue of its replayability, adjustability, and reliableness. Physicallyaccurate forensic animation may shed light on investigating whatexactly happened at a specific crime scene, causes and effects thatembrace the issue of who is at fault or guilty in the case of trafficaccidents. In addition, dynamic simulation can be utilized toperform experiments that are impossible or expensive in the realworld.

The quality of physics required in a simulation is applicationspecific. In the cognitive and affective domains of learning wherethe focus of training is more on attitudes, high physical fidelity isnot always necessary. However, in medical simulation such as sur-gery planning and training, high quality physical fidelity is soimportant that without it the skills acquired in the virtual worldmay not be transferred to the real one. In forensic animation, thereconstruction of vehicle accidents requires rigid body dynamicsand the bombing reconstruction involves breakable rigid bodydynamics. The simulation of the biomechanical movement of thehuman body is an important part of visualization for many majorcrime types, and some animated pathology sequences may requiredynamic simulation of deformable bodies such as muscles. We willdiscuss this further in Crime Types and Level-of-Detail.

Natural Language Visualization

Natural language visualization is a novel research area thatintegrates natural language processing (NLP) and 3D animationgeneration to automate the processes of generating human anima-tion from natural language input (9). Forensic visualization isalready benefitting from the outcomes of research to facilitate thereconstruction processes, which is traditionally conducted by policeinvestigators, forensic reconstructionists, and professional animatorsto create a virtual simulation that fits all the existing facts.

CarSim

CarSim (2) is an automatic text-to-scene conversion system thatvisualizes car accidents from written reports of motor vehicle acci-dents. It understands the accident conditions by automaticallyextracting pieces of information from texts and presenting visuallythe settings and the movements of the vehicles in 3D scenes. Car-Sim has been applied to a corpus of French and Swedish texts forwhich it can currently synthesize visually 35% of the texts. CarSimis also being ported to English.

CarSim represents accidents by applying information extractiontechniques to input texts, which reduce the text content to formal-ized templates that contain road names, road configuration, numberof vehicles, and sequence of movements of the vehicles involved.The visualizer reproduces approximately 60% of manually createdtemplates. CarSim’s NLP module combines regular expressionmatching with dependency parsing to carry out the linguistic analy-sis of the texts. A regular expression grammar is used to identifyproper nouns. CarSim focuses on collision verbs, which are vital inthe domain of motor vehicle accidents. The visualization modulerecreates the 3D scene and animates the vehicles. It represents boththe entities and the motions symbolically, without taking intoaccount physics laws in the real world.

FIG. 2—Motion tracking techniques.

MA ET AL. • 3D ANIMATION IN FORENSIC VISUALIZATION 1229

Human Animation from Natural Language

CONFUCIUS (9) is a framework of language visualization using3D animation techniques with NLP to achieve high-level animationgeneration. It is able to visualize single sentences, e.g., Johnhanded a gun to Nancy, into 3D animation, speech, and soundeffects. The system uses lexical visual semantic representation toconnect linguistic semantics to visual semantics of action verbs andto represent verb meanings for action execution (animation). CON-FUCIUS gives promising results on word sense disambiguation(70% accuracy) with regard to the data set on which it was tested.

Although originally targeted to application in storytelling, thesystem has a potential of reproducing crime scenarios based on textdescriptions.

Forensic Visualization Processes and Contributionsof Computer Technologies

Computer vision technologies automate the processes of obtain-ing original data and generating 3D models from it. Dynamic simu-lation automates the process of generating conclusive data fromoriginal data. Natural language visualization animates 3D modelsbased on original and conclusive data. Integration of the bespokentechnologies provides a potential for automating the processes ofreconstructing crime scenarios from evidence that requires minimalintervention from human experts such as crime scene investigators,forensic reconstructionists, and 3D artists.

Figure 3 illustrates the main processes of forensic visualizationand where the above-mentioned technologies may contribute to theprocesses. The double line arrows indicate technology contributions,and the single line arrows indicate information flow betweenprocesses.

The data flow from ‘‘Data collection’’ to ‘‘Animation’’ providesphysical features of objects and humans that may affect animation,for example, weight and resilience of object, locations after theincident, and ground friction. It also includes the crime types thatrequire different level-of-detail (LOD) in the animation.

The flow from ‘‘Data analysis’’ to ‘‘Animation’’ provides anumber of scenarios to be created based on hypothesis. The infor-mation flow from ‘‘Data collection’’ to ‘‘3D modeling’’ providesmeasurement of physical properties of objects and humans thataffect creating and scaling of their 3D models, such as size, propor-tional information, color, texture, and other visual properties.

The double line arrows indicate contributions of computer tech-nologies to the forensic visualisation processes. The double linearrow from ‘‘Dynamic simulation’’ to ‘‘Data analysis’’ means per-forming experiments in dynamic simulation to formulate or ruleout a hypothesis and assist deduction, or even discover unexpectedsituation. As the research only focuses on reconstruction of forensicscenes, knowledge-based expert systems used for deductive reason-ing are not discussed in this article.

Crime Types and Level-of-Detail

Level-of-detail (LOD) is a useful concept for managing graphiccomplexity across many scales. In many VR systems, a virtualobject often has multi-resolution representations of polygonal mesh,either refining or simplifying according to certain criteria such asthe distance of the object to the camera, so that when it is close tothe camera higher LOD mesh is presented, and when it is far awayfrom the camera a low LOD model is used. This has proved aneffective optimization strategy in computer graphics. LOD could besupported by having multiple articulations, a.k.a. Levels ofArticulation (LOA), when apply to jointed systems like virtualhumans. For instance, a low LOA virtual human may be based ona 6-joint skeleton, and a higher LOA virtual human can have 18 oreven 71 joints (e.g., H-Anim standard LOA1 and LOA2 as shownin Fig. 4).

H-Anim standard provides four LOAs for applications whichrequire different levels of detail. Some applications such as medicalsimulation and design evaluation require high fidelity to anthropog-eometry and human capabilities, whereas games, training, and visu-alized living communities are more concerned with real-timeperformance. Most majority of forensic visualisation containshuman figures and is concerned with accurate simulation of

FIG. 3—Computer technologies contribute to the forensic visualizationprocesses. FIG. 4—Joints and segments of H-Anim LOA2.

1230 JOURNAL OF FORENSIC SCIENCES

humans. Level 2 of Articulation (LOA2) of H-Anim, which has 71joints, is considered sufficient for human modeling in forensicreconstruction (9). This level ensures enough joints for humanmovements in forensic visualisation, e.g., it includes enough handjoints for grasp postures. Figure 4 illustrates the joints of LOA2.The dots denote joints and lines segments.

Specific incident reconstruction deals with traffic accidents,bombings, homicides, and accidents of any severity. The require-ment of LOD is closely related to the type of crime involved. Fig-ure 5 shows the relationships between main crime types and LODand physical fidelity needed in the animation.

High physical fidelity and low LOD animation (the overlappedarea of the two oval shapes), e.g., traffic accident reconstruction,has been admitted and presented in courtrooms already in theUnited Kingdom and U.S.A. (10,11). However, high LOD anima-tion involving human figures is rarely admitted in courtrooms,except in the case of pedestrian collision, because of the risk of itsoverpersuasion power.

Risk of Faulty Interpretation

The issue of creating bias by only watching one scenario ofvisualization brings the problem of admissibility of displaying CGanimation in courtroom. The judge and jury may not be aware ofthe error or uncertainty involved in measuring and reconstructingthe scene, and hence they may be subconsciously biased toward abelief in the presented digital display. According to Lederer andSolomon (12), people are five times as likely to remember some-thing they see and hear rather than something they hear alone; andthey are twice as likely to be persuaded if the arguments are but-tressed with visual aids.

Despite the many benefits of CG visualization that we havementioned earlier, it is absolutely essential to verify authenticity,fairness, and relevance before the visualization is used as evidenceand presented in court. The original data collected and used in eachprocess (the circles in Fig. 3), the accuracy of the methods (thesquares in Fig. 3), and the final visualization must be proved.Another solution to reduce levels of prejudice is visualizing andshowing different scenarios of both sides for the defense and forthe prosecution in court. How this can be done in practice is out ofthe scope of this article. It also could be used during the investiga-tion process to analyze and eliminate hypothesis.

Discussion

In summary, CG animation in VR ⁄ AR has the ability to commu-nicate highly complex, technical spatial, and temporal evidential

information of incident scenes and bring about increased accuracyor speed of the forensic process and can hence reduce the costsinvolved. The research investigates the state-of-the-art in forensic3D animation and identifies various technologies that are beginningto be used for or have a potential to be applied in forensicvisualization, such as motion tracking, computer vision, dynamicsimulation, natural language visualization, and AI. We also analyzethe forensic visualization processes and discussed the areas towhich these computer technologies may contribute. Having lookedat the relationships between major crime types and LODs, it isrecognized that high LOD animation involving human characters,which is appropriate for many major crime types, had only alimited use in courtrooms because of admissibility (1), but it couldbe helpful for crime investigation and informal briefs.

References

1. Burton AM, Schofield D, Goodwin LM. Gate of global perception:forensic graphics for evidence presentation. Proceedings of the 13thAnnual ACM International Conference on Multimedia; 2005 Nov 6–11;Singapore. New York, NY: Association for Computing Machinery,2005;103–11.

2. Egges A, Nijholt A, Nugues P. Generating a 3D simulation of a caraccident from a formal description: the CarSim System. In: GiagourtaV, Strintzis MG, editors. Proceedings of the International Conference onAugmented, Virtual Environments and Three-Dimensional Imaging;2001 May 30–June 1; Mykonos, Greece. Thessaloniki, Greece: Informat-ics and Telematics Institute - CERTH, 2001;220–3.

3. Ma M, McNeill M, McDonough S, Crosbie J, Oliver L. Physics fidelityof virtual reality in motor rehabilitation. In: Richir S, Klinger E, editors.Virtual: a real success. Proceedings of the 8th International Conferenceon Virtual Reality (VRIC—Laval Virtual 2006); 2006 April 26–30;Laval, France. Laval Cedex, France: Laval Virtual, 2006;35–41.

4. Sauter PM. Introduction to crime scene reconstruction using real-timeinteractive 3D Technology, http://www.pmsmicro.com/forensicsciences-chapter_4d.pdf (accessed June 11, 2009).

5. Chisum WJ, Turvey BE. Crime reconstruction. London ⁄ Burlington ⁄ SanDiego: Elsevier Academic Press, 2006.

6. Forensic Student Meeting in-world, http://pacificrimx.wordpress.com/2008/06/14/forensic-student-meeting-in-world/ (accessed June 12, 2009).

7. Zhou H, Hu H. Human motion tracking for rehabilitation—a survey.Biomed Signal Process Control 2008;3(1):1–18.

8. Se S, Jasiobedzki P. Instant scene modeler for crime scene reconstruc-tion. Proceedings of IEEE Computer Society Conference on ComputerVision and Pattern Recognition; 2005 June 20–26; San Diego. Washing-ton, DC: IEEE Computer Society, 2005;123.

9. Ma M. Automatic conversion of natural language to 3D animation[Ph.D. thesis]. Derry, UK: School of Computing & Intelligent Systems,University of Ulster, 2006.

10. Schofield D. Animating and interacting with graphical evidence: bring-ing courtrooms to life with virtual reconstructions. Proceedings of Inter-national Conference on Computer Graphics, Imaging and Visualisation;2007 Aug 14–17; Bangkok, Thailand. Los Alamitos, CA: IEEE Com-puter Society, 2007;321–8.

11. Noond J, Schofield D, March J, Evison M. Visualising the scene: com-puter graphics and evidence presentation. Sci Justice 2002;42(2):89–95.

12. Lederer FI, Solomon SH. Courtroom technology—an introduction to theonrushing future. Fifth National Court Technology Conference (CTC5),National Centre for State Courts, 1997, http://www.ncsconline.org/D_Tech/ctc/showarticle.asp?id=80 (accessed June 11, 2009).

Additional information and reprint requests:Minhua Ma, Ph.D.School of ComputingUniversity of DerbyDerbyU.K.Email: m.ma@derby.ac.uk

FIG. 5—LOD, physical fidelity, and crime types.

MA ET AL. • 3D ANIMATION IN FORENSIC VISUALIZATION 1231