EISWIER Automation in Construction 5 (1997) 393406
Programming construction robots using virtual reality techniques
R. Navon a92, A. Retik b33 a Technion. Department of Civil Engineering, National Building Research Institute, Technion City, 32000 Haifa, Israel
b Department of Civil Engineering, University of Stratchylde, Glasgow G4 ONG, UK
The paper describes a new approach to programming construction robots, using virtual reality (VR) techniques. The new approach is needed because both traditional and new methods of programming industrial robots, described in the paper, have specific drawbacks, which become crucial in the construction arena. This is because of the ever-changing environment of construction and its nature, a prototype or one-of-a-kind, industry. As a result, construction robots need much more programming than their industrial counterparts, which is labor intensive using known methods and is not compensated by mass production. The VR approach is demonstrated with the Multi-Purpose Interior Finishing Robot (MPIR) for a masonry task, accompanied by a detailed description of the VR-based programming model and approach.
Keywords: Automation; 13uilding; Construction; Programming; Robot; Virtual reality
The objective of this paper is to suggest a new method for robot programming with the aid of vir- tual reality (VR) technology, which allows the fol- lowing to be achieved: 1. Learn geometry and spatial arrangements of a
Discussion is open until August 1997 (please submit your discussion paper to the Editor on Construction Technologies and Engineering, M.J. Skibniewski).
Tel: + 972-4-8292600; Fax: + 972-4-8324534; e-mail: cvronie@ tx.technion.ac.il; Internet: http://techunix.technion.ac.il/ - cvronie.
3 Tel.: + +44-141-552-4400; fax: + +44141-552-0112; e- m ail: a.retik@ stratch.ac.uk; Internet: http//www.strath.ac.uk,/Departments/Civeng/conman/cm.html.
Explore the best way for a robot to perform a task using human reasoning. Check and verify structural and organizational safety issues of a robot, including location of materials. Train the robot for its designated tasks, in the same way as workers would have been trained to perform new tasks. Robots are considered as flexible manufacturing , .
machines because they can easily be (re)programmed to perform different tasks. Yet, often the complexity of the programming process limits more widespread use of robotic technology [ 11. Researchers and practi- tioners alike are constantly looking for new methods of programming robots to save programming time and costs. This section reviews some common and novel programming techniques used or developed for
0926-5805/97/$17.00 Copyright 0 1997 Elsevier Science B.V. All rights reserved.
394 R. Nauon, A. Retik/Automation in Construction 5 (1997) 393-406
manufacturing robotics. These techniques are exam- ined for their potential use for construction robots, which operate in an ever-changing environment, pro- ducing one-of-a-kind or at best small-batch products.
1. I. Native programming languages
Currently, most commercial robots are pro- grammed in textual robot programming languages (RPLs) similar to BASIC, PASCAL, etc. Popular examples include VAL, RAIL and AML . These languages include positioning instructions, process parameters and other instructions . RPLs contain primitives to express manipulator motion in world coordinates as a location (x, y, z> and orientation (roll, pitch, yaw) 141. Most languages also contain higher level control commands.
A program in RPL can be written when the robot is in its designated work location (on-line program- ming). The advantage of doing so is that program- ming is relatively easy, but during programming the robot and the dependent equipment are idle, thus increasing programming costs significantly. Off-line programming is supposed to solve this problem but it is very difficult to carry out without visual aids. Lees and Leifer  state that describing a robots motion through space is not the strength of RPLs (in both the on- and off-line modes). It is difficult for a human to envisage the position and orientation of a robot by merely looking at a string of numbers and it is all but impossible to envisage the robots spatial relationship with the rest of the objects in its workspace. Since the basic components of a robot programming language are so counter-intuitive, a complete manipulation program is even more so.
This technique is most unsuitable for program- ming of construction robots due to the complex nature of the construction work. In order to program a construction robot using this technique, the pro- grammer would have to visualize in his/her mind the entire project, including the location of all robot workstations, the locations of the raw materials, etc. In addition, he/she would have to visualize the robots reaction to various commands and its interac- tion with the changing environment. No less impor- tant is the fact that programming this way is labor intensive - because of the one-of-the-kind nature of the construction work - and consequently expensive.
1.2. Programming with the aid of graphic simulators
Off-line programming with simulators enables the programmer to write a program without recourse to the robot itself. It is done in a computer-graphic environment, enabling the programmer to preview, debug and verify the program before the robot is installed, or even bought. Simulators enable the de- velopment of typical tasks, examine different altema- tives to perform them, test different control strate- gies, explore raw material supply alternatives, etc. Once a program is written, the programmer can visualize the robots task being performed, check the planned paths suitability for the performance of specific tasks, check that the robot does not collide with other components in the work cell, etc. When the programmer is satisfied with the program, it is translated into the robots native language and down- loaded to it.
One of the most important characteristics of graphic simulators is that they are able to measure cycle times, which is an important parameter influ- enced by the selected paths, by the control algo- rithms, by the material supply method, etc. Thus higher quality programs can be developed with the aid of graphic simulators, taking into account robot productivity as well. However, simulators still re- quire the users to work with text-based languages and consequently only solve part of the problems associated with native robot programming languages
La. Graphic simulators also aid in selecting the best
robot for a given task. Robot selection is based on a number of parameters, such as cycle time, work envelope, path limitations, joint singularities, re- quired floor area, etc. Most of those parameters cannot be checked with a reasonable level of accu- racy and certainty without a graphic simulation sys- tem.
Construction robot programming using graphic simulators offers a meaningful improvement to RPL techniques by enabling the programmer to see both the environment and the robot during programming. While this solves the visualization problem, it still does not decrease the amount of work involved in programming the robot because the commands re- main very basic (positioning instructions, manipula- tor motion instructions, end effector operations, etc.).
R. Nauon, A. Retik/Automation in Construction 5 (1997) 393-406 395
Consequently, this technique is not suitable for con- struction robot programming.
1.3. Additional programming methods
prehensive solution to the complex problem of pro- gramming construction robots, it is merely suggest- ing an additional idea, which can be used either on its own or with other methods, some of which were reviewed above.
Programming by example, or by human demon- stration, is an intuitive method. The programmer demonstrates how the task is performed using a human/robot teaching device that measures the hu- mans forces and positions [l]. The data gathered from the human is used to generate the robots program.
2. VR iu building construction
The goal in automatic robot programming is to get the robot to perform a task by telling it what needs to be done, rather than by explicitly program- ming it [51. An approach to automatic robot program- ming was suggested by Rogalinski .
Visual robot programming is a term applied to systems which allow programming in a two- or three-dimensional manner . A visual demonstra- tion of the task is performed not in the real world but within the digitized images that the robots vision system sees. The programmer indicates directly in the digitized images (using a mouse) robot actions, constituent parts, grip points, relative orientations, approach routes, etc. Complex task-planning by the robot is avoided by keeping the human in the control loop, thus the human problem solving skills support the robots understanding and execution of the task.
VR is an advanced computer graphics technology dealing with visualization . As a computer graph- ics application it can be used for a wide variety of uses, e.g. [9-121. The use of object-oriented tech- niques for creating virtual environments was the key for a breakthrough in the credibility and applicability of VR technology [ 131. Various definitions exist: one which is both descriptive and short is by Pimentel and Teixeira [ 141 who define VR as the place where humans and computers make a contract. Similarly, Larijani 1131 defines VR as the conver- gence of computer simulation and visualization that attempts to eliminate separation between a user and a machine. From their viewpoint, VR is an interface between humans and computers.
The suitability of the additional techniques, which are still under development, for construction pur- poses has to be examined when they become com- mercial, or as a separate study. Some of them, such as programming by example, seem unsuitable a pri- ori.
Clearly, the above-mentioned programming meth- ods and techniques for construction robots are un- suitable which makes them cost ineffective. Other methods must be developed.
There are two major approaches to creating such an interface: immersive and non-immersive. In im- mersive VR the interface acts as a contact point between a user and machine (system) where the users movements are translated into commands that direct the machines operation and, in addition, the machines simulated condition is communicated to the user through his/her senses (currently mainly through vision). A flight simulator is probably the oldest example of a VR system of this kind. To achieve effective immersion, a VR system requires not only navigation and manipulation abilities, but also a close correspondence between input and out- put devices such as a glove, a head-mounted display, a pressure-sensitive floor, etc.
The research underlying this paper is planned as a A virtual world can also be explored without multi-stage program. In the first stage the basic tools immersion. Desktop and projection VR systems re- were built and experimented with using two tech- tain the navigation and manipulation features, giving niques, as will be explained below. The objective of a user the ability to move around the virtual world this paper is to pmpose the idea of using VR for and manipulate its components using a spaceball, 3D robot programming. as a stimulating idea and to mouse or a simple joystick. Moreover, such non-im- provide some insights from the preliminary experi- mersive applications can have advantages over ments. The paper does not intend to present a com- real VR systems especially in big virtual worlds
396 R. Naaon, A. Retik/Automation in Construction 5 (1997) 393-406
like a typical construction site. Being able to over- look the site from outside (an aerial view) is often more beneficial than to be a small part of the site [15,16].
Visualization and simulation of the construction process using non-immersive projection VR, as demonstrated in [ 161, may assist a construction plan- ner or a user to improve his/her perception of a project, as well as to integrate other involved parties in the planning process. In large-scale projects not only the construction process itself can be monitored, but also all the auxiliary activities, including on-site plant and equipment. In addition the different loca- tions of the construction equipment and the tempo- rary facilities can be checked by placing real three-dimensional models into a virtual construction site. Then both location changes and operational movements can be verified by simulating a projects time schedule. In such cases, especially in projects where heavy equipment is used, clashes and interfer- ences may be more easily identified [ 171.
VR can be applied in building design and con- struction for a wide variety of uses as proposed in [ 18-231 and others. During the design stages, immer- sive virtual reality systems provide the best way to learn and experience a design to be constructed. Then, during the planning and scheduling of the virtual project, a construction manager is presented with an opportunity to test different construction methods applying value engineering techniques and checking constructability aspects of the construction process to select the best alternative.
Another important use of an immersive VR sys- tem is its ability to provide training facilities for construction staff. The training which can be done with VR, due to its real time capabilities, is the most realistic situation that a person could encounter with- out actually taking part in the task itself.
Comparing the immersive and non-immersive types of VR, a distinction can be drawn between their potential uses within construction applications. Immersive VR is the better solution for training purposes as it provides a far clearer and more exact representation of the real site environment, whereas non-immersive VR would be ideal for simulating site operations as it would allow activities and equipment within the site to be modeled.
There are already several research projects inves-
tigating the use of VR as a tool for design and control of robots. For example, the VERDEX project  evaluated the use of VR as an interaction tool between human operators and semi-autonomous robots to be deployed in hazardous environments, disaster areas and space.
Coiffet  presents three main difficulties in the current robotic-applications structure and points out where VR can deliver either an improvement or a new efficient approach. These difficulties are: * the difference in behavior between what is fore-
cast and what is really happening as a result of incorrectness in the matter of robot environment modeling;
- the impossibility of understanding environments by mobile autonomous robots;
- the complexity of teleoperation, which requires...