www.elsevier.com/locate/compind

Computers in Industry 58 (2007) 46–56

Virtual hands and virtual reality multimodal platform to design

safer industrial systems

Mamy Pouliquen a,b,*, Alain Bernard a, Jacques Marsot c, Laurent Chodorge b

a IRCCyN – Institut de Recherche en Communications et en Cybernetique de Nantes, Ecole Centrale de Nantes,

1 rue de la Noe, BP 92101, 44321 Nantes Cedex 3, Franceb CEA List, 18 route du Panorama, 92260 Fontenay aux Roses, France

c INRS – Institut National de Recherche et de Securite, avenue de Bourgogne, BP 27, 54501 Vandoeuvre Cedex, France

Received 5 August 2005; received in revised form 24 February 2006; accepted 13 April 2006

Available online 6 June 2006

Abstract

To face with the competitiveness in product design, industrials come up with the solution to use virtual reality (VR) techniques. Coupled with a

dynamic simulation, those techniques lead to natural user interactions with virtual environments (VE). Our research focuses on how to model the

hands of the operator because they allow him to interact with the environment.

In this paper, we address the problem of VR applications to design for a better integration of safety and health requirements. After reviewing the

industrial applications using VR, we present our virtual hands which are coupled with a virtual press-brake by using a system of motion capture and

a force feedback device. Thus, the operator can interact in real-time with the VE. Our simulation tool is also interfaced to a product model that

allows configuring the machine itself. As a result, we are able to estimate the risk level of this machine tool.

# 2006 Elsevier B.V. All rights reserved.

Keywords: Design process; Human–computer interactions; Virtual reality; Physically based animation; Risk prevention

1. Introduction

The evolution of the market demands the reduction of time-

to-market owing to the increasing pressure on product design

[1]. Industrials have to face a challenge that can be divided into

three actions:

� u

*

01

do

sing the newest techniques in design;

� r

educing the time-to-market as much as possible;

� d

ecreasing the design cost by investing the least possible.

These constraints have lead to the introduction of new tech-

niques into the design process [2]. Thus, the digital mock-up

has modified the design cycle. The use of CAD tools allows

visualizing a system in 3D space for a project review for ex-

ample. Since the nineties, industrials have taken advantage of a

new tool to face with competitiveness: virtual reality (VR). This

helps the designers to assess different concepts before the

manufacturing stage [3,4] or to train for maintenance [5] like

Corresponding author. Tel.: +33 2 40 37 69 57; fax: +33 2 40 37 69 30.

E-mail address: Mamy.Pouliquen@irccyn.ec-nantes.fr (M. Pouliquen).

66-3615/$ – see front matter # 2006 Elsevier B.V. All rights reserved.

i:10.1016/j.compind.2006.04.001

assembly/disassembly process. As a result, this new way to

explore the numerical data speeds up the qualification and the

introduction of the new products, decreases cost production and

helps to communicate better with the clients about the new

products.

The current challenge is to take into account the human

being in order to simulate better the interactions between men

and machines and also to generalise the use of the ergonomic

qualification in the design stage [6]. Even if the development of

VR offers new possibilities to better simulate and understand

the human/system/environment interaction [7] thanks to haptic

interaction coupled with dynamic simulation (that is the

simulation of the physical behaviour of the objects and

environment), human modelling is still an open issue. As the

hand is the main interface with the environment, we focus on

the modelling of virtual hands for VR applications such as risk

prevention (e.g. to estimate the degree of hazard of a machine

tool) or maintenance (e.g. to estimate the workspace required

for the operator’s hands and tools).

In this paper, we propose the use of VR-techniques to

simulate better the interactions between man and machine, and

also to estimate the risk level of the working situation that

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–56 47

means {operator–machine–environment}. Not only do we

present a virtual hand to take into account the operator, but we

also propose a method to design safer systems by introducing a

risk index in our dynamic simulation tool.

The rest of the paper is organised as follows:

� S

ection 2 reviews previous and related work;

� S

ection 3 overviews the physically based model of the hands;

� S

ection 4 deals with the control of the hand;

� S

ection 5 presents the dynamic simulation tool: the virtual

press-brake;

� S

ection 6 shows the virtual reality multimodal platform;

� S

ection 7 concludes the paper and presents some discussions.

2. Previous and related work

A set of parameters is essential to obtain a realistic

simulation with VR-techniques:

� a

natural user interaction which means a good model of the

hands and the use of devices such as data gloves and haptics;

� re

al-time frame rates that require algorithms able to achieve

kilohertz rates for collision detection and multi-contact

resolution;

� a

ccurate dynamic models.

The trade-off between all these conditions should improve

the immersion feeling. In our case, we focus on natural user

interactions: that means the simulation of grasping tasks. First,

we present the different models for the hand. Then, we sum up

the previous studies of risk prevention that use VR. We

conclude with the description of our needs and the specifica-

tions of our model.

2.1. Previous models for the hand

The hand is a complex organ. It is made of bones, tendons,

muscles, fat tissues, blood vessels and so on. That is the reason

why there is no complete and sophisticated model of the human

hand for simulation. In this paper, we focus on hand modelling

and animation for grasping tasks. Actually, most of the models

to grasp and to manipulate virtual objects are rigid models.

Rijpkema and Girard [8] studied strategies for grasping. He

identified different approaches based on the biomechanics of

the human hand and on the geometry of the object to define the

main ways of grasping. Then Boulic Rezzonico and Thalmann

[9] presented a model to manipulate virtual objects by using

several sensors fixed on the joints of the hand. In that case, the

measured data and inverse kinematics allow the user to grasp

objects. After rigid virtual objects, Hui and Wong [10]

introduced the handling of deformable objects in real-time

by using a CyberGlove. The grasping is done by solving the

contact equations between the rigid fingers and the finite

element object. Schmidl and Lin [11] proposed a method to

manipulate virtual objects with geometry-driven physics.

Handling is realised by using collision detection and kinematics

methods. It is not effective in our context. The most relevant

model for our research was proposed by ElKoura and Singh

[12]. A data driven approach is adopted to generate the motion

of a virtual hand playing the guitar. In our approach, instead of

using a muscle model to control the virtual hand, we drive it via

a data glove. The best model is proposed by Barbagli et al. in

Ref. [13]. It allows grasping an object by the fingertips of the

thumb and the index. To get a stable posture, they have taken

into account the normal forces and the friction torques; thus, the

object cannot rotate around the axis made by both interaction

points of the tips. Moreover, thanks to a compliance when there

is a contact they simulate the deformation of the pulp of the

fingers. Recently, Pollard and Zordan [14] has presented a

physically based simulation of grasping by using motion

capture and example-based techniques. Key positions of the

hand are saved and replayed automatically by an algorithm.

Then the virtual hands are controlled by human motion data via

control laws and inverse dynamics. As a result they can

simulate in real-time two-hand interactions with a good

physical realism.

Our model is based on Refs. [12–14]. But in our case, we use

motion capture data and data gloves to animate our virtual

hands and we take into account the forces and torques exerted

on the hand via control laws.

2.2. Risk prevention and virtual reality

Everyday human–machine interaction is far from an error-

free process. Accident risk is always present and it can be

explained by the differences between the work situation

defined by the designer and the real one. Moreover, the degree

of hazard mainly depends on the interactions between the

operator and the environment. Consequently, safety issues are

to be considered in every stage of design since decisions will

often have an impact on the safety of a product or production

[15].

In the late nineties, experiments carried out by the National

Institute for Occupational Safety and Health (NIOSH) [16] and

the Finnish Institute of Occupational Health (FIOH) [17]

involving handling and working at a height gave the first

interesting results of safety analysis procedures with VE. At the

same time, Bell and Scott Fogler [18] has performed several

simulations for hazard analysis in chemical engineering.

Recently, Maatta [19] has shown in his PhD dissertation that

VR-techniques have improved the safety analysis in case

studies like a steel converter plant or a coil conveyor system

and, that visualization by computer modelling can be

effectively used for the safety analysis purpose. In the same

concept, INRS, the French National Research and Safety

Institute, is studying how to combine the occupational risk

prevention viewpoint with VR-techniques to achieve a better

integration of prevention at the work equipment design stage

[20]. On the one hand, the purpose is to demonstrate that, by

providing the designer with virtual operating resources, VR

enables him to check, by successive iteration, that procedures,

operating instructions, etc., foreseen in relation to equipment

operation, neither introduce specific risks nor degrade the

system level of safety. On the other hand, the same VE can be

M. Pouliquen et al. / Computers48

used to train a workforce in assembly, operational and

maintenance tasks to increase their skill levels and safety

awareness. Indeed, in addition to the technical design of a

machine, its use must also be designed. As a result, training

represents an essential component of the occupational risk

prevention system.

2.3. Problematics

In our case, a specific application case (e.g. folding of sheet-

metal plates) has been retained because of the advantage it

offers in prevention terms. Then, we simulate a press-brake to

study means of enhancing the safety analysis procedure in the

design phase of a machine system with VE. Accidents with this

industrial machine tool often cause severe damage for the

operators. Moreover, it is difficult to improve the safety of such

a system owing to the wide range of bending applications

realised. To analyse the degree of hazard of the human–

machine interaction in this case, we have to model the hands of

the operator. As a result, we need multibody hands to interact

with the virtual press-brake and to grasp the virtual sheet-metal

work piece.

Our main objective is to make the simulation of the

interaction with the virtual press-brake the most realistic

possible and to study the benefit of VR to estimate the degree of

hazard of given configurations of the virtual press-brake [21].

As VR-techniques lead to a retroaction loop, this simulation

should give results to be used to improve the final tool machine.

The interaction with the VE should be a good metaphor for the

real one, and in the process, levels of residual risks originating

in design should thereby be reduced.

In next section, we propose a model for the hands of the

operator. This is a physically based model as we take into

account the anatomy and the biomechanics of the human hand.

3. Physically based virtual hands

The human hand is a fertile area of research in many

disciplines such as medical fields, computer graphics or

robotics. In our simulation, the virtual hands allow the

operator to interact with the sheet-metal work piece and the

virtual press-brake. As a result, we need a multibody system

to grasp and manipulate the work piece. To achieve a good

feeling of immersion, it is essential that virtual hands behave

as in real life. That is the reason why our model takes

biomechanics into account to define a human-like kinematics

model. In this section, we describe the model used for our

virtual hands.

3.1. Biomechanics and kinematics

The skeleton is made up of 27 bones (Fig. 1) [22–24]:

� [

8] short bones called carpals;

� [

5] metacarpals;

� [

14] phalanges divided into three types: proximal (5),

intermediate (4) and distal (5).

The hand is articulated by 28 degrees of freedom (DOF): 6

DOFs for the wrist and 22 DOFs for all the joints of the fingers

(Fig. 2) [22–24].

Each finger, except the thumb, is modelled by three

phalanges linked by four hinges: three hinges for the flexion/

extension movements and a hinge for the abduction/adduction

movements.

The thumb is more complex and owing to its high mobility,

we have used a ball and socket joint that means 3 DOFs.

3.2. Interdependence relationships

The movements of fingers are highly constrained because

some postures are unreachable. The actions of the muscles

and tendons and the anatomy of the human hand can be

translated by constraints and limits to take into account

between the joints of a virtual hand [23,25]. There are static

constraints and dynamic constraints. The first ones are the

angular limits of the range of finger motions, in flexion/

extension—noted ( f /e) and in abduction/adduction—noted

(a/a). As a result, hand articulation is limited within a

boundary as for instance the limits of the joint between the

carpal and the first phalange, called uMCP( f /e), which is given

by the following relation:

0� � uMCPð f=eÞ � 90� (1)

As for the dynamic constraints, they describe the correlations

among different joints. On the one hand, they symbolize the

dependencies between the phalanges of a finger (Eq. (2)). For

example, the motions of the DIP joint and PIP joint are

dependent (see Fig. 1 for the localisation of the joints). They

can be described as:

uDIPð f=eÞ ¼ 2

3� uPIPð f=eÞ (2)

with, respectively, uDIP( f/e) and uPIP( f/e) the angles at the DIP

joint and PIP joint in flexion/extension motion.

On the other hand, these constraints translate the relation-

ships between the joints of the nearest fingers. For instance, the

flexion (respective extension) of the proximal phalange of the

index finger or that of the ring finger forces the flexion

(respective extension) of the proximal phalange of the middle

finger. We obtain the following equation:

umiddleMCP ð f=eÞ� supðuindex

MCP ð f=eÞ � 25; uringMCPð f=eÞ

� 45; umiddleMCP ð f=eÞminÞ (3)

with, respectively, uindexMCP ð f=eÞ, u

ringMCPð f=eÞ and umiddle

MCP ð f=eÞ the

angular positions at the MCP joints of the index, ring and

middle fingers, and with umiddleMCP ð f=eÞmin the minimal angle of

the proximal phalange of the middle finger.

Our model of a virtual hand is based on the skeleton of the

human hand as detailed before. Moreover, during the

displacement of the hand, the palm remains most of the time

in a plane. As a result, we have modelled it as a rigid structure.

We also use simple primitives for the phalanges and hinges for

in Industry 58 (2007) 46–56

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–56 49

Fig. 1. The skeleton of the hand.

the joints. This rigid model has been performed by VORTEXTM

[26] (Fig. 3).

All the data of the virtual hand (sizes of the phalanges and

palm, constraints between the phalanges, positions) are stored

in a XML file. These parameters are loaded in our dynamic

simulation. We can modify them in order to take into account

the anthropometric differences between the human beings.

Moreover, friction has been integrated between finger pads and

the VE which enables the grasp and manipulation of virtual

objects without sliding.

In order to reduce the complexity of our model, we do not

integrate the actions of the muscles, nor the deformations of the

finger pads. A deformable model for the fingers has already

been tested and validated (see Ref. [27] for more details). In the

area of interaction techniques, natural manipulation of objects

still needs considerable research. In this paper, we focus on the

animation of rigid virtual hands in real-time. In the next section,

we present the different control laws tested.

4. Animation of the virtual hands

After modelling the hand, it is essential to animate it in a

realistic way. We can perform this task by two different

methods:

� B

y integrating a musculoskeletal model. The hand is

controlled by the contractions of the muscles and the action

of the tendons.

� B

y using a system of motion capture.

After preliminary results, the complexity of the musculos-

keletal model was a bottleneck for the dynamics of the

simulation: the constraint of real-time was not respected.

Consequently, we have chosen to control the hand by using the

data from a motion capture system. Thus, the position of the

virtual hand is given by specific devices: a data glove for

the motion of the fingers and optical trackers for the palm

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–5650

Fig. 2. The degrees of freedom of the hand.

Fig. 3. The hand modelled with VORTEX.

(or wrist). As the data glove measures the position of the

fingers, it is no longer necessary to integrate the static and

dynamic constraints cited in Section 3.2.

We present now the coupling between the data glove and the

trackers in the VE.

4.1. Coupling between virtual and real worlds

Firstly, it is important to differentiate the real environment

from the virtual one. In the first, the operator moves and exerts

forces. In the latter, those interactions have to be modelled

during the simulation for the sake of immersion.

Thus a controller allows both environments to exchange data

in order to simulate the interactions between them. It is used to

link the data glove and the trackers to the physical PC running

the real-time simulation of the virtual world. We can compute

the position of the devices according to the operator’s

movements within the virtual world by means of the controller

(Fig. 4).

4.2. Hand control

In this section, we present the algorithm implemented in

order to achieve a stable simulation of the virtual hands

interacting with the environment with a motion capture

system. At first, we present the control of the palm (or wrist).

Then, we explain two methods tested to control our virtual

fingers.

4.2.1. Control of the palm

The control law is based on a passive approach. It emulates a

virtual coupling between the motion capture devices and the

real-time simulation, which is done in position. The forces are

computed from the position errors thanks to a virtual spring/

damper system that is equivalent to a proportional–derivative

control. The input data of our controller are the 3D Cartesian

coordinates of the tracker of the palm.

We control the palm in position and orientation by

calculating the desired force and the desired torque with

Eqs. (4) and (5):

Fd ¼ kFPðxd � xÞ þ bFPðxd � xÞ (4)

where xd and xd are, respectively, the desired position and

desired linear velocity, and x and x the current ones.

Td ¼ kTPðud � uÞ þ bTPðud � uÞ (5)

where ud and ud are, respectively, the desired orientation and

desired angular velocity, and u and u the current ones.

kFP and bFP are, respectively, the stiffness and damping

coefficients of the virtual object for the position control. They

depend on the mass of the palm. kTP and bTP are the same

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–56 51

Fig. 4. The controller.

coefficients for the orientation control and they depend on the

inertial matrix of the palm.

4.2.2. Control of the fingers by the joints

We have implemented an angular control for our virtual

fingers. We have coupled the user’s hand with a 22-sensor

CyberGlove [28]. This data glove tracks the finger displace-

ments by giving the angular position at each joint. Thus, by

measuring the angle of every phalange of the user’s fingers and

by knowing the angular position of the phalanges of the virtual

fingers at every time step, we are able to calculate the error in

position. By using Eq. (5), we can estimate the force to exert on

the joint to reach the desired angular position.

4.2.3. Control of the fingers by the tips

Then we have tested a Cartesian control of the virtual

fingers. The fingertips are tracked via a motion capture system.

At each time step we know the 3D Cartesian position of the

fingertips of the user and the position and orientation of his/her

palm. The principle of our algorithm is based on the control in

position and/or in orientation of our virtual fingers. We describe

the controller bloc diagram for a finger (Fig. 5).

As the kinematics has been previously described in Section

3, we are able to estimate the torques to reach the desired

configuration. We can define the direct geometric model Hi of

the fingers and the kinematics wrench Ti. Then we calculate the

Jacobian matrix Ji for every phalange. The control is equivalent

to a 3D virtual spring/damper system (spring K and damper B)

between the desired positions and the tracked fingertips [29].

As a result, we can calculate the wrenches exerted on the tips of

a finger noted BW(Bi, Ti) for the damper and KW(Ki, Hi) for the

Fig. 5. The fingertip control scheme.

spring. We obtain the torques to apply to the joints of the virtual

fingers. For every joint of a finger, the torque powered by the

force exerted on the virtual finger is given by Eq. (6):

Ti ¼ JTi WT

i (6)

With the virtual coupling, the equation becomes:

Ti ¼ JTi

KWTi þ JT

iBWT

i ¼ JTi

KWTi � ðJT

i BiJiÞu (7)

with u the angular velocity of a joint.

This coupling is equivalent to a proportional–derivative

control of force error at the fingertips.

4.2.4. Conclusion

We have tested both methods in our environment (see next

section). In both cases, the user managed to interact with the

virtual press-brake or the virtual sheet-metal in a stable way.

For the sake of simplicity, we have chosen to use the control of

the fingertips. This method is more general than the control by

the joints. It allows us to control whatever hand skeleton – of a

child or adult – with whatever optical devices with the same

implemented equations.

5. The dynamic simulation tool

In this section, we first present an overview of our dynamic

simulation tool. Then, we describe the different work situations

that allow to configure the machine and its environment. Both

of them help to estimate the degree of hazard of our simulation.

5.1. The virtual press-brake

The simulation is a sheet-metal press-brake bending

application. Thus, we have modelled a virtual press-brake by

adhering to the specifications of the real one.

The objective is to fold a flat metal sheet form using a virtual

press-brake. To perform this, we have simulated the whole

press-brake operating part: punch, back gages, work piece

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–5652

Fig. 6. The virtual press-brake.

Fig. 7. The virtual machine shop.

support and the like. We have also modelled the safety devices

such as light curtains, laser beams or side guards (Fig. 6).

Finally, we have represented the whole machine shop where

the operator works to improve the feeling of immersion (Fig. 7).

In order to interact with the virtual press-brake, we use a

haptic interface and motion capture devices. Thanks to these

interfaces we can simulate the grip of the work piece and the

contact with other physical components of the press-brake.

The press-brake is industrial equipment that can be used to

create a wide range of pieces: from small ones about a few

centimetres in size for which the fingers are very close to the

punch, to several-meter-large sheets requiring two operators. It

is relevant to take that diversity of tasks into account for the

conception of such a machine in order to offer maximal safety

in whatever task cited before. We have therefore created

different work situations (that is to say scenarii) defining

different configurations of the press-brake by using all of the

safety devices or some of them.

5.2. Work situation generic model

Recent research studies on the design of complex systems

and their modelling have enabled us to propose a data-based

tool (MOSTRA1), based on the work situation system model

described in Ref. [30].

Application of MOSTRA must allow the designer to

consider both viewing multi-point and multi-occupational data

and their interdependencies, in particular through the notion of

risk: see Ref. [31] for more details. A first software

demonstrator of this model has been developed under

AccessTM. By coupling it with the VR application, it is

possible to:

� c

m

‘‘I

N

as

onfigure the VE corresponding to the required ‘‘work’’

situation and, therefore, to choose involving protection

devices or operating modes for example;

1 MOSTRA: MOdele de Situation de TRAvail [work situation model]. This

odel was developed within the framework of a project undertaken by the

ntegration of Risk Prevention at the design stage group’’ of the French

ational Research and Safety Institute (INRS) and jointly financed by CNRS

part of the ‘‘Production systems Program’’ (PROSPER).

� a

Se

ccess different attributes and links of a ‘‘work’’ situation

object by selecting it in the VE;

� a

llocate dynamically the different risk assessment parameters

associated with the ‘‘work’’ situation and display a risk index

in the scene in real-time for the simulated situation.

As a result, we can simulate different configurations from a

basic virtual press-brake without safety devices to the same

press-brake with safeguards, specific tools, specific safety

devices (light curtains or laser systems). Light curtains protect a

wide area in front of the operator, while laser systems protect

locally the fingers of the operator. The choice of these systems

depends of the type of production (high or low rate of output)

and the sizes of the pieces [32]. In Fig. 8, the position of the

virtual hands in a dangerous area leads to an automatic release

of safety devices.

The objectives of this dynamic simulation tool are the

evaluation of the safety level of the simulated situation. To

reach them, this tool was parameterized according to the

methodology described by AISS2 [33]. Thus, it allows the

designer to check if the technical choices, such as procedures or

operating instructions foreseen in relation to equipment

operation in a VE, do not degrade the level of safety of the

future work situation. The overall risk index for a work

situation, the ‘‘R’’ index, is calculated from three parametric

families, which must be evaluated or measured:

� th

ose concerning machine-intrinsic risks: the ‘‘M’’ index;

� th

ose concerning work station environment impact: the ‘‘E’’

index;

� th

ose concerning the ability of someone to control these risks:

the ‘‘P’’ index.

These factors depend on the simulated application and on

different parameters. Some of them are static because they are

inherent in the working situation. For example, the energy

level, the safety level of the facilities or the skills of the

operator are predefined at the beginning of the simulation.

Nevertheless, they can be modified during the simulation loop

and then they are updated in the database called MOSTRA.

2 AISS: Association Internationale de Securite Sociale [International Social

curity Association].

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–56 53

Fig. 8. Example of laser beams as safety devices.

Other parameters are dynamic such as parameters linked to the

task or to the operator, the posture of the operator or the

exposure frequency in the dangerous area. They are calculated

at every time step.

As a result, we can estimate the global risk level of the

simulation and compare it with predefined degrees of hazard.

This coefficient R and the three risk factors M, E and P are

displayed in the right upper side of the screen with coloured

gauges (Fig. 9). The colour index depends on pre-determined

threshold values. The degree of hazard changes with the actions

of the operator while manipulating the work piece sheet-metal.

Every time step, the displayed indexes of the current situation

show to the operator the improvement of the safety or the

increase of hazard [34].

We have explained how the degree of hazard can be

estimated in real-time for a given configuration of the virtual

press-brake. The VR platform and the applications are

described in the next section.

6. VR platform and applications

After describing the human–machine interactions, we

present the VR platform and the sheet-metal press-brake

bending application.

Fig. 9. Estimation of the degree of hazard in a given configuration.

6.1. The VR platform

The INRS large workspace VR platform is a multimodal

immersive system (Fig. 10). It is organised as follows:

� A

real-time physical simulator. The virtual objects evolution

is governed by VORTEXTM which is a real-time physics

engine provided by CM Labs [26]. The dynamics and physics

calculations are performed by this software.

� A

real-time graphical simulator. Visualization and rendering

are performed by VIRTOOLSTM [35]. This software

manages VR displays and peripheral devices to conceive

an application.

� A

real-time optical motion capture system supplied by

VICON [36] including four cameras. This system uses optical

3D measurement sensors to give accurate positions of

markers in 3D-space.

� A

screen of 2.5 m � 2 m, provided by Barco [37] with

stereoscopic retro-projections (we use passive stereo in our

case) to display the simulation.

� A

cluster of PCs dedicated to the stereoscopic screen display.

Fig. 10. The VR platform of INRS.

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–5654

Fig. 11. The modified Virtuose 6D-RVTM.

Fig. 12. The dynamic simulation of a sheet-metal press-brake bending applica-

tion.

� A

Virtuose 6D-RVTM interface with six degrees of freedom

made by Haption [38] that gives force and torque feedback. It

enables the user to touch and handle objects located in the

VE.

� A

CyberGlove with 22 sensors provided by Immersion

Corporation [28] that gives the angular positions of all the

joints of the fingers in flexion/extension and abduction/

adduction motions.

� A

stereo sound system to create the noise generated by a

press-brake.

All the simulation has been implemented using C/C++.

6.2. The folding application

In order to interact with the VE, we use a haptic interface.

Thanks to that device, we can simulate the grip of the work

piece and the contact between metal-sheet and the other

physical components of the press-brake.

A plastic sheet has been clamped on the handle of the haptic

interface. The operator can then simulate the manipulation of

metal-sheet by grasping the plastic sheet (Figs. 11 and 12).

A controller is also used to link the Virtuose 6D-RVTM to the

real-time simulation of the virtual world. Most of the time,

controllers use the haptic device to link the virtual and real

environments. The forces powered by the physical simulation

are directly applied to the haptic device. In our case, we use the

Fig. 13. Using a data glove to inter

same control laws as the previous ones described in Section 4.2.

The controller computes the position of the device according to

the operator’s movements and it manages force feedback

according to the interactions within the virtual world. As a

result, the virtual coupling, a virtual spring and damper, leads to

the achievement of a stable simulation.

6.3. Human–machine interactions

The hands of the operator are tracked by a motion capture

system and data gloves. Thanks to the virtual couplings

described in Section 4, the virtual hands are controlled in real-

time. The operator is able to interact with the virtual press-

brake. He also can grasp and manipulate the sheet-metal piece

(Figs. 13 and 14).

7. Discussion and future work

In the area of interaction techniques, natural manipulation of

objects still needs considerable research. In this paper, we have

presented human–machine interactions in the case of a sheet-

metal press-brake bending application. The use of virtual

reality techniques allows us to model a press-brake and virtual

hands to perform this folding application. The real operator

moves within a large virtual environment and interacts with the

virtual press-brake and the metal sheet via a haptic interface

and motion capture devices (such as trackers and data gloves).

Thus, he/she can manipulate physically a plastic sheet similarly

to real operating conditions by respecting folding scenarii.

act with the virtual metal-sheet.

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–56 55

Fig. 14. The folding application with the virtual hands.

We have proposed a model of a virtual hand which is

physically based thanks to the respect of the kinematics and

biomechanics of the human hand. To achieve a stable

simulation in a continuous time, we have implemented an

algorithm to control our virtual hands in the environment that

acts like a virtual spring and damper. We have also tested two

methods for the fingers – articular and Cartesian controls – and

both of them gave good results.

Our result is promising. This simulation seems to highlight the

contribution of virtual reality for designing and testing safety

devices on industrial equipment without endangering the human

operator. Thus, the association of VR-techniques to a model of

interactive physically based hands leads the designers to have a

first experiment of the machine before making the first prototype.

As a result, the integration of risk prevention can be done in the

firsts stages of the design cycle and human–machine interactions

can be better estimated. Work is underway to test our simulation

by professional operators and to evaluate the relevance of VR-

techniques. This validation phase is essential to compare the

virtual environment to the real one in our study case.

Another development focuses on the virtual hands. Our

objective is to improve the current model to simulate

deformations under contact forces in real-time. We wish to

take into account the physiology and the musculoskeletal

model of the human hand but till now the computation time is

too high to be interactive. Moreover, the study of those data is

still an open issue and some of them remain unknown. Such a

more realistic model could be used for other simulations such as

risk prevention of musculoskeletal disorders induced by typing

or pneumatic drill use.

Acknowledgments

This work is a partnership between INRS (French National

Research and Safety Institute), CEA LIST (Atomic Energy

Commission) and IRCCyN (Communications and Cybernetic

Research Institute of Nantes).

The authors would like to thank the courtesy of the Work

Equipment Engineering Department (INRS) for the photos of

the virtual press-brake.

They would also like to thank the anonymous reviewers for

theirs comments which greatly helped in improving this paper.

References

[1] Y. Xie, On the modern design theory, in: Keynote Paper of the CIRP

Design Seminar, Shanghai, China, 2005.

[2] A. Bernard, Virtual engineering: methods and tools, in: Keynote Paper of

International Conference on Advanced Research in Virtual and Rapid

Prototyping, Leira, Portugal, 2003.

[3] M. Weyrich, P. Drews, An interactive environment for virtual manufactur-

ing: the virtual workbench, Computers in Industry 38 (January (1)) (1999)

5–15.

[4] W. Dangelmaier, M. Fischer, J. Gausemeier, M. Grafe, C. Matysczok,

B. Mueck, Virtual and augmented reality support for discrete manu-

facturing system simulation, Computers in Industry 56 (May (4))

(2005) 371–383.

[5] J.R. Li, L.P. Khoo, S.B. Tor, Desktop virtual reality for maintenance

training: an object oriented prototype system (V-REALISM), Computers

in Industry 52 (2) (2003) 109–125.

[6] J.-M. Burkhardt, Realite virtuelle et ergonomie: quelques apports reci-

proques, Le travail humain 66 (1) (2003).

[7] G. Zachmann, A. Rettig, Natural and robust interaction in virtual assembly

simulation, in: ISPE International Conference on Concurrent Engineering,

California, USA, 2001.

[8] H. Rijpkema, M. Girard, Computer animation of knowledge-based human

grasping, Computer Graphics 25 (4) (1991) 339–348.

[9] R.S. Boulic Rezzonico, D. Thalmann, Multi-fingers manipulation

of virtual objects, in: Proceedings of the ACM Symposium on

Virtual Reality Software and Technology, Hong-Kong, Japan, (1996),

pp. 67–74.

[10] K.C. Hui, N.N. Wong, Hands on a virtually elastic object, The Visual

Computer 18 (2002) 150–163.

[11] H. Schmidl, M.C. Lin, Geometry-driven physical interaction between

avatars and virtual environments, Computer Animation and Virtual

Worlds Journal (2004).

[12] G. ElKoura, K. Singh, Handrix: animating the human hand, in:

ACM SIGGRAPH Symposium on Computer Animation, San Diego,

USA, 2003.

[13] F. Barbagli, A. Frisoli, K. Salisbury, M. Bergamasco, Simulating human

fingers: a soft-finger proxy model and algorithm, in: Proceedings of IEEE

Haptics, Chicago, USA, (2004), pp. 9–17.

[14] N.S. Pollard, V.B. Zordan, Physically-based grasping control from exam-

ple, in: Proceedings of Eurographics/ACM Siggraph Symposium on

Computer Animation, Los Angeles, USA, 2005.

[15] J. CoVan, Safety Engineering, Wiley, New York, 1995, ISBN: 0-471-

55612-2.

[16] B. Dotson, H. Hsiao, S.S. Chiou, J.J. Dipasquale, Occupational traumatic

injury prevention using virtual reality, in: Proceedings of National Occu-

pational Injury Research Symposium, Morgantown, USA, 1997.

[17] H. Laitinen, Usability Through Product Design: New Design Methods and

Industrial Applications, Research Project, Dept. of Occupational Safety,

FIOH, Vantaa, Finland, 1997.

[18] J.T. Bell, H. Scott Fogler, A virtual reality safety and hazard analysis

simulation, in: Proceedings of American Society for Engineering Educa-

tion Annual Conference, American Society Engineering Education, St.

Louis, USA, 2000.

[19] T. Maatta, Virtual environments in machinery safety analysis, Ph.D.

Dissertation, VTT Publications 516, 2003.

[20] J. Ciccotelli, Realite virtuelle, une aide a la decision pour la conception de

systemes surs, in: Proceedings of European Conference on System

Dependability and Safety Decision Making and Risk Management, Lyon,

France, 2002.

M. Pouliquen et al. / Computers in Industry 58 (2007) 46–5656

[21] J. Marsot, J. Ciccotelli, F. Gardeux, Virtual machine for risk prevention, in:

Proceedings of 35th International Symposium on Robotics (ISR’2004),

Paris, France, March, 2004.

[22] H. Rouviere, A. Delmas, 2002. Anatomie humaine descriptive, topogra-

phique et fonctionnelle – Membres, systeme nerveux central, in french,

Tome 3, 15eme edition, Editions MASSON, ISBN 2294003934.

[23] A. Vardy, Articulated Human Hand Model with Inter-joint Dependency

Constraints, Computer Science 6572, Computer Graphics Project Report,

1998.

[24] J. Sobotta, Atlas of Human Anatomy, in: R. Putz, R. Pabst (Eds.),13th ed.,

Head, Neck, Upper Limb, vol. 1, Lippincott Williams & Wilkins,

Hardcover, 2001, ISBN: 0781731755, p. 833.

[25] J. Landsmeer, Studies in the anatomy of articulation, Acta morphologica

Neerlando-Scandinavia 3 (1961) 287–303.

[26] Vortex: www.cmlabs.com.

[27] M. Pouliquen, C. Duriez, C. Andriot, A. Bernard, L. Chodorge, F.

Gosselin, Real-time finite element finger pinch grasp simulation, in:

Proceedings of World Haptics Conference, Pisa, Italy, March, (2005),

pp. 323–328.

[28] Immersion Corporation: www.immersion.com.

[29] S. Stramigioli, Modelling and IPC control of interactive mechanical

systems: a coordinate free approach, INCIS Serie, Springer ed., 2001.

[30] Ch. Guillevic, ‘‘Psychologie du travail’’, Editions Nathan, collection Fac

Psychologie, Paris, 1991, 225 pp.

[31] R. Hasan, A. Bernard, J. Ciccotelli, P. Martin, Integrating safety into the

design process: elements and concepts relative to the working situation, in:

Safety Science – Special issue�Safety in design, vol. 41 (2–3), 2003,

pp. 155–180, ISSN 0925-7535.

[32] J. Gillot, J. Baudoin, J.P. Bello, J.C. Blaide, Hydraulic Press-Brakes

Designed to Work Cold Metal – Improvement of the Safety on the

Novation of Machines in Use – Technical Requirements for Users, Safety

Experts and Renovators, ED 927, INRS, Paris, Juillet, 2004, 64 pp. (in

French).

[33] AISS – Association Internationale de Securite Sociale [International

Social Security Association], Calculez vous-meme vos risques accidents

– Appreciation du risque mecanique au poste de travail – AISS Prevention,

series no. 2034 (F), 1998, 33 pp., ISBN 92-843-2130-1, 2003, DTE 175,

32 pp.

[34] J. Marsot, J. Ciccotelli, Dynamical indexation of risk in a virtual work

environment, in: Proceedings of Virtual Concept 2005 Biarritz, France,

November 8–10, 2005.

[35] Virtools: www.virtools.com.

[36] Vicon: www.vicon.com.

[37] Barco: www.barco.com.

[38] Haption: www.haption.com.

Mamy Pouliquen received a master of mechanical

engineering in 2002 at the Ecole Centrale in Nantes

(France). She is currently a PhD student in

mechanics at the Communications and Cybernetic

Research Institute of Nantes (IRCCyN – Ecole

Centrale of Nantes – France). She works on the

interactive realistic and physical simulation of the

human hand for human–computer interactions,

accident prevention and job safety training in virtual

environments.

Prof. A. Bernard was graduated in 1982 at ENS

Cachan. He contributed to LURPA laboratory

(Research Laboratory for Production), with Prof.

Bourdet, since 1983 and obtained his PhD in

1989, on 3D feature-based manufacturing of forging

dies. As an assistant professor, he worked from 1990,

for six years, with Prof. Bocquet in Ecole Centrale

Paris (Research laboratory on Mechanical Engineer-

ing and Logistics) on product, technology and pro-

cess modeling. He also created, in 1993, the rapid

prototyping and reverse engineering platform of Ecole Centrale Paris, the

CREATE (European Rapid prototyping Center for Assistance, Transfert and

Experiment). He is the vice-president of AFPR (French Rapid Prototyping

Association) and its representative in GARPA (Global Alliance of Rapid

Prototyping Associations). From September 1996 to October 2001, he has

been Professor in CRAN laboratory (Research Center for Automatic Control of

Nancy) in Nancy, where he managed a research group (ICF) on mechanical and

production engineering. His main research topics are related to reverse engi-

neering, knowledge-based systems for Computer-Aided process planning

(applied to machining, rapid prototyping and laser digitizing), and product

and process modeling. His actual position is in Ecole Centrale de Nantes (Head

of the ‘‘Industrial products and systems engineering’’ department) and for

research activities, in IRCCyN (Research Institute for Communications and

Cybernetics of Nantes) and more exactly head of the ‘‘Virtual Engineering for

industrial engineering’’ project.

Jacques Marsot, Dipl-eng in electromechanics, 44

years old, INRS since 1993. Responsible in the

‘‘Working Equipment Engineering’’ Department of

the Laboratory ‘‘Dependability of Machinery and

Components’’ in charge of studies related to methods

used in design engineering (Virtual reality, concur-

rent engineering, etc.) and also assessment of safety

components.

degree from the Ecole Nationale des Ponts et

Chaussees in 1993, together with a Masters of

Science in Robotics from the University of Paris

Laurent Chodorge (1968) received his engineering

VI. He entered CEA in 1994, as a research engi-

neer. He was first responsible for the numeric and

physical validation of a new multi-physics simula-

tion code, in the Defense Division. He joint in 2000

a robotics and virtual reality Lab, inside the

Applied research Division. There, he managed

several projects in VR and interactive simulation. He notably worked on

the design of an engineering software tool for the interactive simulation of

interventions in nuclear environments (5 people team). Since 2006, he is the

head of the Interactive Simulation Lab. Around 20 high qualified staff

members work in the unit, including 8 PhD students. LSI develops interactive

software for virtual reality, and works closely with industry. Application

domains are virtual prototyping for car and plane conception, nuclear

hazardous planification.