ISO/IEC TC JTC 1/SC 29 N - IPSJ/ITSCJ · Web viewFor a convex corner , , where is the angle between the two crease edges spanning the sector, for concave corners,. For concave corners,

INTERNATIONAL ORGANISATION FOR STANDARDISATIONORGANISATION INTERNATIONALE DE NORMALISATION

ISO/IEC JTC1/SC29/WG11

CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC1/SC29/WG11 N4415December 2001, Pattaya

Title: PDAM of ISO/IEC 14496-1 / AMD4Source: MPEG-4 SystemsEditors: Mikaël Bourges-Sévenier (Mindego) - Editor

Iver Grini (Octaga)

Michael Steliaros (Superscape)Dominique Curet (France Telecom R&D)

Status: Draft

Document type: Document subtype: Document stage: Document language:

/tt/file_convert/5aa3f6857f8b9ada698eea94/document.doc

ISO/IEC TC JTC 1/SC 29 N

Date: 2002-01-30

ISO/IEC 14496-1:2001/PDAM 4

ISO/IEC TC JTC 1/SC 29/WG 11

Secretariat:

Information technology — Coding of audio-visual objects — Part 1: Systems, AMENDMENT 4: Multi-users worlds and advanced animation effects

Élément introductif — Élément central — Partie 1: Titre de la partie

Warning

This document is not an ISO International Standard. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an International Standard.

Recipients of this document are invited to submit, with their comments, notification of any relevant patent rights of which they are aware and to provide supporting documentation.

ISO/IEC 14496-1:2001/PDAM 4

Copyright notice

This ISO document is a working draft or committee draft and is copyright-protected by ISO. While the reproduction of working drafts or committee drafts in any form for use by participants in the ISO standards development process is permitted without prior permission from ISO, neither this document nor any extract from it may be reproduced, stored or transmitted in any form for any other purpose without prior written permission from ISO.

Requests for permission to reproduce this document for the purpose of selling it should be addressed as shown below or to ISO's member body in the country of the requester:

[Indicate the full address, telephone number, fax number, telex number, and electronic mail address, as appropriate, of the Copyright Manger of the ISO member body responsible for the secretariat of the TC or SC within the framework of which the working document has been prepared.]

Reproduction for sales purposes may be subject to royalty payments or a licensing agreement.

Violators may be prosecuted.

© ISO/IEC 2002 — All rights reserved 3

ISO/IEC 14496-1:2001/PDAM 4

Contents Page

Foreword.......................................................................................................................................................... viIntroduction.................................................................................................................................................... viiOrganization of this document.....................................................................................................................vii1 Amendment to ISO/IEC 14496-1:2001 (N4266)..................................................................................11.1 SFVec4f and MFVec4f editings (16 modifs)......................................................................................11.2 4D interpolators................................................................................................................................... 91.2.1 CoordinateInterpolator4D................................................................................................................... 91.2.2 PositionInterpolator4D....................................................................................................................... 91.3 Extended BIFS-Updates..................................................................................................................... 91.3.1 ExtendedUpdate................................................................................................................................ 101.3.2 PROTOlistInsertion........................................................................................................................... 111.3.3 PROTOlistDeletion............................................................................................................................ 111.3.4 MultipleFieldReplacement................................................................................................................121.3.5 MultipleIndexedFieldReplacement..................................................................................................121.3.6 GlobalQuantizationConfiguration....................................................................................................121.4 FlexMux tool...................................................................................................................................... 131.4.1 The MPEG-4 FlexMux related descriptors (modified 8.2.2.2)........................................................131.4.2 The Second FlexMux tool definition (Modified 12.2.4)...................................................................161.4.3 FlexMux packet specification (in-band signalling).........................................................................182 Animation Framework eXtension (AFX)..........................................................................................192.1 Introduction....................................................................................................................................... 192.2 Conceptual organization of tools....................................................................................................202.3 Geometry tools.................................................................................................................................. 222.3.1 NURBS geometry.............................................................................................................................. 222.3.2 Subdivision Surfaces........................................................................................................................ 282.3.3 Wavelet Subdivision Surfaces.........................................................................................................602.3.4 MeshGrid representation.................................................................................................................. 672.3.5 Particle systems.............................................................................................................................. 1312.4 Solid representation....................................................................................................................... 1372.4.1 Overview.......................................................................................................................................... 1372.4.2 Node specification..........................................................................................................................1652.4.3 Language specification..................................................................................................................1682.4.4 Rendering of implicit surfaces.......................................................................................................1712.4.5 Examples......................................................................................................................................... 1712.5 Texture tools.................................................................................................................................... 1752.6 Modeling tools................................................................................................................................. 2462.6.1 Introduction..................................................................................................................................... 2462.6.2 Nonlinear global deformation........................................................................................................2462.6.3 Free-Form Deformations (FFD)......................................................................................................2482.7 Animation tools............................................................................................................................... 2502.7.1 Animators........................................................................................................................................ 2502.8 Biomechanical tools.......................................................................................................................2652.8.1 Introduction....................................................................................................................................... 2652.8.2 Bone-based animation....................................................................................................................2663 Multi-User Worlds (MUW)...............................................................................................................2933.1 Glossary of Abbreviations..............................................................................................................2933.2 Introduction..................................................................................................................................... 2933.2.1 Multi-users world technologies are important..............................................................................2933.2.2 Approaches for multi-user world technology...............................................................................293

4 © ISO/IEC 2002 — All rights reserved

ISO/IEC 14496-1:2001/PDAM 4

3.2.3 Proposal Overview.......................................................................................................................... 2943.3 Technical Description..................................................................................................................... 2953.3.1 The Multiuser Nodes....................................................................................................................... 2953.3.2 Encoding of MUNodes.................................................................................................................... 2993.3.3 Behaviour of existing MPEG-4 nodes when shared.....................................................................3003.3.4 The MUCommandStream................................................................................................................3023.3.5 Descriptors for MUCommandStreams..........................................................................................3033.3.6 The MUCommandStream in detail.................................................................................................3063.3.7 The message Encoding.................................................................................................................. 3133.3.8 Description of entities mentioned.................................................................................................3283.3.9 Tables............................................................................................................................................... 328Annex A........................................................................................................................................................ 330Multi-User concepts..................................................................................................................................... 330A.1 High level architecture.................................................................................................................... 330A.1.1 MUTech Session Controller (MSC)................................................................................................331A.1.2 MUTech Bookkeeper (MBK)...........................................................................................................332A.1.3 MUTech Message Handler (MMH)..................................................................................................332A.1.4 MUChannelManager (MCM)............................................................................................................333A.1.5 MUTech Instance............................................................................................................................. 334A.1.6 AV Stream Server............................................................................................................................ 334A.1.7 MUSession....................................................................................................................................... 335A.2 Rationale for supporting for transfer of pilotship........................................................................335A.2.1 With the Pilot on the server............................................................................................................335A.2.2 With the Pilot on client 1................................................................................................................. 336A.2.3 With the Pilot on client 2................................................................................................................. 337Node Coding Tables.................................................................................................................................... 339B.1 Node Tables..................................................................................................................................... 339B.2 Node Definition Type Tables..........................................................................................................357References.................................................................................................................................................... 365


ISO/IEC 14496-1:2001/PDAM 4

Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Standards is normally carried out through ISO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization.

International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.

The main task of technical committees is to prepare International Standards. Draft International Standards adopted by the technical committees are circulated to the member bodies for voting. Publication as an International Standard requires approval by at least 75 % of the member bodies casting a vote.

Attention is drawn to the possibility that some of the elements of this Amendment may be the subject of patent rights. ISO shall not be held responsible for identifying any or all such patent rights.

Amendment 4 to International Standard ISO/IEC 14496-1:2001 was prepared by Technical Committee ISO/IEC JTC 1, Subcommittee SC 29, .


ISO/IEC 14496-1:2001/PDAM 4

Introduction

This document is a working draft of the ISO/IEC 14496-1/AMD 4. This document specifies:1. The Animation Framework eXtension (AFX)

2. The Multi-User Worlds (MUW)

Organization of this document

This document is organized as follow:

Clause 1 specifies new BIFS field types

Clause 2 specifies the AFX framework,

Clause 3 specifies the MUW framework.


ISO/IEC 14496-1:2001/PDAM 4

Information technology — Coding of audio-visual objects — Part 1: Systems, AMENDMENT 4: Multi-users worlds and advanced animation effects

1 Amendment to ISO/IEC 14496-1:2001 (N4266)

1.1 SFVec4f and MFVec4f editings (16 modifs)

Adding the new SFVec4f/MFVec4f types requires 16 modifications to the current specification numbered MOD_1 to MOD_16 below:

MOD_1 Insert this section 9.3.7.29 SFVec3f and before 9.3.7.30 SFImage sections. Hence SFImage section becomes 9.3.7.31

9.3.7.30 SFVec4f9.3.7.30.1 Syntaxclass SFVec4f(FieldData field) {

if (field.isQuantized)QuantizedField qvalue(field);

else {GenericFloat value1(field.useEfficientCoding);GenericFloat value2(field.useEfficientCoding);GenericFloat value3(field.useEfficientCoding);GenericFloat value4(field.useEfficientCoding);

}}9.3.7.30.2 SemanticsAn SFVec4f field typically holds of a 4-Dimensional vector that consists of 4 values (x, y, z, w), where (x ,y ,z) are the 3D coordinates of the vector and w is a scaling factor to represent the infinity.

If the field’s isQuantized bit is TRUE, the QuantizedField scheme shall be used. Otherwise each component of the SFVec4f is coded using the GenericFloat scheme.

MOD_2 Replace table 19 with


ISO/IEC 14496-1:2001/PDAM 4

Category Description0 None1 3D position2 2D positions3 Drawing order4 SFColor5 Texture Coordinate6 Angle7 Scale8 Interpolator keys9 Normals10 Rotations11 Object Size 3D (1)12 Object Size 2D (2)13 Linear Scalar Quantization14 CoordIndex15 SFVec4f16 Reserved

MOD_3 Replace table 20 withquantType Condition1 lqp.position3DQuant == TRUE2 lqp.position2DQuant == TRUE3 lqp.drawOrderQuant == TRUE4 lqp.colorQuant == TRUE5 lqp.textureCoordinateQuant == TRUE6 lqp.angleQuant == TRUE7 lqp.scaleQuant == TRUE8 lqp.keyQuant == TRUE9 lqp.normalQuant == TRUE10 lqp.normalQuant == TRUE11 lqp.sizeQuant == TRUE12 lqp.sizeQuant == TRUE13 Always TRUE14 Always TRUE15 lqp. position3Dquant = TRUE

lqp.scaleQuant = TRUE16 Always TRUE

MOD_4 Replace table 21 with


quantType nbBits1 lqp.position3DNbBits2 lqp.position2DNbBits3 lqp.drawOrderNbBits4 lqp.colorNbBits5 lqp.textureCoordinateNbBits6 lqp.angleNbBits7 lqp.scaleNbBits8 lqp.keyNbBits9,10 lqp.normalNbBits11,12 lqp.sizeNbBits13 fct.defaultNbBits14 This value is set according to the number

of points received in the last received coord field of the node. Let N that number, then:

)1(logCeilnbBits 2 Nwhere the function Ceil returns the smallest integer greater than its argument

15 lqp.position3DnbBitslqp.scaleNbBits

16 0

MOD_5 Replace table 22 withquantType fieldType floatMin1 SFVec3fType lqp.position3Dmin2 SFVec2fType lqp.position2Dmin3 SFFloatType max(fct.floatMin[0],lqp.drawOrderMin)4 SFFloatType lqp.colorMin

SFColorType lqp.colorMin, lqp.colorMin, lqp.colorMin5 SFVec2fType lqp.textureCoordinateMin6 SFFloatType Max(fct.floatMin[0],lqp.angleMin)7 SFFloatType lqp.scaleMin

SFVec2fType lqp.scaleMin, lqp.scaleMinSFVec3fType lqp.scaleMin, lqp.scaleMin, lqp.scaleMin

8 SFFloatType Max(fct.floatMin[0],lqp.keyMin)9 SFVec3fType 0.010 SFRotationTyp

e0.0

11,12 SFFloatType lqp.sizeMinSFVec2fType lqp.sizeMin, lqp.sizeMinSFVec3fType lqp.sizeMin, lqp.sizeMin, lqp.sizeMin

13,14 NULL15 SFVec4fType lqp.position3Dmin

lqp.scaleMin16 NULL

3

MOD_6 Replace table 23 withquantType fieldType floatMax1 SFVec3fType lqp.position3Dmax2 SFVec2fType lqp.position2Dmax3 SFFloatType min(fct.floatMax[0],lqp.drawOrderMax)4 SFFloatType lqp.colorMax

SFColorType lqp.colorMax, lqp.colorMax, lqp.colorMax5 SFVec2fType lqp.textureCoordinateMax6 SFFloatType min(fct.floatMax[0],lqp.angleMax)7 SFFloatType lqp.scaleMax

SFVec2fType lqp.scaleMax, lqp.scaleMaxSFVec3fType lqp.scaleMax, lqp.scaleMax, lqp.scaleMax

8 SFFloatType min(fct.floatMax[0],lqp.keyMax)9 SFVec3fType 1.010 SFRotationType 1.011,12 SFFloatType Lqp.sizeMax

SFVec2fType Lqp.sizeMax, lqp.sizeMaxSFVec3fType Lqp.sizeMax, lqp.sizeMax, lqp.sizeMax

13,14 NULL15 SFVec4fType lqp.position3Dmax

lqp.scaleMax16 NULL

MOD_7 Insert at end of 9.3.3.1.1To quantize an SFVec4f field of value (x, y, z, w), the position3D and scale quantizers of the QuantizationParameter node are used at the same time. Specifically, position3DQuant and scaleQuant should be set to TRUE. position3DMin, position3DMax, and position3DNbBits are the quantization parameters used to quantize the 3D part of the vector (x, y, z) and scaleMin, scaleMax, and scaleNbBits are the quantization parameters used to quantize the scaling factor w.

As two quantizers are used, they also apply to fields using position3D and scale quantizers. As a result, if a node using SFVec4f quantization is followed by a node using SFVec3f and scale fields, these two types of fields will be quantized using the quantization parameters for the node using SFVec4f fields. Otherwise, two QuantizationParameter nodes should be used: one for the node using SFVec4f fields and one for the node using SFVec3f and scale fields.

BIFS-Anim uses quantization of SFVec4f fields the same way it is described above for BIFS-Commands.

SFVec4f fields are only used in solid modeling detailed later in this specification that needs 4D geometry. These fields are not used in previous versions of MPEG-4 (up to amendment 3). As a result, BIFS decoders not supporting amendment 4 and above do not need to implement SFVec4f decoding.

MOD_8 Append at end of 9.4.2.111.2


SFVec4f fields cannot be routed to Valuator node.

MOD_9 Replace 9.3.7.12.2 byIf intraMode is 1, the size of the interval between two intras is first specified. Independent of the intraMode, the number of Bits used in Predictive mode CompNbBits and the CompMins are coded. The function getNbComp() is a function that returns the number of components of the quantizing bounds, and depends on the object. For instance it returns 4 for 3D positions, 3 for 3D positions, 2 for 2D positions, and 3 for rotations. See Table 17. The values CompNbBits and CompMin are stored in the field.aqp AnimationQP structure and are used for the compensation process as defined in Table 31 and subclause 9.3.4.

MOD_10 Replace 9.3.7.13.2 byArrayQP fullfills the same purpose as InitialArrayQP, but in this case, the parameters are optionnaly set. If they are not set in the stream, they are set by default, in reference to the InitialArrayQP or the latest received value of the parameter.

If IntraMode is 1, the size of the interval between two intras is first specified. In any case, the number of Bits used in Predictive mode (CompNbBits) and the CompMins are coded. . The function getNbComp() is a function that returns the number of components of the quantizing bounds, and depends on the object. For instance it returns 4 for 4D positions, 3 for 3D positions, 2 for 2D positions, and 3 for rotations. See Table 17. The values CompNbBits and CompMin are stored in the field.aqp AnimationQP structure, and are used for the compensation process as defined in Table 31 and subclause 9.3.4.

Predictive encoding of 4D values is done per component, extending the scheme for 3D values. As for BIFS-Commands and BIFS-Anim, position3D and scale quantizers parameters are used. (PredictiveMFField spec)

MOD_11 Replace 9.3.7.17.1 byclass SFField(FieldData field, NodeData protoNodeData) {

switch (field.fieldType) {

case SFNodeType:SFNode nValue(field.fieldType, protoNodeData);break;

case SFBoolType:SFBool bValue;break;

case SFColorType:SFColor cValue(field);break;

case SFFloatType:SFFloat fValue(field);break;

case SFInt32Type:SFInt32 iValue(field);break;

case SFRotationType:SFRotation rValue(field);break;

case SFStringType:SFString sValue;break;

5

Mikael Bourges-Sevenier, 03/01/-1,

Should be the same cross reference as in the current text.











case SFTimeType:SFTime tValue;break;

case SFUrlType:SFUrl uValue;break;

case SFVec2fType:SFVec2f v2Value(field);break;



case SFImageType:SFImage imageValue(field);break;

case SFCommandBufferType:SFCommandBuffer commandValue(field);break;

case SFScriptType:SFScript scriptValue(protoNodeData);break;

}}

MOD_12 Append at end of 9.2.3.3 SFVec4f/MFVec4f

MOD_13 Replace 9.3.5.7.1 byInitialAnimQP(animFieldQP aqp) {

aqp.useDefault=FALSE;uint(4) type;aqp.animType = type;switch(aqp.animType) {

case 4: // Colorcase 8: // BoundFloats

bit(1) aqp.useDefaultcase 1: // Position 3Dcase 2: // Position 2Dcase 15: // Position 4Dcase 11: // Size 3Dcase 12: // Size 2Dcase 7: // Floats

if (!aqp.useDefault) {for (i=0;i<getNbBounds(aqp);i++) {

bit(1) useEfficientCodingGenericFloat aqp.Imin[i](useEfficientCoding);


}for (i=0;i<getNbBounds(aqp);i++) {

bit(1) useEfficientCodingGenericFloat aqp.Imax[i](useEfficientCoding);

}}break;

case 13: // Integersint(32) aqp.IminInt[0];

break;}unsigned int(5) aqp.INbBits;

for (i=0;i<getNbBounds(aqp);i++) {int(INbBits+1) vqaqp.Pmin[i] = vq-2^aqp.INbBits;

}

unsigned int(4) aqp.PNbBits;

}

MOD_14 Replace 9.3.8.8.1 byclass AnimQP(AnimFieldQP aqp) {

bit (1) IMinMax ;if (IMinMax) {

aqp.useDefault=FALSE;switch(aqp.animType) {

case 4: // Colorcase 8: // BoundFloats

bit(1) aqp.useDefaultcase 1: // Position 3Dcase 2: // Position 2Dcase 15: // Position 4Dcase 11: // Size 3Dcase 12: // Size 2Dcase 7: // Floats

if (!aqp.useDefault) {for (i=0;i<getNbBounds(aqp);i++) {

bit(1) useEfficientCodingGenericFloat aqp.Imin[i](useEfficientCoding);

}for (i=0;i<getNbBounds(aqp);i++)

bit(1) useEfficientCodingGenericFloat aqp.Imax[i](useEfficientCoding);

}break;

case 13: // Integersint(32) aqp.IminInt[0];

break;

}

bit (1) hasINbBits;if (hasINbBits)

unsigned int(5) aqp.INbBits;

7

bit (1) PMinMax ;if (PMinMax) {

for (i=0;i<getNbBounds(aqp);i++) {int(INbBits+1) vqaqp.Pmin[i] = vq-2^aqp.INbBits;

}}

bit (1)hasPNbBits;if (hasPNbBits)

unsigned int(4) aqp.PNbBits;

}

MOD_15 Append at end of table 33 (in 9.3.7.36.2)0bx101011 SFVec4f

0bx101100 MFVec4f

MOD_16 MPEG-J updatesV.5.373 org.iso.mpeg.mpegj.scene.SFVec4fFieldValue

V.5.373.1 Syntax

public interface SFVec4fFieldValue extends org.iso.mpeg.mpegj.scene.FieldValue

All Superinterfaces: org.iso.mpeg.mpegj.scene.FieldValue

V.5.373.2 Description

An interface used for obtaining SFVec4f values.

Member SummaryMethods

float[] getSFVec4fValue()

V.5.373.3 Methods

GetSFVec4fValue()

V.5.374 public float[] getSFVec4fValue()

Obtain the SFVec4f value.

Returns: a float array containing the value of the SFVec4f field.


1.2 4D interpolators

1.2.1 CoordinateInterpolator4D

CoordinateInterpolator4D {eventIn SFFloat set_fraction

exposedField MFFloat key [] exposedField MFVec4f keyValue [] eventOut MFVec4f value_changed}

As CoordinateInterpolator, this node linearly interpolates 4-dimensional values.

1.2.2 PositionInterpolator4D

PositionInterpolator4D {eventIn SFFloat set_fraction

exposedField MFFloat key [] exposedField MFVec4f keyValue [] eventOut MFVec4f value_changed}

As PositionInterpolator, this node linearly interpolates 4-dimensional values.

1.3 Extended BIFS-Updates

Replace in section 9.3.6.4.1:class InsertionCommand() { bit(2) parameterType ; switch (parameterType) { case 0: NodeInsertion nodeInsert(); break; case 1: ExtendedUpdate extendedUpdate(); break; case 2: IndexedValueInsertion idxInsert(); break; case 3: ROUTEInsertion ROUTEInsert(); break ; }}

Insert in section 9.3.6.4.2:If parameterType is 1, an ExtendedUpdate is expected.

Add section 9.3.6.18 and following

9

1.3.1 ExtendedUpdate

1.3.1.1 Syntax

class ExtendedUpdate() { bit(8) updateType ; switch (updateType) { case 0: PROTOlist protos(); break; case 1: PROTOlistDeletion listDelete(); break; case 2: // remove all existing PROTOs break ; case 3: MultipleIndexedFieldReplacement mifReplacement(); break; case 4: MultipleFieldReplacement mfReplacement(); break; case 5: GlobalQuantizationConfiguration globalQuantizer(); break; }}

1.3.1.2 Semantics

There can be up to 256 extended update types.

If updateType is 0, the insertion of a list of PROTOs is signalled.

If updateType is 1, the deletion of a list of PROTOs is signalled.

If updateType is 2, the deletion of all previously defined PROTOs is signalled.

If updateType is 3, a MultipleIndexedFieldReplacement is signalled.

If updateType is 4, a MultipleFieldReplacement is signalled.

If updateType is 5, a GlobalQuantizationConfiguration is signalled.

1.3.2 PROTOlistInsertion

1.3.2.1 Syntax

class PROTOlistInsertion () { PROTOList list();}

PROTOList is defined in 9.3.7.2.1.

1.3.2.2 Semantics

This element signals the insertion of a list of PROTOs.


Inserting a new PROTO with id=PID means that, from that point in time on, there can be a PROTO instance referring to a PROTO declaration with id=PID.

When an existing PROTO already has the same id as a newly inserted PROTO, it means that new PROTO instances refer to the new definition, while previous PROTO instances are unchanged.

1.3.3 PROTOlistDeletion

1.3.3.1 Syntax

class PROTOlistDeletion () { bit(1) listDescription; if (listDescription) { bit(1) morePROTOs; while (morePROTOs) { bit(BIFSConfiguration.protoIDbits) protoID; bit(1) morePROTOs; } } else { // vector bit(5) len; bit(len) numPROTOs; for (i=0; i < numPROTOs; i++) bit(BIFSConfiguration.protoIDbits) protoID; }}

1.3.3.2 Semantics

This command can delete a selection of the existing PROTO declarations. This only deletes the definition, i.e. existing instances are not influenced by the deletion, but no new instance of the deleted PROTOs can be created.

1.3.4 MultipleFieldReplacement

1.3.4.1 Syntax

class MultipleFieldReplacement () { bit(BIFSConfiguration.nodeIDbits) nodeID; nodeData = getNodeFromID(nodeID); bit(1) maskAccess; if (maskAccess) MaskNodeDescription mnode(nodeData); else ListNodeDescription lnode(nodeData);}

1.3.4.2 Semantics

This element allows the modification of some fields of a node. This element is meant to replace a set of FieldReplacement commands operating on the same node. The fact that the nodeID is not repeated allows for better compression.

1.3.5 MultipleIndexedFieldReplacement

1.3.5.1 Syntax

class MultipleIndexedFieldReplacement () {bit(BIFSConfiguration.nodeIDbits) nodeID;

NodeData node = GetNodeFromID(nodeID);int(node.nINbits) inID;bit (5) lenPosition;bit (5) lenNum;

11

bit (lenNum) numPositions; for (i=0; i < numPositions; i++) {

bit(lenPosition) position;SFField value(node.field[node.in2all[inID]]);

}}

1.3.5.2 Semantics

This element allows the modification of some values of a multiple field. This element is meant to replace a set of IndexedValueReplacement commands operating on the same field. The fact that the nodeID and inID are not repeated allows for better compression.

1.3.6 GlobalQuantizationConfiguration

1.3.6.1 Syntax

class GlobalQuantizationConfiguration () {SFNode gqp();

}

1.3.6.2 Semantics

This element allows the setting of the global quantizer defined in 9.4.2.89.2. The use of the SFNode structure allows to give this global quantizer a DEF ID, or to reset global quantization by setting it to a NULL node, or to use a QuantizationParameter node already present in the scene.

In the context of this command, a node of any other type than QuantizationParameter shall be treated as a NULL node, resetting global quantization. The isLocal field of QuantizationParameter is meaningless in this context.

Replace in section 9.4.2.89.2:“In all cases, the quantization is applied only in the scope of a single BIFS command. That is, if a command in the same access unit, or in another access unit inserts a node in a context in which the quantization was active, no quantization will be applied, except if a new QuantizationParameter node is defined in this new command.”

to“For each scene, by default, there is no quantization. A global quantizer can be defined or modified by using a GlobalQuantizationConfiguration command, see subclause 9.3.6.23. The global quantizer applies to all subsequent BIFS access units where no QuantizationParameter node apply. The global quantizer may be DEFed and then USEd within the scene. The global quantizer may be set to a USE of an existing QuantizationParameter present in the scene.

Note: If DEFed, the global quantizer may be ROUTEd and/or modified by Script, MPEG-J and FieldReplacement commands.

Unless the QuantizationParameter node is present in a GlobalQuantizationConfiguration command, the quantization is applied only in the scope of a single BIFS command. That is, if a command in the same access unit, or in another access unit inserts a node in a context in which the quantization was active, no quantization will be applied, except if a new QuantizationParameter node is defined in this new command.

1.4 FlexMux tool

1.4.1 The MPEG-4 FlexMux related descriptors (modified 8.2.2.2)


Replace the existing table1 List of Class Tags for Descriptors, by the following table 1 :

Table 1 - List of Class Tags for Descriptors

Tag value Tag name0x00 Forbidden0x01 ObjectDescrTag0x02 InitialObjectDescrTag0x03 ES_DescrTag0x04 DecoderConfigDescrTag0x05 DecSpecificInfoTag0x06 SLConfigDescrTag0x07 ContentIdentDescrTag0x08 SupplContentIdentDescrTag0x09 IPI_DescrPointerTag0x0A IPMP_DescrPointerTag0x0B IPMP_DescrTag0x0C QoS_DescrTag0x0D RegistrationDescrTag0x0E ES_ID_IncTag0x0F ES_ID_RefTag0x10 MP4_IOD_Tag0x11 MP4_OD_Tag0x12 IPL_DescrPointerRefTag0x13 ExtensionProfileLevelDescrTag0x14 profileLevelIndicationIndexDescrTag0x15-0x3F Reserved for ISO use0x40 ContentClassificationDescrTag0x41 KeyWordDescrTag0x42 RatingDescrTag0x43 LanguageDescrTag0x44 ShortTextualDescrTag0x45 ExpandedTextualDescrTag0x46 ContentCreatorNameDescrTag0x47 ContentCreationDateDescrTag0x48 OCICreatorNameDescrTag0x49 OCICreationDateDescrTag0x4A SmpteCameraPositionDescrTag0x4B-0x5F Reserved for ISO use (OCI extensions)0x60 FLEXmuxChannelDescrTag0x61 FLEXmuxBufferSizeDescrTag0x62 FLEXmuxTimingDescrTag0x63 FLEXmuxCodeTableDescrTag0x64 FLEXmuxIdentDescrTag0x65 SegmentDescrTag

13

0x66 MediaTimeDescrTag0x67-0xBF Reserved for ISO use0xC0-0xFE User private0xFF Forbidden

note to the editor: There is an editorial error with the ‘ ExtendedProfileLevelDescrTag ’, as in the related paragraph it is called extension and not extended.

FLEXmux related Descriptor definitions: (modifies 12.2.9)

note to the editor : according to the modifications to the table of tags for descriptors, the following descriptors can build a 12.3 paragraph that can be added after the FlexMux buffer model paragraph (12.2.9), and before the File Format paragraph(12.3)

12.3 FLEXmux Descriptors:Directly derived from the FlexMux descriptor classes, hereafter are defined the FLEXMux descriptors pointed to by the “List of Class Tags for Descriptors” table.

12.3.1 FLEXmuxChannel Descriptor12.3.1.1 Syntaxclass FLEXmuxChannelDescriptor extends BaseDescriptor

: bit(8) tag= FLEXmuxChannelDescrTag {for (i=0; i<( sizeOfInstance-1); i += 3) {bit(16) ES_ID;bit(8) FlexMuxChannel;}

}

12.3.1.2 SemanticsES_ID – this 16-bit field specifies the identifier of an ISO/IEC 14496-1 SL-packetized stream.

FlexMuxChannel - This 8-bit field specifies the number of the FlexMux channel used for this SL-packetized stream.

12.3.2 FLEXmuxBufferSize Descriptor12.3.2.1 Syntaxclass FLEXmuxBufferSizeDescriptor extends BaseDescriptor

: bit(8) tag= FLEXmuxBufferSizeDescrTag {DefaultFlexMuxBufferDescriptor()for (i=0; i<( sizeOfInstance-3); i += 4) {

FlexMuxBufferDescriptor()}

}


12.3.2.2 SemanticsDefaultFlexMuxBufferDescriptor - The default size of multiplex buffers for each individual channel in a FlexMux stream is signalled by the DefaultFlexMuxBufferDescriptor class(see clause 11.2).

FlexMuxBufferDescriptor. The exact size of multiplex buffers for each channel in a FlexMux stream can be signalled by the FlexMuxBufferDescriptor class (see clause 11.2)

12.3.3 FLEXmuxTiming Descriptor12.3.3.1 Syntaxclass FLEXmuxTimingDescriptor extends BaseDescriptor

: bit(8) tag= FLEXmuxTimingDescrTag {FlexMuxTimingDescriptor()

}

12.3.3.2 Semantics

FlexMuxTimingDescriptor – This descriptor class defines FCR_ES_ID, FCRResolution, FCRLength, FmxRateLength , (see clause 11.2)

12.3.4 FLEXmuxCodeTable Descriptor12.3.4.1 Syntaxclass FLEXmuxCodeTableDescriptor extends BaseDescriptor

: bit(8) tag= FLEXmuxCodeTableDescrTag {

for(i =0; i < sizeOfInstance; i += sizeof ( MuxCodeTableEntry () ) )

{

MuxCodeTableEntry ()

}

}12.3.4.2 SemanticsMuxCodeTableEntry () – This class defines the FlexMux configuration of one FlexMux channel (see clause 11.2)

Several FLEXmuxCodeTableDescriptor may be used with different instances of the MuxCodeTableEntry class.

11.3.5 FLEXmuxIdent Descriptor12.3.5.1 Syntaxclass FLEXmuxIdentDescriptor extends BaseDescriptor

: bit(8) tag= FLEXmuxIdentDescrTag {FlexMuxIDDescriptor ()

}

12.3.5.2 Semantics

15

FlexMuxIDDescriptor – This class defines MuxID, Muxtype , Muxmanagement (see clause 11.2)1.4.2 The Second FlexMux tool definition (Modified 12.2.4)

12.2.4FlexMux packet specification

12.2.4.1 Syntax

class FlexMuxPacket (MuxCodeTableEntry mct[], FlexMuxTimingDescriptor FM,FlexMuxIDDescriptor mde) {

unsigned int(8) index; if (mde == NULL | mde.Muxtype == 0) { bit(8) length; } else if (mde.Muxtype == 1) { length = 0; bit(1) nextByte; bit(7) length; while(nextByte) { bit(1) nextByte; bit(7) sizeByte; length = length<<7 | sizeByte; } } if (index<238) {

SL_Packet sPayload;} else if (index == 238) {

bit(FM.FCR_Length) fmxClockReference;bit(FM.fmxRateLength) fmxRate;

} else if (index == 239) {bit(8) stuffing[length];

} else {bit(4) version;const bit(4) reserved=0b1111;multiple_SL_Packet mPayload(mct[index-240]);

}}

12.2.4.2 Semantics

MuxCodeTableEntry, FlexMuxTimingDescriptor, FlexMuxIDDescriptor – classes of descriptors (see clauses 12.2.5 to 12.2.10)

Muxtype - the type of the Multiplexing tool used to generate the FlexMux stream.

The values 0 and 1 are defined in the table 2-39:“Multiplexing type table”.

length – the length of the FlexMux packet payload in bytes. This is equal to the length of the single encapsulated SL packet in Simple Mode and to the total length of the multiple encapsulated SL packets in MuxCode Mode. If the FlexMuxIDDescriptor is not used, or if it is used and if the Muxtype is designing the first FlexMux tool, the length field is on one byte. If the FlexMuxIDDescriptor is used and if the Muxtype is designing the second FlexMux tool, the length calculation relies on the combination of the nextByte and sizeByte fields that can be spread over several bytes.

nextByte– The upper bit of one (to four) byte(s) where the seven lower bits allow the calculation of the FlexMux packet length. When set to 1 it indicates that the byte, to which this bit belongs to, is followed by another byte. When set to zero it indicates


that there is no byte following to allow the calculation of the FlexMux packet.

SizeByte- The seven lower bits (following the nextByte bit) that allow the calculation of the FlexMux packet length.

12.2.10 FlexMuxID Descriptor

12.2.10.1 Syntax

aligned(8) class FlexMuxIDDescriptor {bit(8) MuxID;bit(4) Muxtype;

bit(4) Muxmanagement;}

12.2.10.2 Semantics

MuxID – the ID of the FlexMux stream.

Muxtype – the type of the Multiplexing tool used to generate the FlexMux stream.

Indicated type value shall comply with the following Table 1 - Multiplexing type table.

Muxmanagement – the mode of management used by the Multiplexing tool, to

generate the Flexmux stream. Indicated mode value shall comply with the Table 2 Multiplexing management modetable.

Table 1 - Multiplexing type tabletype Multiplexing tool0 FlexMux tool 1 FlexMux_2 tool2-7 ISO/IEC 14496-1 Reserved8-15 User Private

Table 2 Multiplexing management mode table

type management mode0 Static1 Dynamic2-7 ISO/IEC 14496-1 Reserved8-15 User Private

17

1.4.3 FlexMux packet specification (in-band signalling)

1.4.3.1 11.2.4.1 Syntax

class FlexMuxPacket (MuxCodeTableEntry mct[], FlexMuxTimingDescriptor FM,FlexMuxIDDescriptor mde) {

unsigned int(8) index;if (mde == NULL | mde.Muxtype == 0) {

bit(8) length;} else if (mde.Muxtype == 1) {

length = 0;bit(1) nextByte;bit(7) length;while(nextByte) {

bit(1) nextByte;bit(7) sizeByte;length = length<<7 | sizeByte;

} } if (index<238) {

SL_Packet sPayload;} else if (index == 238) {

bit(FM.FCR_Length) fmxClockReference;bit(FM.fmxRateLength) fmxRate;

for (i=0; i<(length-FM. FCR_Length-FM.fmxRateLength) ; i++) { FlexMux_descriptor() }

} else if (index == 239) {bit(8) stuffing[length];

} else {bit(4) version;const bit(4) reserved=0b1111;multiple_SL_Packet mPayload(mct[index-240]);

}}

11.2.4.2 Semantics

“FlexMux descriptor”-one of the FlexMux desciptors.

Note: They are defined in the M4092 document, “Technology under Consideration for Inclusion into ISO/IEC 14496-1:2001 AMD 2”.

2 Animation Framework eXtension (AFX)

2.1 Introduction

“Most people think the word ‘animation’ means movement.

But it doesn’t. It comes from ‘animus’ which means ‘life or to live’.

Making it move is not animation, but just the mechanics of it.”

Frank Thomas and Ollie Johnston


The Animation Framework extension (AFX – pronounced ‘effects’) provides enhanced visual experiences in synthetic MPEG-4 environments. The framework defines a collection of interoperable tool categories that collaborate to produce a reusable architecture for interactive animated contents. In the context of AFX, a tool represents functionality such as a BIFS node [1], a synthetic stream [2], or an audio-visual stream [2,3].

AFX utilizes and enhances existing MPEG-4 tools without sacrificing backward compatibility by offering:

Higher-level descriptions of animations (e.g. inverse kinematics)

Enhanced rendering (e.g. multi-texturing, procedural texturing)

Compact representations (e.g. piecewise curve interpolators, subdivision surfaces)

Low bitrate animations (e.g. using interpolator compression and dead-reckoning)

Scalability based on terminal capabilities (e.g. parametric surfaces tessellation)

Interactivity at user level, scene level, and client-server session level

Compression of representations for static and dynamic tools

Compression of animated paths and animated models is required for improving the transmission and storage efficiency of representations for dynamic and static tools.

This document describes new techniques to speed up rendering of large scenes as well as to improve their visual quality. These methods were designed with multi-user environments in mind. Special considerations for network communication for such environments need to be considered when using DMIF as defined by Systems group.

Figure 1 shows the position of the Animation Framework in MPEG-4 Terminal architecture. It extends the existing BIFS tree and may use a dedicated decoder for animation streams and compressed object representations. For efficiency, it may access terminal capabilities to adapt shape and texture rendering so to maintain a reasonable user experience.

AFX defines an extensibility model for the scene nodes and propose to think about nodes in terms of software components. The interface of these components is specified in the bitstream but the implementations of the components may be carried in another stream and it can be in any language. For security reasons, this module is directly connected to IPMP system.

19

Animation FrameworkeXtension

Audio DBAudio

Decode

IPMP DB

Video DBVideo

Decode Video CB

Com

posite

Elementary Stream Interface

BIFS DB

Audio CB

IPMP System(s)

OD DB ODDecode

BIFSDecode

IPMP-ES

DecodedBIFS BIFS Tree

IPMP-Ds

DMIF

DM

UX

Possible IPMPControl Points

Render

AFX Decode

DecodedAFXAFX DB

Figure 1 - Animation Framework and MPEG-4 Systems.

2.2 Conceptual organization of tools

Modeling is a way of describing the real world, but non-realistic objects can be described using the same toolset. Building blocks of a lower hierarchical category, together with a way of combining them, are used to define the higher hierarchical categories. This is expressed in the AFX by having 6 categories of tools relying on each other that fulfill the different requirements:

Geometric models build the appearance of the objects

Modeling tools provide an easier control of the large amount of data

Physical models add meaningful engineering ideas to the animation

Biomechanical models do semantic grouping and coordination of lower tools.

Behavior models add the communicative movements,

Cognitive models add learning capabilities.


Cognitive

Behavior

Biomechanical

Physics

Modeling

Geometry

AnimationAnimation

TerminalTerminal

Animation Framework eXtension – AFX ‘effects’

Figure 2 - Models in computer games and animation [44].1. Geometric models. They capture the form and appearance of an object. Many characters in animations and

games can be quite efficiently controlled at this low-level; familiar tools for generating motion include key framing, and motion capture. Due to the predictable nature of motion, building higher-level models for characters that are controlled at the geometric level is generally much simpler.

2. Modeling models. They are an extension of geometric models and add linear and non-linear deformations to them. They capture transformation of the models without changing its original shape. Animations can be made on changing the deformation parameters independently of the geometric models.

3. Physical models. They capture additional aspects of the world such as an object’s mass inertia, and how it responds to forces such as gravity. The use of physical models allows many motions to be created automatically and with unparallel realism. The cost of simulating the equations of motion may be important in a real-time engine and in many games, a physically plausible approach is often preferred. Applications such as collision restitution, deformable bodies, and rigid articulated bodies use these models intensively.

4. Biomechanical models. Real animals have muscles that they use to exert forces and torques on their own bodies. If we already have built physical models of characters, they can use virtual muscles to move themselves around. Funge [44] refers the character’s ability to exert some control over their motions as actuated (or animate) objects. These models have their roots in control theory.

5. Behavioral models. After simple locomotion comes character’s behavior. A character may expose a reactive behavior when its behavior is solely based on its perception of the current situation (i.e. no memory of previous situations). Reactive behaviors can be implemented using stimulus-response rules, which are very used in

21

games. Finite-States Machines (FSM) are often used to encode deterministic behaviors based on multiple states. Goal-directed behaviors can be used to define a cognitive character’s goals. They can also be used to model flocking behaviors.

6. Cognitive models. If the character is able to learn from stimuli from the world, it may be able to adapt its behavior. These models are related to artificial intelligence techniques.

The models rely on their predecessors. For example, an autonomous agent (category 5) may respond to stimuli from the environment he is in and may decide to adapt his way of walking (category 4) that can modify physics equation (for example skin modeled with mass-spring-damp properties) or have influence on some underlying deformable models (category 2) or may even modify the geometry (category 1). If the agent is clever enough, it may also learn from the stimuli (category 6) and adapt or modify his behavioral models.

In the remaining of this document, this conceptual organization of tools is followed in the same spirit an author will create content: the geometry is first defined with or without solid modeling tools, then texture is added to it. Objects can be deformed and animated using modeling and animation tools. Finally, avatars need biomechanical tools.

2.3 Geometry tools

In this section, some parametric surfaces are described: NURBS, subdivision surfaces, wavelet subdivision surfaces, MeshGrid representation, and particle systems.. These surfaces share the same concepts:

they are made of a control polygon,

they are defined by mathematical equations

there exists very fast algorithms (even in hardware)

to be rendered, they often need to be sampled. This sampling could be relative to the position of the camera in the scene: less polygons are generated if far away, more if close. As a result, one obtains dynamic level of details (efficient algorithms can also refine locally). This is a desirable property for fast adaptive rendering.

2.3.1 NURBS geometry

A Non-Uniform Rational B-Spline (NURBS) curve has for equation [75, 99]:

Eq. 1 – NURBS curve definition.


where are the rational basis functions, the control points, are the weights, and are the pth-degree B-spline basis functions defined on the non-periodic and non-uniform knot vector

.

Let’s define:

be the number of knots,

the number of control points

the degree of the curve (order is ), then

The B-spline basis functions are defined recursively using the Cox de Boor formula:

From these definitions, it is easy to see that a curve with no interior knot, i.e. , is a Bézier curve

of degree .

NURBS curves can represent all types of curves and, for example, from classical mathematics it is known that all the conic curves can be represented using rational functions (including circle, parabola, hyperbola, and so on).

Finally, let’s present some interesting properties:

Strong convex hull property: the curve is contained in the convex hull of their control points

Affine invariance: rotations, translations, scalings, and shears are applied to the curve by applying them to control points

Endpoint interpolation: and

The control polygon represents a piecewise approximation to the curve. As a general rule, the lower the degree, the closer a NURBS curve follows its control polygon.

is infinitely differentiable on the interior of knot spans and is differentiable at a knot of multiplicity .

23

Local approximation: if the control point is moved or the weight is changed, it affects only that portion of

the curve on the interval .

NURBS surfaces are defined by a bivariate tensor product of two NURBS curves:

Eq. 2 – NURBS surface definition.The B-spline basis functions are defined as previously and can have different degrees in u and v directions.. The resulting surface has the same interesting properties as for NURBS curves.

The key benefits from the use of NURBS geometry [75] primitives include:

Complex smooth curves and surfaces can be described.

Representation uses a small set of control points.

NURBS are able to represent conic shapes exactly.

Representation results in a huge compression advantage compared to equivalent highly detailed, piecewise linear sets of lines or triangles.

Changing only a few control points can smoothly animate shapes.

Computer modeling programs are often based on NURBS.

NURBS are scalable to client 3D graphics rendering speed, screen resolution and terminal CPU performance.

2.3.1.1 NurbsSurface

2.3.1.1.1 Node specification

The NurbsSurface Node describes a 3D NURB Surface patch. Analogous 3D Surface nodes are the ElevationGrid and IndexedFaceSet node. NurbsSurface extends the definition of IndexedFaceSet. Thus, the control polygon as well as normals, texture coordinate, color per vertex, and so on, has the same semantic as for IndexedFaceSet. The control polygon is specified in coord and coordIndex of IndexedFaceSet node.

If an implementation does not support NurbsSurface, it is still able to display the control polygon as a set of quads, which is the coarsest approximation of the NURBS surface.


NurbsSurface { #%NDT=SFGeometryNodeexposedField SFColorNode color NULLexposedField SFTextureCoordinateNode texCoord NULLfield SFBool ccw TRUEexposedField MFVec4f controlPoint []field SFBool solid TRUE

field SFInt32 uDimension 0 # [0, ) field SFInt32 vDimension 0 # [0, )field MFFloat uKnot [] # (-,) field MFFloat vKnot [] # (-,)field SFInt32 uOrder 3 # [2, 32) field SFInt32 vOrder 3 # [2, 32)exposedField SFInt32 uTessellation 0 # (-,) exposedField SFInt32 vTessellation 0 # (-,)

}

uDimension and vDimension define the number of control points in the u and v dimensions.

uOrder and vOrder define the order of surface. From a mathematical point of view, the surface is defined by polynomials of the degree order-1. The order of the curves uOrder and vOrder must be greater than or equal to 2. An implementation may limit uOrder and vOrder to a certain number. The most common orders are 3 (quadratic polynomial) and 4 (cubic polynomial), which are sufficient to achieve the desired curvature in most cases. The number of control points must be at least equal to the order of the curve. The order defines the number of adjacent control points that influence a given control point.

controlPoint define a set of control points of dimension uDimension * vDimension. This set of points defines a mesh similar to the grid of an ElevationGrid, whereas the points do not have a uniform spacing. Depending on the order, this hull is approximated by the resulting surface. The number of uDimension points defines a polyline in u-direction followed by further u-polylines with the v-parameter in ascending order. The number of control points must be equal to or greater than the order. A closed B-Spline surface can be specified by repeating the limiting control points and by specifying a periodic knot vector.

The control vertex corresponding to the control point P[i, j] on the control grid is:

P[i,j].x = controlPoints[i + ( j × uDimension)].x

P[i,j].y = controlPoints[i + ( j × uDimension)].y

P[i,j].z = controlPoints[i + ( j × uDimension)].z

P[i,j].w = = controlPoints[i + ( j × uDimension)].w

where uDimensioni 0 and uDimensionj 0 .

25

A weight value that must be greater than zero is assigned to each controlPoint. The ordering of the values is equivalent to the ordering of the control point values. If the weight of a control point is increased above 1 the point is more closely approximated by the surface. However the surface is not changed if all weights are multiplied by a common factor. The number of values must be identical to the number of control points. If the length of the weight vector is 0, the default weight 1.0 is assumed for each control point.

As a result of the lack of a 4D Coordinate field type in VRML, the control points and the corresponding weight values are held in separate fields. This separation also allows independent animation of the controlPoint fields using a CoordinateInterpolator node.

uKnots and vKnots define the knot vector. The number of knots must be equal to the number of control points plus the order of the curve. The order must be non-decreasing. By setting successive knot values equal, the degree of continuity is decreased, which implies that the surface gets edges. In general, the curve or surface is of continuity C k-1-m

at a knot point, where k is the order and m is the number of consecutive knots being equal. If k is the order of the curve, k consecutive knots at the end or the beginning of the vector cause the curve to interpolate the last or the first control point respectively. Within the knot vector there may not be more than k-1 consecutive knots of equal value. If the length of a knot vector is 0, a default uniform knot vector is computed.

uTessellation and vTessellation give hints to the surface tessellator, u/v Tessellation > = u/v Order sets an absolute number of subdivision steps, 0 lets the browser choose a suitable tessellation. Interpretation of values below 0 is implementation-dependent.

For an implementation subdividing the surface into an equal number of subdivision steps, tessellation values could be interpreted in the following way: if a tessellation value is greater than 0, the number of tessellation points is tessellation+1; if a tessellation value is smaller than 0, the number of tessellation points is (-tessellation * (u/v)dimension)+1 if a tessellation value is 0, the number of tessellation points is (2 * (u/v)dimension)+1

For implementations doing tessellations based on chord length, tessellation values <0 could be interpreted as the maximum chord length deviation in pixels. Implementations doing fully automatic tessellation may ingore the tessellation hint parameters.

A point on a NURBS surface is defined by:

jiumi

vmj vkjuki

jiumi

vmj jivkjuki

wvBuB

VwvBuBvuQ

,0 0 ,,

,0 0 ,,,

)()(

)()(),(

where:

u,v parameters of the surface


B basis functions

k orders in u and v direction (uOrder vOrder)

V mesh of control points (controlPoint[] .x controlPoint[] .y controlPoint[] .z)

w controlPoint[] .w

The basis functions are defined as follows:

},...,{

)()()(

01

)(

0

1,11

1,1

,

11,

um

riiri

riri

iri

iri

iii

uuU

uBuuuu

uBuu

uuuB

otherwiseuuu

uB

U is the knot vector containing a nondecreasing sequence of real numbers (uKnot, vKnot)

Stepping through the u and v domains and evaluating the equation for points on the surface can produce a grid of sample points. Stepping through the u domain at two fixed v values can generate triangle strips.

2.3.1.2 NurbsCurve


NurbsCurve { #%NDT=SFGeometryNodeexposedField SFColorNode color NULLexposedField MFVec4f controlPoint []field MFFloat knot [] # (-,)field SFInt32 order 3 # [2, 32)exposedField SFInt32 tessellation 0 # (-,)

}

The NurbsCurve node is defined analogous to the NurbsSurface node. The dimension field is given implicit through the length of the controlPoint array. The NurbsCurve is displayed as a curved Line similar to the IndexedLineSet primitive.

2.3.1.3 NurbsCurve2D


The NurbsCurve2D is the 2D version of NurbsCurve.

27

NurbsCurve2D {exposedField SFColorNode color NULLexposedField MFVec3f controlPoint []field MFFloat knot [] # (-,)field SFInt32 order 3 # [2, 32)exposedField SFInt32 tessellation 0 # (-,)

}2.3.2 Subdivision Surfaces

2.3.2.1 Overview

Subdivision surfaces, originally introduced by Catmull and Clark [24] and Doo and Sabin [39] in 1978, have recently emerged as a useful tool for modeling free-form surfaces. A number of other subdivision schemes have been devised over the years, including Loop’s [60], Dyn et al’s (known as the “butterfly” scheme) [41] or Kobbelt’s [50]. Also, classical algorithms have been improved by recent refinements, such as Zorin’s modification of the butterfly scheme [102], DeRose et al’s modification of Catmull-Clark’s scheme [37], and Superscape’s extended Loop scheme, which is described below.

Subdivision is a recursive refinement process that splits the facets or vertices of a polygonal mesh (the initial “control hull”) to yield a smooth limit surface. The refined mesh obtained after each subdivision step is used as the control hull for the next step, so all successive (and hierarchically nested) meshes can be regarded as control hulls. The refinement of a mesh is performed both on its topology, as the vertex connectivity is made richer and richer, and on its geometry, as the new vertices are positioned in such a way that the angles formed by the new facets are smaller than those formed by the old facets.

Figure 3: Nested control meshes.Figure 3 shows an example of how subdivision would be applied to a triangular mesh. At each step, one triangle is shaded to highlight the correspondence between one level and the next: each triangle is broken down into four new triangles, whose vertex positions are perturbed to have a smoother and smoother surface.

2.3.2.2 Piecewise Smooth Surfaces

Subdivision schemes for piecewise smooth surfaces include common modeling primitives such as quadrilateral free-form patches with creases and corners.


A somewhat informal description of piecewise smooth surfaces is given here. For simplicity, let’s consider only surfaces without self intersection. For a closed C1-continuous surface in 3 , each point has a neighborhood that can be smoothly deformed into an open planar disk D. A surface with a smooth boundary can be described in a similar way, but neighborhoods of boundary points can be smoothly deformed into a half-disk H, with closed boundary (Figure4). In order to allow piecewise smooth boundaries, two additional types of local charts are introduced: concave and convex corner charts, Q3 and Q1.

Figure 4 - The charts for a surface with piecewise smooth boundary.Piecewise-smooth surfaces are constructed out of surfaces with piecewise smooth boundaries joined together. If two surface patches have a common boundary, but different normal directions along the boundary, the resulting surface has a sharp crease.

Two adjacent smooth segments of a boundary are allowed to be joined, producing a crease ending in a dart (cf. [50]). For dart vertices an additional chart Q0 is required; the surface near a dart can be deformed into this chart smoothly everywhere except at an open edge starting at the center of the disk.

It is important to observe that convex and concave corners, while being equivalent topologically, are not differentially equivalent. That is, there is no C1 non-degenerate map from Q1 to Q3. Therefore, a single subdivision rule can not produce both types of corners [106]. In general, any complete set of subdivision rules should contain separate rules for all chart types.

2.3.2.3 Tagged meshes

Before describing the set of subdivision rules, the description of the tagged meshes is described, which the algorithms accept as input. These meshes represent piecewise-smooth surfaces with features described above.

The complete list of tags is as follows. Edges can be tagged as crease edges. A vertex with incident crease edges receives one of the following tags:

Crease vertex: joins exactly two incident crease edges smoothly.

Corner vertex: connects two or more creases in a corner (convex or concave).

Dart vertex: causes the crease to blend smoothly into the surface; this is the only allowed type of crease termination at an interior non-corner vertex; due to unresolved practical and theoretical issues, crease termination at vertices without smooth blending is not included.

29

Figure 5 - Features: (a) concave corner, (b) convex corner, (c) smooth boundary/crease, (d) corner with two convex sectors.Boundary edges are automatically assigned crease tags; boundary vertices which do not have a corner tag are assumed to be smooth crease vertices.

Crease edges divide the mesh into separate patches, several of which can meet in a corner vertex. At a corner vertex, the creases meeting at that vertex separate the ring of triangles around the vertex into sectors. Each sector of the mesh is labeled as convex sector or concave sector indicating how the surface should approach the corner.

In all formulas k denotes the number of faces incident at a vertex in a given sector. Note that this is different from the standard definition of the vertex degree: faces, rather than edges are counted. Both quantities coincide for interior vertices with no incident crease edges. This number is referred as crease degree.

The only restriction that on sector tags is that concave sectors consist of at least two faces. An example of a tagged mesh is given in Figure 6.

Figure 6 - Crease edges meeting in a corner with two convex (light grey) and one concave (dark grey) sectors. The subdivision scheme modifies the rules for edges incident to crease vertices (e.g. e1) and corners (e.g. e2).

2.3.2.4 Subdivision Stencils

In order to reduce the angles between adjacent triangles, the position of each new vertex introduced by the subdivision process is calculated as a function of some old vertices in the vicinity of the considered edge. In fact, the position of old vertices themselves may be altered as well for that same purpose. As what is sought is a low-pass filtering of the 3D shape, usually some linear combination is chosen: a new vertex position is then simply a weighted average of those of


some nearby old vertices. It is important to stress that only some finite number of such nearby old vertices is used in the blend in most schemes, which are consequently said to have finite (or compact) support. Some attractive features of subdivision surfaces are derived from this property.

The stencil of the subdivision scheme refers to the set of old vertices used to compute a new vertex. The term mask, or the set of (averaging) rules, is used to refer to the set of values of coefficients. It determines the convergence of the process and the smoothness degree of the limit surface.

Different masks are necessary for different types of vertices discussed in more detail in the next section.

2.3.2.4.1 Loop subdivision

Figure 7 shows the stencils used for repositioning old vertices and inserting new ones by splitting edges in the case of Loop’s scheme [60].

Vertex rules are used to update positions of old vertices, edge rules are used to compute positions of newly inserted vertices.

For untagged interior and dart vertices rules shown in Figure 7a are used; for boundary/crease vertices rules in Figure7b are used. Positions of corner vertices are never changed.

31

(b)(a)

(c)

(d) (e)


Figure 7 - Stencils of Loop’s subdivision scheme. (a) repositioning interior old vertices (b) repositioning boundary or crease old vertices , (c) splitting interior edges not adjacent to boundary extraordinary vertices, (d) splitting boundary and crease edges, (e) splitting interior edges adjacent to an extraordinary (valence not 4) vertex on the boundary/crease or corner vertex . The values of beta and gamma are defined in the text.

The coefficient can be chosen in different ways.

Loop’s original proposal was =

2π2cos41

83

851

kk. In particular, this is = 3/16 for k = 3, and = 1/16 for

k = 6. Warren proposed later to keep = 3/16 for k = 3, but to have simply = k83 for k > 3, which also amounts to

= 1/16 for k = 6 [94].

A basic implementation should implement Warren’s and Loop’s versions for backward compatibility with existing models.

Edge rules are more complex. The rule for an edge point inserted on an edge e is chosen depending on the tag of the edge and the tags of adjacent vertices and sectors. There are several cases:

1. In the absence of tags, the standard edge rules are applied (Figure 7c).

2. If a crease edge e is split, and no endpoint is a dart vertex, the rule shown in Figure 7d is used.

3. If one of the endpoints is a dart vertex, the standard rule (Figure 7c) is applied.

4. If one endpoint v is tagged as corner or smooth crease and the edge e is not a crease edge, we use a modified rule shown in Figure 7e, with parameter chosen as kk cos4/12/1)( . For a crease vertex,

kk / , where k is the crease degree of the vertex in the sector where the edge e is.

5. If both endpoints are tagged as corner or smooth crease, then the average of the masks given by the rules above is taken.

At a corner vertex v, e is differentiated whether it is in a convex or concave sector. For a convex corner , kk / ,

where is the angle between the two crease edges spanning the sector, for concave corners, kk /)2( .

For concave corners, one may use kk / and for convex corners kk / . The simplest choice of k is

)2/(3 k for concave corners and )2/( k for convex corners which is the default; the best choice is to use k/ for

33

convex corners (respectively k/)2( for concave corners) where is the angle between edges of the control

mesh bounding the sector; k for concave and convex corners can be also a user-defined parameter changing in the ranges specified above.

Application of the stencils is followed by application of flatness modification described in the next section. This modification is optional for all cases excluding concave corner; in which case it is necessary to ensure tangent plane continuity of the surface.

2.3.2.4.1.1 Flatness modification

With k defined as in section 2.3.2.3, let the vector of control points be mk

mmc

m pppp 10 ,,, (Figure 8). The superscript indicates the subdivision level the subscript indicates the point number in the sector. After subdivision step in a neighborhood of a corner vertex is modified as follows:

)()1( 22

11

00 xxxspspnew aaa

where s is the flatness parameter. ),( pl ii a , 210 ,, lll are limit position and tangent masks defined in the next

section, and 210 ,, xxx are the right eigenvectors also defined in the next section. The valid range of the flatness

parameter is 10 s for all vertex types except concave corner, in which case it is 1211

s , with lambda

defined as ))/cos((cos4/12/1 kk . Geometrically, the modified rule blends between control point positions before flatness modification and certain points in the tangent plane, which are typically close to the projection of the original control point. The limit position 0a of the center vertex remains unchanged.


Figure 8 - Neighborhoods of a vertex on different subdivision levels. The subdivision matrix relates the vector of control points mp to the control points on the next level 1mp . For a neighborhood of k triangles

mk

mmc

m pppp 10 ,,, .

The flatness modification is always applied at concave corner vertices; the default value for the flatness parameter is 1. Better visual results are obtained for )4/(11 s . In other cases, s can be taken to be 0 by default and the modification step need not be applied.

Optionally, a user-specified flatness value can be associated with any vertex and used to control how fast the surface approaches the tangent plane.

A compliant implementation is required to support s =0 and s = 1, but an implementation may support arbitrary s values.

2.3.2.4.1.2 Limit position and tangent masks

For each type of vertex (interior, smooth crease, corner, dart) there are masks for computing the limit position of this vertex and two tangent vectors at this vertex. The limit position masks are denoted 0l and limit tangent position masks are denoted 1l and 2l respectively. A subscript c denotes the coefficient corresponding to the center vertex. Once two tangent vectors are obtained by applying the masks, the normal can be computed as the cross-product of the two. In addition we define right eigenvectors in each case which are necessary for flatness modification. Recall that dominant right eigenvector 0x is the vector consisting of ones

Untagged interior or dart vertex of degree k. In all cases i is in the range 10 k , and

35

,cos2,sin2,cos,sin

0

)3/8(1)3/8(,

)3/8(11

21

21

2121

00

kiki

kiki

cccc

ic

ik

lik

l

ixix

llxx

kl

kl

Smooth crease vertex of crease degree k. Let kk /

11,0,6/1,3/2 0001

0 killll ikc

For k=1,

1,2/1,0,2/1,1

1,1,0,3/2,3/122

21

212

11

1

22

21

212

11

1

llllll

xxxxxx

cc

cc

Otherwise 0121 ccc lxx ,

11,sin221

61

22

322

1,0,2/1,2/1

0,cos,sin

2

3122

0

312

1110

21

kiik

l

bak

ll

bak

l

killl

kiixix

ki

k

c

ik

kiki

where

)1arccos(coscoscos

sin2

cos

cos1sin

2cos

32

)cos41

21(3

)cos1(41

31

k

k

k

k

k

k

k

k

k

aba

Convex/concave corner vertex of crease degree k with parameter k . Let kk .


killl

killl

kiik

xi

xxx

kill

ic

ikc

ki

kicc

ic

1,0,1,1

10,0,1,1

0,sin

)sin(,

sinsin

,0

0,0,1

220

2

111

2121

00

2.3.2.4.2 Catmull-Clark subdivision

The Catmull-Clark scheme was described in [27]. This scheme can be applied to general polygonal meshes, but after the first subdivision step all polygons in a mesh become quads. The masks for a mesh consisting only of quads are shown in Figure 9 and Figure 10. The scheme produces surfaces that are C2 everywhere except at extraordinary vertices, where they are C1.

Figure 9 - Catmull-Clark rules with k defined as in section 2.3.2.3 (a) face rule; (b) regular edge rule (c)

boundary/crease edge rule (d) interior vertex rule k23

and k4

1 (e) smooth boundary/crease vertex rule

There are three types of rules: vertex rules used to update old vertex positions, edge rules used to insert new vertices for each edge, and face rules used to insert new vertices for each face.

37

(a)

(b)

(c)

(d)

(e)

Figure 10 - Modified rule for odd vertices adjacent to a boundary extraordinary vertex (Catmull-Clark scheme), the coefficient gamma is defined in the text.

Let’s start by describing rules that are used only on the first step.

a face control point for an n-gon is computed as the average of the corners of the polygon;

an edge control point for an untagged edge as the average of the endpoints of the edge and newly computed face control points of adjacent faces; for a crease/boundary edge it is the average of the endpoints;

The vertex rule for untagged interior and dart vertices is:

11

02

1 12 ji

k

i

ji

jc

jc qp

kp

kkp

where jip are k control points at level j sharing edges with j

cp , and 1jcq are face control points for faces

incident at the vertex. Note that face control points on level j+1 are used.

For boundary/crease vertices, it is the spline rule identical to the one used for the Loop scheme. The corner vertex positions are never changed.

Next, let’s describe the rules applied once the mesh is converted to quadrilateral (Figure 9 and Figure 10). The vertex and face rules are essentially the same; the masks are shown in Figure 9. As in the case of the Loop scheme edge rules are more complex.

1. In the absence of tags, the standard edge rules are applied (Figure 11b).

2. If a crease edge e is split and no endpoint is tagged as a dart vertex , rules of Figure 11c are used.

3. If one of the endpoints is a dart vertex, the standard edge rule is used (Figure 11b).


4. If one endpoint v is tagged as a corner or smooth crease and the edge e is not a crease, a modified rule shown in Figure 6 is used, with parameter chosen as kk cos4/18/3)( . Exactly like for Loop scheme, for a

smooth crease vertex, kk / , where k is the crease degree of v in the sector where e is. For a corner use

kk / , where for concave corners and for convex corners.

5. If both endpoints are tagged as corner or smooth crease, then the average of the masks given by the rules above is taken.

Application of the stencils is followed by application of flatness modification described in the next section. This modification is optional for all cases excluding concave corner; in this case it is necessary to ensure tangent plane continuity of the surface.

2.3.2.4.2.1 Flatness modification

The flatness modification is similar to the one used for Loop scheme, but the vector of control points has to include one more group of points ],,[ 10

mk

m qq as shown in the Figure 11. The superscript indicates the subdivision level the subscript indicates the point number in the sector.

Figure 11 - Neighborhoods of a vertex on different subdivision levels. The subdivision matrix relates the vector of control points mp to the control points on the next level 1mp . For a neighborhood of k quadrilaterals

mk

mmk

mmc

m qqpppp 1010 ,,,,,, .

The rule itself coincides with the rule for the Loop scheme:

39

)()1( 22

11

00 xxxspspnew aaa

where s is the flatness parameter. ),( pl ii a , 210 ,, lll are limit position and tangent masks defined in the next

section, and 210 ,, xxx are the right eigenvectors also defined in the next section..

The valid range of the flatness parameter is 10 s for all vertex types except concave corner, in which case it is

1211

s , with lambda defined as

944483264161

81

21 24222 cccc and

kc

2cos .

The default flatness parameter can be set to zero for all cases except concave corners, in which case it is set to 1. Better visual results are obtained for the values close to )4/(11 s .

Optionally, a user-specified flatness value can be associated with any vertex and used to control how fast the surface approaches the tangent plane.

2.3.2.4.2.2 Limit position and tangent masks

Interior or dart vertex of degree k. In all cases i is in the range 10 k , and kk /2 .

14)1cos(cos,

14)1sin(sin

,cos4,sin4

,)14(

)1cos(cos,)14(

)1sin(sin

cos1sin10

)5(1,

)5(4,

5

21

21

21

21

2121

000

kkqi

kkqi

kpikpi

kkqi

kkqi

kpikpi

cccc

qipic

iiliil

ilil

iixiix

ixix

llxx

kkl

kkl

kkl

where )2cos92/cos(cos16/116/5 kkk and ))14(

cos12( 2

kk

Smooth crease vertex of crease degree k. Let kk /

11,0

,6/1,3/2000

0

001

0

killl

lll

piqiq

pkpc


For 1k ,

0110033602/12/10

18/518/218/218/1

20

21

20

2

10

11

10

1

20

21

20

2

10

11

10

1

qppc

qppc

qppc

qppc

llllllllxxxxxxxx

Otherwise 0212 ccc xxl ,

10,)cos3(

))1sin((sin4

11,)cos3(

sin4

)cos21(),1(cos4

11,0,2/1,2/1

10,)1cos(cos

10,)1sin(sin

0,cos,sin

1

1

110

1

2220

220

2

1

21

kik

iil

kik

il

RllRl

killlll

kiiix

kiiix

kiixix

k

kkqi

k

kpi

kpkpkc

piqiqpkp

kkqi

kkqi

kpikpi

where )cos3(sin1cos

kk

k

kR

Convex/concave corner vertex of crease degree k with parameter k . Let kk .

Left eigenvectors are the same as for Loop with zeroes everywhere except lc, lp0 and lpk.

10,sin

)1sin()sin(

10,sin

)1sin(sin

0,sin

)sin(,

sinsin

2

1

21

kiikik

x

kiii

x

kiik

xi

x

kkqi

kkqi

kpi

kpi

2.3.2.4.3 Extended Loop subdivision

Loop’s subdivision [60] requires that the control hull is a mesh of triangles, and generates triangles at all subdivision levels. Arbitrary polygons can be broken up into triangles, but this process introduces a “kinking” artifact (Figure 13a) that is visible if the original control hull (Figure 12a) contains material boundaries. In a traditional animation setting, the expensive step of splitting the original control hull into several pieces is an acceptable workaround for this problem, but compromises the utility of this approach for reducing storage size requirements.

The Superscape extended Loop subdivision is a major refinement of Loop subdivision that eliminates this “kinking” and allows arbitrary polygons to be used for the control hull by storing a visibility state with each edge. When arbitrary polygons are broken down into triangles, a record must be kept of which edges were originally present (the visible

41

edges) and which are introduced only to break the polygons down into triangles (the invisible edges). This edge visibility tagging is used to eliminate the disproportionate influence exerted by invisible edges on the positioning of new vertices, as well as a means for providing the mesh creator with direct control over the final triangulation.

(a)

(b)

Figure 12: A simple control hull before (a) and after (b) triangulation.Invisible edges are shown dotted.Consider Figure 12a, which is a view from the rear onto the roof portion of a car. The two vertices at the top center of the rear windscreen appear to be equivalent, but after triangulation (cf. Figure 12b) the right hand vertex has two invisible edges radiating from it. As is evident from Figure 13a, using unmodified Loop subdivision, such edges exert a disproportionate influence on that vertex, and result in a distortion of the final mesh; the “kinking” effect mentioned earlier.

(a)

(b)

Figure 13: The same control hull subdivided according to Loop’s (a) and extended Loop (b) rules.If edge visibility is taken into consideration, this unwanted influence can be eliminated, as shown in Figure 13b. In addition, the new mesh appears to be made of polygons similar in shape to the originals. This is an added benefit of propagating edge visibility information from one level of subdivision to the next, and is essential for the correct operation of the process.


2.3.2.4.3.1 Stencils.

Extending Loop to handle arbitrary meshes requires the addition of edge visibility tags for edges that are introduced during triangulation and are not part of the original control hull. The introduction of edge visibility information will clearly affect the vertex and edge subdivision rules.

(b) (c)(a)

(d) (e) (f)

Figure 14: Stencils of extended Loop subdivision for repositioning interior(smooth and dart) (a), (b) and boundary/crease old vertices and splitting interior (d), (e) and boundary/crease edges.Figure 14 shows the extended Loop subdivision stencils used for repositioning old vertices (as with standard Loop, corner vertices are never moved) and splitting edges both on the surface interior and boundaries/creases. Solid lines indicate original control hull (visible) edges while dashed lines indicate invisible edges. It is clear from the stencils (a), (b) and (c) that repositioning old vertices with extended Loop is identical to standard Loop if invisible edges are ignored, with one exception (b). If a vertex has two or less visible edges in its neighborhood, then the invisible edges are included in the calculation. If all edges are visible then the stencils are identical to standard Loop subdivision.

As with standard Loop subdivision, =

2π2cos41

83

851

kk, where k is the number of visible edges (Figure

14a) or the valence (visible + invisible edges) if the number of visible edges is 2 (Figure 14b).

The stencil for splitting boundary/crease edges (Figure 14f) is also identical to standard Loop, while splitting interior edges depends on whether the edge in question is visible (Figure 14d – left) or not (Figure 14d – right). As with the old vertex repositioning stencils, the edge split stencils try to employ vertices at the “other end” of visible edges only, ignoring invisible edges completely. Identifying visible edges at each side of the edge endpoint involves a clockwise

43

and/or counter-clockwise search around the endpoint neighborhood and may result in the same vertex for both sides or no visible edges in which case the endpoint is assigned the full weight (Figure 14e). Again if all edges are visible, the stencils are identical to standard Loop subdivision.

2.3.2.4.3.2 Visibility propagation.

The visibility of the new edges needs to be determined, since it will be required on the next round of subdivision. New edges radiate from a new point (see Figure 15), and their visibility is determined according to the neighborhood of the point.

E 2

E 3

E 1

E

E 4

e 2

e 1

e 3 e 4

Figure 15: New edges created when splitting an edge in two.Two edges will be the two halves of the original edge E - these simply keep the visibility of the original edge. Of the other four new edges, two edges will be to the right hand side of the original edge, and two more to its left. If we label these edges clockwise as e1, e2, e3 and e4, these connect to other new points created by dividing the original edges labeled E1, E2, E3 and E4. The visibility of the new edges is initialized to invisible and is changed to visible if the lookup of the combination of E, E1, E2, E3 and E4 in Table 3 indicates that it should be. Clearly the visibility of each of e1, e2, e3

and e4 is not uniquely determined by splitting E alone; e1, for example, also depends on the visibility lookup for splitting E1.

Original edges visible New edges visible

Original edges visible New edges visible

E E1 E2 E3 E4 e1 e2 e3 e4 E E1 E2 E3 E4 e1 e2 e3 e4


Table 3 - New edge visibility lookup table.

Note that Table 3 is unchanged if we swap left and right, so it is unimportant which side of the original edge we call “right” to start with.

2.3.2.4.3.3 Vertex normal calculation

In order to perform operations such as lighting on a surface, it is necessary to determine the normal to that surface at each vertex. This is the direction at right angles to the limit surface at that point. In order to calculate this normal, it is necessary first to obtain two tangent vectors, t1 and t2, which are directions that lie along the surface rather than at right angles to it. Given these, a vector cross product will result in the desired normal:

21 ttn .

2v

3v 4v

5v0v

kv1v

0v

kv

4v

3v2v

1v

(b)(a)

Figure 16: (a) interior, and (b) boundary/crease vertex neighborhood used for normal calculation.

45

For interior smooth and dart vertices of valence k (Figure 16a), the tangents are defined as the weighted averages of the positions of the neighborhood vertices (vi where i [1, k]):

kkk

kk

vcvcvcvcvctvcvcvcvct

113423122

3322111

,

where:

kici

2cos .

For boundary, crease and corner vertices (Figure 16b), each sector between pairs of boundary/crease edges will have a different pair of tangent vectors which are defined as:

4,4,2223,2,2

332211

43210

02

210

2

11

kvwvwvwvwkvvvvvkvvkvvv

t

vvt

kk

k

,

where, for1

k , wi is defined as:

12),)1sin(()2cos2(

,1,sinkii

kiiwi

.

2.3.2.5 Extensions

2.3.2.5.1 Normal control

In addition to the flatness modification which insures that the surface is tangent plane continuous at concave corner vertices and provides additional shape control at other vertex types, a similar modification mechanism can be used to control the tangent plane of the surface as shown in Figure 17.

Specifically, if the user prescribes tangent vectors or the normal for a sector at a corner or smooth boundary vertex or a normal for an interior vertex then the following modification can be applied after each subdivision step for both Loop and Catmull-Clark surfaces,

))()(( 222

111 xxtpp new aaaa


where the notation used is the same as in the sections describing flatness modification above, 1aand 2a denote the prescribed tangents and t is a user-specified parameter 10 t , by default set to 1. If only the normal is prescribed, the tangents are computed as 2,1),( inniii aaa .

The parameter t determines how fast the surface approaches the prescribed tangent plane.

Figure 17 - Normal interpolation. (a) Surface with convex corners. (b) Prescribed directions: at each corner we tilt the normal for one surface sector slightly inwards. (c) Smooth surface. (d) Same control mesh but all normals vertical.

2.3.2.6 Node specification

2.3.2.6.1 SubdivisionSurface node

The SubdivisionSurface node is based on the existing IndexedFaceSet node with all fields relating to face normals and the creaseAngle field removed since normals are generated by the subdivision algorithm at run-time and their behaviour at facet boundaries is controlled by edge and vertex crease tagging. Some new fields are introduced, specific to the subdivision process. The control hull is specified as a collection of polygons and must be a manifold mesh. No “dangling” edges or vertices are permitted, and each polygon may share each of its edges with at most one other polygon.

The SubdivisionSurface node is defined as follows:

SubdivisionSurface { #%NDT=SFGeometryNodeeventIn MFInt32 set_colorIndexeventIn MFInt32 set_coordIndexeventIn MFInt32 set_texCoordIndexeventIn MFInt32 set_creaseEdgeIndexeventIn MFInt32 set_cornerVertexIndexeventIn MFInt32 set_creaseVertexIndex eventIn MFInt32 set_dartVertexIndex

47

exposedField SFNode color NULLexposedField SFNode coord NULLexposedField SFNode texCoord NULL

exposedField SFInt32 subdivisonType 0 # 0 – Loop, 1 – Catmull-ClarkexposedField SFInt32 subdivisonSubType 0field MFInt32 invisibleEdgeIndex [] # [0, )

# extended Loop if non-empty

exposedField SFInt32 subdivisonLevel 0 # [-1, )

# IndexedFaceSet fieldsfield SFBool ccw TRUEfield MFInt32 colorIndex [] # [-1, ) field SFBool colorPerVertex TRUEfield SFBool convex TRUEfield MFInt32 coordIndex [] # [-1, )field SFBool solid TRUEfield MFInt32 texCoordIndex [] # [-1, )

# tagsfield MFInt32 creaseEdgeIndex [] # [-1, )field MFInt32 dartVertexIndex [] # [-1, )field MFInt32 creaseVertexIndex [] # [-1, )field MFInt32 cornerVertexIndex [] # [-1, )

# sector informationexposedField MFNode sectors[]

}

The SubdivisionSurface node represents a 3D shape formed through surface subdivision of a control hull constructed from vertices listed in the coord field. The coord field contains a Coordinate node that defines the 3D vertices referenced by the coordIndex field. SubdivisionSurface uses the indices in its coordIndex field to specify the polygonal faces of the control hull by indexing into the coordinates in the Coordinate node. An index of “-1” is used as a separator for the control hull faces. The last face may be (but does not have to be) followed by a “-1” index. If the greatest index in the coordIndex field is N, then the Coordinate node shall contain N+1 coordinates (indexed as 0 to N). Each face of the SubdivisionSurface control hull shall have:

a. exactly three non-coincident vertices for Loop subdivision;

b. at least three non-coincident vertices for Catmull-Clark and extended Loop subdivision;

c. vertices that define a planar polygon;d. vertices that define a non-self-intersecting polygon.

Otherwise, the results are undefined.


The SubdivisionSurface node is specified in the local coordinate system and is affected by the transformation of its ancestors.

The subdivisionType field specifies the major subdivision rules:

0. Loop scheme (section 2.3.2.4.1),

1. Catmull-Clark scheme (section 2.3.2.4.2).

The subdivisionSubType field specifies a sub category of subdivision rules in the major subdivision rule. For this specification, for subdivisionType 0 (Loop), subdivisionSubType 0 indicates Loop and 1 Warren’s Loop.

A subdivisionType value of zero and a non-empty invisibleEdgeIndex array imply that the extended Loop scheme (section 2.3.2.4.3) is used.

The invisibleEdgeIndex field is used to select co-ordinate pairs from coord to define which edges should be tagged as invisible. Consecutive pairs of co-ordinates define an edge; this pair of vertices must belong to one facet as defined by the coordIndex field and may be “interior” to the facet (e.g. in the case of a quadrilateral, the invisible edge would be one of its diagonals). All non-triangular facets should have enough tags to fully describe their triangulation. Note that a non-empty invisibleEdgeIndex array is only allowed when subdivisionType is 0; the results are undefined in other cases.

The subdivisionLevel field specifies the number of times the surface should be subdivided. If this field is “-1” then the terminal should apply enough iterations of subdivision until it reaches a “visually smooth” surface.

The fields color, coord, texCoord, ccw, colorIndex, colorPerVertex, convex, coordIndex, solid, and texCoordIndex have the same semantic as for the IndexedFaceSet node. These fields define the control hull of the subdivision surface. Textures coordinates and color information, if present, propagate from the control hull to subsequent subdivision levels through linear interpolation.

The creaseEdgeIndex field an indexed line set of creases, indicating where creases appear on the otherwise smooth subdivision surface (section 2.3.2.3). As for the IndexedLineSet node, the line is defined by successive vertex indices defined in the coord field. An index of "-1" indicates that the current crease has ended and the next one begins. The last crease may be (but does not have to be) followed by a "-1". Crease edges must already exist in the control hull topology (i.e. the edge must coincide with a single edge of a polygon as defined by the coordIndex field). Note that if an edge is tagged as both invisible and a crease, the results are undefined.

From the creaseEdgeIndex array, the default behavior (and appropriate subdivision rules) for vertices on crease edges is defined (section section 2.3.2.3) as:

49

vertices with exactly one crease edge in their neighborhood are dart vertices,

vertices with exactly two crease edges in their neighborhood are crease vertices,

vertices with three or more crease edges in their neighborhood are corner vertices.

However, this array may not provide a description general enough for all surfaces and three arrays are provided: cornerVertexIndex, creaseVertexIndex, and dartVertexIndex. The information provided by these arrays override the information in creaseEdgeIndex.

The cornerVertexIndex field indicates the vertex index of corner vertices.

The creaseVertexIndex field indicates the vertex index of crease vertices.

The dartVertexIndex field indicates the vertex index of dart vertices.

Finally, the sectors field may optionally contain an array of SubdivSurfaceSector nodes that describe additional sector information; see section 2.3.2.6.2 for more details. Note that if both the sector and the invisibleEdgeIndex fields are non-empty, the results are undefined.

2.3.2.6.2 SubdivSurfaceSector node

SubdivisionSurface node may have sectors attached to. Its sectors field contains SubdivSurfaceSector nodes.

SubdivSurfaceSector { #%NDT=SFSubdivSurfaceSectorNodefield SFInt32 faceIndex 0 # face index from coordIndexfield SFInt32 vertexIndex 0 # vertex index from coordexposedField SFInt32 tag 0exposedField SFFloat flatness 0exposedField SFFloat theta 0exposedField SFVec3f normal 0 0 0exposedField SFFloat normalTension 0

}

The faceIndex field contains the index of the face and the vertexIndex field contains the vertex index the sector is attached to. The indices refer to point index and face index from coord and coordIndex fields respectively in the parent SubdivisionSurface node.

The tag field indicates the type of sector (Figure 18):


Tag value Semantic

0 No corner sector

1 Convex sector

2 Concave sector

Figure 18 - Concave corners (left) and convex corners (right).The flatness field contains a value between 0 and 1: 0 indicating no modified flatness, and 1 indicating maximal flatness. This parameter is denoted s in Loop and Catmull-Clark rule definitions above (Figure 19).

Figure 19 – Influence of the flatness parameter..

The theta field is an angle used to control the shape of the corner; this angle is denoted k in the mask definitions above (Figure 20).

Figure 20 - Influence of theta.The normal field specifies the desired normal at the vertex (Figure 21).

51

The normalTension field is the influence of the normal between 0 and 1, 0 meaning no influence; denoted t in the formulas for normal modification (Figure 21).

Figure 21 – Prescribed normals and tension parameter.

2.3.2.7 Examples

2.3.2.7.1 Subdivision surfaces and creases

(a)

(b)

(c)

#VRML V2.0 utf8

DEF tensioncube Shape {

Appearance Appearance {

Material Material {

AmbientIntensity 0.2

SpecularColor 0.8 0.8 0.8

Shininess 0.35

Transparency 0

}

}

geometry DEF tensioncube-FACES SubdivisionSurface {

ccw FALSE

solid FALSE

subdivisionType 0

subdivisionLevel 4

invisibleEdgeIndex [


(d)

(e)

(f)

(g)

0, 13, 1, 14, 2, 15, 3, 12,

4, 1, 5, 2, 6, 3, 7, 0,

8, 5, 9, 6, 10, 7, 11, 4,

]

color DEF tensioncube-COLORS Color {

color [

0.0 1.0 0.0,

0.0 1.0 1.0,

0.0 0.0 1.0,

1.0 0.0 1.0,

]

}

coord DEF tensioncube-COORD Coordinate {

point [

-1.0 -1.0 1.0,

-1.0 1.0 1.0,

1.0 1.0 1.0,

1.0 -1.0 1.0,

-0.5 -0.5 -1.0,

-0.5 0.5 –1.0,

0.5 0.5 -1.0,

0.5 -0.5 –1.0,

-1.0 -1.0 -1.0,

-1.0 1.0 –1.0,

1.0 1.0 -1.0,

1.0 -1.0 –1.0,

53

-0.3 -0.3 1.0,

-0.3 0.3 1.0,

0.3 0.3 1.0,

0.3 -0.3 1.0,

]

}

colorPerVertex FALSE

colorIndex [

0, 0, 0,

3, 3, 3,

2, 2, 2,

1, 1, 1,

]

coordIndex [

0, 1, 13, 12, -1,

4, 5, 1, 0, -1,

8, 9, 5, 4, -1,

1, 2, 14, 13, -1,

5, 6, 2, 1, -1,

9, 10, 6, 5, -1,

2, 3, 15, 14, -1,

6, 7, 3, 2, -1,

10, 11, 7, 6, -1,

3, 0, 12, 15, -1,

7, 4, 0, 3, -1,

11, 8, 4, 7, -1,

]


creaseEdgeIndex [

# uncomment for (b), (d)

# 0, 1, 2, 3, 0, -1,

# uncomment for (c), (d)

# 4, 5, 6, 7, 4, -1,

# uncomment for (e), (f), (g)

# 1, 2, 3, -1,

]

cornerVertexIndex [

# uncomment for (f)

# 3,

# uncomment for (g)

# 2,

]

}

}

2.3.2.7.2 Normal Control

2.3.2.7.2.1 Prescribed normal directions on boundary

Prescribed directions: (a) tilted downwards, (b) horizontal, (c) no modification, (d) vertical.

55

2.3.2.7.2.2 Convex Corner with flatness modification

Left to right: maximum flatness, medium flatness, no flatness modification)

2.3.2.7.2.3 Concave corners with flatness modification

Left to right: small flatness, medium flatness, maximum flatness

2.3.2.7.2.4 Surface manipulation with corners

(a) Smooth boundary curves. (b) Concave corners on top, convex corners on bottom. (c) Corners with convex and concave sectors. (d) Creases and corners as for (c) but with prescribed normal direction on concave sectors.


2.3.2.7.2.5 Concave corner with different theta parameter

Shape {

Appearance Appearance {

Texture ImageTexture {url "texture.jpg"}

}

geometry SubdivisionSurface {

subdivideLevel 4

subdivideType 0

coord Coordinate {

point [

0.525731 0 -0.850651,

0.525731 0 0.850651,

-0.525731 0 0.850651,

-0.850651 -0.525731 0,

-0.525731 0 -0.850651,

-0.850651 0.525731 0,

0.850651 0.525731 0,

0.850651 -0.525731 0,

0 -0.850651 0.525731,

0 -0.850651 -0.525731,

57

0 0.850651 -0.525731,

0 0.850651 0.525731,

]

}

texCoord TextureCoordinate {

point [

0 0,0 1,1 1,

0 0,0 1,1 1,

0 0,0 1,1 1,

0 0,0 1,1 1,

]

}

coordIndex [

0, 4, 10, -1,

10, 4, 5, -1,

5, 4, 3, -1,

3, 4, 9, -1,

1, 2, 8, -1,

8, 2, 3, -1,

3, 2, 5, -1,

5, 2, 11, -1,

11, 2, 1, -1,

1, 6, 11, -1,

11, 6, 10, -1,


10, 6, 0, -1,

0, 6, 7, -1,

7, 6, 1, -1,

10, 5, 11, -1,

8, 3, 9, -1,

1, 8, 7, -1,

7, 8, 9, -1,

7, 9, 0, -1,

]

creaseLineIndex [

0, 4, -1,

4, 9, -1,

9, 0, -1,

]

cornerIndex [ 0, 4, 9 ]

sectors [

DEF subSurfSec0 SubdivSurfaceSector {

FaceIndex 18

VertexIndex 2

Tag 2

Flatness 0.7

Theta 4.71239

Normal 0 0 0

NormalTension 0

59

}

SubdivSurfaceSector {

FaceIndex 3

VertexIndex 1

Tag 2

Flatness 0.7

Theta 4.71239

Normal 0 0 0

NormalTension 0

}

SubdivSurfaceSector {

FaceIndex 18

VertexIndex 1

Tag 2

Flatness 0.7

Theta 4.71239

Normal 0 0 0

NormalTension 0

}

]

}

}


2.3.3 Wavelet Subdivision Surfaces

2.3.3.1 Overview

With simultaneous advances of both 3D scanning and hardware renderers, 3D mesh compression is a major issue in multimedia content transmission over networks. MPEG4 provides such a compression scheme, namely 3DMC, based on Topological Surgery and Progressive Forest Splits. The principle is to encode topologically and predictively some groups of connected local simplifications (vertex splits). The method is intended to provide both compression and progressivity. This last is very important since it permits partial transmition and/or reconstruction according to bitrate or terminal rendering capabilities.

However, this method fails to provide both state of the art compression and decoding convenience. Recent scalable compression methods based on geometrical wavelets have proven to palliate 3DMC main lacks:

First, 3DMC is progressive but not hierarchical. The details can be added locally, but there is no obvious way to relate the coarsest versions of the mesh and further details added over it. Thus it is not well suited for animation, as hierarchical methods can be. Moreover, on-demand transmission of refinements and view-dependent reconstruction are quite difficult, since details are not well-localized and cannot be added independently.

The second main drawback of using 3DMC with recent 3D scenes is the compression ratio: state-of-the art methods have proven to perform considerably better compression than what was possible at the time 3DMC was adopted.

Finally the 3DMC decoder is quite complex compared to recent multiresolution techniques.

Wavelet methods elegantly address these weaknesses by providing a hierarchical, efficiently compressed representation enabling fast and selective reconstruction of meshes in the context of 3D model animation.

2.3.3.2 Scope

Second generation wavelets have proven to be a powerful tool for scalable representation of 3D surfaces. One important property of wavelets is that they are well localized in space, which enables efficient selection of significant wavelets according to the position of the refinements they induce, as well as their frequency or magnitude. This selection can of course be applied at the reconstruction stage, but in a client/server configuration, the server can also use this information to transmit only the necessary refinements to the terminal.

In the classical subdivision surfaces setting, objects are described by a coarse mesh (called the base mesh) along with a subdivision rule that represents an almost-everywhere smoothing: the subdivision rule is iterated so that the resulting sequence of meshes converges to the initial surface. Hence, given this rule, no extra information has to be transmitted to get the smooth object.

61

In a nutshell, the principle of geometrical wavelets is to locally average the application of a given subdivision rule over the base mesh. Information is thus added at various stage of the subdivision rule iteration [63]. Hence a subdivision surface can be regarded as a wavelet representation with all coefficients set to 0.

It's important to note that the initial models (those to be encoded) suitable for each of these two techniques are from two basically different kinds: subdivision surfaces are adapted to models with large regular areas, while wavelets are useful for objects with many chaotic "high frequency" peaks. For example, representing a noised object with subdivision surfaces will require too complex a base mesh, which has to contain high frequencies.

Though wavelet representation performs high decorrelation of the surfaces geometry, their encoding can lead to further considerable gain in compression. The state-of-the-art techniques for efficiently encoding both classical and geometrical wavelets are zero-trees. Zero-trees provide progressive encoding of the significance maps, i.e. parts of the object which are pertinent for a given threshold. The objective is to mark the largest amount of non-pertinent coefficients with minimum information.

Zero-trees have initially been developed in the context of 2D image coding [83,90], and extended to 3D shapes [57]. The original algorithms can be directly applied provided that a hierarchy is expressed between coefficients, as it is canonically the case in the 2D setting. For example in [57], this hierarchy corresponds to the degrees of subdivision of edges with the same orientation. One drawback of the impressive compression ratios obtained by zero-trees is that the order of decoding is imposed.

Suppose now that a server transmits geometric data to a terminal with unknown / limited processing power. As the portion of a 3D scene actually viewed is generally a small part of the whole scene, the most part of the corresponding bitstream may not be needed by the client.


With the traditional zero-tree embedded representation, the client can start navigating in the 3D scene just after the reception of the base mesh, which is then refined during the walkthrough. But as the order of arrival of the coefficients is imposed by the zero-tree representation, the terminal may have to load many useless wavelet coefficients before the ones it actually needed.

The terminal should then be able to ask the server some parts of the bitstream representation of the wavelet coefficients, and the server should be able to select the relevant parts of this bitstream from information about the client (e.g. view direction, position, viewed facets, …) provided through the back-channel. The main problem is that enabling the change in the order of arrival of the coefficients would imply non-negligible loss in term of compression. Thus, a tradeoff has to be made on the granularity of adaptively: the bitstream should be composed of isolated zero-trees on areas independent enough to have comparable compression ratios, but small enough compared to the whole model to let the corresponding groups of wavelet coefficients have local influence on the base mesh.

2.3.3.3 Definition of adaptive 3D-mesh decoding

2.3.3.3.1 Architecture

Scenegraph

Decoder

Cache

backChannel

Control

ViewPoint

baseMesh

Adaptative 3D-mesh + texturing info

detailsStream

Figure 22 Simplified abstract decoder block-diagram

2.3.3.3.2 Walkthrough

First the BIFS Node is decoded. The baseMesh is decoded and instantiated

The texture is decoded and instantiated.

Now it is possible to render the node at its coarsest definition.

When the baseMesh has been instantiated, the following streams, if they exist, are set up :

DetailsStream

backChannel

Data from detailsStream are received; they are cached depending on the viewpoint & quality fields. Depending on viewpoint & quality fields, the adaptive mesh is built using the baseMesh, its TextureCoordinate and available

coefficients that have been gathered both in MeshStream.

63

Depending on the frequency field and on the viewpoint, data is sent through the backChannel. This contains information about viewpoint and cache state.

Depending on the viewpoint and on the cache state, data is removed from the cache. When the coordinates of the baseMesh are animated, modified, interpolated, the adaptative mesh is re-built.2.3.3.3.3 Node specification

WaveletSubdivisionSurface {field MFString backChannel []exposedField SFBaseMeshNode baseMesh NULL

field MFString detailsStream []

exposedField SFFloat frequency 1.0exposedField SFFloat fieldOfView 0.785398 # pi/4

exposedField SFInt32 quality 1 }

The WaveletSubdivisionSurface node is used to represent adaptively multi-resolution polygonal models. It allows representing at any moment a 3D-model at a resolution that depends on the distance of the object with respect to the viewpoint. Transmission of such a model consists of two parts: a base mesh which describes the model at the coarsest level of detail, and series of refinements (wavelet coefficients) which can be carried on one or more elementary streams (see the bitstream definition).

The backChannel field specifies, if present, an upstream channel that allows a server to monitor which series of refinement are needed at the terminal. Thus in an interactive scenario (client-server interaction), refinements are sent according to the client state (see the back-channel bitstream definition).

The baseMesh field shall specify either a SubdivisionSurface node or an IndexedFaceSet node (see Annex B.2). If baseMesh is a SubdivisionSurface, the stream containing the mesh shall consist of a single access unit, and it will be explicitly instantiated as a SubdivisionSurface. The structure of this SubdivisionSurface will be preserved, namely deletion and insertion of new faces and/or vertices is explicitly forbidden. However, replacing the values of the coordinates is allowed. If baseMesh is an IndexedFaceSet, it is required to be manifold.

NOTE — Instantiating explicitly baseMesh as a SubdivisionSurface or IndexedFaceSet allows animating the node.

The detailsStream field shall specify the data source for wavelet coefficients. The terminal is responsible for maintaining an internal representation of the mesh to be rendered consisting of the baseMesh to which the detailsStream data must be applied.

The frequency exposedField gives a preferred frequency at which the terminal shall give its state by the means of the backChannel stream if it exists.

NOTE — If the backChannel does not exist this field has no effect.

The quality exposedField allows to parametrize the quality of the rendered object. The range of values for this field is [0..2] where 0 stands for low quality, 1 for normal quality, 2 for best quality. Those values don’t have a more explicit semantic, that is, it is a hint for the terminal.


The WaveletSubdivisionSurface node is seen by the player as a SubdivisionSurface or IndexedFaceSet node container. That is, the player has access to the baseMesh field.

2.3.3.3.4 detailsStream : the wavelet coefficients stream

will be defined later..

2.3.3.3.5 backChannel : the monitoring channel.

The backChannel is the stream conveying the information necessary to monitor the state of the terminal (cache state, viewpoint).

2.3.3.3.5.1 Functionality

The backChannel is an optional upstream elementary stream that allows the client to request for specific chunks of the wavelet coefficients so that only relevant parts of the mesh are refined. It also allows the server to know the client state and send in advance chunks that may be useful.

2.3.3.3.5.2 Stream Specification

2.3.3.3.5.2.1 ES Configuration. DecoderSpecificInfo

The configuration is defined as follows.

Stream Configuration : Syntax

class WmeshBC_DecoderConfig extends DecoderSpecificInfo : bit(8) tag=DecSpecificInfoTag {bit(5) viewpointNbBits

}

Semantics

ViewpointNbBits : the number of bits used to quantize the viewpoint direction.2.3.3.3.5.2.2 BackChannelFrame.

Syntax

class BackChannelCommandFrame() {Position position;Viewpoint viewpoint;bit(1) reserved bit(1) hasMeshCacheStateif (hasMeshCacheState){

CacheState meshCacheState;}else{

CacheDelta meshCacheDelta;}

}

65

Semantics

This is the payload of an access unit (AU).

First the position and the viewpoint direction are sent. The position is described in the local coordinate system of the WaveMesh node.

2.3.3.3.5.2.3 Position

Syntax

class Position () {EfficientFloat xEfficientFloat yEfficientFloat z

}

Semantics

The Position syntax describes the position of the user with respect to the local coordinate system of the user.

2.3.3.3.5.2.4 Viewpoint

Syntax

class Viewpoint () {int(1) directionint(2) orientationint(viewpointNbBits) vq[0]int(viewpointNbBits) vq[1]

}

Semantics

The Viewpoint syntax describes the viewing direction of the user with respect to the local coordinate system of the user. It uses the quantization scheme defined in 9.3.7.25 in ISO/IEC 14496-1 with a fixed number of bits defined in the decoder specific info.

2.3.3.3.5.2.5 CacheState

Syntax

class CacheState () {bit(2) reservedint i=0;for (i=0;i<numTrees;i++){

int(8) numWaveletCoefficientsint(5) numBitPlane

}}

Semantics


This element specifies for all the faces of the base mesh the number of coefficients as well as the number of bitplanes received concerning those coefficients.

2.3.3.3.5.2.6 DeltaCache

Syntax

class DeltaCache () {bit(2) reservedint i=0;int(5) numFaceBits;int nbBits=numFaceBits;

for (faceNb =0;i<numTrees;i++){

if (baseMeshFace[faceNb].cacheStateHasChanged()){

int (nbBits) faceNb;int(8) numWaveletCoefficients ;int(5) numBitPlane;

}}

}

Semantics

This element specifies for all the faces of the base mesh the number of coefficients as well as the number of bitplanes received concerning those coefficients.

2.3.4 MeshGrid representation

2.3.4.1 General Description

The aim of the MeshGrid format is to provide a compact storage, and an animation-friendly and multi-scalable representation. The features and the functionality of the MeshGrid representation can be exploited (1) for storing surface representations of objects, (2) for transmission of these representations over bandwidth limited transmission channels, (3) in view-dependent decoding or visualization scenarios requiring the adaptation of the complexity of the scene depending on the processing power in order to meet the Quality of Service (QoS) constraints, and (4) in editing and animation operations.

The basic idea of the MeshGrid representation is (1) to define a regular 3D grid of points, called reference-grid, and (2) attach a wireframe description of the surface of an object to the reference-grid. The wireframe description, which is called the connectivity-wireframe1, keeps the connectivity information between a set of vertices located on the surface of the object. The reference-grid stores the spatial distribution of the vertices from the connectivity-wireframe. Each reference-grid point is identified by a discrete position (u,v,w), and by a coordinate (x(u,v,w),y(u,v,w),z(u,v,w)).

The MeshGrid representation contains several novel ideas that are briefly introduced here:

1 Notice that a distinction is made between the connectivity-wireframe from the MeshGrid representation and the classical wireframe representation. The connectivity-wireframe is similar to the classical wireframe in the sense that it consists of a set of vertices and lines or curves connecting those vertices. Yet, the connectivity-wireframe is different from the classical wireframe because the direct polygonal representation may consist of non-planar polygons ranging from triangles to heptagons, which have to be split in triangles for the final polygonal representation. Only this final representation is a classical or typical wireframe or surface mesh representation

67

It combines a dual representation of the surface description of an object, i.e. (1) the connectivity-wireframe, and (2) the reference-grid, the both of them having specific scalability and animation features.

The connectivity-wireframe is constructed in such a way that it inherits particular connectivity properties allowing deriving unambiguously the polygonal representation of the surface.

It stores the connectivity-wireframe and its associated reference-grid instead of the polygonal representation in order to achieve a more compact stream.

The connectivity-wireframe has a regular structure, each vertex being connected to exactly four other vertices, but still flexible to fit complex surfaces, since its polygonal representation yields surface elements ranging from triangles to heptagons. Non-planar polygons are split into triangles.

The connectivity-wireframe is efficiently encoded by using a new type of 3D extension of Freeman chain-code. The reference-grid is a smooth vector field defined on a regular discrete 3D space, i.e. each reference-grid point is identified by a discrete position (u,v,w), and by a coordinate (x,y,z), and is efficiently compressed using an embedded 3D wavelet-based multi-resolution intra-band coding algorithm.

The MeshGrid representation can be efficiently exploited for QoS since it allows for three types of scalability in both view-dependent and view-independent scenarios, including: (1) resolution scalability, i.e. the adaptation of the number of transmitted vertices, (2) shape precision, i.e. the adaptive reconstruction of the reference-grid positions, and (3) vertex position scalability, i.e. the change of the precision of known vertex positions with respect to the reference-grid.

In addition to the vertex-based animation, typical for IndexedFaceSet meshes, the MeshGrid representation also supports specific animation capabilities, such as (1) rippling effects by changing the position of the vertices relative to corresponding reference-grid points, and (2) reshaping the regular reference-grid and its attached vertices, which can be done on a hierarchical or non-hierarchical basis; When reshaping or animating the reference-grid on a hierarchical basis, any changes applied to the reference-grid points, corresponding to a lower resolution of the mesh, will be propagated to the reference-grid points, corresponding to the higher resolutions of the mesh, while applying a similar deformation to the reference-grid points corresponding to a higher resolution of the mesh will only have a local impact. For the vertex-based animation, each vertex will retrieve its 3D coordinates from the coordinates of the grid points it is attached to and from the specified offset.

Since the MeshGrid representation is upwards compatible with the IndexedFaceSet, it allows specifying textures, color, normal vector, descriptions.

Building a MeshGrid representation of the surface of an object requires two steps (see section 2.3.4.2): (1) design an appropriate reference-grid according to the topology of the object, and (2) build the connectivity-wireframe for the specified reference-grid.

This document explains three methods to obtain a MeshGrid representation of an object: (1) for an object expressed as a scalar field (e.g. discrete 3D data) or scalar functions (e.g. implicit surfaces) one can apply a surface extraction method, called TriScan described in section 2.3.4.2, (2) IndexedFaceSet representations can be converted, by remeshing using a similar TRISCAN surface extraction approach as described in section 2.3.4.7, and (3) quadrilateral meshes can be imported in a lossless manner by deforming the reference grid towards the surface mesh. In general, the remeshing of IndexedFaceSet representations can’t be done in a lossless way.

Also, some post-processing algorithms could be envisaged to obtain the corresponding connectivity-wireframe and reference-grid descriptions from other surface extraction methods, such as the Marching Cubes algorithm [62] applicable to objects represented as scalar fields, or the “Polygonization of Implicit Surfaces” algorithm [24], which directly target the polygonal representation of the surface instead of its connectivity.


2.3.4.2 Designing the MeshGrid components: reference-grid and connectivity-wireframe

In the MeshGrid representation, the reference-grid is defined by the intersection points between the 3 sets of reference-surfaces. The discrete position (u,v,w) of each reference-grid point represents the indices of the reference-surfaces intersecting in that point, while the coordinate (x,y,z) of the reference-grid point is equal to the coordinate of the computed intersection point. This concept is shown in Figure 23 for a particular case of reference-surfaces, i.e. all reference-surfaces are planar, the reference-surfaces from one set are parallel to each other, and the 3 sets of reference-surfaces are orthogonal to each other. The reference-surfaces from each set are displayed in a different color in Figure 23.

(a) (b) (c)

Figure 23: A regular reference grid visualized as 3 sets of planar reference-surfaces in (a), as a 3D grid of points in (b), and as both representations in (c).In the general case the 3 sets of reference-surfaces don’t’ have to satisfy the orthogonality and parallelism constraints and may consist of curved reference-surfaces. Based on the fact that each reference-surface has a certain ordering inside the set, the following constraints are imposed: (1) the reference-surfaces from one set keep their ordering at the surface of the object, (2) they do not self-intersect, and (3) the reference-surfaces from one set intersect the reference-surfaces from the other two sets.

Although any type of wireframe, be it triangular, quadrilateral, or polygonal could eventually be attached to the reference-grid, the connectivity-wireframe must have particular connectivity properties.

The connectivity-wireframe addresses three issues: (1) storing the connectivity information between the vertices of the mesh, (2) the possibility to derive the normal vector of the surface by computing the cross product between the connectivity vectors (explained in this section) and (3) deriving the discrete position of a vertex vN from the discrete position of another vertex vP connected to vN if for both vertices the discrete border direction is known. Both aspects (1) and (3) are required for encoding the connectivity-wireframe in a similar way as the Freeman chain-code, but extended to 3D, as explained in section 2.3.4.4.1. Deriving the polygonal representation from the connectivity-wireframe (see section 2.3.4.2.1) needs (1) the connectivity information between the vertices, and (2) for some connectivity cases, the discrete border direction of the vertices.

The Freeman chain code is a typical encoding method for the discrete space. In our explanations, the discrete space is represented by the discrete (u,v,w) positions of the reference-grid.

The relation between vertices of the connectivity-wireframe and reference-grid positions is very tight since each vertex v from the connectivity-wireframe is located on the intersection line (curve) l between two reference-surfaces S1 and S2, belonging to two different sets. This line (curve) l is called a reference-grid line, and passes implicitly through the series of reference-grid points common to both reference-surfaces S1 and S2. The exact position of each vertex v on the reference-grid line l, respectively the coordinates (x,y,z) of each vertex v, are given by the intersection point of line l with the object’s surface. A line l might intersect the surface of the object at different positions, but in each intersection

69

point another vertex will exist. For a closed surface, the number of intersection points between a grid line and the surface is even. The orientation of the surface along the reference-grid line l in one of the directions {u,v,w} at a vertex (i.e. an intersection point between the reference-grid line l and the object’s surface) is called the discrete border direction. As illustrated in Figure 24, there are six discrete border-directions named: Front (positive u orientation), Back (negative u orientation), Left (positive v orientation), Right (negative v orientation), Up (positive w orientation) and Down (negative w orientation).

The connectivity-wireframe stores besides the vertices the connectivity information, called connectivity vectors, between these vertices. A connectivity vector has three constraints: (1) given a starting vertex VP, a connectivity vector -L- from VP will be located inside one of the reference-surfaces S1 or S2 passing through VP, (2) a connectivity-vector will connect two vertices VP and VN that are lying the closest to each other at the same side of the object’s surface, (3) the orientation of a connectivity vector inside reference-surface S, e.g. from vertex VP to VN, is defined by a counter-clockwise (CCW) or a circular clockwise (CW) scanning direction around a central point inside the object. The CCW scanning direction is imposed when the vertices VP and VN linked by the connectivity vector, are located on the external surface of the object, respectively the CW scanning direction is imposed when the vertices VP and VN are located on the internal surface of the object. This aspect is explained in section 2.3.4.5, and illustrated in Figure 65 and Figure 66.

Since a vertex V is located on a reference-grid line, which in its turn is defined by the intersection of two reference-surfaces S1 and S2 from different sets as previously explained, there will be two connectivity-vectors going from vertex V to other vertices (outgoing), one connectivity vector being located inside reference-surface S1, while the other one being located inside reference-surface S2. Similarly, vertex V will be referred to by two other vertices, and there will be two connectivity vectors pointing to vertex V (incoming), one connectivity vector being located inside reference-surface S1, while the other one lying inside reference-surface S2. The two incoming connectivity vectors are labeled -LP1- and -LP2-, while the two outgoing connectivity vectors are labeled -LN1- and -LN2-. The -LP1- and -LN1- (respectively -LP2- and -LN2-) connectivity-vectors are oriented in the same direction and belong to a curve C1 (respectively C2). Curves C1 and C2 belong to reference-surfaces from a different set, i.e. S1 and S2, and intersect each other in node V.

The 4 neighboring vertices with V are named P1, P2, N1 and N2, and are connected with V via -LP1-, -LP2-, -LN1- and -LN2-. Node N1 (respectively N2) follows node V (is the next one) on curve C1 (respectively C2), and node P1 (respectively P2) precedes node V (is the previous one) in curve C1 (respectively C2) for the imposed scanning orientation.

Figure 24 shows the basic concepts of the MeshGrid representation: (1) the object is shown as a cube, (2) the reference-grid point, consisting of only one grid point in this case, is located at the intersection position between three reference-surfaces, one reference-surface from each set, passing through the middle of the object, and colored in green, cyan and yellow respectively, (3) the reference-grid lines, three in this case, shown in green, cyan and yellow respectively, (4) the vertices shown as white dots, located at the intersection positions between a reference-grid line and the surface of the object, and (5) the connectivity vectors illustrated by the thick red (-L1-), and blue arrows (-L2-) attached to the vertices. The example from Figure 24 is a very simple illustration of the MeshGrid concept, since the object is only intersected by three reference-surfaces. Notice that the indices (1, 2) of the connectivity vectors (-L1-) and (-L2-), given their orientation, have been chosen in such a way that the cross product defined by equation returns the normal vector oriented outwards the surface of the object at the position of vertex V.

N = (LN1 – LP1)x(LN2 – LP2)

According to equation , Figure 24 illustrates the correct identification of the -L1- and -L2- connectivity vectors attached to vertices located on one of the 6 discrete border faces, given the imposed scanning orientation for a connectivity path C: i.e. CCW for external, respectively CW for internal surfaces. More details about the scanning orientation are given in section 2.3.4.5. In order to satisfy equation the connectivity vectors of two consecutive vertices may change their ordering (L1 L2 and L2 L1). This implies that when walking from a node V to one of its neighbors Ni with i={1, 2} via link -LNi- (respectively to Pi via -LPi-), it is possible to return from that position back to node V on one of the -LP1- or -LP2- links (respectively -LN1- or -LN2-) depending whether their indices have changed or not. The index of the connectivity vector between two vertices is determined by the position of the border-face, the two neighboring vertices belong to. The following table shows all the possible connectivity cases between vertices, located on each of the 6 border faces, and its neighboring vertices, as illustrated in Figure 24. The rows and the columns headers specify the position of the


border face the node belongs to. The column header refers to a node V and its two outgoing links (-LN1- and -LN2-), while the rows indicate the incoming links (-LP1- and -LP2-) for a node that is neighboring with V. The links use the same coloring scheme as displayed in Figure 24 (-L1- in red color and -L2- in blue color). As long as two consecutive vertices stay on the same border face, illustrated by the cases on the diagonal of the table, the connectivity vectors keep their indices, otherwise they may change. The exclamation marks indicate cases that are not possible.

Back Front Left Right Down Up-LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 -

Back -LP1 - -LP2 - ! ! ! -LP1 - -LP1 - ! -LP2 - ! ! -LP2 -Front ! ! -LP1 - -LP2 - ! -LP2 - -LP2 - ! -LP1 - ! ! -LP1 -Left -LP2 - ! ! -LP2 - -LP1 - -LP2 - ! ! ! -LP1 - -LP1 - !

Right -LP1 - ! ! -LP1 - ! ! -LP1 - -LP2 - ! -LP2 - -LP2 - !Down ! -LP1 - -LP1 - ! -LP2 - ! ! -LP2 - -LP1 - -LP2 - ! !

Up ! -LP2 - -LP2 - ! -LP1 - ! ! -LP1 - ! ! -LP1 - -LP2 -

Table 4: The connectivity vectors between adjacent vertices in the connectivity-wireframe, and the way they change the indices.

Figure 24: The 6 discrete orientations of the border, labeled as: Front, Back, Left, Right, Up, Down. Any vertex belonging to the surface is located relative to a discrete position inside the body in one of these 6 orientations. Each vertex has two outgoing connectivity links colored in red and blue towards other vertices. The orientation of the connectivity links is given by the direction of the border-tracking engine. The choice of the colored connectivity links is chosen in such a way that the cross product between the red and the blue arrow resulting in a normal vector oriented outwards to the surface.From the previous explanations one can conclude that every vertex V from the connectivity-wireframe knows its coordinates (x,y,z), the discrete position (u,v,w), the discrete border direction, and the two incoming (-LP1-, -LP2-) respectively two outgoing (-LN1- and -LN2-) connectivity vectors from, respectively to, two other vertices. The regular

71

structure of the connectivity-wireframe in the MeshGrid representation, i.e. each vertex connected to four other vertices, is illustrated for a discrete sphere in Figure 25(a), and for a continuous sphere in Figure 25(b). The vertices are displayed as yellow dots, while the connectivity vectors, colored in red, green and blue, connect the vertices. Starting from a vertex Vs, the path obtained by following the connectivity vectors always in the same direction to the neighboring vertices, will finally lead back to the vertex Vs if the surface of the object is closed. An example of an open surface is shown in Figure 72. The coloring for the connectivity vectors, respectively the path, is the same as the color of the reference-surface (see Figure 23) containing them. For the continuous sphere from Figure 25(b), several vertices will overlap if more than two reference-surfaces intersect in the same position, and can be eliminated in a post-processing step. The overlapping vertices might give the wrong idea that a vertex is connected with more than four vertices.

(a) (b)

Figure 25: MeshGrid representations in (a) of a discrete and in (b) of a continuous sphere.Another specific property of the connectivity-wireframe is that the border direction of two consecutive vertices may have only three different orientations. As illustrated in Figure 26, with respect to the scanning orientation, the border direction between two consecutive vertices can be the same – case V1 and V2, it can be rotated 90° CCW – case V3

and V4, or it can be rotated 90° CW – case V5 and V6. As described in section 2.3.4.4.1, the coding of the connectivity-wireframe using a new type of Freeman chain code extended to 3D, encodes the relative discrete border direction between consecutive vertices. Notice that there exists a direct relation between the discrete position (u,v,w) of two consecutive vertices and their discrete border direction. This aspect is exploited when reconstructing the connectivity-wireframe from the bit-stream and attaching it to the reference-grid (see section 2.3.4.4.1).


Figure 26: MeshGrid representation of a discrete object showing the relative positions between two connected vertices.

2.3.4.2.1 Rules for deriving the surface primitives from the connectivity-wireframe.

Deriving the polygonal representation from the connectivity-wireframe is necessary for display purposes.

In order to unambiguously identify the shape and the orientation of a patch P from the connectivity-wireframe, such that the right surface primitive can be inserted in the opening, a set of connectivity rules has been designed. A patch corresponds to the shortest circuit formed by navigating from one node to one of its neighbors following the connectivity vectors.

The polygonal representation consists of the union of the surface primitives, one surface primitive for each patch P from the connectivity-wireframe.

The connectivity rules define connectivity cases that allow identifying the polygonal surface, fitting a connectivity patch inside the connectivity-wireframe.

The following notations have been used to describe the connectivity cases and to define the connectivity rules:

V a starting vertex when describing the connectivity

N1 the next vertex reachable from V via -LN1-

N2 the next vertex reachable from V via -LN2-

P1 the previous vertex reachable from V via -LP1-

P2 the previous vertex reachable from V via -LP2-

Figure 27 shows images for the connectivity cases corresponding to the triangle primitive. There are 4 distinct connectivity cases, as shown in Figure 27 (a), (b), (c) and (d), which are identical for both rows of images. The first row, rendering the discrete model, shows the different cases that correspond to concavities of the shape. The second row, rendering the surface extracted from the discrete model, illustrates the cases found for local convexities of the shape. Table 5 gives the connectivity rules for the triangle primitive.

73

(a) (b) (c) (d)

Figure 27: Images for the connectivity cases corresponding to the triangle primitive.

Triangle Comments

(a) 1 2 2N N N= Detected by the pentagon case (b)

(b) 2 1 1N N N= Detected by the pentagon case (a)

(c) 1 1 2N N P= or 1 1 1N N N V= Detected three times. The first time the rule is matched, a flag is set for all three vertices of the triangle, to avoid duplicates.

(d) 2 2 1N N P= or 2 2 2N N N V= Detected three times. The first time the rule is matched, a flag is set for all three vertices of the triangle, to avoid duplicates.

Table 5: Connectivity cases for the triangle primitiveFigure 28 shows images for the connectivity cases corresponding to the rectangle primitive. There are 3 distinct connectivity cases, as shown in Figure 28 (a), (b) and (c). The square shapes are obtained by the case from (a), while the rectangles are generated by any of the three cases. Table 6 gives the connectivity rules for the rectangle primitive.


Discrete model (a) (b) (c)

Figure 28: Images for the connectivity cases corresponding to the rectangle primitive.

Rectangle Comments

(a), (b), (c)

1 2 2 1N N N N= Generates both square (all the vertices are located in the same reference-surface) and rectangles.

(b) 1 2 2 2N N N P= or 1 2 2 2N N N N= Detected by hexagon case (e)

(c) 2 1 1 1N N N P= or 2 1 1 1N N N N= Detected by hexagon case (f)

Table 6: Connectivity cases for the rectangle primitiveFigure 29 shows images for the connectivity cases corresponding to the pentagon primitive. There are 4 distinct connectivity cases, as shown in Figure 29 (a), (b), (c) and (d), which are identical for both rows of images. The first column shows the discrete model used for each row, followed by the renderings of the surface extracted from the discrete model. The cases corresponding to the first row, correspond to convexities in the shape, while the cases from the second row illustrate the cases found for local concavities in the shape. Table 7 gives the connectivity rules for the pentagon primitive.

Discrete model (a) (b) (c) (d)

Figure 29: Images for the connectivity cases corresponding to the pentagon primitive.

75

Pentagons Comments

(a) 1 1 1 2 1N PN N N= if !1 1 1 1N PN N= Detects triangle 2 1 1N N N=

(b) 2 2 2 1 2N PN N N= if !2 2 2 2N PN N= Detects triangle 1 2 2N N N=

(c) 1 2 2 2 1N N N N N= Detected by hexagon case (a)

(d) 2 1 1 1 2N N N N N= Detected by hexagon case (b)

Table 7: Connectivity cases for the pentagon primitive.Figure 30 shows images for the connectivity cases corresponding to the planar hexagon primitive. There are two distinct cases, as shown in Figure 30 (a) and (b), the same for both border-tracking algorithms.

Figure 31 shows images for the connectivity cases corresponding to the arbitrary hexagon primitive. There are two distinct cases, the same for both border-tracking algorithms. Following table gives the connectivity rules for the hexagon primitive.

Figure 32 shows images for the connectivity cases corresponding to the folded hexagon primitive. There are 4 distinct connectivity cases, as shown in Figure 32 (e), (f), (g) and (h), which are identical for both rows of images. The first column shows the discrete model used for each row, followed by the renderings of the surface extracted from the discrete model. The cases corresponding to the first row correspond to the 90° border-tracking engine, while the cases from the second row illustrate the cases found for the 45° border-tracking engine.

Discrete model (a) (b)

Figure 30: Images for the connectivity cases corresponding to the planar hexagon primitive.

Discrete model (c) and (d) (b)


Figure 31: Images for the connectivity cases corresponding to the arbitrary hexagon primitive.

Discrete model (e) (f) (g) (h)

Figure 32: Images for the connectivity cases corresponding to the folded hexagon primitive.

Hexagons Comments

(a) 1 1 1 2 2 2N PN N PN= Detected three times. The first time the rule is matched, a flag is set for all three vertices of the triangle, to avoid duplicates.

(b) 1 2 2 2 1 1N N N N N N=

(c) 2 2 1 1 1 2PN N PN N=

(d) 1 2 1 2 1 2N N P N N P= if !1 2 1 2N N P N= and !2 1 2 1N N P N=

(e) 1 2 2 2 2 2N N N N PN= if !2 2 2 2N PN N= Detects rectangle 1 2 2 2N N N N=

(f) 2 1 1 1 1 1N N N N PN= if !1 1 1 1N PN N= Detects rectangle 2 1 1 1N N N N=

(g) 1 2 2 2 2 1N N N N N N=

(h) 2 1 1 1 1 2N N N N N N=

Table 8: Connectivity cases for the hexagon primitive.Figure 33 shows images for the connectivity cases corresponding to the heptagon primitive. There are 8 distinct connectivity cases, as shown in Figure 33 (a)-(h), which are the same for both border-tracking algorithms. The discrete model, displayed in the first column, is different for each of the border-tracking engines. The cases for the 90° border-tracking engine are shown in the first row, while the second row shows the same connectivity cases but for the 45° border-tracking engine. Following table gives the connectivity rules for the heptagon primitive.

77

Discrete model (a) (b) (c) (d)

(e) (f) (g) (h)

Figure 33: Images for the connectivity cases corresponding to heptagon primitive.

Heptagons Comments

(a) 1 2 2 2 1 2 2N N N N N PN= if !2 1 2 2 2 1N N PN N N= and !2 1 2 2 2 2N N PN N N=

Detects pentagon 1 2 2 2 1N N N N N=

(b) 2 1 1 1 2 1 1N N N N N PN= if !1 2 1 1 1 2N N PN N N= and !1 2 1 1 1 1N N PN N N=

Detects pentagon 2 1 1 1 2N N N N N=

(c) 1 2 1 1 2 2 1N N PN N N N= if !1 2 1 1 2 1 2 2N N PN N N N N= and !1 2 1 1 2 1 1 2N N PN N N N N=

(d) 2 1 2 2 1 1 2N N PN N N N= if !2 1 2 2 1 2 1 1N N PN N N N N= and


(e) 2 2 2 1 1 2 2N PN N N N N= if !2 2 2 1 2 1N PN N N N= Detects pentagon 1 2 2 2 1N N N N N=

(f) 1 1 1 2 2 1 1N PN N N N N= if 1 1 1 2 1 2N PN N N N= Detects pentagon 2 1 1 1 2N N N N N=

(g) 2 1 1 1 1 2 2N N N N N N N= if !1 2 2 1 1N N N N P=

(h) 1 2 2 2 2 1 1N N N N N N N= if !2 1 1 2 2N N N N P=

Table 9: Connectivity cases for the heptagon primitive.

2.3.4.2.2 Building the MeshGrid

As explained in section 2.3.4.2 the connectivity-wireframe of an object is based on a specified reference system, i.e. the reference-surfaces and the corresponding reference-grid. The connectivity-wireframe can be used standalone since for each vertex both the coordinates and the connectivity are known. For the purpose of having a compact stream and flexible animation possibilities, one can separate the surface description of the object into two parts linked together: i.e. (1) the connectivity between the vertices, and (2) the spatial distribution of the vertices. We call this dual representation the MeshGrid representation.

Each reference-grid point P is identified by a discrete position (u,v,w), and by a coordinate (x,y,z). When attaching the connectivity-wireframe to the reference-grid, we actually attach each vertex to a reference-grid point that is identified by the discrete position (u,v,w) of the vertex. More specific, a vertex V is located on the reference-grid line segment between two grid positions P1 and P2, vertex V being attached either to P1 or P2. Grid positions P1 and P2 are the closest to vertex V according to the discrete border direction as explained in section 2.3.4.2. One of the reference-grid positions P1 (P2) lies inside the object, while the other reference-grid positions P2 (P1), lies outside the object. Reference-grid positions with similar properties as P1 and P2 are called border reference-grid positions since the surface of the object passes between those positions. For the MeshGrid representation we have chosen to attach the vertices to border reference-grid positions of type P1, i.e. reference-grid positions located inside the object. Note also, that it is allowed to attach several vertices to the same reference-grid point.

Since the reference-grid points already have 3D coordinates, we do not need to keep the coordinates of the vertices. The coordinate V(x,y,z) of a vertex can be computed as the sum between the coordinate of the reference-grid point P1

it is attached to, and an offset vector off, as shown in equation . The offset vector is equal to the product between the vector 1 2P P

uuurand the scalar offset, as given by equation , while the scalar offset is computed with equation .

1 2 1( ( , , ) ( , , )) /( ( , , ) ( , , ))offset V x y z P x y z P x y z P x y z

2 1( , , ) ( ( , , ) ( , , ))off x y z P x y z P x y z offset

1( , , ) ( , , )V x y z P x y z off

Since vertex V is lying on the line segment in between the reference-grid points P1 and P2, equation yields scalar offsets in the range [0, 1).

Hence, one can conclude that in the MeshGrid representation, the description of each vertex from the connectivity-wireframe needs to store besides the connectivity-information to other vertices, the discrete position (u,v,w) of its corresponding reference-grid point, and a scalar offset relative to this reference-grid point. In addition a vertex description may contain the direction of the border or other attributes as explained in section 2.3.4.2. Note that defining the offset for the vertices as a relative value instead of an absolute one, has the advantage that the coordinates of the

79

vertices can still be recomputed from the reference-grid coordinates after having applied arbitrary deformations to the reference-grid.

2.3.4.3 The hierarchical multi-resolution MeshGrid representation

Building several MeshGrid representations with different resolutions of the same object is done by diversifying the reference-system, e.g. changing the number and/or position of the reference-surfaces. The reference-system can be tuned using two parameters: (1) the number of reference-surfaces intersecting the object, and (2) the distribution of the reference-surface. Both parameters have similar weighting factors on the precision of the generated surface. A reference-system consisting of many reference-surfaces that are badly distributed may generate a MeshGrid representation whose polygonal representation is less precise, and possibly consisting of more triangles, than in case of a well distributed reference-system (e.g. the density of the reference-surfaces should be higher in areas where the curvature of the object’s surface is important). Since the method of generating the connectivity-wireframe (see section 2.3.4.5) does an implicit sampling at the reference-grid positions, making an analogy with the Nyquist’s sampling theory implies that any changes in shape that are smaller than the local step, will not be defined in the connectivity-wireframe. This is the reason why the reference-system has to be chosen carefully when generating low-resolution MeshGrid representations.

In order to generate a hierarchical MeshGrid representation of an object we need to enforce that vertices found in a lower level are available in all higher levels that follow. However, each level will alter the connectivity between the existing vertices. An example of a hierarchical MeshGrid is shown in Figure 34, while Figure 35 illustrates the reference-surfaces corresponding to each resolution level of the hierarchical MeshGrid.

In the hierarchical connectivity-wireframe, vertex lnv is connected on level l with vertex l

mv via the blue colored line.

This connection is replaced in level l+1 by another line (green color) to vertex 1lpv , and replaced again in level l+2 by

the by the red line that connects it to vertex 2lqv . Note that the level of a vertex indicates the position in the hierarchy

when it first appears.

The hierarchical MeshGrid model imposes the following constraint on the reference-system: the reference-system of any level is a super-set of the reference-system of the lower levels. Figure 35 shows the changes in the reference-system when generating a hierarchical MeshGrid with 3 levels: the first level in Figure 35(a) consists of 3 reference-surfaces colored in blue, the second level in Figure 35(b) has in addition other reference-surfaces (colored in green), and the third level in Figure 35(c) contains besides the reference-surfaces of the previous levels also other reference-surfaces colored in red. Each higher level will add an extra reference-surface in between two existing reference-surfaces from the previous levels, while keeping the reference-surfaces of the previous levels.

Figure 34: The hierarchical connectivity-wireframe of the MeshGrid representation.


(a) (b) (c)

Figure 35: The changes in the reference-system when generating a hierarchical MeshGrid with 3 levels: the first level in (a) consists of 3 reference-surfaces shown in blue color, the second level in (b) has in addition other reference-surfaces that are shown in green color, and the third level in (c) contains besides the reference-surfaces of the previous levels other reference-surfaces shown in red color.

2.3.4.3.1 Building the hierarchical connectivity-wireframe

A hierarchical connectivity-wireframe can be obtained from the single-resolution connectivity-wireframe corresponding to the highest resolution MeshGrid representation of the surface of the object. This approach is called the top-down method. A similar bottom-up method is applied when decoding a multi-resolution MeshGrid bit-stream and increasing the resolution of the connectivity-wireframe each time a new resolution level is decoded. Notice that the top-down or bottom-up approaches only make sense if the reference-system has a hierarchical structure as illustrated in Figure 35.

Any lower resolution connectivity-wireframe contains only a part of the vertices of any higher connectivity-wireframe, and the connectivity between its vertices will be altered with respect to any higher resolution connectivity-wireframe, as shown in Figure 34.

For the top-down approach, building the hierarchical connectivity-wireframe with L resolution levels consists of generating all L single-resolution connectivity-wireframes. To build a single-resolution connectivity-wireframe Wl of level l a single-resolution connectivity-wireframe Wl+n of at least level (l+1), and the reference-system for both Wl and Wl+n

connectivity-wireframes, are required.

The first phase of the top-down approach is to remove those vertices from the connectivity-wireframe Wl+n that are not intersected by two reference-surfaces of the target connectivity-wireframe Wl. For any vertex V removed from connectivity-wireframe Wl+n the connectivity vectors are restored by bypassing vertex V. If the 4 neighboring vertices with V are named P1, P2, N1 and N2, and are connected with V via -LP1-, -LP2-, -LN1- and -LN2-, that bypassing vertex V means that vertex P1 will be connected with vertex N1, and vertex P2 will be connected with vertex N2.

The removal of vertices during the first phase usually introduces inconsistencies according to the constraints imposed upon the connectivity-wireframe (see Figure 24), i.e. two connected vertices cannot have opposite discrete border directions (i.e. cannot lie on opposite border faces). They can only have the same discrete border direction, or discrete borders oriented perpendicular to each other. The second phase of the top-down approach removes the inconsistencies of the connectivity-wireframe Wl.

The hierarchical connectivity-wireframe is the result of merging the single-resolution connectivity-wireframes Wl

obtained for each level l. During the merging procedure all the vertices V from lower resolution levels will be found in the higher resolution levels, and all vertices V will store in a hierarchical way the connectivity corresponding to the higher resolution levels as illustrated in Figure 34.

81

2.3.4.3.2 Hierarchical multi-resolution MeshGrid Features

The MeshGrid representation allows three types of scalability that can be exploited simultaneously in progressive transmission schemes with variable detail that depend on the bandwidth of the connection and the capabilities of the terminal equipment, in both view-dependent and view-independent scenarios. The view-dependent scenario is implemented by means of regions-of-interest (ROIs). A ROI is defined as a part of the reference-grid. ROI based coding implies subdividing the entire MeshGrid in smaller parts and encoding the surface description and reference-grid description of each of these parts separately. The different features of the hierarchical multi-resolution MeshGrid are discussed next.

2.3.4.3.2.1 Scalability Features

The MeshGrid multi-resolution representation can be efficiently exploited for “Quality of Service” (QoS) since it allows for three types of scalability in both view-dependent and view-independent scenarios, including: (1) mesh resolution scalability, i.e. adapting the number of transmitted vertices on a ROI basis in view-dependent mode or globally otherwise, and (2) shape precision i.e. adaptive reconstruction of the reference-grid positions on a ROI basis in view-dependent mode or globally otherwise, (3) vertex position scalability, i.e. increasing the precision of known vertex positions with relation to the reference-grid (the value of the offset) on a ROI basis in view-dependent mode or globally otherwise.

2.3.4.3.2.1.1 The mesh resolution scalability

The mesh resolution scalability is illustrated in Figure 36 for three different resolutions of an object. Different mesh resolutions represent different connectivity-wireframe descriptions of the same object. The first row of images show a shaded surface of its polygonal representation with the outline of the connectivity-wireframe drawn on top of the shaded surface, while the second row of images display the object as a wireframe of the polygonized representation. The mesh resolution is increasing from Figure 36 (a) to (c). Table 10 shows the correspondence between the number of vertices, the number of triangles and the stream size of the single-resolutions respectively multi-resolution MeshGrid representation for the object from Figure 36; the multi-resolution version containing all three single-resolution descriptions.

Another example of a multi-resolution MeshGrid is illustrated for two objects (Figure 74 and Figure 75) originally represented as single-resolution IndexedFaceSet models that have been remeshed and represented in the MeshGrid format. The object from Figure 74, a 3D eight model, has been converted to a multi-resolution MeshGrid with 4 mesh resolutions, i.e. a hierarchical connectivity-wireframe consisting of 4 resolution levels and the corresponding reference-grid. The original object represented as an IndexedFaceSet is shown Figure 74(a), while Figure 74(b)-(e) illustrate increasing resolutions of the MeshGrid representations of the object. The first row of images display the object as a wireframe, while the second row of images display the same object as a shaded surface of its polygonal representation obtained from the MeshGrid representation. In a similar way, the bone model from Figure 75(a) represented as an IndexedFaceSet, has been converted to a multi-resolution MeshGrid, as shown for increasing mesh resolution levels in Figure 75(b)-(e). The images display the shaded surface of the polygonal representation of the MeshGrid object.

A multi-resolution MeshGrid surface description obtained from a single-resolution quadrilateral mesh is illustrated in Figure 72. The different mesh resolutions are displayed as a shaded surface of the polygonal representation with the outline of the connectivity-wireframe, corresponding to that resolution level, drawn on top of the shaded surface (see Figure 72 (d)-(f)).

Notice that the multi-resolution MeshGrid representation of a connectivity-wireframe allows, in different resolutions of the same object, topological changes, visible especially when displaying the polygonal representation of the connectivity-wireframe. As illustrated in Figure 74 for increasing resolutions of the eight model, the lowest mesh resolution level (Figure 74(b)) does not have any holes. They only appear at the second resolution level (Figure 74(c)), and continue to exist for the higher resolution levels.


Table 10: The number of vertices, triangles and the size of the stream for different resolutions of the MeshGrid object from Figure 36.

Figure 36: Scalability in resolution

2.3.4.3.2.1.2 The scalability in shape precision

The scalability in shape precision is a particular feature of the MeshGrid representation and is supported by the reference-grid. The reference-grid contains the description about the volume of the object. The connectivity-wireframe standalone (not as part of the MeshGrid representation) contains the vertices, the connectivity between those vertices and at least the 3D coordinates of those vertices. When generating the MeshGrid representation, i.e. attaching the connectivity-wireframe to the reference-grid, the coordinates of the vertices from the connectivity-wireframe are obtained from the coordinates of the reference-grid points and the offsets associated to the vertices, as explained in section 2.3.4.2.2. This implies that the 3D distribution of the connectivity-wireframe for a certain mesh resolution level, when keeping its connectivities unchanged, but allowing the position of the connected vertices to vary, is given by the distribution of the reference-grid points to which vertices are attached to.

The minimal description for a reference-grid consists of the coordinates of the eight corners of the reference-grid that in general define a non-regular deformed box. For this minimal description one obtains a uniform distributed reference-grid, by using trilinear interpolation based on the coordinate values of the corners. In case the 8 corners define a regular box (i.e. faces are two by two parallel and perpendicular) it suffices to apply three times (once for each coordinate) a linear interpolation to distribute the reference-grid points in a uniform way. An example illustrating this aspect is shown in Figure 38. The uniform distributed reference-grid is displayed in Figure 38(c) and the shaded polygonal representation of the connectivity-wireframe for the uniform distributed reference-grid is shown in Figure38(a). A reference-grid with non-uniformly distributed grid-points is displayed in Figure 38(d), and the corresponding shaded polygonal representation in Figure 38(b). Notice that the connectivity between the vertices did not change.

83

This scalability aspect can be exploited for progressively refining non-uniform distributed reference-grid points. Such an example is shown in Figure 37 for a MeshGrid object. The coordinates of the reference-grid points are progressively refined from left, Figure 37(a), to right, Figure 37(d). The first row of images show a shaded surface of its polygonal representation with the outline of the connectivity-wireframe drawn on top of the shaded surface, while the second row of images display the object as a wireframe of the polygonized representation.

Figure 37: Scalability in shape precision

1.2111689663129606414

%Size deformedSizeTriangleVertex

1.2111689663129606414

%Size deformedSizeTriangleVertex

Figure 38: Progressively refining the reference-grid


2.3.4.3.2.1.3 The vertex position scalability

The vertex position scalability is a particular feature of the MeshGrid representation and is possible because the connectivity-wireframe vertices have one degree of freedom relative to the reference-grid points, i.e. the offset value that can vary between [0,1). This feature can either be exploited for animation, as discussed in section 2.3.4.3.2.3, or as a scalability option. As explained in section 2.3.4.2.2, the offset is a relative value.

When generating the stream for a MeshGrid surface description, the vertices’ offset values are quantized on a specified number of bits, and are encoded bit-plane by bit-plane. The number of bit-planes is equal to the number of quantization bits, and a bit-plane contains for each vertex offset the value of the quantisation bit corresponding to that bit-plane. The topic is discussed in detail in section . When receiving the stream, the vertices offset values are refined, using the progressive quantization approximation scheme, bit-plane by bit-plane. The default offset value is 0.5, which is set when the offset has no value specified yet. The progressive refinement of the vertices offset values is illustrated in Figure 39 for a sphere. The first row of images show a shaded surface of its polygonal representation of the sphere with the outline of the connectivity-wireframe drawn on top of the shaded surface, while the second row of images display the sphere as a wireframe of the polygonized representation. The result for a default offset (0.5) is shown in Figure 39(a), notice the local effect due to discretisation. The refinement of the vertices offset initially quantized on 5 bits, is illustrated for an increasing number of refinement bits from left, Figure 39(b), to right, Figure 39(d).

0 bits 1 bits 2 bits 5 bits

Figure 39: Vertex position scalability

2.3.4.3.2.2 Bit-stream polyvalence

A view-dependent multi-resolution stream is versatile since it allows in both view-dependent and view-independent scenarios, single resolution or multi-resolution decoding. The stream description of the MeshGrid representation is discussed in section 2.3.4.4.

For the same multi-resolution stream, the view-independent scenario consists of decoding all the view-dependent information for a certain resolution level, while in the view-dependent scenario, one can retrieve the description of parts

85

from a certain resolution level and decode them. In this case, different parts of the recovered MeshGrid may have different resolution levels. From the same multi-resolution stream, single resolution decoding relies on locating the appropriate resolution level of the MeshGrid description in the stream and decoding it. In the multi-resolution scenario the decoding will start at a certain resolution level (it does not have to be the first resolution level) and continue from that level on till a certain higher resolution level. This approach is very different from classical multi-resolution representations that typically need the decoding of the lower resolution versions of a mesh, before being able of enhancing some portions of it by incrementally decoding additional parts of the bitstream.

2.3.4.3.2.3 Animation flexibility

The MeshGrid representation is very flexible for animation purposes, since it allows in addition to the vertex-based animation typical for IndexedFaceSet represented objects, specific animation possibilities, such as (1) rippling effects by changing the position of the vertices relative to corresponding reference-grid points and (2) reshaping the regular reference-grid. The latter form of animation can be done on a hierarchical multi-resolution basis: i.e. deforming the reference-grid for a lower resolution mesh, will propagate the action to the higher levels, while applying a similar deformation at the higher resolution levels will only have a local impact. The vertices of the wireframe will be updated, each time the grid point, it is attached to, moves.

2.3.4.3.2.3.1 Vertex-based animation

Vertex animation applies to the vertices of the connectivity-wireframe. This type of animation requires that each vertex retrieves its initial coordinates from the grid and its associated offset, and stores, updates these coordinates during the animation, without further referencing the grid.

2.3.4.3.2.3.2 Rippling effects, changing the offsets

Changing the vertices’ offsets allows limited movements, oscillations (rippling effects) around the reference-grid points they are attached to. Such an example is given in Figure 40 (the sphere). Both images show different frames during the animation. This type of animation is easier to perform if all the vertices’ offsets have the same value, preferably equal to 0.5. Obtaining a constant offset of 0.5 for all the vertices, as illustrated in Figure 72 and Figure 73, is feasible by positioning the boundary grid positions in such a way that the vertices lie at half the distance in between two boundary grid positions.

If those offsets are not equal to 0.5 everywhere than one may run into problems e.g.: for an overall vertex movement of 0.3, vertices already having an offset of 0.8 will get an offset equal to 1.1, exceeding the theoretical accepted range of [0, 1), which might give artifacts.


Figure 40: Shows two frames from a sequence of animations of a sphere: rippling effects.

2.3.4.3.2.3.3 Animation of the reference-grid points

Animating or deforming the reference-grid points will affect the global shape of the object. This type of animation will not change the connectivity between the vertices; it will only change the coordinates of the vertices when they are updated from the corresponding reference-grid positions. The multi-resolution MeshGrid consists of a hierarchical reference-grid and a hierarchical connectivity-wireframe. Animating the hierarchical reference-grid can be done on a hierarchical basis, more precisely any change in the coordinates of reference-grid points at a certain resolution will trigger a local update of the coordinates of the reference-grid points belonging to a higher resolution level. The new positions of the reference-grid points belonging to a higher resolution level ( l+1) can be computed via interpolation methods from the new positions of the neighboring reference-grid points at a lower resolution level ( l), similar to the ones used in subdivision surfaces. The position of a reference-grid point 1

2lnP + is computed according to the “Dyn’s four

point scheme for curves” [40] (see Figure 41) as follows:

12 2 3 2 1 2 1 2 3

1 12 2

l l l l ln n n n nP w P w P w P w P+

- - + +æ öæ ö æ ö ÷ç ÷ ÷ç ç= - ´ + + ´ + + ´ - ´ ÷÷ ÷ç ç ç÷ ÷ç ç ÷÷ç è ø è øè ø

where l represents the hierarchical level of the grid point. The weight w defines the smoothness of the limit curve. In general w is taken equal to 1/ 16, which corresponds to fitting a Catmull-Rom or Cardinal spline curve through the points. For 1/ 16w = the filter coefficients are as follows:

1 9 9 1, , ,16 16 16 16æ ö÷ç- - ÷ç ÷çè ø

2 3lnP - 1

2 2lnP +

-

2 1lnP - 2 1

lnP +

12 2lnP +

+ 2 3lnP +

12lnP +

Figure 41: Dyn’s four point scheme for curves applied for the hierarchical MeshGrid.Applications of this type of animation are: morphing, face and body animations, rescaling, shape matching, …

An example of a sequence of frames obtained by animating the reference-grid, is illustrated in Figure 42(a)-(c). The first row of images shows the shading of the polygonal representation of the surface, while the second row of images displays the polygonal representation of the surface inside the reference-grid. Another example of reference-grid animation is the morphing of a face as illustrated in the sequence of images from Figure 43.

87

(a) (b) (c)

Figure 42: Reference-grid animation of the eight model

Figure 43: Morphing of a human head


2.3.4.4 View-dependent coding of the MeshGrid representation

A bit-stream containing the information about the MeshGrid representation of an object is called a MeshGrid stream. There are several possible types of MeshGrid streams, e.g. single-resolution MeshGrid streams, multi-resolution MeshGrid streams, both types of streams with or without view-dependent decoding features. The most general and flexible MeshGrid stream that has been implemented, is the view-dependent multi-resolution MeshGrid stream that supports all the modes, i.e. view-dependent, view-independent, multi-resolution and single-resolution decoding.

View-dependent decoding of the MeshGrid stream can be enabled by coding the MeshGrid representation in separate local surface descriptions (LSDs), instead of coding the entire MeshGrid at once. To define the LSD we exploit the fact that the MeshGrid representation contains the reference-grid, which is regular in the space (u,v,w). Due to its regular nature, it is straightforward to divide this reference-grid into a number of 3D blocks (called ROIs), and it is possible to encode the surface locally in each of these ROIs (see Figure 45). In principle, it is possible to define the ROI size as small as one: i.e. we encode the surface locally at the level of the grid positions in that case. Yet, such an encoding is far from optimal, because of the overhead as explained next.

The MeshGrid stream will in general consist of three parts: (1) a connectivity-wireframe description, (2) a reference-grid description, and (3) a vertices-refinement description (i.e. refining the position of the vertices relative to the reference-grid – the offsets). Each of these parts can be specified according to the type of stream in single-resolution, multi-resolution, view-dependent or view-independent mode, as explained in the following sections. A minimal stream may however only consist of the description of the connectivity-wireframe, which is mandatory for every stream. In that case a default equally distributed reference-grid, upon which the MeshGrid is based, will be derived from its 8 corners that where specified in the header of the stream, and the vertices’ offset will get the default value of 0.5.

The multi-resolution connectivity-wireframe is constructed in such a way that each single-resolution connectivity-wireframe from the multi-resolution description is the direct super-set of the preceding lower resolution connectivity-wireframe, and a super-set of all lower resolution connectivity-wireframes. In this way the multi-resolution connectivity-wireframe is truly hierarchic. By super-set we mean that a connectivity-wireframe from a higher resolution contains all the vertices of any lower resolution connectivity-wireframe. This redundancy in the multi-resolution connectivity-wireframe makes that a higher resolution mesh does not need any lower resolution connectivity-wireframe in order to display itself. Yet, there is no redundancy in the reference-grid description and in the vertices-refinement description, which means that a higher resolution mesh needs the information of the reference-grid and vertices-refinement of the preceding resolutions, or otherwise default values will be used to fill in the missing reference-grid positions and the final positions of the vertices.

For the definition of the ROIs for view-dependent coding, it is possible to use the same subdivision for all the resolution levels or to define a different subdivision at each resolution level. The subdivisions don’t have to be present at each resolution level, and should be defined in a hierarchical way for the different resolution levels. It might be preferred, to code the surface for view dependent purposes at the highest resolution levels only. Two predefined modes are possible, such as: (1) the size of the ROIs is the same for all resolution levels, this mode being called constant size, and (2) the size of the ROIs is divided by 2 for each next resolution level, this mode being called constant density.

2.3.4.4.1 Coding the connectivity description

The peculiarity of the MeshGrid representation is that the polygonal representation can be derived from the connectivity-wireframe. It has been proven that coding the connectivity is more efficient than coding the polygonal representation of the same surface description. In the following we show how the connectivity-wireframe can be encoded efficiently.

Although there are four connectivity-vectors for each boundary node in the connectivity-wireframe, we only need to encode either the outgoing connectivity vectors (-LN1- and -LN2-) or the incoming connectivity vectors (-LP1- and –LP2-). In our implementation we store the outgoing connectivity vectors since they are defined in the scanning direction.

89

There are two aspects that we have to keep in mind: (1) we have to encode the connectivity between the vertices, and (2) the position (u,v,w) of the reference-grid point the vertices are attached to. We will show that our encoding method of the connectivity allows to derive the reference-grid position when decoding the connectivity.

According to the border-tracking rules, two connected vertices may have three relative discrete border positions (see Figure 24 and Figure 26). They can either have the same discrete border direction, either be rotated 90° clockwise, or be rotated 90° counter clockwise, with respect to the border-scanning direction. For the starting vertex V (the ancestor) with its two outgoing links - LN1- and - LN2- pointing to node V1 (sibling) respectively V2 (sibling), we need to store in the stream its position (u,v,w), i.e. the reference-grid point it is attached to, and the discrete direction of the border as shown in Figure 24 and Figure 26. Each of the connections to vertices V1 and V2 can be encoded with 2 bits each, since the relative discrete direction between two consecutive vertices may have only three values, as previously mentioned. The fourth value has to be used to indicate a broken connectivity when dealing with open meshes. This procedure is illustrated later on in this section. Every sibling node will become an ancestor node when it is its turn to be streamed and it will have two sibling vertices referred by its outgoing links - LN1- and - LN2-.

The relation between reference-grid positions of connected vertices and the discrete border direction is derived from the connectivity rules explained in section 2.3.4.2. Table 11 explores the aforementioned relationship between the positions (u,v,w) of reference-grid points of two consecutive vertices and the orientation of the border at the node positions. It is to be recalled that there are 6 possible choices for the border directions, i.e. “Left” -L-, “Right” -R-, “Front” -F-, “Back” -B-, “Up” -U- and “Down” -D-.The results from the table are visually illustrated in Figure 24 and Figure 26. The column header of the table has the labels of the border directions related to a node playing the role of an ancestor, while the row header displays the same border directions, but related to a sibling node. The table gives all the possible combinations between the directions of the ancestor node and the sibling vertices. Combinations that are not possible have been shown with an exclamation mark. It is to be noticed that, whatever the direction of the ancestor node is, there are always three valid border directions (with different combinations) for the sibling vertices. Moreover, the same relationship can be found between the direction of the ancestor node and the directions of a sibling vertex, i.e. one sibling direction is the same as the ancestor’s direction, another is rotated 90° clockwise, and the other is rotated 90° counter clockwise. All the six directions can be encoded in 3 bits, but since there are always three possibilities for every direction, they can be encoded with 2 bits. They could for example be encoded with 0 if the direction does not change, with 1 if the relative difference is 90° clockwise, and with 2 for the third case. The value 3 will be used to indicate a broken connectivity vector, which may appear in case of an open connectivity-wireframe (e.g. non-closed objects as shown in Figure 72).

For each valid case, the table gives the position (u,v,w) in the reference-grid of the sibling node relative to the ancestor’s node (the table shows only the position coordinate in the reference-grid of the sibling that differs from the ancestor’s position), and the connectivity vector from the sibling back to the ancestor. It is to be noticed that a link may change its local ordering from -L1- (respectively -L2-) to -L2- (respectively -L1-) when it leaves the ancestor and arrives to a sibling. It is to be recalled that the normal vector to the surface is computed as the cross product between -L1- and -L2-, and each border face assigns the directions for -L1- and -L2- such that the normal direction points outward. Figure24 illustrates the physical model used for the MeshGrid representation, the -L1- direction being shown by the red arrows in the centers of the border faces, while the -L2- directions are shown by the blue arrows. The same color convention is used in the following table.

Back Front Left Right Down Up-LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 - -LN1 - -LN2 -

Backv -1 w +1

! ! !v -1

same!

same! !

w +1 u +1 u +1

-LP1 - -LP2 - -LP1 - -LP1 - -LP2 - -LP2 -

Front ! ! w -1 v +1 ! same v +1 !

w -1 ! ! sameu -1 u -1

-LP1 - -LP2 - -LP2 - -LP2 - -LP1 - -LP1 -

Left same ! !u +1 w -1 u +1 ! ! !

w -1 same !v +1 v +1-LP2 - -LP2 - -LP1 - -LP2 - -LP1 - -LP1 -


Rightu -1

! ! same ! ! u -1 w +1 ! same w +1 !v -1 v -1

-LP1 - -LP1 - -LP1 - -LP2 - -LP2 - -LP2 -

Down !u -1 same ! same ! !

v +1 u -1 v +1 ! !w +1 w +1-LP1 - -LP1 - -LP2 - -LP2 - -LP1 - -LP2 -

Up !same u +1

!v -1

! !same

! !v -1 u +1

w -1 w -1-LP2 - -LP2 - -LP1 - -LP1 - -LP1 - -LP2 -

Table 11: Relation between the discrete border directions of two connected vertices and their discrete coordinates (u,v,w).The first case, i.e. the direction does not change from the ancestor to the sibling, is shown in the table on the first diagonal. Specific for this case is that only one index of the position in the reference-grid {u,v,w} differs from the ancestor to the sibling, and that the ordering of the connectivity vectors does not change. Flat areas that are parallel with the reference-surface generate this case.

The second case, i.e. the border direction for the sibling is rotated 90° counter clockwise relative to the ancestor, is specific for convexities in the surface, and is characterized by the fact that the position in the reference-grid of the sibling does not differ from the position in the reference-grid of the ancestor. It is only the offset that moves the vertices into their final position on the boundary of the surface. The indices of the connectivity vectors may change in order to comply with the definition of the normal vector external to the surface.

In the third case, i.e. the border direction for the sibling is rotated 90° clockwise relative to the ancestor, is specific for concavities in the surface. There are two indices of the position in the reference-grid {(u,v), (v,w), (u,w)} of the sibling node that differ from the ancestor node. The indices of the connectivity vectors may change as well.

2.3.4.4.1.1 View-dependent coding of the connectivity description

The coding of a multi-resolution connectivity-wireframe consists basically in encoding sequentially each single-resolution connectivity-wireframe apart. In case of view-dependent coding, each single-resolution will be split into local surface descriptions LSDs, and the connectivity-wireframe from each LSD will be coded separately. In view-dependent mode the only issue is to define artificial boundaries in the connectivity-wireframe (artificially cut the links of the connectivity-wireframe). The boundaries of a LSD are defined by the ROIs that subdivide the entire reference-grid into disjoint 3D blocks. In order to avoid coding twice the connectivity of the vertices at the border in between two adjacent ROIs, the domain of the ROI [min, max) is closed for the minimum value (i.e. includes the vertices), and open for the maximum value (i.e. it does not include the vertices).

Next to encode the LSD, we start from a certain vertex VS, lying inside the ROI containing the local surface, and determine all the other vertices, belonging to the ROI, which can be reached through a connection path from the starting vertex VS. The connection paths are explored via the outgoing connections (as previously mentioned) of the vertices in an iterative way by using a FIFO buffer (see Figure 44).

91

1 2

3 5

4

8

9 6

10

7 FIFO Buffer

1

2

3

4

5

6

7

8

9

10

23

4

5

678910

3456


78910

5678

9

10

89

10

93

Figure 44: Coding the LSD with a FIFO bufferThe scenario is as follows: The position of the 1st vertex (starting vertex VS) is stored in a FIFO. We consume the first vertex from the FIFO and put vertex 2 reached via the outgoing connectivity vector 1 and vertex 3 reached via outgoing connectivity vector 2 in the FIFO. Next we pop vertex 2 from the FIFO and place its connected vertices 4 and 5 in the FIFO. In case an outgoing connectivity vector is leading to a vertex, that was already visited before, or to a vertex that lies outside the ROI, the visited vertex will not be put in the FIFO. When consuming vertex 3 from the FIFO, we only put the connected vertex 6 in the FIFO, since the first connectivity vector leads to vertex 5, which has been already visited. The scenario will stop when all vertices in the FIFO are consumed and no new vertices are reached inside the ROI.

In order to obtain a stream, which can be decoded unambiguously, it suffices to encode the absolute position and orientation of the starting vertex VS and the outgoing connectivities of all the connected vertices including the starting vertex VS, as soon as they are visited. This way of encoding ensures that the vertices will be visited in the same order during the decoding and that their absolute position and directions can be derived from their ancestor. In the example, vertices 2 and 3 are coded relative to the starting vertex 1 via its outgoing connectivity vectors, while vertices 4 and 5 are coded relative to the starting vertex 2 via its outgoing connectivity vectors.

Some times, one starting vertex will not be sufficient to visit all the vertices from the local surface. In that case it might be necessary to repeat the same scenario for different starting vertices. We call a starting vertex and its connected vertices a patch. Hence a local surface will in general consist of one or more patches. For optimal coding, one should try to encode the local surface in a minimal number of patches, since each starting vertex of a patch introduces some overhead (i.e. its absolute position and absolute orientation have to be coded extra). A LSD can be coded in a minimal number of patches, by first determining all the singular vertices (i.e. a vertex which has no incoming connectivities inside the ROI containing the local surface). Since a singular vertex cannot be reached, it has to be considered as a starting vertex. If after coding the patches starting from all these singular vertices, there is still a remaining part of the local surface not coded (or there are no singular vertices) for that patch, only one starting vertex is needed to completely encode the rest of the vertices. In order to find one of those particular starting vertices, a backtracking


algorithm can be applied. Yet in general, for surfaces with a shape of moderate complexity, it is possible to describe the local surfaces in one patch and no extra overhead is encountered.

In this approach different local surfaces are connected to each other via the outgoing connectivity vectors of the vertices lying at the edge of a ROI.

In order to have more flexibility for the view-dependent decoding, the ROI sizes should be chosen small enough compared with the size of the entire reference-grid. The smaller the size of the ROI the more overhead is involved in the coding of the connectivity-wireframe because of the increasing number of starting vertices. The torus example from Figure 45 shows visually the size of the ROI compared to the surrounding box of the object. High overhead is caused by very small ROIs.

Figure 45: An example of view dependent coding based on ROIs with a constant density of grid planes at each resolution level. If the number of grid planes of the ROI exceeds the number of grid planes of the mesh, then the surface is coded in a global way. From top to buttom the rows correspond to the following ROI sizes: (1) 1x1x1, (2) 3x3x2, (3) 6x6x4, (4)12x12x8.One of the curves in the following chart (labeled torus) illustrates the coding overhead for ROI sizes normalized with respect to the volume of the reference-grid (sizeU*sizeV*sizeW), computed from the data of Table 12.

95

0

1

2

3

4

5

6

7

9.54E-0

7

3.8147

E-06

3.0517

6E-0

5

0.0002

4414

1

0.0019

5312

5

0.0156

250.1

25 1

torus

humanoid

blob

Figure 46: Coding overhead for ROI sizes normalized with respect to the volume of the reference-grid (sizeU*sizeV*sizeW), computed from the data of Table 12.

Torus Humanoid Blob

Vertices 7856 Vertices 7836 Vertices 28326

Stream View-indep. 8878 (bytes)

Stream View-indep 8855 (bytes) Stream View-indep

31906(bytes)

View-dependent coding

ROI SzStream

Sz Ratio ROI SzStream

Sz Ratio ROI SzStream

Sz Ratio

1.1.1 23125 2.604753 1.1.1 52159 5.890012 1.1.1 177268 5.555815

3.3.2 11582 1.304573 2.1.3 18528 2.092259 2.2.1 79719 2.4985

6.6.4 9766 1.100023 4.2.9 10915 1.232567 4.4.2 42378 1.328183

12.12.8 9190 1.035143 8.4.18 9790 1.105528 8.8.4 34789 1.090334

24.24.16 8915 1.004168 16.8.36 9540 1.077297 16.16.8 32723 1.025582


48.48.32 8884 1.000676 32.16.72 9449 1.06702 32.32.16 32345 1.013735

64.32.144 9313 1.051663 64.64.32 31991 1.002641

128.128.64 31917 1.000321

Table 12: This table shows the size of compressed meshes with and without view-dependent coding, and the corresponding ratio between the view-dependent stream and the globally coded mesh for three examples. Different ROI sizes were considered for each mesh coding.

2.3.4.4.2 Coding the reference-grid description

The general representation of the Reference-Grid description is a vector field (x, y, z) defined on a regular discrete 3D space (u, v, w). In the simplest case (regular, discrete reference grid) each vector corresponds to its position in the discrete 3D space, i.e. x(u,v,w) = u; y(u,v,w) = v; z(u,v,w) = w. However, in the general case, when dealing with curved reference surfaces, each component of the vector can be an arbitrary function of (u,v,w): i.e. x(u,v,w) = f(u,v,w); y(u,v,w) = g(u,v,w); z(u,v,w) = h(u,v,w). Yet in general, the vector field will have a smooth nature. This implies that it can be highly compressed, and the transform-based coders are the most appropriate since they de-correlate the signal before coding.

Although there are many possible 3D coding algorithms reported in the literature, there are several critical functionality requirements that have to be taken into account when designing the reference-grid coding algorithm. The most critical constraint is that the hierarchical nature of the connectivity-wireframe is matched with the hierarchical grid-representation, that is, for each resolution of the connectivity-wireframe one should be able to decode the corresponding grid-resolution. Therefore, resolution scalability is a critical issue that must be supported by the employed codec. Furthermore, the quality-scalability functionality should be addressed as well, since the user should be able to decode each grid-resolution up to a certain rate-constrained or user-defined quality-level. And finally, ROI decoding should be also supported in order to address the view-dependent functionality.

Among the many possible alternatives existing in the literature, the wavelet-based coders exploiting the intra-band dependencies yield state-of-the-art rate-distortion performance and ensure several functionalities, including quality and resolution scalability, and ROI coding/decoding [6, 68, 70, 71]. The wavelet-based coding algorithm used for the coding of the reference-grid is part of the intra-band wavelet coders family [6, 68, 70, 71] and supports the above-mentioned functionality requirements.

This algorithm is described in the following three sub-sections. Section 2.3.4.4.2.1 describes the particular type of filters and down-sampling/up-sampling operations used for the wavelet decomposition and reconstruction. The 3D coding algorithm is described in section 2.3.4.4.2.2, while the third section describes the particular design options needed to address ROI decoding supporting the view-dependent functionality.

2.3.4.4.2.1 The 3D Wavelet Decomposition

Denote by lnC the u, v, or w coordinate of the thn reference-grid point l

nP defined at the resolution level l , with

1 l L , and L indicating the number of resolution levels of the grid. In order to encode the coordinates lnC at all the

levels l , 1 l L , the coding algorithm performs a wavelet decomposition of the 3D volume defined by the reference-grid coordinates at the highest resolution L

nC . The wavelet decompositions are performed for each component of the vector field.

97

lnC 2l

qC 1lpC 2l

rC lmC

(a) (b)

3

2

1 lnC

2lqC

2lqC

1lpC

2lrC

lmC

2lrC

1lpC

(c)

Figure 47. Coding example of a deformed hierarchical reference-grid (a) regular grid, (b) deformed grid, and (c) coding of the grid points of a higher resolution based on the grid points of the immediate lower resolution.In order to introduce the particular type of wavelet decomposition/reconstruction implemented in the reference-grid coding algorithm and to describe the wavelet analysis/synthesis filters, let us consider the simple one-dimensional example shown in Figure 47. In this example, the coordinates l

nC and lmC should be encoded/decoded at the

resolution level l . At the higher resolution level 1l , an additional grid-point coordinate 1lpC should be encoded as

well. At this resolution level, instead of encoding directly the value 1lpC , the algorithm encodes the relative coordinate-

difference 1lpC , computed as the difference between 1l

pC and the average between lnC and l

mC , as follows:


H

1djA f

jD f

djA f

2

2 G

2

2

H~

G~ 1

ˆdjA f

Decomposition Reconstruction

Figure 48. Wavelet decomposition and reconstruction of a 1D signal.

1 1 2l l l lp p n mC C C C .

Similarly, two additional relative coordinate-differences 2lqC and 2l

rC given by:

2 2 1 2l l l lq q n pC C C C ,

2 2 1 2l l l lr r m pC C C C

are encoded at the resolution level 2l , and so on.

In a decoding-phase scenario of the resolution level 1l , all the grid-point coordinates from all the coarser resolutions, including l

nC and lmC are already decoded. Then, the coordinates at the higher resolution levels 1l and 2l can be

derived immediately with the reconstruction formulas below:

1 1 2l l l lp p n mC C C C

and

2 2 1 2l l l lq q n pC C C C ,

2 2 1 2l l l lr r m pC C C C .

In general, it can be shown that the relative coordinate differences at each resolution level can be calculated in a simple manner with a pyramidal algorithm implementing a wavelet transform [65]. The analysis low-pass and band-pass wavelet filters are respectively:

1| 0H n n

0.5,1, 0.5 | 1,0,1G n n

The synthesis low-pass and band-pass wavelet filters are:

0.5,1, 0.5 | 1,0,1H n n

1| 0G n n

The block structure of a 1D wavelet decomposition and wavelet reconstruction [65] is depicted in Figure 48. In this figure ,H G and ,H G are the low-pass and band-pass analysis and synthesis filters respectively, while d

jA f n and

99

jD f n are the discrete approximation and the detail signals respectively at the resolution 2 , 1j L j . Filtering

the approximation signal 1djA f n with ,H G followed by the down-sampling operation ( 2 ) – which retains every

second sample from the filtered signals – yields djA f n and jD f n . At the first decomposition level ( 1j ) the

input approximation signal 0dA f is actually the original 1D-signal f n , that is 0

dA f n f n . Repeating recursively the decomposition algorithm shown in Figure 48 yields a set of 1L signals, which represents the wavelet decomposition on L levels of the 1D-signal f n :

1 1, , ,...,dL L LA f D f D f D f

To reconstruct the original signal from its wavelet representation we need to perform the wavelet reconstruction, illustrated on the left side of Figure 48. The up-sampling operation ( 2 ) – which inserts a zero every second sample – is followed by filtering with the low-pass and band-pass synthesis filters respectively. Summing the resulted filtered signals yields the reconstructed approximation at the higher resolution, 1

ˆdjA f n . If the filters ,H G and ,H G satisfy

the perfect-reconstruction condition [65], then 1 1ˆd d

j jA f n A f n for any value of j .

Basically, the analysis low-pass and band-pass filters given by – followed by the down-sampling operation decompose the signal into two different subbands, as follows:

1

1 1 1

2

12 1 2 2 22

d dj j

d d dj j j j

A f n A f n

D f n A f n A f n A f n

This equation shows that the low-pass filter H and the band-pass filter G are applied on the even and odd samples respectively of 1

djA f n . The up-sampling operation followed by filtering with the synthesis low-pass and band-pass

filters reconstruct the original signal as follows:

1

1

2

12 1 12

d dj j

d d dj j j j

A f n A f n

A f n A f n A f n D f n

These two equations correspond to the filtering with H and G of the even and odd samples respectively of the low-

pass and band-pass components djA f n and jD f n . From and it can be easily demonstrated that the perfect

reconstruction condition is satisfied, that is, 1 1ˆd d

j jA f n A f n for any value of j .

Notice from – that the analysis and synthesis low-pass and band-pass filters are very short. The benefits of these filters are (1) that they are computationally very fast, (2) they can be used to encode/decode small reference-grid sizes and (3) since the analysis low-pass filter H is 0 1H , there is no dependency between the values in the lowest frequency subband and the other subbands at the same or higher resolution levels. Point (3) implies that the exact values of the grid positions for a certain MeshGrid resolution can be computed from the coded grid information stored at that resolution level only: i.e. decoding the information from subbands at higher grid resolution levels will not influence the values of the grid positions at lower resolution levels.

MeshGrid targets resolution scalability, and decoding any j-th grid resolution is equivalent with decoding the corresponding approximation d

jA f . However, the grid can be decoded at any resolution j if and only if the corners of


the grid are stored in any , 0djA f L j . There are some situations in which this constraint is not satisfied with the

classical implementation of the pyramidal algorithm [65] discussed above. As a consequence, the wavelet decomposition and reconstruction is not so straightforward as it might seem from the decomposition and reconstruction equations , . To solve this problem, our customized implementation of the pyramidal algorithm involves in some situations non-uniform down-sampling and up-sampling operations, coupled with analysis/synthesis filters that are different than the ,H G , ,H G given by – . In order to illustrate these situations, we make use of the graphical representation given in Figure 49, in which the three-level wavelet decomposition of a 1D-signal f n is depicted. The low-pass filtering and the band-pass filtering are indicated with blue and green arrows respectively. The common situation illustrating the decomposition equation is shown in Figure 49(a). At the first decomposition level, since

0 1H , the even samples of f n are simply copied to form the subband 1dA f , while applying the band-pass filter

on the odd samples of f n yields the detail signal 1D f . The procedure is applied recursively, until the lowest

resolution subband 3dA f contains only the first and the last samples of f n (see Figure 49(a)). In three dimensions,

this is equivalent to decompose the reference grid until the low-pass (LLL) subband at the lowest resolution level corresponds to a cube with a size of 2x2x2, consisting of the 8 corner points of the grid. We notice that in this example every discrete approximation , 3 0d

jA f j contains the corners of the grid, therefore the above-mentioned constraint is satisfied. However, this constraint is not satisfied anymore if the standard pyramidal algorithm is performed for even-length signals. In order to solve this problem, we adopted a non-uniform down-sampling operation, as illustrated in the example shown in Figure 49(b). The low-pass filtering is performed on the samples 6 , 9f f

instead of 6 , 8f f as it would be the case in the classical implementation. The band-pass filtering operation is

indicated with red arrows in Figure 49(b)-(c) and is performed with the filters 11G n and 2 1

1 1G n G n on the

samples 7f and 8f respectively. The filters 11G n and 2

1G n are given by:

11

1 2,0,1, | 2, 1,0,13 3

G n n

2 11 1

2 1,1,0, | 1,0,1,23 3

G n G n n

0 1 2 3 4 5 6 7 8

0 2 4 6 8 1 3 5 7

0 4 8 2 6 1 3 5 7

0 8 4 2 6 1 3 5 7

0 1 2 3 4 5 6 7 8

0 2 4 6 9 1 3 5 7

0 4 9 2 6 1 3 5 7

0 9 4 2 6 1 3 5 7

9

8

8

8

0 1 2 3 4 5 6 7 8

0 2 4 6 8 11 1 3 5

0 4 11 2 6 8 1 3 5

0 11 4 2 6 1 1 3 5

9

7

7

7

10 11

9 10

9 10

9 10

(a) (b) (c)

1j

2j

3j

f n

Figure 49. Graphical illustration of the wavelet decomposition for odd and even-length signals.These filter coefficients ensure that by calculating the one level decomposition of a regular grid, the detail signal at the first resolution 1D f contains only values of zero. This can be easily demonstrated by calculating analytically the detail

101

coefficients obtained by filtering with 11G n and 2

1G n . Assume that the signal f n is a regular grid of the form

f n n . Filtering with 11G n in some arbitrary position 0f n yields:

1 1 1 1 101 0 1 0 0 1 0 1 0 1 0 1

0 0 0

2 2 1 1 0 1 12

1 22 1 03 3

nD f n G n f n G f n G f n G f n G

n n n

,

while filtering with 21G n yields:

2 2 2 2 201 0 1 0 0 1 0 1 0 1 0 1

0 0 0

1 1 0 1 1 2 22

2 11 2 03 3

nD f n G n f n G f n G f n G f n G

n n n

.

where x is the integer part of x . The G filter given in satisfies also this property; therefore the wavelet decomposition of a regular grid produces only wavelet coefficients with values of zero in the detail signal 1D f . This property is very useful, since in this way regular grids (which represent the most common situation) can be compressed very efficiently with a few non-zero wavelet coefficients. In order to ensure that all the details jD f , 1L j in the

wavelet decomposition of a regular grid contain only values of zero, we need to construct filters kjG adapted for each

resolution level j .

As long as the length of the discrete approximation signal djA f is odd, there is no need to perform a non-uniform

down-sampling operation, and the classical pyramidal algorithm can be used. If at some resolution level , 0r L r the length of d

rA f is even, then apart of the common situation in which G given by is used, we need also to use the

analysis band-pass filters 1 21 1,r rG G and to apply non-uniform down-sampling for the last samples. This operation has

to be repeated for all the resolution levels ,p L p r . The filter coefficients of 1 21 1,p pG G depend on the length of

the discrete approximation signal dpA f (whether it is an odd or an even number) and on the resolution level p . For

example, we notice from Figure 49(b)-(c) that in order to calculate 2D from 1A , apart from the common situation in

which G is used (green arrows), we need one additional band-pass filter 12G if the length of 1

dA f is odd (red arrow),

and two additional band-pass filters 1 22 2,G G if the length of 1

dA f is even (cyan arrows).

In general, it can be shown that the additional filters used to derive the detail 1pD f starting from the discrete

approximation dpA f are:

12 1,0,1, | 2, 1,0,1pG n c c n

24 3,1,0, | 1,0,1,2pG n c c n

if the length of dpA f is even, and:


16 5,1, | 1,0,1pG n c c n

if the length of dpA f is odd. The constants 1 6,...,c c verify the relations:

1 2 3 4 4 21 , 1 , 2c c c c c c

5 61c c

The constants 2 6,c c are given by:

2 6

x nc c

y n

where ,x n y n satisfy the recurrence

1 11 2 , 1 2n nx n x n y n y n

if the length of dpA f is even, and:

11 2 , 1 2n nx n x n y n y n

if the length of dpA f is odd, with 0 1, 0 3x y and n r p .

2.3.4.4.2.2 Intra-band wavelet coding algorithm

The wavelet coefficients resulted after applying the 3D wavelet decompositions on each component of the vector field are encoded using an embedded, progressive in multi-resolution intra-band coding algorithm. There are many wavelet-coding algorithms reported in the literature, and they can be divided basically into two classes: inter-band coders [90], exploiting the zerotree model of the wavelet coefficients, and intra-band coders [6, 68, 70, 71], exploiting the clustering property of the wavelet coefficients. The coders from the first category exploit the residual dependencies between the wavelet coefficients by grouping them in trees growing exponentially across the scales, and by classifying them jointly. As a consequence, these coders inherently encode an image in a quality-progressive manner (i.e. all the lower quality versions of the original data are embedded at the beginning of the bit-stream), and addressing resolution scalability with such coders is simply not possible. However, the coders from the second category do not have this limitation, and can provide both resolution and quality scalability. Since resolution-scalability is a critical functionality requirement, we opted for an intra-band wavelet coder in MeshGrid. The wavelet coefficients are coded subband by subband by an octree-based coder (called Cube-Splitting), which is a 3D extension of the SQP (SQuare Partitioning) coder described in [70]. This coder has the advantage that is not very computing demanding and yields competitive compression performance [70]. In order to address the view-dependent functionality, each subband is split into a lattice of blocks that are coded independently of each other by using the Cube-Splitting algorithm. This aspect will be detailed in the next section.

As mentioned above, in the context of volumetric encoding, the SQP technique was extended to a third dimension: from square splitting towards cube splitting. Figure 50 illustrates the encoding process; during the first significance pass the wavelet coefficients, newly identified as significant, are registered using a recursive tree structure of cubes (Figure 18.a-c.). If a cube, having initially the size of the currently coded block contains a significant coefficient, it is divided into eight sub-cubes. The descendent “significant” cube(s) is (are) then further divided until the significant wavelet coefficients are isolated. The result is an octree-structured description of the data significance against a given threshold (Figure 18.d). As it might be noticed, equal importance weights are given to all the branches. When a significant coefficient is isolated, also its sign - for which two code symbols are preserved - is immediately encoded.

103

After the significance pass, the threshold is lowered (typically dyadic), and the refinement pass is initiated for the next bit-plane, refining all the coefficients marked as significant in the octree. Thereafter, the significance pass is restarted to update the octree by identifying the new significant coefficients for the current bit-plane. During this stage, only the non-significant nodes are re-encoded, and the significant ones are ignored since the decoder already received this information. The complete significance procedure can thus actually be seen as a tree growing process. The described procedure is repeated, until the entire block is coded or the desired bit-rate is met.

SGN

NSSGNNS

NSNS

NSNS

NS

NSSGN

NS

NSNS

NSNS

NS

(a) (b) (c)

(d)

Figure 50. When a significant wavelet coefficient is encountered, the cube (a) is spliced in eight sub-cubes (b), and further on (c) up to the pixel level. The result is an octree structure (d) (SGN = significant cube; NS = non-significant cube). In the next significance pass, the new significant cubes are further refined.

2.3.4.4.2.3 Addressing ROI decoding for view-dependent functionality

The ROI functionally is a procedure that allows for only a specific part(s) of the image (defined by the client) to be reconstructed losslessly before reconstructing perfectly the rest of the image. Basically, there are two different types of ROI functionality that might be relevant to different applications. The first, known as “ROI during encoding” or “a priori ROI” is relevant to applications in which the geometry and relative importance of different regions in the image is known at the time when the image is compressed. This type of ROI can be used for the storage of large amount of data. In this case the selection algorithm compresses directly the region of interest in a lossless manner, while the areas outside the ROI are compressed at very high compression ratios. A second scenario for this type of the ROI is to losslessly compress and transmit first the region of interest, followed by a complete refinement (up to the lossless stage) of the background information. This first approach for ROI handling is somehow straightforward from the technical point of view, since a simple and common solution implies the shifting of the wavelet coefficients that correspond to the ROI [6, 90, 94, 103]. In this way, one assigns to these coefficients a higher priority than to the coefficients belonging to the background, therefore, in an embedded coder the ROI is coded first.

The second approach for ROI functionality, known as “ROI during decoding” or “a posteriori ROI” is referred to applications in which the geometry and importance of the regions is known only after the image has been decompressed, say during interactive browsing of the compressed image stored on a remote database. Using this


procedure we preventively apply lossless compression on the input image and then we employ the ROI functionality on the compressed buffer.

As mentioned in the previous section, multi-resolution coding is supported by losslessly encoding the subbands starting from the coarsest approximation d

LA f and ending with the highest frequency detail 1D f . In order to offer the view-dependent functionality, the CS coder should be modified to support ROI during decoding (or simply ROI decoding), as explained in the following.

For simplicity, let us consider the two-dimensional case, and assume that the spatial-domain reference-grid at some resolution-level , 1k L k is decomposed into a lattice of blocks of some user-defined dimensions ,k ku v , as shown in Figure 51(a). Denote the lattice decomposition operation as ,k ku v L .

Each block in the spatial domain is identified with an index resulting from the raster scanning of the entire lattice of blocks. Assume that the user is interested to decode some spatial-domain block indicated in green in Figure 51 (a). This block defines the spatial-domain ROI ( kSROI ) at the particular resolution-level k . As mentioned in section 2.3.4.4.1.1, for every resolution-level k , the lattice dimensions ,k ku v are identical with the sizes of the blocks used to encode the connectivity wireframe in LSD. In this way, the lattice decomposition ,k ku v L defines the same spatial-domain ROIs for the reference-grid as well as for the connectivity wireframe.

The blocks of coefficients in the wavelet domain that correspond to kSROI are depicted in Figure 51 (b), and they define the wavelet-domain ROI at the resolution-level k denoted as kWROI .

It is important to mention that kWROI depicted in Figure 51(b) does not delineate all the wavelet coefficients needed to reconstruct losslessly the corresponding spatial-domain ROI. Typical wavelet coders [6, 71, 83], supporting the ROI functionality extend the wavelet-domain ROI to account for the size of the synthesis wavelet filters, and to avoid the border effects in the decoded ROI. Moreover, these coders implement the classical pyramidal algorithm, involving uniform down-sampling operations and fixed synthesis wavelet filters. As shown in section 2.3.4.4.2.1, MeshGrid uses different filters at the boundaries and non-uniform down-sampling operations, therefore the calculation of kWROI should account for these two aspects. The exact correspondence between the spatial-domain ROI and the wavelet-domain ROI can be derived via an algorithm that will be explained in the next section.

v u 1 2 3 4 5

6 7 8 9 10

11 12 13 14 15

16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

(a) (b)

Figure 51. Lattice decomposition of the original image and the corresponding decomposition in the wavelet domain.The example in Figure 51 indicates that the lattice decomposition of the spatial-domain translates to a similar lattice decomposition of the subbands of each resolution level. As a consequence, given the lattice-dimensions ,k ku v at

105

the resolution-level k , one can derive that the lattice decomposition ,j ju v WL of the subbands at some resolution

level ,j L j k is 2 jju u and 2 j

jv v .

Moreover, the example in Figure 51 shows that in order to address the ROI decoding functionality for the multi-resolution CS coder, we need to decompose the subbands of each resolution level into a set of code-blocks that are coded independently of each other. The splitting into code-blocks of each subband is identical with the lattice decomposition ,j ju v WL of the wavelet subbands at the resolution level , 0j L j .

In order to illustrate the multi-resolution ROI decoding scenario, let us consider the example given in Figure 52. Assume that at the resolution-level 1k the decoder requests the spatial domain ROI formed by the tiles with indices 8, 9, 13, as shown in Figure 52(a). The encoder (1) identifies the corresponding wavelet-domain tiles (forming the wavelet-domain ROI) in all the wavelet subbands from all the resolution levels , 1j L j k , (see Figure 52(b)), (2) selects from the entire bit-stream the bit-streams corresponding to each tile, and (3) sends them to the decoder in the order given by their indices. At the higher resolution-level k , the decoder requests another spatial-domain ROI, indicated in green in Figure 52(c). Once again, the encoder identifies the corresponding wavelet-domain ROI formed in all the wavelet subbands from the current and from the lower resolution levels, ,j L j k . As illustrated in Figure52(d) some wavelet-domain tiles have already been transmitted at the lower resolution-level 1k , therefore they need not to be transmitted.

2v 2u 1 2 3 4 5

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

(a) (b)

v u 1 2 3 4 5

6 7 8 9 10

11 12 13 14 15

16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

(c) (d)

Figure 52. Multi-resolution ROI decoding scenario.

In general, if kWROI is the wavelet-domain ROI requested at the resolution level k, then the transmitted wavelet-domain tiles are determined according to the following relation:

\transmittedk k j

L j kWROI WROI WROI

2.3.4.4.2.3.1 Calculation of the Wavelet-domain Region of Interest

An example illustrating the spatial-domain ROI and the correspondent wavelet-domain ROI for a one-dimensional signal is given in Figure 53. The spatial domain ROI at the highest resolution 0SROI is indicated with the red rectangle.


The corresponding regions of interest at the resolution-levels 1... 4 are also indicated with red rectangles. Notice from this example that the samples 0 and 17 represent the boundary samples of the signal, and are by default decoded at the lowest resolution. Notice also that in order to decode losslessly sample 9 (representing the boundary of 0SROI ) we need also sample 8. This indicates that at each resolution one should compensate for the possible border effects by adding the necessary samples needed to correctly reconstruct the boundaries of 0SROI .

4WROI

1j

2j

3j

f n 0 1 2 3 4 5 6 7 8

0 2 4 6 8 10 1 3 5

0 4 8 2 6 10 14 1 3

0 8

9

7

5

10 11

9 11

7 9

12 13 14 15 16 17

12 14 17 13 15 16

12 17 11 13 15 16

2 6 10 14 1 3 5 7 9 11 13 15 16 4 12 17

0 17 2 6 10 14 1 3 5 7 9 11 13 15 16 4 12 8 4j

0SROI

1WROI

2WROI

3WROI

Figure 53. Example of ROI defined in the spatial and wavelet domain.Let us generalize in the following, and describe the algorithm used to calculate the wavelet-domain ROI, given a rectangular ROI defined in the spatial domain. Let ,X a b denote a reference-grid point, where

, 0, 1 0, 1a b W H and ,W H are the vertical and horizontal dimensions of the reference-grid. Let

0, 1 0, 1R W H denote a rectangular region of interest. We assign to R the background region S , as

0, 1 0, 1 \S W H R . In order to losslessly reconstruct the spatial-domain ROI, the encoder determines the

mapping between the ROI defined in the spatial domain X and corresponding regions in the wavelet domain WX . This correspondence between domains is determined via an algorithm, called the wavelet mask, which will be explained in the following.

The Wavelet Mask is a binary image indicating which wavelet coefficients have to be completely transmitted and losslessly reconstructed such that the receiver reconstructs the desired spatial-domain region perfectly.

In order to derive the correspondence of the ROI from the spatial domain to the wavelet domain, we calculate directly the mapping in the wavelet domain of the top-left corner of the spatial-domain ROI and the mapping of its size. Let

,LX i j and ,RX p q denote the top-left and bottom-right corners respectively of the ROI in X . If i and j are even, then their mapping in WX over the four subbands from the first level of the wavelet decomposition are the pixels with the coordinates ,m n given by the following equation:

, ,2 2i jm n

If i and j are odd, then their mapping in WX is given by:

107

1 1, ,2 2

1 1, ,2 2 2

1 1, ,2 2 2

1 1, ,2 2 2 2

LL LL

LH LH

HL HL

HH HH

i jm n

i W jm n

i j Hm n

i W j Hm n

nHL

mHL

X[ i, j]

ROI

LL LH HL HH

Figure 54. Representation in wavelet domain of the ROI. The top_left and down_right corners of the ROI are indicated with red and green dots respectively.

Let xL and yL denote respectively the width and height of the ROI in original image. If i and j are even, then the mapping of xL and yL in the wavelet domain over the four different subbands of the first level are given by:

11 , ,2 2

1, ,

2 2

1 , ,2 2

, ,2 2

yLL LLxx y

yLH LHxx y

yHL HLxx y

yHH HHxx y

LLWL WL

LLWL WL

LLWL WL

LLWL WL

If i and j are odd, then the mappings of xL and yL in the wavelet domain are mirror versions of , as follows:


, ,2 2

1 , ,2 2

1, ,

2 2

11 , ,2 2

yLL LLxx y

yLH LHxx y

yHL HLxx y

yHH HHxx y

LLWL WL

LLWL WL

LLWL WL

LLWL WL

For i even and j odd we use the expressions in the first column of and the second column of . The opposite combination should be used for i odd and j even. Figure 55 shows the representation in the wavelet domain of a region of interest of an arbitrary dimension derived by using the mapping operations described by -.

WROI X[ i, j]

ROI

Figure 55. Wavelet ROI obtained by applying the mapping operations described by -.However, this version of the wavelet mask is not adequate to obtain a lossless reconstruction of the ROI, because so far we did not take into account the border effects. In fact, the ROI area in the wavelet domain depends also on the width of the low-pass and high pass synthesis filters. Filtering a coefficient on the border of the ROI area in the wavelet domain with the previous mask yields a border distortion, as shown in Figure 56.

X 1

Analysis Filtering Synthesis Filtering

Xb

Figure 56. Filtering process on the border.

As we remark, in order to achieve a lossless reconstruction of the border coefficient Xb , the coefficients 1X are necessary in the inverse filtering process. In order to derive which coefficients should be added in the wavelet mask, we analyze the particular case of the ,H G given by – , as shown in Figure 57.

109

One can observe that the coefficients necessary to losslessly reconstruct 2X n and 2 1X n are L n to 1L n

and H n . This shows that we do not need any additional band-pass components to compensate for the border effects. However, we must update the mask if synthesis low-pass filtering is performed at the borders. Therefore, if i and j are even there is no need to extend the wavelet mask, while if they are odd, then we need to extend the wavelet domain ROI with 1 sample. The mapping operations given by equations – are modified as follows:

' '

' '

' '

' '

1, 1,

, 1,

1, ,

, ,

LL LL LL LL

LH LH LH LH

HL HL HL HL

HH HH HH HH

m m n n

m m n n

m m n n

m m n n

Remind that ,RX p q defines the bottom-right corner of the ROI. Also if p and q are even there is no need to extend the wavelet mask, while if they are odd, then we need to extend the wavelet domain ROI with 1 sample, in a similar way as described above:

' '

' '

' '

' '

1, 1,

, 1,

1, ,

, ,

LL LL LL LL

LH LH LH LH

HL HL HL HL

HH HH HH HH

p p q q

p p q q

p p q q

p p q q

n-2 n-1 n n+1 n+2 n-3 n-2 n-1 n n+1 n+2 n+3

2n 2n+1

X

Low-Pass High-Pass

Figure 57. The inverse wavelet transform performed with the filters ,H n G n , indicating the wavelet coefficients that are needed in the inverse filtering process.

Figure 58 illustrates an example of the new dimensions of the wavelet mask obtained by applying – to compensate for the border effects. To derive the wavelet-domain ROI mask for a wavelet decomposition with L levels, we apply recursively the procedure described above on the ROI mask from the LL subband.


LLn LL LHn n

'LL LLm m

'HL HLn n

'LH LHm m

L

L

L

L L

L

L L

Figure 58. Wavelet Mask calculated for one level wavelet decomposition.The illustration of the wavelet domain ROI calculated for an arbitrary dimension spatial-ROI using the previously described algorithm is given in Figure 59. The intersection between this binary mask and the set of tiles defined in each subband in the wavelet domain indicates the tiles from all the subbands that are needed to reconstruct losslessly the spatial-domain ROI.

Figure 59. Wavelet Mask calculated using - for an arbitrary size ROI.

111

ISO/IEC 14496-1:2001/PDAM 4

2.3.4.4.3 Coding the vertices’ refinement description

The vertices are located at the intersection positions between the grid-lines and the object contour, at a certain ratio in between two reference-grid points. This ratio has a default value of 0.5, but can be modified by the refinement description to fit a known position. The ratio can vary between [0,1). The update of the vertex position is done by a successive approximation method spanning all the levels in the hierarchy. For each resolution level, except the highest one, the most significant bit from the vertex-precision value is used to update the position of the vertex towards the next level. The used bit is consumed from the vertex-precision value, so that the next bit becomes the most significant one. When the most significant bit is 1, than the existing vertex will migrate from its current grid-position to the new inserted grid-position as shown in Figure 60. When it is 0, the position of the vertex is not affected. At the highest resolution level it is possible to consider more than one bit to update the refinement positions. We can summarize that a vertex that appears at a lower level needs more vertex-precision bits than a vertex located on a higher level. The number of precision bits required for a vertex can be computed as follows: nr bits = (maximum levels – vertex level) + (number bits at the highest level). Because of the possible vertex migration effect, the most logical way is to insert one vertex-refinement bit for each vertex of the current level after the mesh-description of that level. The remaining Vertices-Refinement description can be found after the highest-level Mesh-Description in order to further refine the position of all the vertices.

a

b

c

d

ef

g

h

a

b

c

d

ef

g

h

a

b

c

d

ef

g

h

a

b

c

d

ef

g

h

Figure 60: Demonstration of vertex migration via the Vertex-Refinement bits. The labeled vertices from the lower resolution Mesh-Description (left image) will migrate (some of them) to new reference-grid positions in the next higher resolution level (right image). For the sequence of vertices abcdefgh, the precision bits are: 00110110, which means that vertices a, b, e, and h will not migrate while the others do.

2.3.4.5 Surface extraction in MeshGrid format using the TriScan method

TriScan is an automatic surface extraction method for object representations expressed as scalar fields (e.g. discrete 3D data as illustrated in Figure 61(a)) or scalar functions (e.g. implicit surfaces as shown in Figure 61(b)-(c)). There are no limitations on the shape of the objects; all surface topologies are correctly handled. The aim of the TriScan method is to obtain the connectivity-wireframe of an object that can be kept standalone or represented as a MeshGrid.

(a) (b) (c)


ISO/IEC 14496-1:2001/PDAM 4

Figure 61: Shaded surfaces of triangulated MeshGrid representations of discrete and analytical models.The TriScan technique can be classified in the group of contour oriented methods for surface extraction methods, but distinguishes itself from ( )1 other contour oriented methods for surface extraction ([17, 44, 50]), performing the contouring in only one set of reference surfaces, and ( )2 from local methods extracting surface primitives using a lookup table ([57, 76, 77, 96]), although there are similarities between the types of surface primitives generated with TriScan (from the connectivity-wireframe) and the ones obtained by means of the local methods (e.g. the Marching Cubes algorithm [62]).

Similar to the explanations from section 2.3.4.2, the TriScan approach requires specifying a reference-system upon which the connectivity information between the vertices can be derived. The reference-system, i.e. the three sets of reference-surfaces, defines the reference-grid as illustrated in Figure 23. Inside any reference-surface RS, exists a 2D grid of reference-points which are the result of the intersections between reference-surfaces from a different set than the reference-surface RS belongs to. The reference-surfaces define the density and the distribution of the tessellation. Their position and shape can be chosen in an interactive or automatic way, such that the density is higher in areas where the curvature of the object is large, or is expected to change when animating the mesh.

The TriScan algorithm applies the contouring of the object in each of the reference-surfaces intersecting the object, such that by merging the connectivity information from each contour one obtains the 3-D connectivity-wireframe. The obtained connectivity-wireframe implicitly contains the information for the polygonization of the objects’ surface, if needed in a later stage. Building the MeshGrid representation is straightforward since both the connectivity-wireframe and the associated reference-grid are known as explained in section 2.3.4.2.2.

In order to obtain the contouring of the object in each reference-surface, the TriScan method uses a border-tracking algorithm. The border-tracking algorithm is visiting the border reference-grid positions and evaluates for each such border position a decision function based on the object description. Such a decision function should return (1) a binary value (e.g. decision functions for discrete data), one symbol used to indicate that the border reference-grid position is inside the object, and another symbol when it is outside the object, or (2) a ternary value that in addition to the previous mentioned two symbols has a symbol for the exact position of the border (e.g. decision functions from implicit surfaces). The decision function does an implicit sampling of the object representation at the positions specified by the reference-grid.

The border-tracking algorithm explained in the next section is designed to work on discrete representations. It is the task of the decision function to evaluate the original representation (discrete, mathematical, …) of the object and to return a binary or ternary value, such that the border-tracking can make the contouring.

2.3.4.5.1 The border-tracking algorithm

The discrete representation model employed for the border-tracking algorithm is illustrated in Figure 62. The reference-grid positions (u,v,w) are identified by the green dots from the center of each voxel. The voxel has a size of 1 relative to the distance (the step) between the reference-grid positions. Notice that the voxel faces from Figure62(a) are located at an offset of 0.5 relative to the center of the voxel, i.e. the reference-grid position. The discrete model employed for the border-tracking algorithm is compatible with the explanations in section 2.3.4.2.2 about the MeshGrid representation more precisely in the way the vertices are attached to the reference-grid positions, i.e. there exists a relative offset in the range [0,1) with a default value 0.5 between a vertex and the reference-grid point it is attached to. Figure 62(b) illustrates the discrete representation as it looks inside a reference-surface cutting the object. The discrete object is displayed in gray, while the part external to the object is displayed in white. The reference-grid positions contained in the reference-surface are displayed as green dots in the center of the voxels.

The following terminology is used to explain the border-tracking approach: (1) an object voxel is a voxel belonging to the object, (2) an external voxel is a voxel external to the object, (3) a border voxel is a voxel whose adjacent voxels are a mixture of object voxels and external voxels. Notice that a border voxel can be an object voxel or an external voxel.


Figure 62: The discrete model: Voxel v in (a) has the

coordinates (u,v,w), with its centre position shown as a green dot, and is delimited by six planes. In (b) a slice through the volume is taken.The border-tracking algorithm employs a connectivity model to determine the adjacent (connected) voxels to a given voxel. Two connectivity models can be used as illustrated in Figure 63 for the 2D and the 3D space: (1) the 6-connectivity model in 3D (Figure 63(a)) and its corresponding 4-connectivity model in 2D (Figure 63(c)), and (2) the 18-connectivity model in 3D (Figure 63(b)) and its corresponding 8-connectivity model in 2D (Figure 63(c)).

(a) (b) (c) (d)

Figure 63: Voxel adjacency. 3D case: (a) 6-connected and (b) 18-connected. 2D case: (c) 4-connected, and (d) 8-connected. It is feasible to design both a 2D and a 3D border-tracking algorithm. The current section discusses the 2D border-tracking algorithm; its 3D version is a generalization.

Border-tracking methods are iterative algorithms that, starting from an initial position on the surface of the body, follow that boundary such that, in the 2D approach a complete revolution around the object has been performed, and respectively in the 3D approach the whole object’s surface has been covered. For each iteration, the algorithm checks the neighborhood of the current discrete position, based on a specified connectivity model, and decides which discrete direction to explore for the next move (which is the next discrete position that has to be visited). The original boundary of the object can be approximated by connecting the successive border positions (vertices) found during the procedure.

The principle upon which the border-tracking methods are based was already known by the ancient Greeks: keep always right when walking through a labyrinth and you will always find the exit (entrance). The rule states that you should always turn in the same direction, be it right or left, when the followed path gives you the choice, and that you should always keep the border at the same side of the path.

The following conventions have to be made such that the border-tracking algorithm obtains the contouring according to the specifications discussed in section 2.3.4.2, and illustrated in Figure 24:


u v

w

(u+0.5, v+0.5, w+0.5)

Voxel v(u,v,w)

(u-0.5, v-0.5, w-0.5)

0 0

…

… j …

i-1

i

…

m j-1

n

(a) (b)

We have to decide whether the algorithm will track the border from outside or from inside the body. If we consider the object being the labyrinth than it would make sense to track the border from a path inside the body (see Figure 65 and Figure 66).

The position of the boundary relative to the direction followed by the path has to be set. In our implementation we keep the border at the right side of the followed path, such that the connectivity between the vertices is defined in counter clockwise CCW orientation for an external contour, and in clockwise CW orientation for an internal contour as shown in Figure 65 and Figure 66, and imposed for the connectivity-wireframe in section 2.3.4.2.

The neighborhood where the algorithm traces the border has to be set according to the chosen connectivity (adjacency) model (see Figure 63).

The border-tracking algorithm can handle both external and internal contours or surfaces. It only needs a correct initialisation, i.e. specify a position inside the discrete object and the direction of the border relative to that position. The border-tracking rules depend in addition on the connectivity model that is considered. The shape of the contouring result may depend on the connectivity model that was chosen as illustrated in Figure 70. Whatever the chosen connectivity model is, all the border positions are restricted to 4 orientations in case of a 2D contour (see Figure 65 and Figure 66), respectively 6 orientations in case of a surface (see Figure 24).

(a) (b)

Figure 64: External and internal contours obtained with the same contouring algorithm (same conventions): Global circular direction is CCW for an external contour in (a), (b), and CW for an internal contour in (b).Notice that although the “global” scanning of an external contour occurs in CCW direction, this still may “locally” result in an opposite scanning direction, as illustrated in Figure 64(a). In Figure 64(b), the scanning direction of the interior contour is globally CW. The inner and outer contours are scanned with the same strategy, namely: an observer inside the object looking in the direction of the scanning should see the contour on its right side.

The border-tracking algorithm needs as a starting point the position of a border voxel VO belonging to the object. In order to avoid the ambiguity of which boundary to track, in case the starting border voxel VO has two opposite border faces located on different boundaries (e.g. thin walls being one voxel thick), it is necessary to specify in addition the position of the border face with respect to the center of voxel VO. Any of the border voxels belonging to the object can be the starting point of the border-tracking algorithm.

Based on the chosen connectivity model, two border-tracking algorithms can be designed: (1) the 90 tracking algorithm based on the 4-connectivity model, and (2) the 45 tracking algorithm based on the 8-connectivity model.

The contouring of the object from Figure 62(b) using the 90° scanning algorithm is illustrated in Figure 65, respectively using the 45° scanning algorithm is illustrated in Figure 66. Since the object has a cavity both the external and the internal contour will be found by the border-tracking algorithm. The path followed by the border-tracking algorithm is shown with a dotted line passing through the centers of the voxels (reference-grid positions). The successive steps in tracking the border are shown by the black arrows. Each arrow indicates a new step, and its orientation shows the search directions for the boundary. Continuous black arrows reveal that the border is located in that direction, while dotted arrows are the steps in the followed path. The contour vertices are displayed as yellow dots and are positioned in the centers of the border faces. The final contour is shown in green, with arrows connecting the successive vertices, one for each border face encountered in the followed path. In Figure65(a) the algorithm tracks the external boundary of the object, while in Figure 65(b) the same algorithm tracks the


internal boundary. The global circular direction resulting from the tracking determines whether the algorithm found an external (CCW) or internal (CW).

N

S

W E

X (Y, Z)

Y (Z, X) N

S

E W

CCW CW

(a) (b)

Figure 65: The 90° border-tracking algorithm applied on a slice through a 3D volume: Tracking an external contour in CCW direction in (a), and an internal contour in CW direction in (b).

X, Y, Z

Y, Z, X N

S

W E W

N

S

E

CCW CW

(a) (b)

Figure 66: The 45° border-tracking algorithm applied on a slice through a 3D volume: Tracking an external contour in CCW direction in (a), and an internal contour in CW direction in (b).The 90 tracking algorithm is looking for the border in orthogonal directions ( -N-, -W-, -S-, -E-). The border-tracking rules are the following:

1. Store the first contour vertex on the border corresponding to the starting position, and begin the search, jumping to a voxel located in a direction rotated 90 CCW relative to the starting direction of the border.


2. When arriving to an external voxel VE as result of a step from an object voxel VO, store a contour vertex on the border face located between VE and VO, and return back to VO, rotate the search direction 90 CCW and jump to the next voxel.

3. When arriving to an object voxel VO as result of a step from another object voxel VO, rotate the search direction 90 CW and jump to the next voxel.

4. Finish the contouring when the starting border position has been found.

The 45 tracking algorithm is looking for the border in both orthogonal directions (-N-, -W-, -S-, -E-), and diagonal directions (-NW-, -SW-, -SE-, -NE-). The border-tracking rules are the following:

1. Store the first contour vertex on the border corresponding to the starting position, and begin the search, jumping to a voxel located in a direction rotated 45 CCW relative to the starting direction of the border.

2. When arriving to an external voxel VE as result of a step in a diagonal direction from an object voxel VO, return back to VO. Rotate the search direction 45 CCW and jump to the next voxel.

3. When arriving to a external voxel VE as result of a step in an orthogonal direction from an object voxel VO, store a contour vertex on the border face located between VE and VO, and return back to VO. Rotate the search direction 45 CCW and jump to the next voxel.

4. When arriving to an object voxel VO as result of a step in a diagonal direction from another object voxel then store a contour vertex on the border face located between VE and VO, where VE is positioned a step away from VO in a direction rotated 135 CW from the current search direction. Voxel VE has already been detected in a previous iteration. Continue the procedure by rotating the search direction 90 CW and jumping to the next voxel.

5. When arriving to an object voxel VO as result of a step in an orthogonal direction from another object voxel then store a contour vertex on the border face located between VE and VO, where VE is positioned a step away from VO in a direction rotated 90 CW from the current search direction. Voxel VE has already been detected in a previous iteration. Continue the procedure by rotating the search direction 45 CW and jumping to the next voxel.

6. Finish the contouring when the starting border position has been found.

The next section will make a comparison between both 2D border-tracking approaches.

2.3.4.5.1.1 Comparison between the 90° and 45° border-tracking algorithm

Both border-tracking algorithms obtain identical contouring results as illustrated in Figure 65 and Figure 66, except if a particular connectivity case is encountered as explained next.

A connectivity case table has been generated for the 2D border-tracking algorithms. The connectivity cases are derived for a group of four adjacent voxels, resulting in 16 (24) possible arrangements. Only unique connectivity cases are kept, since configurations that can be obtained by rotating the unique ones do not reveal any new contouring possibilities. These cases are illustrated in Figure 67. The same color and symbol conventions were used for the illustration of the connectivity cases (i.e. object voxels are gray squares, external voxels are white squares, the center of a voxel is marked with a white dot, contour vertices are yellow dots, green arrows are parts of the contour connecting two successive vertices, dotted arrows show the path triggered by the tracking rules).

Cases where no border voxel is present (i.e. all are object voxels or external voxels) do not contribute to the contouring (case 1 and 6). Cases 2, 3 and 5 yield the same contouring result (green arrow) for both border-tracking algorithms. In case 5a a different path (black dotted arrow) is followed by the 90° border-tracking algorithm with respect to the 45° border-tracking tracking algorithm (case 5b), yet still obtaining the same contouring result. The particular connectivity case that is handled completely different by the two border-tracking approaches is number 4. The reason for this, is that the voxels in configuration 4 are considered by the 4-connectivity rule as being disconnected, while the 8-connectivity rule regards them as being connected. The result of the 90° border-tracking


algorithm (4-connectivity) is case 4a, while the result of the 45° border-tracking algorithm (8-connectivity) is case 4b.

1 2 3 4a 4b 5a 6 5b

Figure 67: Case table for the 90° and 45° 2D border-tracking algorithms. Cases that can be derived via rotations are not displayed. The green arrows show the local contouring of the configurations, and each one of them connects two vertices (yellow dots). The dotted arrows indicate the path followed by the border-tracking algorithm when contouring adjacent border voxels belonging to the object. Cases 4a and 5a have been contoured with the 90°approach, while 4b and 5b with the 45°approach. The rest of the cases give similar contouring results for both algorithms.An example illustrating the difference between the contours obtained with both border-tracking approaches when the particular voxel configuration (case 4) is present, is shown in Figure 68. Notice that the 90° border-tracking algorithm sees only one contour as shown in Figure 68(a), while the 45° border-tracking algorithm generates an external and an internal contour for the same object, illustrated in Figure 68(b).

(a) (b)

Figure 68: Example of contouring with the 90° tracking algorithm in (a), and with the 45° border-tracking algorithm in (b).A more detailed view of the particular voxel configuration (case 4) from the example of Figure 68 is shown in Figure69. The dotted arrows, colored in red and blue, from the images indicate the path followed by the 90° border-tracking algorithm, respectively 45° border-tracking algorithm. For each step the search directions for the border are displayed as solid black arrows in Figure 69(a) and solid red, blue arrows in Figure 69(b) to differentiate between external and internal contours.

The reasoning and the rules triggered by the 90° border-tracking algorithm when contouring the object (shown Figure 69(a)) are described next. The starting voxel for the algorithm is voxel -1- and its border located in direction -N-.

Store a contour vertex on the border face of voxel -1- direction -N- (black arrow). Set the border search direction to -W- (black arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -1- direction -W- (black arrow). Set the border search direction to -S- (black arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -1- direction -S- (black arrow). Set the border search direction to -E- (blue dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -2-. Set the border search direction to -S- (black arrow) and test for the border in that direction.

…


Arrive at voxel -3-. Set the border search direction to -N- (blue dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -4-. Set the border search direction to -E- (black arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -4- direction -E- (black arrow). Set the border search direction to -N- (black arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -4- direction -N- (black arrow). Set the border search direction to -W- (black arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -4- direction -W- (black arrow). Set the border search direction to -S- (red dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -3-. Set the border search direction to -W- (black arrow) and test for the border in that direction.

…

Arrive back to voxel -2- and store the new contour vertex on the border face of voxel -2- direction -N- (black arrow). Set the border search direction to -W- (red dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -1-. Set the border search direction to -N- (black arrow) and test for the border in that direction.

A border position is found. Finish the contouring since the contour vertex found on the border face of voxel -1- direction -N- is the same as the starting point.

The reasoning and the rules triggered by the 45° border-tracking algorithm when contouring the external boundary of the object (shown Figure 69(b)) are described next. The starting voxel for the algorithm is voxel -1- and the border located in direction -N-.

Store a contour vertex on the border face of voxel -1- direction -N-. Set the border search direction to -NW- (red arrow) and test for the border in that direction.

No object voxel is found. Set the border search direction to -W- (red arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -1- direction -W- (red arrow). Set the border search direction to -SW- (red dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -2-. Store the new contour vertex on the border face of voxel -2- direction -N-, border that was detected in the previous step. Set the border search direction to -NW- (red arrow) and test for the border in that direction.

No object voxel is found. Set the border search direction to -W- (red arrow) and test for the border in that direction.

A new border position is found. Store the new contour vertex on the border face of voxel -2- direction -W- (red arrow). Set the border search direction to -SW- (red arrow) and test for the border in that direction.

No object voxel is found. Set the border search direction to -S- (red dotted arrow) and test for the border in that direction.


No border is found and the current position is voxel -3-. Store the new contour vertex on the border face of voxel -2- direction -W-, border that was detected in the previous step. Set the border search direction to -SW- (red arrow) and test for the border in that direction.

…

Arrive at voxel -4-. Store the new contour vertex on the border face of voxel -4- direction -N-, i.e. the border that was detected in the previous step. Set the border search direction to -W- (red dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -1-. Finish the contouring since the contour vertex found on the border face of voxel -1- direction -N- is the same as the starting point.

The reasoning and the rules triggered by the 45° border-tracking algorithm when contouring the internal boundary of the object (shown Figure 69(b)) are described next. The starting voxel for the algorithm is voxel -1- and the border located in direction -S-.

Store a contour vertex on the border face of voxel -1- direction -S-. Set the border search direction to -SE- (blue arrow) and test for the border in that direction.

No object voxel found. Set the border search direction to -E- (blue dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -4-. Store the new contour vertex on the border face of voxel -4- direction -S-, border that was detected in the previous step. Set the border search direction to -SE- (blue dotted arrow) and test for the border in that direction.

…

The path will not pass via voxel -3-, it will arrive at voxel -2-. Store the new contour vertex on the border face of voxel -2- direction -E-, border that was detected in one of the previous steps. Set the border search direction to -NE- (blue dotted arrow) and test for the border in that direction.

No border is found and the current position is voxel -1-. Finish the contouring since the contour vertex found on the border face of voxel -1- direction -S- is the same as the starting point.

1 2

4

1 4

2

N

S

W E

3 3

(a) (b)

Figure 69: Contouring detail for the 90° (a) and 45° (b) border-tracking algorithm.


2.3.4.5.2 Obtaining the connectivity-wireframe

The TriScan algorithm consists of two steps: (1) contouring the object in each of the reference-surfaces intersecting the object, and (2) generating the 3D connectivity-wireframe of the surface of the object.

As illustrated in Figure 24, every border face BF of a border voxel is intersected by two reference-surfaces RS1 and RS2, with RS1 and RS2 belonging to different sets. Inside each reference-surface RS1 and RS2 the border-tracking will make the contouring and obtain a contour C1 respectively C2. Each of the contours will contain at least one common vertex V located in the center of the border face BF. This can be generalized as follows: each vertex, located in the center of a border face BF, will belong to two contours C1 and C2 obtained by the border-tracking algorithm in reference-surface RS1 and RS2 from different sets. By merging the connectivity information for each vertex found inside those two reference-surfaces RS1 and RS2, one obtains the connectivity-wireframe. As explained in section 2.3.4.2, the indices of the connectivity vectors -LP1-, -LP2-, -LN1-, and -LN2 between a vertex V and its 4 neighboring vertices P1, P2, N1 and N2 will be chosen to satisfy equation . The obtained connectivity-wireframe satisfies the constraints imposed in section 2.3.4.2.

Since the connectivity-wireframe obtained with the TriScan method is based on a reference-system and its implicit reference-grid, migrating to the MeshGrid representation is straightforward as explained in section 2.3.4.2.2.

The MeshGrid representation and the corresponding connectivity-wireframe obtained with the TriScan method from a discrete object are illustrated in Figure 70 and Figure 71. Figure 70(a)-(b) and Figure 71(a)-(b) show the contouring result based on the 4-connectivity model, respectively Figure 70(c)-(d) and Figure 71(c)-(d) the contouring result based on the 8-connectivity model. The discrete object is displayed as a union of voxels, the vertices, located in the center of the voxel faces, are rendered as yellow dots. The connectivity-wireframe consists of the vertices and the connectivity between them. Although both figures show the same contouring result of the object, Figure 70 emphasizes the contouring in the three sets of reference-surfaces, each contour yielding the color associated to the reference-plane it belongs to, while Figure 71 stresses the connectivity vectors between the vertices of the connectivity-wireframe by coloring the -L1- connectivities in red, respectively the -L2- connectivities in blue. Figure 71 displays in addition the red and blue arrows indicating the scanning direction in the same way as illustrated in Figure 24. The red and blue arrows have been chosen in such a way that their cross product yields a vector that is normal to the surface of the object and oriented outwards (see equation ).

The orientation of the connectivity links is given by the border-tracking direction, as shown in Figure 65 when using the 4-connected model, respectively in Figure 66 for the 8-connected model.

(a) (b)


(c) (d)

Figure 70: Contouring a discrete object. In (a) and (b) the 4-connectivity (2D approach), respectively the 6-connectivity (3D approach) model was used. A slightly different shape, in (c) and (d), was generated when the 8-connectivity (2D approach), respectively the 18-connectivity (3D approach) model was used instead.

(a) (b)

(c) (d)

Figure 71: Contouring a discrete object. The left column shows the result obtained using the 4-connectivity (2D approach), respectively the 6-connectivity (3D approach) model. A slightly different shape, in the right column, was generated when the 8-connectivity (2D approach), respectively the 18-connectivity (3D approach) model was used instead. Notice the red and blue colored lines and arrows, symbolizing the L 1

and L2 connectivity links, and chosen in such a way that the cross product between L1 and L2 gives the normal vector at the surface in that position.

2.3.4.6 Storing Quadrilateral meshes in MeshGrid format

A quadrilateral mesh has the property that each vertex has 4 neighbors. The MeshGrid representation has the same constraint on the vertices’ connectivities as the quadrilateral meshes. A quadrilateral mesh is in fact a subset of the possible mesh representations supported by the MeshGrid format, since the latter may combine: triangles, quadrilaterals, pentagons, hexagons and heptagons.


There are three steps involved in the representation of quadrilateral meshes in the MeshGrid format: (1) defining the constrained reference-grid (see Figure 72(a)), (2) deforming and distributing the reference-grid to fit the quadrilateral mesh (see Figure 72(b)-(c)), and (3) deriving the connectivity-wireframe from the quadrilateral mesh and attaching it to the reference-grid (see Figure 72(d)-(f) displaying the multi-resolution MeshGrid).

Since the surface description is already known when building the MeshGrid representation for quadrilateral meshes, we have to define the 3D reference-grid in a constrained way: i.e. the number of reference-surfaces in the (u,v,w) directions are defined by the number of quadrilaterals in the mesh. This approach is contrary to the one explained for the TriScan method, where one has the freedom to choose the reference-surfaces cutting the object.

The reference-grid will be deformed to fit the volume of the quadrilateral mesh, and then uniformly distributed in order to achieve an efficient encoding. The vertices of the quadrilateral mesh can be used as boundary conditions for the deformation of the bounding reference-surfaces and for the distribution of the reference-grid points. Figure73 illustrates the distribution of the reference-grid; for illustration purposes the deformed and distributed grid has been shown in one slice only.

The quadrilateral mesh already contains the connectivity information between the vertices. In order to derive the connectivity-wireframe for the defined reference-surfaces, one retrieves the connectivity information between the vertices of the quadrilateral mesh and stores it for the corresponding vertices in the connectivity-wireframe as a connectivity vector on either -L1- or -L2- links as explained in section 2.3.4.2.

Once the connectivity-wireframe has been derived, it will be attached to the reference-grid as explained in section 2.3.4.2.2.

Notice that since the reference-grid is distributed to fit the volume of the quadrilateral mesh, the boundary grid positions can be chosen in such a way that the vertices lie at half the distance in between two boundary grid positions, thus yielding an offset of 0.5 (see Figure 73, the vertices are located on the reference-grid lines between the outside reference-grid points and immediate inside reference-grid points), which is the default value. This has two advantages: (1) a more compact stream can be obtained since default offset values do not have to be stored, (2) more easy and straightforward animation of the vertices relative to the reference-grid (rippling effects) is possible as explained in section 2.3.4.3.2.3.2.

Quadrilateral meshes find their applications for example in subdivision surfaces. Quadrilateral meshes can be generated in an automatic way from structured light scanners, which typically project a known light pattern onto the 3D surface of the object to be modeled. This light pattern is distorted by the surface relief and will be captured by the scanner. The quadrilateral meshes converted to the MeshGrid representation and illustrated in Figure 72 and Figure 73 have been acquired from a structured light scanner. The quadrilateral mesh from Figure 72 has been produced via an elliptical projection grid, while for obtaining the mesh from Figure 73 a cylindrical projection grid has been employed. The MeshGrid representation allows to convert a single-resolution quadrilateral mesh to a multi-resolution representation, as shown in Figure 72 (d) to (f) from the lowest to the highest resolution.

Figure 73 (a) shows the original surface, while Figure 73(b) shows the converted surface with the connectivity on it. A slice from the deformed reference grid is displayed in Figure 73(c) and (d) inside the mesh, and stand-alone, respectively.

Quadrilateral meshes can be converted to the MeshGrid representation in a lossless way. When encoding a bit-stream, as explained in section 2.3.4.4, the encoding of the connectivity is always lossless, while the reference-grid can be encoded losslessly if the quantization is high enough.


(a) (b) (c)

(d) (e) (f)

Figure 72: A spherical mapped quadrilateral mesh obtained from a light field scanner, visualized at different resolutions of the MeshGrid representation


(a) (b)

(c) (d)

Figure 73: Experiments with a cylindrical mapped scanner mesh obtained from a light field scanner.

2.3.4.7 Importing IndexedFaceSet models to the MeshGrid format.

Arbitrary IndexedFaceSet models have to remeshed in order to be brought to the MeshGrid format. For this conversion one has to go through the following steps: (1) define the appropriate sets of reference-surfaces, (2) derive analytical contours by intersecting the polygons of the IndexedFaceSet model with the reference-surfaces, (3) generate the MeshGrid representation from these analytical contours.


IndexedFaceSet models can be brought to the MeshGrid representation by remeshing using the TriScan method. (see Figure 75). The obtained MeshGrid representation might be optimized in terms of coding efficiency, by using the reference surfaces in an appropriate way (see Figure 74). In general, the remeshing can’t be done in a lossless way.

(a) (b) (c) (d) (e)

Figure 74: Remeshing of an eight model. Initial representation is an IndexedFaceSet (a), converted to a multi-resolution MESHGRID (b till e). The Reference-Grid has been set non-equidistant to better encode the topology and shape of the model

V T Sz

Original 3897 7798 269421

Resolution 1 24 50

Resolution 2 168 352



Multi-resolution 5476

V T Sz

Original 3897 7798 269421

Resolution 1 24 50




Multi-resolution 5476

VV TT SzSz

OriginalOriginal 38973897 77987798 269421269421

Resolution 1Resolution 1 2424 5050




Multi-resolutionMulti-resolution 54765476

(a)


(b) (c) (d) (e)

Figure 75: Remeshing of a femur bone. Initial representation is an IndexedFaceSet (a), converted to multi-resolution MESHGRID representation (b till e).

2.3.4.8 The MeshGrid Node

2.3.4.8.1 Node interface

MeshGrid { eventIn MFInt32 set_coordIndex

eventIn MFInt32 set_normalIndex eventIn MFInt32 set_texCoordIndex eventIn MFInt32 set_colorIndex exposedField SFCoordinateNode gridCoord NULL exposedField MFFloat vertexOffset []

exposedField MFInt32 vertexLink []exposedField SFCoordinateNode coord NULLexposedField SFNode normal NULLexposedField SFNode texCoord NULLexposedField SFNode color NULLexposedField SFInt32 displayLevel 0exposedField SFInt32 hierarchicalLevel 0eventOut SFInt32 nrLevels 1eventOut MFInt32 nrSlices [ 0 0 0 ]eventOut SFInt32 nrVertices 0

field SFBool solid TRUE field MFInt32 coordIndex []field MFInt32 normalIndex []

field MFInt32 texCoordIndex [] field MFInt32 colorIndex []}2.3.4.8.2 Semantic

The MeshGrid node is derived from the IndexedFaceSet node, thus it inherits the fields of the latter. As illustrated in the node interface only a number of the inherited fields are useful for the MeshGrid representation. The reference-grid points are specified by means of the gridCoord field of type SFCoordinateNode. This node is readable and writeable, in order to enable the updating of the reference-grid positions during grid based animation or progressive decoding of the reference-grid. When animating a hierarchical reference-grid, the hierarchicalLevel field, specifies for which resolution level the grid positions will be modified, this in order to allow updating or interpolating the reference-grid points belonging to resolution levels, higher than the specified hierarchicalLevel value. Since there is no field to set interpolation on or off, setting the hierarchicalLevel value larger or equal to the highest resolution level in the MeshGrid representation, will disable interpolation of the reference-grid points.


The connectivity-wireframe is defined using the vertexLink field, while the vertices refinement values are specified in the vertexOffset field. The first four values from the vertexLink field specify the absolute position (3 values) of the starting vertex and its absolute orientation (1 value), while the following pairs of values correspond to the connectivity-links to the other vertices, as explained in section 2.3.4.4.1. Several of such sequences can be specified one after another in the vertexLink field, in case the connectivity-wireframe has been divided into patches (view-dependent coding), when dealing with open meshes or when specifying a multi-resolution representation.

The vertexOffset field is of type MFFloat and specifies for each vertex the relative distance between that vertex and the reference grid point the vertex is attached to. Again several sequences of these values can be specified one after another, when dealing with multi-resolution representations, open meshes and patches. This field is exposed in order to enable small movements of vertices relative to the grid positions, they are attached to.

It is also necessary to know the number of reference slices in the u, v, w directions for the reference grid (nrSlices field of type MFInt32) and the number of resolution levels of the mesh (nrLevels field of type SFInt32). Number of vertices (nrVertices field of type SFInt32) is an event out, and is known after decoding the mesh. The displayLevel field of type SFInt32 specifies at which resolution level the mesh is displayed. The polygonal description of the mesh, will be generated when the MeshGrid representation is decoded. The vertices are stored in the coord field, and the indices of the vertices, defining the polygons are stored in the coordIndex field. Vertex animation, independent of the gridCoord and the vertexOffset are possible by changing the coord field.

2.3.4.9 Examples

The following is an example of the textual description of the MeshGrid node.

Shape { appearance Appearance { material Material {

diffuseColor 1.0 0.0 0.0ambientIntensity 0.1

} }

geometry MeshGrid {nrSlices [ 3 3 3 ]nrLevels 1displayLevel

hierarchicalLevel 0gridCoord Coordinate {

point [ -1.000000 -1.000000 -1.000000,

0.000000 -1.000000 -1.000000,

1.000000 -1.000000 -1.000000,

-1.000000 0.000000 -1.000000,

0.000000 0.000000 -1.000000,

1.000000 0.000000 -1.000000,

-1.000000 1.000000 -1.000000,

0.000000 1.000000 -1.000000,

1.000000 1.000000 -1.000000,

-1.000000 -1.000000 0.000000,

0.000000 -1.000000 0.000000,


1.000000 -1.000000 0.000000,

-1.000000 0.000000 0.000000,

0.000000 0.000000 0.000000,

1.000000 0.000000 0.000000,

-1.000000 1.000000 0.000000,

0.000000 1.000000 0.000000,

1.000000 1.000000 0.000000,

-1.000000 -1.000000 1.000000,

0.000000 -1.000000 1.000000,

1.000000 -1.000000 1.000000,

-1.000000 0.000000 1.000000,

0.000000 0.000000 1.000000,

1.000000 0.000000 1.000000,

-1.000000 1.000000 1.000000,

0.000000 1.000000 1.000000,

1.000000 1.000000 1.000000

]}

vertexLink [

1 1 1, 32

2 2

2 2

2 2

2 2

2 2

2 2

]

vertexOffset [

0.500000

0.200000


0.100000

0.400000

0.200000

0.500000

]

}}

The following example loads the MeshGrid representation from a binary file.

Shape

{

appearance Appearance { material Material {

diffuseColor 1.0 0.0 0.0ambientIntensity 0.1

} }

geometry CompressedMG {

url "cuboctahedron.mgh"

displayLevel 0

hierarchicalLevel 0

}

}

2.3.5 Particle systems

2.3.5.1 Overview

This type of geometry tools can be used to produce interesting computer graphics effects such as fire, smoke, rushing water, and so on. Particle systems provide a collection of rendering primitives that evolve over time.


Adapted from X3D specification [23, 49], particle systems are specified as::

Particles {exposedField SFBool enabled TRUEexposedField SFFloat particleRadius 0.1


exposedField SFFloat particleRadiusVariation 0exposedField SFFloat particleRadiusRate 0 exposedField SFVec3f emitterPosition 0 3 0exposedField SFVec3f emitVelocity 2.5 5 2.5 exposedField SFFloat emitVelocityVariation 0.0exposedField SFFloat creationRate 500# particles / secondexposedField SFFloat creationRateVariation 0 exposedField SFInt32 maxParticles 500 exposedField SFTime maxLifeTime 5 exposedField SFFloat maxLifeTimeVariation 0 exposedField SFVec3f gravity 0 -9.8 0 exposedField SFVec3f acceleration 0 0 0 exposedField SFColor emitColor 1 1 1 exposedField SFFloat emitAlpha 1.0 exposedField SFFloat emitColorVariation 0.0exposedField SFColor fadeColor 0.25 0.25 0.25 exposedField SFFloat fadeAlpha 1.0 exposedField SFFloat fadeRate 0.25 # factor / secondexposedField SFFloat minRange 1exposedField SFFloat maxRange 100exposedField SFInt32 primitiveType 2exposedField SFNode primitive NULL exposedField SFParticleInitializer init NULLexposedField MFObstacle obstacle []

}

enabled specifies whether the particle system emits new particles. This field could be routed from the isActive eventOut of a TimeSensor node. minRange and maxRange give the visibility range of the generated primitives. creationRate is the number of primitives generated per second. maxParticles is maximum number of primitives in existence at a single time. primitiveType specifies the type of the generated primitives:

0. point: each particles is represented as a point

1. line: each particle is represented as line between the old and new particle position

2. quadrangle: the particle is rendered as a screen aligned quadrangle. The parent shapes nodes will normally specify a textured Appearance node; the texture will be mapped 1:1 on each quadrangle

3. shape: primitive will contain a geometry node. For each particle the geometry node will be displayed using the current particles position, radius and color values.

4. coordinate: primitive will contain an Coordinate node which is updated with the current particle positions

5. face set (faces of a shape are animated). ??

Particle initializers:

The field init determines the particle emitter geometry.

If init is NULL particles are emitted at location emitterPosition.

The following nodes are special cases of emitters:

ParticleInitBox {exposedField SFVec3f size 1 1 1 # extend of the box


exposedField SFFloat falloff 0 # degree of randomness for # generated values (1: all values generated in center of the box, 0: values generated # randomly distributed across entire image of box

}

ParticleInitCone {exposedField SFVec3f direction 0 1 0 #direction vector of the coneexposedField SFFloat spread 1.4 # angular width of cone in radiansexposedField SFFloat falloff 0 # same as for ParticleInitBox

}

The emitters are translated to the position emitterPosition. Each internally generated particles store the following attributes:

SFFLoat t the current time of life

SFFLoat t1 the time of death

SFVec3f position the current position

SFVec3f velocity the current velocity

SFColor color the color

SFFloat alpha the alpha value

SFColor fadeColor the target color

SFFloat fadeAlpha the target alpha value

SFFloat fade progression from color to target color

The initial attribute values of a particle are computed from the corresponding fields in the particle system with optional random variation.

Particles are destroyed if t exceeds t1 or alpha reaches 0.

obstacle can contain the following nodes.

PlanarObstacle {exposedField SFFloat distance 0exposedField SFVec3f normal 0 1 0exposedField SFFloat reflection 0

}

if reflection is > 0 particles are reflected at this plane and the particles velocity is scaled with factor reflection.

PointAttractor {


exposedField SFFloat radius 100exposedField SFVec3f position 0 0 0exposedField SFFloat rate 1

}

If a particle is in the sphere given by position and radius, the particle is attracted via gravitational force towards position.

2.3.5.3 Example

DEF TS TimeSensor {cycleInterval 10enabled TRUE loop TRUE

}DEF Objects Switch {

choice [DEF PS Particles {

bboxSize -1 -1 -1lodRange 300particleRadius 0.75particleRadiusVariation 0.1particleRadiusRate 1emitterPosition 0 2 0emitterRadius 1.5emitterSpread 0.347705emitVelocity 8 8 8emitVelocityVariation 0.25creationRate 343.115maxParticles 600maxLifeTime 2gravity 0 0 0emitColor 0.95 0.95 0.95fadeColor 0.8 0.9 0.1fadeAlpha 0fadeRate 0.5

},DEF PS-S Shape {

appearance Appearance {texture DEF PS-TEX ImageTexture {url

"fire.png"}textureTransform TextureTransform {rotation

3.14}}geometry USE PS

},DEF PS2 Transform {

children Transform {children USE PS-Stranslation 2 0 15

}translation 0 -2 0

}]whichChoice -1

}DEF _Group Group {

children [DEF Ground Transform {

children DEF _Shape Shape {appearance Appearance {

material Material {diffuseColor 0 0 0emissiveColor 1 1 1transparency 0.2

}


texture ImageTexture {url "gray_ground.jpg"}textureTransform TextureTransform {scale 10

10}}geometry DEF Square IndexedFaceSet {

coord Coordinate {point [-1 0 -1,1 0 -1,1 0 1,-1 0 1]}

texCoord TextureCoordinate {point [0 0,1 0,1 1,0 1]}

coordIndex [0,1,2,3,-1]

creaseAngle 3.14solid FALSE

}}scale 50 50 50

},DEF PS-T Transform {

children USE PS2}

]}DEF PS-translation PositionInterpolator {

key [0,1]keyValue [0 0 0,0 0 0]

}DEF PS-rate ScalarInterpolator {

key [0,0.25,0.5,0.75,1]

keyValue [200,350,200,100,200]

}DEF PS-spread ScalarInterpolator {

key [0,0.25,0.5,0.75,1]

keyValue [0.25,0.35,0.3,0.1,0.25]

}ROUTE TS.fraction TO PS-rate.set_fractionROUTE PS-rate.value TO PS.set_creationRateROUTE TS.fraction TO PS-spread.set_fractionROUTE PS-spread.value TO PS.set_emitterSpread

2.4 Solid representation

2.4.1 Overview

Solid representation functionalities deal with:

reducing the size of files to be transmitted or shared by transmitting constructive commands rather than results;

increasing the rendered objects’ geometrical precision by managing a polynomial representation of volumes based on implicit primitives;

manipulating complex objects combined by solid operations. The algebraic representation is extended to the Arithmetic of forms.


Solid representation includes 3 geometry nodes (Implicit, Quadric and SolidRep) and extensions of the script language for implementation of the Arithmetic of forms.

2.4.1.1 Solid primitives

Any solid object has a three-dimensional form defined by a skin, which delimits the inside from the outside. Each point of this form can be deducted according to a mathematical formula. The surface will thus be an algebraic surface. If the surface equation is unknown or impossible to manipulate, it will be sectioned in smaller pieces until they arrive at known forms that we will call primitives. These primitives will then be assembled in order to constitute the original complex form.

Algebraic surfaces are generally defined by a parametric equation, which makes it possible to calculate the spatial coordinates of each surface point.

For example, the parametric equation of a sphere with radius R and centered at C is as follows:

(Px, Py, Pz) = (Cx, Cy, Cz) + (rcosBcosA, rcosBsinA, rsinB),

where A varies (0,2PI), and B varies (- PI/2, PI/2)

Each surface has its own equation that requires particular processing. On the other hand, because they allow to circulate on the surface, they are well adapted to meshing and thus easier to display according to polygonal rendering techniques.

Algebraic surfaces can also be represented by an implicit equation, which makes it possible to know if an unspecified point of space is inside, tangent or outside the volume delimited by this surface.

For example, the implicit equation of a sphere with radius R and centered at C is as follows:

(Px-Cx) 2 + (Py-Cy) 2 + (Pz-Cz) 2 – R 2 = 0

Thus for each point in space, if the result of the equation is < 0 it is inside, = 0, it will be tangent and > 0, it will be outside.

The implicit equation’s general form makes it possible to represent, in a very compact way, a very broad range of algebraic surfaces, a range as yet unexplored, by the way.

So, with an algebraic surface defined by a quadratic equation (second degree equation), the entire quadrics family is covered, while an equation of the fourth degree covers the huge quartics family.


Because of these possibilities, implicit forms are well adapted to the transmission of solid object representation in exact geometry.

2.4.1.1.1 Implicit surfaces

The implicit equation of the second degree that defines quadrics, is as follows:

where, for each point coordinates (X0,X1,X2,X3), the result is

whether the point is outside, on the surface or inside.

In the above equation, the point coordinates were made homogeneous by the addition of a fourth constant term

X3.

A surface of the second degree thus requires 10 coefficients; a surface of the fourth degree (quartic) requires 35 of them. The initialization of these coefficients makes it possible to numerically define any algebraic surface.

This surface can extend to infinity. For example the cylinder equation represents a cylinder that prolongs to infinity at both its extremities. It is therefore important to bind it (to circumscribe it by finite space) so that is can be processed and displayed.

In 3D modeling, using algebraic surfaces by coefficients is difficult because there is no direct obvious link between these coefficients and the resulting geometry.

In the field of projective geometry, an incidence-based geometry without measurement or parallelism, there are construction mechanisms that allow defining curves, surfaces and volumes from geometrical properties. These construction mechanisms were used as a starting point to resolve algebraic equations, allowing coefficient deduction of the corresponding quadric’s algebraic equation.

2.4.1.1.2 The quadric’s construction mechanism

Based on the principle of projective geometry, a geometrical construction mechanism for quadrics has been defined in [80, 82, 66] according to Pascal’s Theorem.

In fact, the geometric control of any quadric goes via the extension to 3D of the principle for constructing conics, using a construction triangle and a passing point. The quadric will be constructed and controlled with the help of a "construction tetrahedron" and two passing points located in the planes defined by two particular faces of the


tetrahedron. A set of six points will thus allow us to define the quadric from two conic sections that intersect in space at the two contact points on the tangent planes. (Fig. S1).

Fig S1.

The sphere in figure Fig.8 is constructed with the help of two circular sections belonging to two perpendicular planes. The two points P2 and P3 that serve as poles have been sent to infinity respectively along the Y-axis and the Z-axis (fourth coordinate equal to 0).

In general, for any quadric - projective, affine or metric - we shall define just six control points, P0 to P5, such that P0 and P1 are two contact points on two tangent planes to the quadric, P2 and P3 are two poles of the quadric and therefore belong to the two tangent planes, and P4 and P5 are two passing points for the quadric defining two conic sections of the quadric in the planes P0-P1-P2 and P0-P1-P3.

These construction mechanism principles may be extended to surfaces of higher degree (such as fourth degree, quartics).

2.4.1.1.3 Determining the coefficients of the quadric's polynomial

It is possible to determine the 10 coefficients of the quadric's general equation (5.1) from the previously-described quadric's geometrical construction system.

(Reference: [80] – pages 100 – 114)

2.4.1.1.4 Projective Coordinates System

In order to define all the quadrics with the 6 control points, it is imperative to remain in projective space. The coordinates system used in projective space is that of Grassmann-Plücker (one refers then to homogeneous coordinates), in which a point, a line and a plane respectively have 4, 6 and 4 coordinates. It can be interesting to make a parallel with non-homogeneous Euclidean coordinates. In this respect, one uses a representation of the


point in 3 dimensions of the form (X Y Z). Adding to this triplet a fourth coordinate equal to 1 does not modify in any way the correct operation of all geometrical procedures known as Euclidean. The first 3 coordinates will indifferently represent integers or decimal numbers, provided that the fourth coordinate remains equal to 1. By particularizing the fourth coordinate and giving it a 0 value, one goes from a Euclidean coordinates system to an affine coordinates system, in which it becomes possible to control and use the points to infinity. Thus, an observer located in (1 1 1 0) will be positioned at infinity in the direction determined by its Euclidean position (1 1 1 1) and the origin of the coordinate system (0 0 0 1). In addition to the possibility of managing infinity in a natural way, this approach makes it possible to solve the problem of representation and calculation of rational coordinates, which is fundamental to obtain perfect precision whenever the coordinates are initially rational or brought about to become. Let us consider for example the homogeneous coordinates point (1/3 5/6 0 1); it will be represented by the approximate quadruplet (0.333 0.833 0 1) or, much better, by the quadruplet (2/6 5/6 0/6 6/6) (2: 5: 0: 6).

2.4.1.2 Arithmetics of Forms

The solid representation is based on a logical modeling system for solid objects called "Design Logic" [80, 81]. The user combines the geometric primitives with the help of solid operators that enables to specify both the nature of the operations, such as adding or removing matter, and their scheduling.

The description of a solid object takes the form of a solid tree made up of operators that will be described below, and operands that may be either elementary primitives such as a sphere or cylinder, or more complex solid structures. In the first case we talk about the leaves of a volume tree and in the second case about solid sub-trees that display all the characteristics of a general tree.

The first concept to remember in Design Logic is density. In fact, every basic geometric primitive splits space into three regions - external, boundary and internal - coded by the integers 0, 1, 2. This initial ternary coding of space tells us the density of every point in space with respect to any elementary primitive. Using the implicit functions’ property of these primitives that verifies if a point of space is inside, outside or on the surface, it is possible to establish the ternary logic according to the function result.

For example, every point inside a quadric will have density 2 with respect to that quadric. Given a space thus arithmetized, we are able to combine primitives or elementary volumes with the help of arithmetic operators. This brings the capabilities to manage matter that is made up of volumes.

2.4.1.2.1 Solid operations

A complex volume consists of cell solid units assembled with the help of solid operations and producing a solid tree.

The most common operations are those used by the CSG (Constructive Solid Geometry), union, intersection and logical subtraction. These operations were extended to arithmetic operator applied on densities of an object [88].

These operations can easily be implemented for implicit-defined volumes as well as for volumes whose surface is polygonized, as long as their implicit attributes have been preserved (by an implication in a SolidRep node, for example).

An operation between two volumes is applied for each point in space and is independent of the final representation mode: for each point in space, the result is the density obtained from the operation carried out on these volumes’ densities..


2.4.1.2.1.1 General arithmetic operators on densities

The operators below, of purely arithmetic origin, achieve combinations of solid forms by isolating or selecting certain regions in a combinatorial way by means of their densities. The densities are always positive integer.

2.4.1.2.1.1.1 Addition

Arithmetic addition of two volumes

F0 + F1

F1

+ 0 1 2

F0

0 0 1 2

1 1 2 3

2 2 3 4

The addition of volume densities is the simplest operation that allows having multiple density space representing the original primitives as a whole. Encoding of density values must be used in order to obtain a unique density value for each section of matter. An example of a coding method is the Gödel [85] method, which enables whole integer calculation from any series of integers, and then restore (retrieve) this series form the initial number.

Addition

The illustration is a series of three figures:

- The figure at left represents two distinct spheres.

- The center figure represents the two spheres that intersect.

- The line figure represents the two overlapping spheres.


Each figure is made up of three pictures:

- The top picture indicates the two spheres’ respective positions.

- The center picture is the section result of the addition operation.

- The bottom picture shows the integral result of the operation.

A density is associated to each color:

- 0) black

- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.1.2 Multiplication

Arithmetic multiplication of two volumes

F0 * F1

F1

* 0 1 2

F0

0 0 0 0

1 0 1 2

2 0 2 4

While the operation of adding densities applies intuitively to the combinations of solids by way of the physical notions of adding or mixing matter, this is not true of the operation of multiplying two forms. The concept underlying this operation is the logical concept of conjunction. In fact, you will note in the truth table that only regions for which both densities are non-zero will get a non-zero density and hence be accepted (true). The other regions are systematically discarded (false). The truth table above, limited to densities 0 and 1, is in fact none other than the truth table for binary logical conjunction. The multiplication operation will thus allow us to eliminate regions not common to two forms and to dissect the common regions into parts that may be individually selected. Thus, the multiplication of two intersecting forms F0 and F1 allows us to isolate the vertices with density 1, the curved edges with density 2 and the face or region with density 4. The inclusion of F1 in F0 becomes, in the density level assessment, equivalent to the multiplication of F1 by the scalar 2


Lastly, the inclusion of the dual of F1 into F0 allows us to isolate the different densities of an annular region of density 4 and its two borders of density 2. Besides the preceding applications, essentially of a geometric nature, we shall find that the notion of multiplication is a central concept in operations for the selection and logical filtration of densities, and hence also for the associated regions.

Multiplication


- The left figure represents the two separate spheres.


- The right figure represents the overlapping spheres.

Each figure is made up of three pictures:- The top picture indicates the spheres’ respective positions.

- The center picture represents the section result of the product operation.



- 0 )black

- 1 )red

- 2 )yellow

- 3 )orange

- 4 )blue

2.4.1.2.1.1.3 Difference

Positive difference of two volumes:


F0 – F1

F1

- 0 1 2

F0

0 0 0 0

1 1 0 0

2 2 1 0

The positive difference of two forms F0 and F1, returns the difference between the densities if this result is nonnegative, and 0 if the result is negative. The truth table is not symmetric, so the order of the two subtracted forms is essential.

The positive difference of F0 and F1 when they intersect enables us to select the "lune" of density 2 and its two curved edges with density 1. However, the associated vertices with density 0 cannot be distinguished from empty space. The inclusion of F1 in F0 enables us to distinguish an annular region with density 2 and two borders with density 1. You will notice that ( 2 *( F0 - F1)) in this case is equivalent to (F0 *( ! F1)). This suggests the possibility of adapting different combinatorial strategies in Design Logic leading to identical "solid results", and also the possibility of an algebraic control over the equivalence of solid expressions. Finally, the inclusion of F0 in F1 illustrates an efficient way for the user to make a form temporarily invisible. This applies quite generally to any volume expression. By choosing a form F1 that encompasses a solid <vol> the (F1 - <vol>) expression will allow you to nullify all the densities of the volume. By dualizing F1, (( ! F1)- <vol>), you will see F0 reappear.

Difference


- The left figure represents two distinct spheres.



- The right figure represents the two overlapping spheres.



- The center picture is the section result of the difference operation.



- 0) black

- 1) red

- 2) yellow

- 3 )orange

- 4) blue

2.4.1.2.1.1.4 Exponentiation

Exponentiation of volumes:

pow ( F0 , F1)

F1

pow 0 1 2

F0

0 0 0 0

1 1 1 1

2 1 2 4

The Exponentiation operator raises one form to the power of another form. The density of a point with respect to F1 serves as exponent to the density of the same point with respect to F0. Notice that in the truth table, x0 = 0 if x = 0, and 1 otherwise. This convention is more precise than the one adopted by certain other computer languages where x0 = 1 for any x. In the case where F0 and F1 intersect geometrically, ^ makes it easy to isolate the part of F1’s border inside F0. We may also use this operator to give a geometric thickness to "border” regions with density 1, as illustrated by the inclusion of F1 in F0. In this case we may benefit from a coding that allows us to dissociate the interior of density 4, the interior of the border of density 2, the border of density 1 and the exterior of density 0. Lastly, the inclusion of F0 in F1 makes it possible to raise the internal density independently of the other regions.


Exponentiation




- The left figure represents the two overlapping spheres.



- The center picture is the section result of the involution operation.



- 0) black

- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.1.5 Greatest common divisor

the volume given by the greatest common divisor of two volumes

gcd (F0, F1)


F1

gcd

0 1 2

F0

0 0 1 2

1 1 1 1

2 2 1 2

The greatest common divisor operator has a symmetric truth table. This operator allows us to select regions relative to borders. All points belonging to at least one border get density 1. All interior points that do not belong to any border get density 2 and any other point is exterior and gets density 0. The inclusion of F0 in the dual to F1 gives an example of a partitioning of space into regions of density 1 if the points belong to at least one border and density 2 otherwise.

Greatest common divisor





Each figure is made up of three pictures:- The top picture indicates the two spheres’ respective positions.

- The center picture is the section result of the operation.




- 0) black

- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.1.6 Least common multiple

the volume given by the least common multiple of two volumes.

Lcm (F0, F1)

F1

lcm 0 1 2

F0

0 0 0 0

1 0 1 2

2 0 2 2

The least common multiple operator has a symmetric truth table. This operator allows us to isolate points that are interior to one of the forms and not exterior to the other one. These points have density 2. The points with density 1 are the common border points of the forms. The inclusion of F1 into F0 allows us to separate space into two categories: full matter, with density 2, and empty, with density 0. The inclusion of the dual of F1 in F0 allows us to create an annular region with no border.

Least common multiple







- The top picture indicates the spheres’ respective positions.

- The center picture represents the section result of the Kleene implication operation.



- 0) black

- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.1.7 Integral remainder (modulo)

the volume given by the Integral remainder of two volumes.

F0 % F1

F1

% 0 1 2

F0

0 0 0 0

1 1 0 1

2 2 0 0

The Integral remainder operator calculates the integer remainder when the density of a point with respect to F0 is divided by the density of that point with respect to F1. It is a delicate volume operator to manipulate because it may destroy the ternary "coherence" of interior, border and exterior. In fact, you may find configurations where "full" matter with density 2 is directly adjacent to empty matter with density 0. This is illustrated below (Fig 31). in the case where F0 and F1 intersect and where F1 is included in F0. However, this operator is useful for a direct


isolation of "border" regions of density 1, thus transforming a solid model into a surface model (where F0 included in F1).

Integral remainder (modulo)







- The center picture represents the section result of the operation.



- 0 )black

- 1 )red

- 2 )yellow

- 3 )orange

- 4 )blue

2.4.1.2.1.1.8 Absolute difference

the volume given by the absolute difference of two volumes.


Abs (F0 - F1)

F1

Abs

0 1 2

F0

0 0 1 2

1 1 0 1

2 2 1 0

The absolute subtraction between two forms returns the result of the subtraction between the two forms’ densities when this result is nonnegative. Otherwise, the return is the result opposed value. Consequently, every negative density becomes positive. The truth table is hence symmetric and the absolute subtraction of F0 and F1 when they intersect allows us to isolate two "lunes" of density 2 bordered by curved edges of density 1. The "lunes" are geometrically joined together but not logically, because the intersection lens between F0 and F1 with density 0 is linked to the region external to the two forms by the two vertices with density 0.

An interesting application of this configuration is to combine the absolute subtraction and the addition of F0 and F1, so as to obtain the solid expression ( ( F0 + F1) + (abs (F0 - F1))). In this case all regions interior to one or the other form will get density 4, the regions exterior to both forms will get density 0, and all remaining borders will get density 2. Here we have an example of complex solid modeling with interior, border and exterior all having homogeneous densities.

Absolute difference








- The center picture represents the section result of the absolute difference operation.



- 0 )black

- 1 )red

- 2 )yellow

- 3 )orange

- 4 )blue

2.4.1.2.1.1.9 Integral cube root

the volume given by the integral cube root of a volume.

icurt F0

icurt

F0

0 0

1 1

2 1

8 2

2.4.1.2.1.1.10 Integral square root

the volume given by the integral square root of a volume.

isqrt F0


isqrt

F0

0 0

1 1

2 1

8 2

2.4.1.2.1.1.11 Maximum

the volume given by the maximum of two volumes.

Max(F0 , F1)

F1

max

0 1 2 8

F0

0 0 1 2 8

1 1 1 2 8

2 2 2 2 8

This operator is equivalent to the Kleene union for n-ary logic.

2.4.1.2.1.1.12 Minimum

the volume given by the minimum of two volumes.

Min(F0 ,F1)


F1

min 0 1 2 8

F0

0 0 0 0 0

1 0 1 1 1

2 0 1 2 2

This operator is equivalent to the Kleene intersection for n-ary logic.

2.4.1.2.1.2 Ternary logic : the Kleene operators

The following operators use the ternary logic only and , in many cases, are sufficient to carry out complex solid modeling. Four of these operators have been directly borrowed from the system of ternary logic established by S. Kleene in 1952 [54]. The user familiar with the CSG systems will find that Boolean operators are binary subcases of the Kleene operators. But the latter avoid special cases introduced by the way CSG handles boundaries, and they may at any time benefit from the possibility of getting access to the densities of the regions for more sophisticated solid operations.

Note: if used on a n-ary logic, the densities are taken modulo 2 to convert the value into the ternary logic.

2.4.1.2.1.2.1 Kleene union

the volume given by the Kleene union of two volumes.

F0 | F1

F1

| 0 1 2

F0

0 0 1 2

1 1 1 2

2 2 2 2

The first Kleene operator is the union of two forms.The truth table is symmetric and the logical interpretation of the density values is the following: every interior point for one or the other form will get density 2; every exterior point to both forms will get density 0 and finally any other point will get density 1. In other words, all non interior border points get the density 1. This operator allows us to merge the matter of several forms when these forms are either in geometric intersection or in inclusion, but it also allows us to create a solid expression for disjoint forms. For example, the Kleene union of a table and a separated chair gives a solid description of the two pieces of furniture in a single solid expression.


Kleene union







- The center picture represents the section result of the Kleene union operation.



- 0 )black

- 1 )red

- 2 )yellow

- 3 )orange

- 4 )blue

2.4.1.2.1.2.2 Kleene intersection

the volume given by the Kleene intersection of two volumes.

F0 & F1


F1

& 0 1 2

F0

0 0 0 0

1 0 1 1

2 0 1 2

The second Kleene operator is the intersection of two forms. The truth table is again symmetric and the logical interpretation of the density values is the following: every point interior to all forms will get density 2, every point exterior to one or more of the forms gets density 0, and the remaining points - being border points for at least one form and on the border or in the interior for the other forms get density 1. The intersection of the forms F0 and F1 in the figure below allows us to isolate a lens-shaped region of density 2, with two vertices and two curved edges, all four with density 1. In terms of densities, the lens is logically equivalent to a basic primitive form since we find the homogeneous ternary coding: 0 outside, 1 at the border and 2 inside. Finally, the Kleene intersection of forms F0 and F1 when one is included in the other allows us to "protect" the innermost form. In fact, its densities are not altered and furthermore, any non-collision of a geometric element with the external form necessarily implies non-collision with the internal form. However, this type of protection is normally only valid for models achieved exclusively with Kleene operators.

Kleene intersection






- The top picture indicates the spheres’ respective positions.

- The center picture represents the section result of the Kleene intersection operation.




- 0) black

- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.2.3 Kleene implication

the volume given by the Kleene implication of two volumes.

F0 >> F1

F1

>> 0 1 2

F0

0 2 2 2

1 1 1 2

2 0 1 2

The next Kleene operator is the implication and we shall "prefer" its dual .

It is clear that the dual implication of two forms in intersecting position allows you to isolate a lune-shaped region of whose the borders (curved edges and vertices) are uniformly dense (density 1). You may compare this operation with a "homogeneous" removal of matter that you cannot achieve with the positive difference operator. In fact, the latter eliminated the two vertices that had 0 density and only allowed a "heterogeneous" removal of matter. It is certain that, in spite of its theoretical and practical limitations, the objectives of the CSG subtraction operator are very close to those of the dual Kleene implication. Regarding the inclusion of F1 in F0, this allows us to isolate, using the dual implication, an annular region with density 2 bounded by two borders of density 1.

Finally, the inclusion of F1 into F0 results in every point having a density 2, already obtained in a different way in (Addition) , or possibly by doing ( ! ( F0 - F1)) in (Positive difference) in the same case.


Kleene implication

Dual Kleene Implication

The illustrations are a series of three figures:





- The center picture represents the section result of the Kleene operation and of it’s dual.



- 0) black


- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.2.4 Reciprocal Kleene implication

The volume given by the reciprocal Kleene implication of two volumes.

F0 << F1

F1

<< 0 1 2

F0

0 2 1 0

1 1 1 1

2 0 1 2

The last Kleene operator is the converse implication and we shall also "prefer" its dual. Then we benefit from symmetric truth tables. When F0 and F1 intersect, the logical interpretation of the dual converse implication is that all interior non-common points of F0 and F1 get density 2, all border points get density 1 and all remaining points, exterior to the resulting form, get density 0.

Reciprocal Kleene implication


Dual reciprocal Kleene implication






- The center picture represents the section result of the reciprocal Kleene implication and of it’s dual operation.



- 0) black

- 1) red

- 2) yellow

- 3) orange

- 4) blue

2.4.1.2.1.3 The dual operator

the volume given by the ternary dual of the volume

! (F0)


!

F0

0 2

1 1

2 0

The dual operation can be used alone or in conjunction with others ternary operators to consider the matter as empty space and the empty space as matter inside the solid tree.

2.4.1.2.2 Densities filtering

Systematic exclusion of matter with density zero may enable to avoid any filtering. For example, visualization of the surface and interior of a quadric follows from the simple preservation of densities 1 and 2, and discarding density 0 by convention.

Nevertheless, when the combinatorial logic of the designer requires precise control of the densities’ numerical transition states, we may need a totally explicit filtering.

In this case the multivalued nature of the different regions in space allows for a large variety of arithmetic selections of a single volume object.

2.4.1.2.2.1 The Filtering Functions

The simpler way to filter densities is to define the densities to be considered at the display time. This can be done setting the density list in the SolidRep node.

To include the filtering inside the solid tree as shown in the previous example in the functional pseudo-code, a set of filtering functions must be defined at the same level as the solid operators.

To cover all the needs, the functions are:

2.4.1.2.2.1.1 Equality filter

The volume that checks the equality test, and 0 otherwise.

F0 == F1

F1


== 0 1 2 8

F0

0 0 0 0 0

1 0 1 0 0

2 0 0 2 0

8 0 0 0 8

2.4.1.2.2.1.2 Greater than or equal filter

The second volume if the first one is greater than or equal to the second volume, and 0 otherwise.

F0 >= F1

F1

>= 0 1 2 8

F0

0 0 0 0 0

1 0 1 0 0

2 0 1 2 0

8 0 1 2 8

2.4.1.2.2.1.3 Greater than filter

The second volume if the first one is greater than the second volume, and 0 otherwise.

F0 > F1

F1

> 0 1 2 8

F0

0 0 0 0 0

1 0 0 0 0

2 0 1 0 0

8 0 1 2 0


2.4.1.2.2.1.4 Less than or equal filter

The second volume if the first one is less than or equal to the second volume, and 0 otherwise.

F0 <= F1

F1

<= 0 1 2 8

F0

0 0 1 2 8

1 0 1 2 8

2 0 0 2 8

8 0 0 0 8

2.4.1.2.2.1.5 Less than filter

The second volume if the first one is less than the second volume, and 0 otherwise.

F0 < F1

F1

< 0 1 2 8

F0

0 0 1 2 8

1 0 0 2 8

2 0 0 0 8

8 0 0 0 0

2.4.1.2.2.1.6 Even filter

The volume density if the density is even, and 0 otherwise.

Even F0


even

F0

0 0

1 0

2 2

8 8

2.4.1.2.2.1.7 Odd filter

The volume density if the density is odd, and 0 otherwise.

Odd F0

odd

F0

0 0

1 1

2 0

3 3

2.4.1.2.2.1.8 Difference filter

The second volume density if the densities are different, and 0 otherwise.

F0 != F1

F1

!= 0 1 2 8

F0

0 0 1 2 8

1 0 0 2 8

2 0 1 0 8

8 0 1 2 0

2.4.2 Node specification

Implicit { # % NDT = SFGeometryNodeexposedField SFVec3f bboxSize 2.0 2.0 2.0exposedField MFFloat c [ ] #10 coeffs for quadrics,

# 35 coeffs for quartics


exposedField SFBool solid FALSEexposedField SFBool dual FALSEexposedField MFInt32 densities [] # ( 1, 2)

}

where:

bboxSize : bounding box dimension

c coefficients of following polynomials :

for the quadric :

for the quartic :

solid: solid (TRUE) or surface (FALSE)

dual unary duality operation to be applied to volume initialization. Implies solid == TRUE.

densities: 2 values: the first value is the density code associated to the skin and the second value is the density associated to the inside.

The surface is created when the node is created. The origin of the axis system is the bounding box’s center. An implicit implementation will keep the volume in a polynomial form. A polygonal implementation will also generate the mesh according to a fixed or variable precision, up to the limits of the bounding box. The surface will be clipped by the bounding boxes. If solid is set to TRUE, the volume will be closed and intersected with the bounding boxes.

If the dual field is TRUE, the volume is set to solid and the difference operation with bounding box will be carried out (bounding box - solid).

The use of bounding boxes for Implicit nodes makes it possible to limit infinite surfaces and to optimize processing of non polygonal rendering.


The densities are used when including the primitive into a SolidRep node with solid operation (see Solid Operations and language specification). The density value is an Int32 and can be encoded using an encoding scheme like Gödel [85].

As a benefit of the implicit definition, the function IsOutside (Point &p) will return 3 values:

1: if the point is outside the volume defined by the surface

0: if the point is on the surface

-1: if the point is inside the volume defined by the surface

Quadric { #%NDT=SFGeometryNodeexposedField SFVec3f bboxSize 2 2 2exposedField SFVec4f P0 -1 0 0 1exposedField SFVec4f P1 1 0 0 1exposedField SFVec4f P2 0 1 0 0exposedField SFVec4f P3 0 0 1 0exposedField SFVec4f P4 0 1 0 1exposedField SFVec4f P5 0 0 1 1exposedField SFBool solid FALSEexposedField SFBool dual FALSEexposedField MFInt32 densities [] # ( 1, 2)

}

where:

bboxSize bounding box dimension

P0, P1 2 points tangent to the quadric

P2, P3 2 poles of the construction tetrahedron

P4, P5 2 passing points of the quadric

solid solid (TRUE) or surface (FALSE)

dual unary duality operation to be applied to volume initialization. Implies solid == TRUE.

densities 2 values: the first value is the density code associated to the skin and the second value is the density associated to the inside.

Each point is defined using homogeneous coordinates allowing the point to be sent to the infinity. The values are relative to the unitary bounding box (from –1 to +1). If the absolute value of a coordinate is greater than 1, the point is outside the bounding box.

This node creates an Implicit node of the second degree through a geometric interface. The polynomial coefficients will be calculated according to the geometrical construction method using a construction tetrahedron and 2 passing points described above in the quadric’s construction mechanism.

A short cut could also be established in order to direct the process of polygonization towards algorithms that are


simplified according to specified control points. For example, it would be faster to use a simplified polygonization algorithm to generate a cone, a sphere, a cylinder or a cube than to use a general implicit surface algorithm.

The various preset solid cells could use these characteristics. They could be implemented using a prototype definition (PROTO) that would fix the control points or the coefficients if they are known.

A continuous volume deformation could be implemented by moving the control points.

The densities are used when including the primitive into a SolidRep node with solid operation (see Solid Operation and language specification). The density value is an Int32 and can be encoded using an encoding scheme like Gödel [85].

SolidRep { #%NDT=SFGeometryNodeexposedField SFVec3f bboxSize 2.0 2.0 2.0exposedField SFNode solidTree NULLexposedField MFInt32 densityList []

}

where :

bboxSize : bounding box dimensions

solidTree: geometry to be initialized as solid

densityList: set of densities to select for display – default: all

Volume can come from various geometry. If this geometry does not define a closed volume, it will not be considered.

The result of a solid operation is always preserved in a SolidRep node. The SolidRep node is always the root of a solid tree. This solid tree is not processed until the display request. Only the root of the complete tree will be processed (and not the sub-trees).

A SolidRep node can be included in a Transform node. The implicit operation of the children of a Transform node or of any grouping node is the Kleene union operation.

Texture mapping onto implicit surfaces can be done by using any algorithm. The gradient algorithm [104] is an example of an algorithm well suited for implicit surfaces.

2.4.3 Language specification

The SolidRep node can also contain the result of a solid operation on one or more primitives and/or other SolidRep and/or other MPEG-4 standard geometrical nodes .


This node makes it possible to unify the set of VRML volumes as a cell and to allow solidification of the set of 3D geometrical primitives as MPEG-4 standard language.

These operations are accessed through script language, whose selected operators (+, -...) were overloaded for operations between Geometry nodes.

A non implicit solid geometrical node is first of all transformed into a SolidRep node before being used as a solid operation’s operand.

Syntax: SolidRep.solidTree = {Geometry / Int32} < op > {SolidRep/Implicit/Quadric}

The operands can be:

An implicit primitive ( Implicit or Quadric) with “solid = TRUE” and densities;

A SolidRep containing a sub-solid tree;

A non-implicit geometry (only solid primitives are supported like sphere, cylinder, cone, box);

An Int32 value representing the density of the entire space (used as operand for implicit filtering of densities)

An expression with only assignment makes it possible to convert a non-implicit geometrical primitive into a bound implicit solid.

The complete set of solid operations of the Arithmetic of Form from [80, 82, 88] are supported:

<op>

General arithmetic operators [ + ] Addition

[ * ] Product

[ - ] Difference

[ pow ] Exponentiation

[ gcd ] Greatest common divisor

[ lcm ] Least common multiple

[ % ] Integral remainder (modulo)

[ abs (v1-v2) ] Absolute difference


[ icurt ] Integral cube root

[ isqrt ] Integral square root

[ max ] Maximum

[ min ] Minimum

Ternary logic: Kleene operators [ | ]Kleene union

[ & ] Kleene intersection

[>>] Kleene implication

[<<] Reciprocal Kleene implication

[!] Dual (not)

In addition a set of test functions can be applied on SolidRep nodes. These functions are used to filter densities.

[ == ] Equality filter

[ >= ] Greater than or equal to filter

[ > ] Greater than filter

[ <= ] Less than or equal to filter

[ < ] Less than filter

[ even ] Even filter

[ != ] Difference filter

[ odd ] Odd filter

Scripts are used through Script node

The program associated with a Script node is evaluated in three situations:

Upon initialization: when the program is read for the first time;

Upon termination: when the program is withdrawn from the memory. This can happen when the current world is replaced by another or when the node’s URL field is modified;

Upon reception of events: those intervene when a value is received by the node according to an eventIn interface.

A Script node can be included as descending from any grouping node but is independent of the current coordinate system. Typically these nodes are placed at the end of the VRML file’s most external group.


These scripts can be written in various languages, depending on the VRML viewer in use, but typically, most of them support JAVA and JavaScript.

For reference the solid extension was carried out in the JavaScript interpreter named VRMLScript included in the MPEG reference software.

2.4.4 Rendering of implicit surfaces

The display of a volume defined by implicit surfaces requires the establishment of techniques linked to ray tracing if one does not want to lose precision. For Instance SGDL uses this type of technique optimized in order to obtain rendering speeds similar to polygonal techniques. SGDL libraries thus generate the scene’s 3D picture frame by frame. This picture can be merged with the one generated by the rendering of the other elements (for example polygonal) while accessing directly the graphic rendering Z-buffer system.

On the other hand, if one accepts less geometrical precision, it is possible to consider the tessellation of implicit surfaces before the rendering, for example see [24]. Thus, solid are directly inserted in the polygonal scene and are processed in the same way.

2.4.5 Examples

A cell primitive

Dual of quarter cylinder defined by projective hexahedron

# Dual of quarter cylinder

PROTO GDlychxa [

field SFVec3f size 5 5 5

field SFBool dual TRUE

]{

Quadric {

P0 -3 0 0 1

P1 1 0 -1 1

P2 0 1 0 0


P3 0 0 1 0

P4 0 1 0 0

P5 -1 0 1 1

bboxSize IS size

dual IS dual

}

}

A quartic defined by 35 coefficients

Tangle cube

VRML code:

Group {

children Shape {

# tangle cube

geometry Implicit {

bboxSize 5 5 5

solid FALSE

c [

-1,0,0,0,-1,0,0,0,0,0,0,0,0,0,-1,0,

0,0,0,0,0,0,0,0,0,5,0,5,0,0,5,0,

0,0,-10.2]

}

}


Solid operations and densities

Example of density use: a single model carries the characteristic of the matter and can be displayed differently

Egg white : density 1

Egg yolk : density 3

Operation: addition

Full egg with yolk inside

Cut by a box (density 4)




Densities displayed

1,3,4 1,4 1,3,4 1 4

VRML code:

# b: egg cut by a box

Group{

children [

# Primary SolidRep

DEF Le_Solid3 SolidRep{

bboxSize 5 5 5

densityList [1 3 4] # choice of densities to display

dual FALSE

},

DEF Le_Script Script{ # Solid Tree definition

field SFNode Srepout USE Le_Solid3

#primitives

field SFNode White # container

Transform {


translation 0 0 0

children [

Shape {

…

}

geometry

Quadric { P0 1 0 0 1

P1 -1 0 0 1

P2 0 1 0 0

P3 0 0 1 0

P4 0 1 0 1

P5 0 0 1 1

boxSize 1 1.3 0.95

solid FALSE}

}

]

}

field SFNode Yolk # matter inside

Transform {

children [

Shape {

…

geometry

Quadric { P0 1 0 0 1

P1 -1 0 0 1

P2 0 1 0 0

P3 0 0 1 0

P4 0 1 0 1

P5 0 0 1 1

bboxSize 0.7 0.7 0.4


density 3

solid FALSE}

}

]

}

field SFNode CuttingBox # cutting tool

Transform {

translation 0.5 0 0

children [

Quadric { P0 -1 0 0 1

P1 1 0 0 1

P2 0 1 0 0

P3 0 0 1 0

P4 0 1 0 0

P5 0 0 1 0

bboxSize 1 1 1

density 4

solid FALSE}

]

}

directOutput TRUE

url "vrmlscript:

function initialize(){

Srepout.solid = (White + Yolk)- CuttingBox;}

"

} ]

}


2.5 Texture tools

2.5.1.1 Procedural texturing

2.5.1.1.1 Overview

Texture mapping is used extensively in 3D modeling as means for increasing model detail and perceived surface quality while maintaining modeling and rendering complexity to a minimum. The most common form of texture mapping is through the ImageTexture and PixelTexture nodes. The 2D image they describe is subsequently mapped to a 3D model via the geometry node’s texture coordinate mechanism.

Complex textures, however, will usually require large images implying that a significant portion of the available bandwidth is devoted to texture data. The Superscape procedural textures employ a compact parameterization technique that is capable of reproducing a wide range of gradients, man-made and organic texture images which can subsequently be mapped on 3D models using standard tools. The procedural texture parameters can be used to specify many features of the texture map, including:

Colors and how they vary across the texture.

How the texture is divided into similar-shaped cells, and the shape and color variation of any such cells.

Whether the texture has a regular, man-made look, or a less regular, organic appearance.

(a) (b) (c)

(d) (e)


Figure 76: The 5 cell types: (a) Rectangle, (b) Brick, (c) Weave, (d) Hexagon, (e) Ring.

The processes underlying the creation of the textures include the generation of a fractal ‘plasma’ field, subdivision of the texture into cells, spatial distortion of the texture, selection of colors to apply to the texture, and control of how the colors vary within a cell. The five cell types supported are shown in Figure 76.

2.5.1.1.2 Texture Generation

Underlying all procedural textures is a pseudo-random fractal ‘plasma’ (section 2.5.1.1.2.1) which is constructed so that it can be tiled horizontally and vertically without discontinuities between one tile and its neighbor(s).

The generation of the texture is effectively a mapping from a pair of coordinates (x, y) to a color value. The stages in this procedure are as follows:

1. The coordinates are ‘warped’ (section 2.5.1.1.2.2) to produce (xwarp, ywarp) using a set of user-supplied parameters. This can be used to produce large-scale variations in the texture, such as a gradual shift from bottom to top, or ripple effects.

2. The warped coordinates are subsequently distorted to produce (xdist, ydist). This can be used to change a regular cell pattern, such as a simple square grid, into something more ‘organic’, perhaps similar to the scales on crocodile skin. The distortion itself is generated using the plasma so the result is tileable. It also produces the appearance of stretched and compressed regions, rather than the ‘noisy’ image that would result from completely random distortion (Figure 77).

3. The distorted coordinates are subsequently analyzed to determine which cell they lie in (the basic cell types are shown in Figure 76). A record is made of the center of the cell (xcenter, ycenter) and the offset (xoffset, yoffset) of the distorted coordinates from the cell origin. In addition, a polar coordinate representation of the offset is calculated (roffset, aoffset), and eoffset, a variation on roffset representing distance to the nearest cell edge. Repeatable random values are derived from the cell center position, and from the distorted coordinates. Plasma values are looked up corresponding to the cell center (pcenter), the distorted coordinates (pdist), and a fixed offset from the distorted coordinates (poffset).

4. A weighted blend of (x, y, xwarp, ywarp, xdist, ydist, xoffset, yoffset, roffset, eoffset, aoffset, pcenter, pdist, poffset) is used to produce a pair of values which, after a further warping stage, are merged with two further weighted repeatable random values (randcell, randdist). The repeatability is achieved through cell specific seeding of a pseudo-random number generator (section 2.5.1.1.2.3).

5. The resulting coordinates are used to interpolated between four colors, the result of which is the RGB pixel data for the texture at position (x, y).

Figure 77: The effects of distortion: (a) 25%, (b) 50%, (c) 75%, (d) 100%.

The generation of the texture (T) can be represented by the following pseudo-code:


P[] = plasma(width, height, roughness, seed)

for every pixel x, y

xwarp = remap(x, xwarpmap[], xwarpmapsmooth) * width

ywarp = remap(y, ywarpmap[], ywarpmapsmooth) * height

xdist = xwarp + P[(xwarp + width / 2) % width, ywarp] * width * distortion

ydist = ywarp + P[xwarp, (ywarp + height / 2) % height] * height * distortion

Create vector v[] of 16 cell-related values:

x, y

xwarp, ywarp

xdist, ydist

xoffset, yoffset = offset of xdist, ydist from cell bottom-left

roffset = radial distance of xdist, ydist from cell center

eoffset = distance towards nearest edge

aoffset = angle of xdist, ydist around cell center

pcenter = P[xdist, ydist] // per-cell plasma value

pdist = P[xcenter, ycenter] // per-pixel plasma value

poffset = P[xdist + width / 2, ydist + height / 2]

randcell = nonrandom(xcenter, ycenter) // per-cell random value

randdist = nonrandom(xdist, ydist) // per-pixel random value

A =

15

0

15

0

][

][][

i

i

iaweights

iaweightsiv

B =

15

0

15

0

][

][][

i

i

ibweights

ibweightsiv

Awarp = remap(A, awarpmap[], awarpmapsmooth)

Bwarp = remap(B, bwarpmap[], bwarpmapsmooth)

c01 = color[0] + Awarp * (color[1] - color[0]);

c23 = color[2] + Awarp * (color[3] - color[2]);


T[x, y] = c01 + Bwarp * (c23 – c01);

where:

Width horizontal texture size

Height vertical texture size

Roughness amount of large-scale variation in texture

Distortion amount of distortion applied to image

Seed seed for random number generator

Colors[4] colors contributing to texture

Xwarpmap[],xsmooth curve parameters for warping texture horizontally

Ywarpmap[],ysmooth curve parameters for warping texture vertically

Awarpmap[],asmooth curve parameters for warping A

Bwarpmap[],bsmooth curve parameters for warping B

Aweights[16] weights of factors contributing to A

Bweights[16] weights of factors contributing to B

2.5.1.1.2.1 Fractal Plasma Calculation

The plasma is a two-dimensional array with the same dimensions as the final texture ( heightwidth ). It contains a single value per element ( ]1,0[],1,0[],5.0,5.0[),( hjwijiP ) that varies over the plasma in a manner reminiscent of hills and valleys. The generation of the plasma is controlled by a seed for a pseudo-random number generator (section 2.5.1.1.2.3) and by a roughness parameter that controls how widely spaced the peaks and troughs will be (Figure 78).

(a) (b) (c)

Figure 78: The effects of the roughness parameter: (a) 3, (b) 5, (c) 8.


The plasma array (P) is calculated as follows:

plasma(width, height, roughness, seed)

{

srandom(seed);

L = width * height;

P[0, 0] = 0;

w = width;

h = height;

rmult = 1;

while (w > h)

{

step = w / 2;

p1 = 0;

while (p1 < width)

{

v = random() - 0.5;

if (step < roughness)

{

rmult *= 0.5;

v = v * mult + (P[p1, 0] + P[(p1 + w) % width, 0]) / 2;

}

P[p1 + step, 0] = v;

p1 += w;

}

w = w / 2;

}

while (h > w)

{

step = h / 2;


p1 = 0;

while (p1 < height)

{

v = random() - 0.5;


{

rmult *= 0.5;

v = v * rmult + (P[0, p1] + I[0, (p1 + h) % height]) / 2;

}

P[0, p1 + step] = v;

p1 += h;

}

h = h / 2;

}

// here, w==h

while (w > 1)

{

step = w / 2;


rmult *= 0.5;

for (y = 0; y < height; y += w)

{

for (x = 0; x < width; x += w)

{

x1 = x;

x2 = (x + w) % width;

x3 = x + step;

y1 = y;

y2 = (y + w) % height;

y3 = y + step;


v1 = random() - 0.5;

v2 = random() - 0.5;

v3 = random() - 0.5;


{

v1 = v1 * rmult + (P[x1, y1] + P[x2, y1]) / 2;

v2 = v2 * rmult + (P[x1, y1] + P[x1, y2]) / 2;

v3 = v3 * rmult +

(P[x1, y1] + P[x2, y1] + P[x1, y2] + P[x2, y2]) / 4;

}

P[x3, y1] = v1;

P[x1, y3] = v2;

P[x3, y3] = v3;

}

}

w = w / 2;

}

offset & scale so all values lie between -0.5 and +0.5;

}

2.5.1.1.2.2 Warping

Warping is achieved using a user specified 2D curve and is used to produce large-scale variations in the texture, such as a gradual shift from bottom to top, or ripple effects. The curve itself is either a collection of straight-line segments, or a smooth Hermite spline. An example of the effects of warping is shown in Figure 79.


(a) (b) (c)

Figure 79: The effect of warping: (a) linear x wrap, (b) smooth x warp, (c) linear a & b warp.

remap(v, WarpMap[], smooth)

{

n = WarpMap number of elements;

if (n == 0)

return 0.5;

if (n == 1)

return WarpMap[0].y;

if (smooth) // Hermite spline interpolation

{

MinDx = 0.001;

if (f < WarpMap[0].x) // Extrapolate before first knot

{

dy = WarpMap[1].y - WarpMap[0].y;

dx = WarpMap[1].y - WarpMap[0].x;


if (dx < MinDx) dx = MinDx;

f = WarpMap[0].y - (WarpMap[0].x - f) * dy / dx;

return f;

}

if (f > WarpMap[n-1].x) // Extrapolate after last knot

{

dy = WarpMap[n - 1].y - WarpMap[n - 2].y;

dx = WarpMap[n - 1].x - WarpMap[n - 2].x;

if (dx < MinDx) dx = MinDx;

f = WarpMap[n - 1].y + (f - WarpMap[n - 1].x) * dy / dx;

return f;

}

for (i = 0; i < n-1; i++) // Find knot interval

{

if (f >= WarpMap[i].x && f <= WarpMap[i + 1].x && WarpMap[i].x != WarpMap[i + 1].x){

dx1 = (WarpMap.x[i + 1] - WarpMap.x[i]);

f = (f - WarpMap.x[i]) / dx1;

y1 = WarpMap.y[i];

y2 = WarpMap.y[i + 1];

if (i > 1)

{

y0 = WarpMap.y[i - 1];

dx0 = WarpMap.x[i] - WarpMap.x[i - 1];

}

else

{

y0 = y1 - (y2 - y1);


dx0 = dx1;

}

if (i < n - 2)

{

y3 = WarpMap.y[i + 2];

dx2 = WarpMap.x[i + 2] - WarpMap.x[i + 1];

}

else

{

y3 = y2 + (y2 - y1);

dx2 = dx1;

}

if (dx0 < MinDx) dx0 = MinDx;

if (dx2 < MinDx) dx2 = MinDx;

m0 = (y1 - y0) * dx1 / dx0;

m1 = (y2 - y1) * dx1 / dx1;

m2 = (y3 - y2) * dx1 / dx2;

m0 = (m0 + m1) * 0.5;

m2 = (m2 + m1) * 0.5;

f2 = f * f; // f^2

f3 = f * f2; // f^3

// Create a Hermite spline

y = y1 * (2 * f3 – 3 * f2 + 1);


y += y2 * (-2 * f3 + 3 * f2);

y += m0 * (f3 – 2 * f2 + f);

y += m2 * (f3 - f2);

return y;

}

}

}

else // Linear interpolation

{

if (f < WarpMap[0].x) // Extrapolate before first knot

return WarpMap[0].y;

if (f > WarpMap[n - 1].x) // Extrapolate after knot

return WarpMap[n-1].y;

for (i = 0; i < n - 1; i++) // Find knot interval

{

if (f >= WarpMap[i].x && f <= WarpMap[i + 1].x &&

WarpMap[i].x != WarpMap[i + 1].x)

{

f = (f - WarpMap[i].x) / (WarpMap[i + 1].x - WarpMap[i].x);

return WarpMap[i].y + f * (WarpMap[i + 1].y - WarpMap[i].y);

}

}

return WarpMap[n-1].y;

}

}

2.5.1.1.2.3 Pseudo-random Number Generator

The pseudo-random number generator used for the plasma generation is based on the fast 32-bit linear congruential generator described in [78].

static unsigned int seed = 826374438;


srandom(s) // seed the pseudo-random number generator

{

ci = (unsigned int)(s * 0xFFFF);

// Run through three cycles to remove obvious visual artifacts

// that occur when input is too closely related to output

ci = ci * 1664525 + 1013904223;

ci = ci * 1664525 + 1013904223;

ci = ci * 1664525 + 1013904223;

union { unsigned int Int; float Float; } IEEEFloatMaker;

IEEEFloatMaker.Int = (ci & 0x7FFFFF) | 0x3F800000;

seed = IEEEFloatMaker.Float - 1.0;

}

random() // return random value in [-0.5, 0.5)

{

seed = seed * 1664525 + 1013904223;

union { unsigned int Int; float Float; } IEEEFloatMaker;

// Or in 23 bits of mantissa, now [1.0, 2.0)

IEEEFloatMaker.Int = (seed & 0x7FFFFF) | 0x3F800000;

// Return [-0.5, 0.5)

return(IEEEFloatMaker.Float - 1.5);

}


nonrandom(a, b) // return a repeatable, coordinate based, “random” value

{

srandom(a);

srandom(b + random());

return(random());

}


The ProceduralTexture node is specified as follows:

ProceduralTexture {exposedField SFInt32 type 0 # 0 – rectangle, 1 – brick, 2 – weave,

# 3 – hex, 4 – ringexposedField SFInt32 width 128 # [1, 215) & width = 2nexposedField SFInt32 height 128 # [1, 215) & height = 2nexposedField SFInt32 cellWidth 16 # [1, 215) & cellWidth = 2nexposedField SFInt32 cellHeight 16 # [1, 215) & cellHeight = 2nexposedField SFInt32 roughness 0 # [0, 15]exposedField SFFloat distortion 0 # [0, 1]exposedField SFInt32 seed 129093 # (-, )exposedField MFColor color [0.3 0.698 1, 0.8 0.8 0.8, 1 1 1, 0 0 0]

# [0, 1] 4 entriesexposedField MFVec2f xWarpmap [] # [0, 1]exposedField MFVec2f yWarpmap [] # [0, 1]exposedField SFBool xSmooth FALSEexposedField SFBool ySmooth FALSEexposedField MFVec2f aWarpmap [0 0, 1 1] # [0, 1]exposedField MFVec2f bWarpmap [0 0, 1 1] # [0, 1]exposedField SFBool aSmooth FALSEexposedField SFBool bSmooth FALSEexposedField MFFloat aWeights [0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0] # [-1, 1] 16 entriesexposedField MFFloat bWeights [0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0] # [-1, 1] 16 entries

}

The type field specifies the basic cell type (Figure 76), while width and height specify the overall texture dimensions and cellWidth, cellHeight the basic cell size. Note that due to the underlying plasma, there is a requirement that all sizes are a power of two.

The plasma properties are controlled through the roughness, distortion and seed parameters (section 2.5.1.1.2.1).

The color field describes the four colors used to generate the final texture.


The xWarpmap, yWarpmap, xSmooth and ySmooth parameters warp the x and y coordinates prior to the lookup into the plasma, while the aWarpmap, bWarpmap, aSmooth and bSmooth fields provide an extra warping stage prior to the final interpolation between the four color parameters. See section 2.5.1.1.2.2 for details.

The aWeights and bWeights arrays are used to provide a weighted blend between x, y, xwarp, ywarp, xdist, ydist, xoffset, yoffset, roffset, eoffset, aoffset, pcenter, pdist, poffset, randcell, randdist (see section 2.5.1.1.2 for details). Note that for the majority of textures most of these weights will be zero.

Figure 80: Default plasma texture.

The output is an image that can be employed directly as a texture map. Figure 80 shows the basic texture generated by the default values.

2.5.1.1.4 Examples

DEF Brickwork ProceduralTexture {

Type 1

CellWidth 8

CellHeight 8

roughness 2

seed 63530

color [ 0.447 0.43137 0, 0.6549 0.2 0.07843,

0.447 0.43137 0, 0.447 0.43137 0 ]

aWarpmap [ 0 0, 0.14 1, 0.83 1, 1 0 ]

bWarpmap [ 0 0, 0.14 1, 0.83 1, 1 0 ]

aWeights [ 0, 0, 0, 0, 0, 0, 0.48, 0, 0, 0, 0, 0, 0, 0, 0.40, 0.12 ]


bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ]

}

DEF Fabric ProceduralTexture {

type 2

width 256

height 256

cellWidth 4

cellHeight 4

roughness 1

distortion 0.05

seed 114300

color [ 0.898 0.89418 0.95294, 0.34118 0.29418 0.70196,

0 0 0, 0 0 0 ]

aWarpmap [ 0 0, 0.03 1, 0.88 1, 1 0 ]

bWarpmap [ 0 0, 0.48 1, 1 0 ]

aWeights [ 0, 0, 0, 0, 0, 0, 0.56, 0, 0, 0, 0, 0, 0, 0, 0.20, 0.24 ]

bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ]

}

DEF GranitePink ProceduralTexture {

width 256

height 256

cellWidth 8

cellHeight 8

roughness 5

seed 36792

color [ 0.72157 0.5647 0.5647, 0 0 0,

0.91373 0.86275 0.86275, 0.80784 0.55294 0.51765


]

aWarpmap [ 0 0, 0.5 0, 0.5 1, 1 1 ]

bWarpmap [ 0 0, 0.5 0, 0.5 1, 1 1 ]

aWeights [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.90, 0, 0, 0.10 ]

bWeights [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ]

}

DEF Horizon ProceduralTexture {

width 256

height 256

roughness 4

distortion 2

seed 108436

color [ 0.30196 0.69804 1, 0.56078 0.56078 0.56078,

0.05098 0.2549 0.03921, 0.4902 0.67059 0 ]

bWarpmap [ 0 0, 0.55 0, 0.59 1, 1 1 ]

}

DEF Marble ProceduralTexture {

width 256

height 256

roughness 1

seed 22209

color [ 0.8 0.7098 0.6902, 0.95686 0.8902 0.87451,

0.87451 0.37255 0.23529, 0.95686 0.8902 0.87451 ]

aWarpmap [ 0 1, 0.33 0, 1 1 ]

bWarpmap [ 0 0, 0.55 0, 0.6 1, 0.65 0, 1 0 ]

bWeights [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ]

}


2.5.1.2 Depth Image-based Representation

2.5.1.2.1 Overview

A simplest form of Image-based Rendering would be the traditional texture mapping. However, if the texture is mapped to a simple 3D model, the absence of parallax reveals the flatness of the surface when seen by a moving observer, or by a still observer at an oblique angle. To overcome this low quality, more complex geometry can be used. However, for example, texture mapping a tree with thousands of leaves that exhibits appropriate parallax, as the viewpoint changes, would be very difficult to create and slow in rendering.

Image-based Rendering [31] is a technology that can overcome this restriction. To represent the Image-based methods, a set of images or photos of a 3D object are taken from several different camera positions. Each of these images is accompanied by the depth information for each pixel.

A simplest representation of the image-based method is to use the Relief Textures [21] or Depth Image (DI). The depth image is a single image with depth map, which is an array of distances from the pixels in the image plane to the object surface. An advantage of such a representation is that reference images can provide high quality of the object visualization regardless of its polygonal model complexity, and can be compressed by usual image compression techniques without sacrificing much quality. In addition, rendering time is proportional to the number of pixels in the reference and output images and not the object complexity. This representation can be also used for animated objects, by storing sets of compressed video streams instead of images, together with ‘streams’ of depth maps representing object geometry. However, the information of the part of the 3D object that is not visible in any reference image is not known.

This may be overcome by using another popular image-based representation called Layered Depth Image (LDI) [89], which represents an object as a projection on a fixed plane, and for each pixel of projection stores distances and colors of all (not only visible) points of the object that project into this pixel (in other words, it is a multivalued image and depth map).

Instead of using depth maps, the formats such as Qsplat [60] or Binary Volumetric Octree (BVO) uses the octree structure to represent the depths of a 3D model.

2.5.1.2.2 Application – use case scenario

Modern graphics systems are supposed to perform high quality rendering of 3-dimensional objects at interactive speed. At the same time, explosive growth of importance of transmitting 3-dimensional graphics via communication networks in commerce, entertainment, science, engineering, medicine imposes a restriction on the high-quality models – they should be as compact as possible. The two requirements are somewhat contradictory, as far as traditional polygonal models of 3-dimensional objects are concerned. The latter have two major shortcomings: large volume (e.g., realistic models of humans require tens of million of triangles) and difficulty of constructing them. Image-based representations are suggested to partially overcome these difficulties.

2.5.1.2.2.1 An example of image-based representation


Figure 81 - Reference images (left) and depth maps (right) for the ‘Troll’ model. Images and depth maps (regarded as gray scale images) are stored in lossless PNG format. Total amount of data is 137.6 Kbytes. In compressed BVO format (Octree + JPEG reference images) it occupies 50 Kbytes.

Figure 82 - 3D view reconstructed from the image-based data above. Purely software rendering speed is 30 frames per second on Celeron 600 machine (no 3D hardware accelerator is used).

2.5.1.2.2.2 Node specification

2.5.1.2.2.2.1 DepthImage

DepthImage {field SFVec3f position 0 0 10field SFRotation orientation 0 0 1 0field SFVec2f fieldOfView 0.785398 0.785398 field SFFloat nearPlane 10


field SFFloat farPlane 100field SFBool orthographic FALSEfield SFNode diTexture NULLfield SFString depthImageUrl “”

}

The DepthImage node defines a single IBR texture. When multiple DepthImage nodes are related to each other, they are processed as a group, and thus, should be placed under the same Transform node.

The diTexture field specifies the texture with depth, which shall be mapped into the region defined in the DepthImage node. It shall be one of the various types of depth image texture (SimpleTexture or PointTexture).

The position and orientation fields specify the relative location of the viewpoint of the IBR texture in the local coordinate system. Position is relative to the coordinate system’s origin (0, 0, 0), while orientation specifies a rotation relative to the default orientation. In the default position and orientation, the viewer is on the Z-axis looking down the –Z-axis toward the origin with +X to the right and +Y straight up. However, the transformation hierarchy affects the final position and orientation of the viewpoint.

The fieldOfView field specifies a viewing angle from the camera viewpoint defined by position and orientation fields. The first value denotes the angle to the horizontal side and the second value denotes the angle to the vertical side. The default values are 45 degrees in radiant. However, when orthographic field is set to TRUE, the fieldOfView field denotes the width and height of the near plane and far plane.

The nearPlane and farPlane fields specify the distances from the viewpoint to the near plane and far plane of the visibility area. The texture and depth data shows the area closed by the near plane, far plane and the fieldOfView. The depth data are normalized to the distance from nearPlane to farPlane.

The orthographic field specifies the view type of the IBR texture. When set to TRUE, the IBR texture is based on orthographic view. Otherwise, the IBR texture is based on perspective view.


Mikaël Bourges-Sévenier, 03/01/-1,

Node spec SFString, Annex H SFUrl (encoding type)

Figure 83 - Perspective view of the DepthImage

Figure 84 - Orthographic view of the DepthImageThe depthImageUrl field specifies the address of the depth image stream, which may optionally contain the following contents.

position

orientation

fieldOfView

nearPlane

farPlane


orthographic

diTexture (SimpleTexture or PointTexture)

1 byte header for the on/off flags of the above fields

2.5.1.2.2.2.2 SimpleTexture

SimpleTexture {field SFNode texture NULLfield SFNode depth NULL

}

The SimpleTexture node defines a single layer of IBR texture.

The texture field specifies the flat image that contains color for each pixel. It shall be one of the various types of texture nodes (ImageTexture, MovieTexture or PixelTexture).

The depth field specifies the depth for each pixel in the texture field. The size of the depth map shall be the same size as the image or movie in the texture field. It shall be one of the various types of texture nodes (ImageTexture, MovieTexture or PixelTexture). If the depth node is NULL or the depth field is unspecified, the alpha channel in the texture field shall be used as the depth map.

2.5.1.2.2.2.3 PointTexture

PointTexture {field SFInt32 width 256field SFInt32 height 256field MFInt32 depth []field MFColor color []

}

The PointTexture node defines a multiple layers of IBR points.

The width and height field specify the width and height of the texture.

The depth field specifies a multiple depths of each point (in normalized coordinates) in the projected plane in the order of traversal, which starts from the point in the lower left corner and traverses to the right to finish the horizontal line before moving to the upper line. For each point, the number of depths (pixels) is first stored and that number of depth values shall follow.

The color field specifies color of current pixel. The order shall be the same as the depth field except that number of depths (pixels) for each point is not included.


2.5.1.2.2.2.4 OctreeImage

OctreeImage {field SFInt32 octreeresolution 256field SFString octree “”field MFNode octreeimages []field SFString octreeUrl “”

}

The OctreeImage node defines an octree structure and their projected textures. The size of the enclosing cube of the total octree is 1 x 1 x 1, and the center of the octree cube shall be the origin (0, 0, 0) of the local coordinate system.

The octreeresolution field specifies maximum number of octree leaves along a side of the enclosing cube. The level of the octree can be determined from octreeresolution using the following equation: octreelevel = int(log2(octreeresolution-1))+1 )

The octree field specifies a set of octree internal nodes. Each internal node is represented by a byte. 1 in ith bit of this byte means that the children nodes exist for the ith child of that internal node, while 0 means that it does not. The order of the octree internal nodes shall be the order of breadth first traversal of the octree. The order of eight children of an internal node is shown in Figure 85.

Figure 85 - The structure of octree and the order of the childrenThe octreeimages field specifies a set of DepthImage nodes with SimpleTexture for diTexture field. However, the nearPlane and farPlane field of the DepthImage node and the depth field in the SimpleTexture node are not used.

The octreeUrl field specifies the address of the octreeImage stream with the following contents.

header for flags

octreeresolution

octree


octreeimages (Multiple DepthImage nodes)

nearPlane not used

farPlane not used

diTexture SimpleTexture without depth

2.5.1.2.3 Rendering

The formats differ in the methods of rendering.

DI and LDI can be rendered with the aid of prewarping, followed by splatting. They can also be rendered directly onto the screen using simplest splatting (actual pixels).

BVO is rendered with the aid of splats whose size is computed using the Octree leaves position with respect to the viewer. Occlusion is handled with the aid of back-to-front tree traversal, were the traversal order is hierarchically determined in a fast fashion (in a sense, is inherited from higher level nodes to leaves).

Although these methods are highly efficient by themselves, they may also be assisted by hardware supported functions.

2.5.1.2.4 Dependencies

The formats use still and video coding formats of MPEG-4 for textures and depth maps.

2.5.1.3 Light Field Mapping

2.5.1.3.1 Overview

Light Field Mapping (LFM) is a method for progressive encoding, efficient compression and interactive rendering of surface light fields. A surface light field is a 4-dimensional function f(r,s,) that completely defines the radiance of every point on the surface of an object in every viewing direction. The first pair of parameters of this function (r,s) describes the surface location and the second pair of parameters (describes the viewing direction.

Intuitively, the LFM representation can be thought of as a special type of texture map that changes its appearance with the viewing angle. Because it is compact and allows for hardware-accelerated rendering, this representation is ideal for accurate modeling of the appearance of many physical and synthetic objects with complex surface reflectance properties. The system consists of three main components: approximation, compression and rendering that are described briefly.

Approximation StepThe first component approximates the 4-dimensional surface light field function f as a sum of a small number of products of lower-dimensional functions

),(),(),,,(1

k

K

kk hsrgsrf

Equation 1: Describes the approximation of light field data as a sum of a small number of produces of lower-dimensional functions.


The discrete functions gk and hk encode the light field data and are stored in a sampled form as texture maps, called light field maps. The functions gk are called the surface maps and functions hk as the view maps based on their parameterization. By taking advantage of the existing hardware support for texture mapping and composition surface light fields are visualized directly from the proposed representation at highly interactive frame rates. The process of rendering from this representation is called light field mapping.

Compression StepAlthough it is possible to render surface light fields directly from the approximation described in the last section, further compression of the surface light field data is possible. Since the light field maps are in essence collections of images that themselves exhibit redundancy, they can be further compressed using standard image compression techniques.

Rendering Step The sequence of rendering operations for each surface primitive is always the same. It starts with computing the view-dependent texture coordinates for the view maps. Subsequently, it proceeds to evaluate each approximation term and adds them together. Each approximation term is evaluated the same way:

1. The algorithm texture maps the surface primitive using the surface map.

2. It texture maps the surface primitive using the fragment of the view map determined by the view-dependent texture coordinates.

3. It performs pixel-by-pixel multiplication of the results of the two texture mapping operations. The following sections describe the rendering algorithm in more details.

Partitioning of DataSince the geometry of the models is represented as a triangular mesh, an obvious partitioning of the light field function is to split it between individual triangles. Unfortunately, an approximation of surface light field partitioned this way results in visible discontinuities at the edges of the triangles. To eliminate the discontinuities across triangle boundaries the light field data is partitioned around every vertex. The surface light field unit corresponding

to each vertex is called the vertex light field. For vertex vj, it is denoted as ],,,[ qqppv qrf j . Partitioning is

computed by weighting the surface light field function

],,,[],[],,,[ qqppppv

qqppv qrfqrqrf jj

where ],[ ppv qrj is the barycentric weight of each point in the ring of triangles around vertex vj.

In the final step of vertex-centered partitioning each vertex light field is reparameterized to the local coordinate system of the vertex. To this end, the viewing direction angles are defined as the azimuth and elevation angles of the viewing direction vector in the local reference frame of the vertex. The vertex reference frame is defined in such a way that its z-axis is parallel to the surface normal at the vertex. The same letters are used to denote the local parameters of the vertex light field in order to simplify the notation.


Partitioning of light field data into vertex light fields ensures that in the resulting approximation each triangle shares its view maps with the neighboring triangles. Therefore, even though we approximate each vertex light field independently, we obtain an approximation that is continuous across triangles regardless of the number of

approximation terms K in Equation 1. Let ],[ ppv qrg j be the surface map and ],[ qq

v jh be the view map

corresponding to one approximation term of vertex light field ],,,[ qqppv qrf j , i.e.,

],[],[],,,[ qqv

ppv

qqppv jjj hqrgqrf

Equation 2: One term approximation of light field for vertex vj.

Let ],,,[ qqpp qrf i denote the corresponding approximation term of the light field data for triangle i . The following equality holds

3

1

],[],[],,,[j

qqv

ppv

qqppjj

i

i hqrgqrf

Equation 3: One term approximation of light field for triangle i expressed as a sum of the triangle’s 3 vertex light fields.

In above equation index j runs over the three vertices of triangle i and ],[ ppv qrg j

idenotes the portion of the

surface map corresponding to the triangle i . This equality holds for all approximation terms.

Rendering AlgorithmThe rendering algorithm takes advantage of the property of vertex-centered partitioning, described by Equation 3, which says that the light field for each triangle can be expressed independently as a sum of its 3 vertex light fields. It enables a very efficient rendering routine that repeats the same sequence of operations for each mesh triangle. As each light field approximation term is evaluated the same way, the description of how to evaluate one approximation term of one triangle is sufficient to completely describe the rendering algorithm.


Figure 86: Light field maps for one approximation term of one triangle. Vertex reference frames are shown in the left column.

Figure 86 shows 6 light field maps used in Equation 3 to compute one approximation term of light field for triangle

i . The middle column shows surface maps ],[ ppv qrg j

i. In each image, the pixels covered by the shaded

triangle correspond to the points inside triangle i where we sampled the light field function. We will describe the texture coordinates of these points as (s, t). As a result of the weighting applied during the construction of function

],,,[ qqppv qrf j the pixels of the surface maps are weighted, as indicated in the figure by gradient shading, but

this does not alter the rendering algorithm in any way.

The right column shows view maps ],[ qqv jh . In each image, the pixels inside the circle correspond to the

orthographic projection of the hemisphere of viewing directions, expressed in the local coordinate system xyz of


vertex vj, onto the plane xy shifted and scaled to the range (0, 1). We will describe the texture coordinates of these points as (x,y). This projection allows a simple texture coordinate computation

2/)1(2/)1( ydxd yx

Equation 4: Equations describing parameterization of viewing directions.

In the equation above d represents the normalized local viewing direction and vectors x and y correspond to the axes of the local reference frame. Other transformations from 3D directions to 2D maps are possible but the one described here is efficient and accurate.

Based on where the camera is located, the rendering algorithm needs to access a different 2D subset of the 4D light field function. This is done by recomputing the view map coordinates ),( jj v

ivi yx every time the camera moves. To this end, we

apply Equation 4 to vectors jvid that represent the viewing directions to vertex vi expressed in the reference frame of vertex vj.

This results in 3 texture fragments shown in the right column of Figure 86. Note that the texture coordinates are different for each fragment because we are using a different reference frame to compute them. Note also that the surface map texture coordinates ),( jj v

ivi ts do not depend on the viewing angle and they do not need to be recomputed when the camera moves.

Evaluating one complete approximation term is equivalent to multiplying pixel-by-pixel the image projections of the surface map and view map texture fragment pairs for the 3 vertex light fields of the triangle, each shown in a separate row of Figure 86 and adding the results together. The multiple term approximation of each triangle light field is computed by simply adding the results of each approximation term.

Tiling of Light Field MapsTo avoid excessive texture swapping and improve rendering efficiency, we tile individual light field maps together to create tiled texture maps. Through out this document we are going to use term tile to refer to the image containing tiled light field maps. To simplify the problem, we allow a predefined set of triangle sizes during the resampling process, and then tile same-size light field maps together, as shown in Figure 87. We will use the term surface map tile to refer to the image containing tiled surface maps and we will use the term view map tile to refer to the image containing tiled view maps.

Since one triangle requires three surface maps per approximation term, all these maps are arranged in the same texture. The geometry of the model is split into p segments so that the tiled view maps for each segment do not exceed maximum texture size allowed. One view map tile is produced per segment: [V1, V2, ..., Vp]. Let

],,,[ 21iq

iii

SSS be the list of surface map tiles for vertex tile Vi. Note that, in general, each triangle represented by a given tile will have more than one surface map in this tile. For each approximation term, the rendering algorithm is as follows

for i=1,...,p do

load view map tile into texture unit 1

for j=1,...,qi do

load surface map tile ijS into texture unit 2



Excellent!

render all triangles with surface maps in tile ijS

end for

end for

Figure 87: The tiled surface maps (left) and view maps (right)


The root node that describes the Light Field is LFM_Appearance. This node should be present in the appearance field of a Shape node, while its geometry field should contain an IndexedFaceSet.

LFM_Appearance {exposedField MFNode tileList []exposedField MFNode lightMapList []exposedField SFNode blendList NULLexposedField SFNode vertexFrameList NULL

}

The field tileList describes the list of images containing the tiled light maps. Elements of this list should be nodes of type ImageTexture. In general, surface maps will be tiled separately from view maps and that not all of them are necessarily stored in a single image.

The LFM algorithm requires that each vertex of the mesh has a reference frame associated with it. The reference frames can be either computed automatically, as described in Rules for computing the reference frame, or they can be specified explicitly through LFM_FrameList node. For vertices that do not have the reference frame specified through the LFM_FrameList node, the reference frames are computed using the automatic rules. A node describing the reference frames for the vertices is defined as

LFM_FrameList {


exposedField MFInt32 index [ -1 ]exposedField MFVec3f frame [ 1 0 0, 0 1 0, 0 0 1 ]

}

The field lightMapList, used in the definition of the LFM_Appearance node, describes how to access individual light maps from within the image tiles. The number of the elements in the light map list corresponds to the number of the decomposition terms used in the surface light field approximation. The list lightMapList should contain elements of type LFM_LightMap. The format for accessing surface maps is slightly different from the format for accessing the view maps; therefore, each element in the light map list consists of a field describing how to access the surface maps and a field describing how to access the view maps. Note that each one of those two fields is going to be a list as well.

Figure 88 - This figure shows a diagram of the nodes used to define the shape that uses light field mapping appearance. The top of the figure shows the geometry of the object defined as a triangular. The middle of the figure shows the list of images containing the light map tiles. Finally, the bottom of the figure shows the light map list that specifies how to access individual light maps from within the tiles. Note that the first element of the light map list contains only a surface map list and no view map list. This element of the light map list would be used just like a regular texture map. The remaining two elements of the light map list each contain the par of surface map and view map list. In this case we would perform a multiplication of the corresponding surface maps and view maps. We made certain simplification in this diagram. For example, the diagram shows a one-to-one correspondence between the tiles and the surface/view map lists. In practice, this is not always the case, since we can have multiple lists for each tile.

LFM_LightMap {exposedField SFNode surfaceMapList NULLexposedField SFNode viewMapList NULL


exposedField SFVec3f scaleRGB 1 1 1exposedField SFVec3f biasRGB 0 0 0exposedField SFInt32 priorityLevel 0

}

Scale and bias map the default color range [0, 1] of the image into the color range required for proper image synthesis. Let s be the RGB scale vector, let b be the RGB bias vector and let c be the RGB color value. The new color range cnew for color vector c can be computed using vectors s and b as follows

bcsc new

Equation 5 – Scale and Bias map default color range to image synthesis color range..The field priorityLevel is a non-negative integer value specifying the level of importance of a given LightMapList node. The lower the value associated with the node, the more important it is. If the value is 0, the node must be rendered. When the renderer is set to a given priority value, then all nodes with the priorityLevel below that value must be rendered. For example, if the priority value of the render is set to 2, then all nodes with priority level 0, 1, and 2 must be rendered.

The lists surfaceMapList and viewMapList containing information about the surface maps and the view maps are defined as follows

LFM_SurfaceMapList {exposedField MFInt32 triangleIndex []exposedField MFInt32 tileIndex []exposedField MFInt32 viewMapIndex []exposedField SFNode triangleCoordinate NULL

}

where triangleIndex refers to the index of a given triangle as defined by the geometry node, tileIndex refers to the index of the tile as defined by the list of tiles, viewMapIndex indicates the number of view maps that should be used to render a given triangle (it equals –1 if there is no viewMapList in the current lightMap) and triangleCoord is a list of triples of texture coordinates of every triangle that specifies which part of the tile contains the surface map for the current triangle, it contains a TextureCoordinate node.

LFM_ViewMapList {exposedField MFInt32 vertexIndex []exposedField MFInt32 tileIndex []exposedField SFNode textureOrigin NULLexposedField SFNode textureSize NULL

}

where vertexIndex refers to the list of indices of vertices as defined by the geometry node, tileIndex refers to the list of indices of the tiles as defined by the list of tiles. The fields textureOrigin and textureSize are two lists of texture coordinates that specify the origin and the size of the view map, respectively, they should contain TextureCoordinate nodes.



You mean “view maps” instead of viewMap ?


Insert scale/bias equation


I think it’s SFInt32, right?

The field blendList, used in the definition of the LFM_Appearance node, describes how to combine the individual light maps together during rendering. It may contain a LFM_BlendList node

LFM_BlendList {exposedField MFInt32 lightMapIndex []exposedField MFInt32 blendMode []

}

where lightMapIndex refers to the index of the lightMap in lightMapList and tileIndex refers to the index of the tile as defined by the list of tiles and the field blendMode specifies what type of blending operation to use when combining the given LightMapList node with the data in the framebuffer. This field can take on the following values: 0 or LFM_ADD, 1 or LFM_SUBTRACT.

Note the following implementation facts

1. For each triangle, each approximation term consists of 3 surface map/view map pairs.

2. In general, each element in the LFM_LightMap will have a pair of valid pointers, one pointing to a valid LFM_SurfaceMapList node and one pointing to a valid LFM_ViewMapList node. If this is the case, during rendering the corresponding surface maps and view maps get multiplied together and blended with the earlier rendering results.

3. In case, when an element of the LFM_LightMap list contains a valid pointer to the LFM_SurfaceMapList node, but the pointer to the LFM_ViewMapList node is NULL, we simply use the surface maps as ordinary texture maps.

4. Let sm1, vm1, ..., smK, vmK be the first K pairs of surface map/view map for a given triangle. Multitexturing can be supported in the form sm0 + sm1*vm1 + sm2*vm2 + sm3*vm3 + ..., where addition refers to adding (or subtracting, depending on the value of field blendMode) the results of rendering through blending, and multiplication refers to pixel-by-pixel modulation of rendering results. This set of operations is repeated for each mesh triangle of the object.

2.5.1.3.3 Examples

The following example shows the definition of the Shape node that uses LFM_Appearance field. In the example, the light map list has two elements but its first element has the view map list pointer set to NULL, which means that the surface map list in this element will be used as a regular texture map. In practice, we need at least 3 pairs of surface/view map lists for correct light field mapping.

Shape {

geometry IndexedFaceSet {

coord Coordinate { point [ v1, v2, ..., vN ] }

coordIndex [ t1, t2, ..., tM ]

}



Don’t use -1 and 1 but 0 and 1 for binary compression.

appearance LFM_Appearance {

tileList [ ImageTexture { url "surfaceMapTile0.jpg" }

ImageTexture { url "surfaceMapTile1.jpg" }

ImageTexture { url "viewMapTile1.jpg" }

]

lightMapList [

LFM_LightMap {

surfaceMapList LFM_SurfaceMapList {

triangleIndex [

0 1 2 3 .. ]

tileIndex [

0 0 0 1 .. ]

viewMapIndex [

-1 –1 –1 –1 .. ]

triangleCoord TextureCoordinate { point [

0.00390625 0.00390625, 0.00390625 0.12890625, 0.12890625 0.12890625, 0.29296875 0.00390625, 0.29296875 0.12890625, 0.41796875 0.12890625, 0.58203125 0.00390625, 0.58203125 0.12890625, 0.70703125 0.12890625, 0.00390625 0.13671875, 0.00390625 0.26171875, 0.12890625 0.26171875,

.

.

.

] }

}

viewMapList NULL

scaleRGB 1 1 1

biasRGB 0 0 0

}

LFM_LightMap {


surfaceMapList LFM_SurfaceMapList {

triangleIndex [

0 1 2 3 .. ]

tileIndex [

0 0 0 1 .. ]

viewMapIndex [

0 3 6 9 .. ]

triangleCoord TextureCoordinate { point [

0.01562500 0.00390625, 0.14062500 0.00390625, 0.14062500 0.12890625, 0.30468750 0.00390625, 0.42968750 0.00390625, 0.42968750 0.12890625, 0.59375000 0.00390625, 0.71875000 0.00390625, 0.71875000 0.12890625, 0.01562500 0.13671875, 0.14062500 0.13671875, 0.14062500 0.26171875,

.

.

.

] }

}

viewMapList LFM_ViewMapList {

vertexIndex [

0 1 2 3 .. ]

tileIndex [

2 2 2 2 .. ]

textureOrigin TextureCoordinate { point [

0.00830078 0.00830078,

0.02490234 0.00830078,

0.04150391 0.00830078,

0.05810547 0.00830078,

.

.

.

] }


textureSize TextureCoordinate { point [

0.00781250 0.00781250,

0.00781250 0.00781250,

0.00781250 0.00781250,

0.00781250 0.00781250,

.

.

.

] }

}

scaleRGB 2 2 2

biasRGB -1 -1 -1

}

]

blendList LFM_blendList {

lightMapIndex [ 1 2 . . ]

blendMode [ 1 1 . . ]

}

vertexFrame NULL

}

}

2.5.1.3.4 Rules for computing the reference frame

The rule for computing the reference frame of a given triangle consists of two steps:

1. Compute the following vectors

iiiii 213322211 ,, eeevvevve

where v1, v2, v3 are the vectors describing the positions of the 3 vertices of the given triangle.


2. Construct the change of basis matrix R = [ x y z ], where

.,,3

3

1

1 zxyeez

eex

i

i

i

i

The rule for computing the reference frame of a vertex consists of the following steps:

1. Compute the direction of the normal of the vertex to be the average of the directions of the ring of triangles for that vertex

where Rj is the set of triangles contained in the ring around vertex vj.

j

iij

Ri

v213 eee

2. Pick the first edge of the first triangle in the ring as the direction of the x-axis of the vertex reference frame

111ee jv

3. Construct the change of basis matrix R = [ x y z ], where

.,,,3

3

1

1 yzxzxyeez

eex

j

j

j

j

v

v

v

v

2.5.1.4 Synthesized Texture

2.5.1.4.1 Overview

The Synthesized Texture imaging technology represents animated photo-realistic textures using vector graphics approach to describe color information. These vector based parameters can be animated over time, producing very low-bit-rate movies suitable for terminals and networks with limited resources and narrow bandwidth. The technology is known as VIM – VectorIMaging by Vimatix [26].

Synthesized textures are made of 3 base elements and 3 animation enabling elements:

Base Elements:

Characteristic Lines made of line segments and line color profiles. Line segments are defined as symmetrical parabolas. A line color profile is associated with each line segment. There are two types of Lines: Edges and Ridges.

Patches are defined as ellipsoids of few pixels long whose color is significantly different from their surrounding.

Area Color Points describe the low-scale (background) color changes in the areas between Lines.

Animation enabling elements:



TB linked with gloal reference

Sub-textures define a high-level structure enclosing entire regions of the texture. A sub-texture is completely enclosed by lines. It contains Lines, Area Color Points and Patches. A Sub-Texture may belong to an Object or not. A Sub-Texture cannot belong to more than one object.

Skeletons are invisible curves that affect the animation of Sub-Textures. Skeletons are the control points of the Objects. They can be transformed by external skeletons like Skin&Bones or by BIFS interpolators.

Objects (layers) are an aggregation of Sub-Textures that are subject to animation. Rigid transformations applied to an Object result in transforming of all its sub-textures.

It is important to note that although the result is a 2D image, the SynthesizedTexture scene is described using 3D elements. The third dimension is used to define layering information that a content creator can use to create overlapping sub-textures. The color information associated with curves is applied after projection of elements to the 2D image plane. The combination of the Base 3 elements is used by the SynthesizedTexture generator described below to interpolate colors between the elements to produce an image. It is important to note that elements are not just drawn on the screen like in BIFS’ CompositeTexture(2D) nodes.

SynthesizedTexture elements

Line (LN), bounded by 2 Terminal Points (TP)

Line Segment (LS), bounded by 2 Line Points (LP)

Line marked as control line (Skeleton)

Line Color Profile (LC)

Area Color Point (AC)

Color Patch (PA)

Sub-Texture (ST)


The SynthesizedTexture uses symmetric parabolas to describe ‘Line Segments’. For a curve segment, 3 parameters are needed: the coordinates of the end-points and the distance from the line between these end-points and the tip of the parabola, as shown in the Figure below.

Figure 89 - SynthesizedTexture Line componentIn VRML and BIFS, lines and curves can have a single color. In Synthesized textures, color information can be interpolated along the curve hence Line Color Profiles are introduced.

Figure 90 - Line color profile definition.

Figure 91 - A Line with associated Line Color Profiles.Each point along a Line has its own Line Color Profile. Sometimes two adjacent Color Profiles may have the same brightness for example LP2 and LP3. In that case, the color between these two control points is the same. In other cases, the color should be interpolated by the decoder.



2.5.1.4.2.1 SynthesizedTexture node

SynthesizedTexture { #%NDT=SFTextureNodeeventIn MFNode addChildreneventIn MFNode removeChildrenexposedField MFNode children []exposedField SFInt32 pixelWidth 0 # [0, ]exposedField SFInt32 pixelHeight 0 # [0, ]exposedField SFNode background NULLexposedField SFNode viewport NULL

}

The SynthesizedTexture node is similar to CompositeTexture2D and reuse the same fields and semantic. Similar to CompositeTexture2D, SynthesizedTexture is a texture node that produces an image from a scene described using BIFS nodes in children field.

As any texture, the resulting texture can be mapped onto any 2D or 3D surface. As mentioned above, the SynthesizedTexture is rendered while interpolating the color between its elements.

For the fields description please refer to the CompositeTexture2D node. See also an example below.

Example of a Synthesized enabled scene:

Shape {geometry Rectangle { size 3 1 }appearance SynthesizedTexture {

pixelWidth 256 # assuming pixelMetrics = TRUEpixelHeight 256 # this defines a 256x256 texturechildren [

…]

}}

Example of a SynthesizedTexture node hierarchy:

DEF Synthesized1 SynthesizedTexture {

children [

DEF OBJ1 Group {

Transform { # This is the Object's rigid transformation

DEF SUB-TEXT1 Group {

children [

Transform { # This is the Sub-Texture’s rigid transform

# Below are the various Sub-Texture lines/BG/Patch


DEF N1 Curve2D { … } # Sub-texture Lines

DEF N2 Curve2D { … }

DEF BG1 PointSet {...} # Sub-texture Area Color Points

DEF PTCH1 Circle {...} # Sub-texture Patches

}

]

}

}

DEF OBJ2 Group [

...

]

]

}

SynthesizedTexture Objects and sub-textures

The higher level of aggregation within a SynthesizedTexture, Object, is declared using an existing BIFS Group node [1].

The lower level of aggregation, Sub-Texture, is declared also with the existing BIFS Group node.

SynthesizedTexture skeletons.

When sub-textures are connected and rigid-transformation is needed, a higher-level tool is needed: the Skeleton. The bones of the skeleton are made of parabola so they can be flexible and produce non-rigid deformation of objects they apply to. The bones are defined by means of Parabola nodes (of type 2) defined below. As the coordinates (point) of the curves are exposed, they can get routed events from bones based animation’s external skeletons (section 2.8.2) or any other sensor or interpolator.

Area Color Point position and Color

The coordinates of a SynthesizedTexture (background) Area Color Points are declared by the PointSet node [1]. Color information at each point can be specified using PointSet’s color field with Color or ColorProfile node.

2.5.1.4.2.2 Parabola node

To declare the SynthesizedTexture’s color characteristic and control lines, Parabola node is defined:



This section used to say ‘solid color’ or gradient. But a gradient is a texture on a surface, that a point does not have by definition. I modified the text thinking about points. If the author needs surface, then he uses Ellipse or Circle node.

Parabola { #%NDT=SFGeometryNodeexposedField MFVec3f point []exposedField MFInt32 type [] # 0, 1, or 2exposedField SFNode color NULLexposedField MFInt32 colorIndex []exposedField MFBool separatingFlags [] # TRUE, FALSEexposedField MFInt32 contourFlags [] # [0, 2exposedField MFInt32 controlFlags [] # [0, 3]exposedField SFBool colorPerVertex TRUE

}

The point field contains all the points of the poly-curve.

the type field describes how they are connected to each other. Currently, four types are defined:

0: moveTo. Move the drawing pencil to next point LPi, creating disjoint segments.

1: lineTo. Draw a line to the next point LPi.

2: Parabola. Draws a parabola defined by 3 points: LPi, LPi+1, LPi+2 (Figure 92).

d

P1

P2P0

Figure 92 –Parabola type is defined by 3 points (d is the distance from the middle of P0P2 to P1).As for IndexedLineSet, multiple curve segments, potentially disjoint, can be specified. The type field indicates how to draw each curve segment. As a hint, there are as many curves as elements in the type array minus the number of 0 (moveTo) entries.

Curves are not lit and do not participate in collision detection. The width of curves is implementation dependent and each curve segment is solid (i.e., not dashed).

When closed, Parabolas may be textured objects. As for any shape, a texture is specified in the material field of the Shape node containing the Parabola node. The texture is mapped using the bounding box of the Parabola node and the curves define a clipping region.

If the color field is not NULL, it shall contain a Color or ColorProfile node, and the colors are applied to the curve(s) as follows:

If colorPerVertex is TRUE, at each endpoint a curve must have a color in Color or ColorProfile node.


If colorPerVertex is FALSE and color field is not empty, then there must be as many colors in Color or ColorProfile node as curves. If N is the number of curves (i.e. entries in type field), then there must be N entries in Color or ColorProfile.

If the color field is NULL (regardless of colorPerVertex value) and there is a Material defined for the Appearance affecting this Curve, the emissiveColor of the Material shall be used to draw the curves.

The colorIndex Field specifies which color is associated with a point. Hence, colors can be reused multiple times. If colorIndex contains no value, it is assumed that for every point, a color is defined in Color or ColorProfile node in color field. If colorIndex has values, there must be as many values as points.

Each curve may have various properties attached to. These properties are only used when the Curve node is a child of a SynthesizedTexture node. There are three types of flags:

separatingFlags: FALSE separating, TRUE non-separating

contourFlags: 0 none, 1 left, 2 right. Value of 1 means that the sub-texture bounded by this curve is on the curve’s left side.

ControlFlags: 0 none, 1 visible, 2 invisible control line that belongs to a sub-texture and affects only this sub-texture, 3 invisible control line not belonging to any sub-texture but affecting sub-textures. Lines that are flagged as Control (type 3) behave as skeletons for the specific sub-texture.

Figure 93 –Example of open Parabola node.

2.5.1.4.2.3 Ellipse node

SynthesizedTexture’s patches are defined using with Ellipse node:

Ellipse { #%NDT=SFGeometryNodeexposedField SFVec2f radius 1 1

}

An ellipse is a planar geometry node and is defined by two-radius extent in its local coordinate system: radius[0] along x-direction and radius[1] along y-direction. By default, the ellipse is a unit circle (rx = ry = 1).


A patch can be described using Ellipse node as geometric support. For the color information, a new material node is needed: GradientRadial. We also propose a GradientLinear, that is not part of VIM texture’s patch definition.

Example:

2.5.1.4.2.4 Color Profiles

The color along characteristic lines is interpolated based on the color profiles attached to its control points. Color Profiles are new nodes defined as:

ColorProfile { #%NDT=SFColorNodeexposedField MFInt32 type 0 # 0 (Ridge), 1 (Edge)exposedField MFInt32 leftWidth [] # WLexposedField MFInt32 rightWidth [] # WRexposedField MFColor leftBrightness1 [] # LB1exposedField MFColor leftBrightness2 [] # LB2exposedField MFColor rightBrightness1 [] # RB1exposedField MFColor rightBrightness2 [] # RB2exposedField MFColor centralBrightness [] # CB

}

The type field tells whether the profile is of Edge ( 1 = TRUE) or Ridge (0 = FALSE).

The leftWidth and rightWidth are specified in normalized coordinates.

The leftBrightness1, leftBrightness2, rightBrightness1, rightBrightness2 and centerBrightness are defined as RGB color space.

Similar to the Color node, ColorProfile specifies multiple color profiles. Hence, color profile I consists of 6 parameters: leftWidth[i], rightWidth[i], leftBrightness1[i], leftBrightness2[i], rightBrightness1[i], rightBrightness2[i], and centerBrightness[i].



All the specification is in normalized coordinates because the size of the image is specified in SynthetizedTexture so I think we should remain in normalized coordinates.

2.5.1.4.2.5 Linear and radial gradients

The patches are colored with gradients that propagate in specified angles and degrade at the borders of the ellipsoids in a certain fashion that is described below.

Two general types of Color Gradients are described below to fulfill the demand for this type of functionality in BIFS.

A gradient is a material node that generates a texture procedurally.

GradientRadial { #%NDT=SFMaterialNodeexposedField SFVec2f center 0.5 0.5 # cx, cy [0,1]exposedField SFFloat radius 0.5 # r [0,1]exposedField SFVec2f focalPoint 0 0 # fx, fy [0,1]exposedField SFInt32 spreadMethod 0 # [0, 2]exposedField MFFloat key [] # gradient stopsexposedField MFColor keyValue [] # ARGB colorsexposedField MFFloat opacity [1] # alpha value [0,1]

}

center and radius define the outermost circle (cx, cy, r) for the radial gradient. The gradient will be drawn such that 100% of the gradient stop is mapped to the perimeter of this outermost circle. The units for center and radius are in percentages of the bounds of the colored object. Length is x- and y-direction in the local coordinate system of the shape. By default, the center of the outermost circle is in the middle of the shape (0.5 means 50%).

focalPoint (fx, fy) represents the focal point of the gradient. The gradient will be drawn such that 0% of the gradient stop is at (fx, fy). The units are also in percentage of the bounds of the object.

spreadMethod can be pad (0), reflect (1), or repeat (2). It indicates what happens if the gradient starts or ends inside the bounds of the object. Pad means that the last color is used, reflect says to reflect the gradient pattern start-to-end, end-to-start, … repeatedly until the target object is filled, and repeat says to repeat the gradient pattern start-to-end, start-to-end, … until the target object is filled.

tranform is an optional parameter that defines how the coordinate system of the gradient can be transformed from the gradient coordinate system onto the target coordinate system. By default, the gradient coordinate system is the same as the object it is applied to. This allows effects such as skewing the gradient. Only a Transform2D node can be present here.


key and keyValue define the ramp of colors to use on a gradient. At least two values are necessary to define a gradient. key indicates, in percentage, where the keyValue (a RGB color value) will be placed. key represents the percentage distance between the focalPoint (fx, fy) and the edge of the outermost circle (cx, cy, r). opacity for each color value can be specified. By default, colors are 100% opaque. One value of opacity can be specifed meaning all color values have the same opacity, else an opacity must be specified for each color value.

Example:

Shape {geometry Rectangle { size 3 1 }appearance Appearance {

material GradientRadial {key [ 0 0.5 1 ]keyValue [ 0 0 1, 1 0 0, 0 0 1 ]

}}

GradientLinear { #%NDT=SFMaterialNodeexposedField SFVec2f startPoint 0 0 # x1, y1 [0, 1]exposedField SFFloat endpoint 1 1 # x2, y2 [0, 1]exposedField SFInt32 spreadMethod 0 # [0, 2]exposedField MFFloat key [] # gradient stopsexposedField MFColor keyValue [] # RGB colorsexposedField MFFloat opacity [1] # alpha value [0, 1]

}

startPoint and endpoint define the gradient vector, in percentage of the bounds of the object.

key represents a location along the gradient vector, expressed in percentage of its length. At each location, an RGB color is specified in keyValue. opacity for each color value can be specified. By default, colors are 100% opaque. One value of opacity can be specifed meaning all color values have the same opacity, else an opacity must be specified for each color value.

Example:

Shape {geometry Rectangle { size 3 1 }appearance Appearance {

material GradientLinear {key[ 0 1 ]keyValue [ 0 1 0, 1 1 0 ]

}}


2.5.1.4.3 SynthesizedTexture rendering considerations

2.5.1.4.3.1 Characteristic Lines and their Color Cross-Sections

Characteristic lines (Lines) are the lines (either visible or virtual) on the image along which the image visual pattern is consistent.

’Lines’ are represented by their virtual “central lines” and “brightness cross-sections (Color Profile”, or CP). The central line captures in the most accurate way the geometric shape of the Line, while the CP describes the brightness (color) behavior in the orthogonal direction to the line.

The central line is given by a second or third order spline curve (preferably a second order).

CPs are given by a small number of model shape types, each characterized by a small number of parameters. CPs are stored at some predefined “CP control points” on the central line, and interpolated between these control points. The line sections defined between these control points are called Line Segments (LS).

Figure 94 shows an LS of a Line with a CP interpolated along it, at point ‘t’, which is used to calculate point ‘u’ color.

Figure 94 – Color interpolation along a curve with color profiles at end points.

2.5.1.4.3.1.1 Parameters of Lines' Geometry

All the geometric parameters below are described in the coordinate system (x,y) in the image plane, with the unit length taken to be the distance between two neighboring pixels.

The endpoints of the Line and their crossings are called “Terminal Points”, or TP. Other points on the lines are defined as Line Points (LP), and the in-between line sections are the Line Segments (LS). If a Line forms a simple closed curve, one of its points is chosen to be a TP.

All the TPs and LPs on the Synthesized image are described (in a certain order) by their coordinates (x,y) in the above coordinate system.


Lines are represented by special second order splines, that we call P-curves.

A P- curve is a chain of convex arcs S i, i = 1, 2, …, n, starting at the point z0 = (x0, y0) and ending at the point zn = (xn, yn). The arcs Si and Si+1 have the common point zi = (xi, yi), called a vertex, which is an LP. Each Arc is an LS. For a closed curve the initial point z0 and the end point zn may coincide.

Being represented by an ordered list of consecutive points, each P-curve possesses a natural orientation.

Either parabolic or circular arcs Si are used, in such a way that for each [z i-1, zi] the arc Si is completely characterized by the height hi of its center over the segment [z i-1, zi]. This height is taken with the sign plus, if the arc is curved to the left side of the P-curve, and with the sign minus, if the arc is curved to the right.

In particular, if the parabolic arcs Si are chosen, they are taken to be symmetric with respect to the line passing through the center of the straight segment [z i-1, zi] and orthogonal to it. Thus for [z i-1, zi] given, the symmetric parabolic arc Si is completely characterized by the height hi of its center over the segment [zi-1, zi].

Consequently, a P-spline curve is given by the following parameters:

The coordinates (xi, yi) of the vertices zi, i = 0, 1, …, n, and the heights h i, i = 1, …, n, of the arcs Si. It may be more convenient to keep the coordinates (x0, y0) of the starting point z0, and the vector coordinates (v i, wi) of the segments [zi-1, zi], i = 1, … , n, for the rest, vi = xi - xi-1 , wi = yi - yi-1.

The arcs Si of the P-curves, representing poly-links, are called below “links”.

Notice, that for each P-curve, representing Line, its starting point and its endpoint are known in advance, being the TPs.

To summarize, the central lines of the Lines of a Synthesized image are completely described by the following parameters:

a) The list of the TPs and LPs, with their coordinates. Notice that the information, given by TPs is redundant and can be reconstructed from the rest of the data. TPs do not appear in the Nodes.

b) The list of LSs, specified by their vertices. Since the choice of one of two possible orientations of the Line is important, each Line is given by an ordered list of its vertices zi (Line Point), starting at one of the endpoints of Line and continuing following the natural order of the vertices, together with the heights of the corresponding Line Segment. Alternatively, the vectors [zi-1, zi] and the heights can be specified.

2.5.1.4.3.1.2 Parameters of Lines' Brightness Cross-Sections (Color Profiles)

In the basic Color Profile (CP) default configuration, described below, the CPs control points coincide with the vertices (Line Points) of the P-curves, representing the Line. The type of the CP (edge or ridge) does not change along the Line. Each ridge (i.e. a Line with the ridge type CP) is marked either as a “separating” or as a “non-separating” one. Any edge is separating.

The basic Synthesized images representation uses two types of cross-sections: edge CP and ridge CP.

2.5.1.4.3.2 Color profiles (CP)

2.5.1.4.3.2.1.1 Edge CP

EdgeCP is shown on Figure 95, is completely described by the following parameters:


Figure 95 - Edge color profile.- Left width WL

- Right width WR (default assumption is that WL = WR)

- Left brightness LB1


- Right brightness RB1


**The reason for appearance of the “bumps” on both sides of the edge CPis that in most of natural images, either scanned or taken by a video or a still digital camera, the actual edges cross-sections usually contain such bumps. They are caused by the physics of the light propagation, by specifics of scanning procedures and, on the other side, by the human visual perception.

However, for different kind of images, such as synthetic computer-generated images quite different shapes of the cross – sections might appear. This is true also for certain types of scanners, and especially, for other sensors, like infra-red ones. In more advanced profiles of Synthesized representation fairly general shape models of cross-sections can be used, leading to more types of Color Profiles.

The “margin” parameters LB1 and RB1, being important cross-section features, are stored together with other cross-section parameters. However, they have another natural visual interpretation as the background brightness along the edge on its different sides. It is this interpretation, that is used in the recommended reconstruction (decoding) algorithm: the margin cross-section parameters enter the background Area (in between the Lines) reconstruction Procedure.

In most of applications, the “height” of the cross-section bump can be taken to be a certain fixed fraction of the total height of the cross section. This fact is utilized in the “aggregation” level of the Synthesized representation, by encoding of the actual heights as the corrections to the default ones. In an ultimate implementation of this mode the parameters LB2 and RB2 can be eliminated and replaced by their default values, computed through the rest of the parameters.**

2.5.1.4.3.2.1.2 Separating Ridge CP

Separating Ridge CP is shown on Figure 96, is completely described by the following parameters:


- Left width WL

- Right width WR



- Central brightness CB



Figure 96 – Separating ridge color profile.**All the remarks above referring to edges, concerning the nature of the bumps and the redundancy of the side brightness parameters, as well as the interpretation of the margin brightness parameters LB1 and RB1 as the Background values, remain valid also for ridges.**

2.5.1.4.3.2.1.3 Non-Separating Ridge CP

This type of CP has the same parameters as the separating one, besides the margin parameters LB1 and RB1, which are not defined; see Figure 97.


Figure 97 – Non-separating ridge color profile.**On a higher aggregation level various default configurations can be adopted, simplifying data representation. For example, the right and the left width of the ridge are usually roughly equal, so a single width parameter can be used. The right and left brightness values of a non-separating ridge can be identified with the background values at the corresponding points, etc.**

If a color image is represented, all the brightness parameters are defined independently for each color component, while the same width parameters are kept. These color components can be R, G, B, CMYK, YUV or other formats. Below we use the word “brightness” to denote any component of these color formats.

As it was stated above, the Color Profiles (cross-sections) are specified at each LP of the Line.

To summarize, Color Profiles (cross-sections) of characteristic lines are specified by the following parameters:

- Type of a cross-section (edge, ridge or separating ridge) on each LP.

- Width and brightness parameters of the cross-sections, as specified above, at each of the vertices of the P-curves, representing LPs.

In case of a color image, the type and the width of the cross-sections are the same for each color component, while the brightness parameters are specified independently for each color.

Of course, on a higher aggregation level various known visual spatial-color correlations can be used to reduce the data size.

A closed Line can be marked as a boundary contour of the Synthesized image or an Object within it.

2.5.1.4.3.3 Patches

Patches capture fine scale details in the image. They are represented by Gaussian-shaped or parabolic-shaped mathematical models, blended with the background along their elliptic-form margins; see the Figure 98.


Figure 98 – Patches definition.Each patch is specified by the following parameters:

- The coordinates (Cx, Cy) of its center

- The sizes R1 and R2 a of the bigger and the smaller semi-axes of the base ellipse

- The direction a (angle with the x-axis) of the main semi-axis of the base ellipse

- The brightness value CB at the center

- The “margin” brightness value MB

In case of a color image the brightness values at the center of each patch are specified independently for each color separation.

On the aggregation level various default configurations can be adopted, simplifying data representation. For example, for patches smaller than a few pixels in size, the distinction between the sizes of the two semi-axes of the elliptic base is hardly visually appreciated.

Consequently, a single size parameter can be used in this case, while the angle is omitted. In most situations the margin brightness value MB of the patch can be identified with the Background brightness at the patch center.

2.5.1.4.3.4 The Area Color AC (‘Background’)

(This term is used in Imaging in many different situations. Below we use it to denote the part of the Synthesized image, “complementary” to the Lines and patches. It can be called also the “slow scale image component”. In the Synthesized Texture script the term “Area Color” is used as equivalent to the “background” in the present informal text).

Background is determined by the following elements:

1. All the separating Lines are excluded from the background domain.

2. Some of the image subparts, completely bounded by separating characteristic Lines or/and the image borders, are provided with their (single) global background brightness value GB, or a Color Gradient. See the figure below. (In fact, these subparts, called Synthesized Sub-Textures, play an important role on higher representation and animation levels. They are specified by Sub-Texture numbers, and the values GB are attached to the Sub-Textures).


3. A certain number of “background representing points” (AC – Area Color Points) is defined, each point carrying its brightness value. These values are further interpolated between the background representing points in such a way that the interpolation does not “cross” the separating Lines. See the figure below

4. The margin brightness values of the CPs of separating Lines are blended with the background along the margins of these separating lines.

The following parameters participate in the calculation of the background brightness values (as described in detail in the Procedure BB below):

- Geometric parameters of all the separating Lines.

- Margin brightness values (LB1 and RB1 above) of the CPs of all the separating Lines.

- List of the “background representing points” (ACs), each one given by its (x,y) coordinates and its brightness value.

- Single background global brightness values GB (or a Color Gradient) for some of the Sub-Textures. (In the parameters structure these values are associated with the Sub-Texture numbers, which, in turn, are associated with the Lines, bounding the Sub-Texture).

**On the aggregation level various default configurations can be adopted, simplifying data representation. In particular, a regular grid of background representing points can be used, to eliminate the need to store coordinates of these points. (Notice, however, that in a general structure, allowing, in particular, for geometric transformations and animations of the images, all the geometric parameters must be represented in a geometrically invariant form, without reference to a fixed grid).**


2.5.1.4.3.5 Crossings

Crossings of Lines are represented by crossings of their central lines and by blending of their brightness values near the crossings (Figure 99).

Figure 99 – Crossings.In the data structure described above, crossings are represented by TPs (Terminal Points).

At each TP, CPs are given for each of the Lines starting (or ending) at this TP. No compatibility conditions between these CPs are assumed in the basic Synthesized profile.

**However, in most practical situations the side brightness (color) values on appropriate sides of the adjacent characteristic lines at their crossing are roughly equal. This fact is used on the higher aggregation level to reduce the data size (Figure 100)

Figure 100 – Same brightness at crossing of multiple characteristic lines.Splitting is used on the higher aggregation level to reduce the data size and to preserve the image visual continuity along the characteristic Lines. **


2.5.1.4.3.6 Ranges and Resolutions

2.5.1.4.3.6.1 The Range of Synthesized Parameters

A typical range of all the above parameters depends on the type of the images to be represented, on their resolution, on the specifications of the display and on visual conditions. Below we refer to the usual PC screen applications.

For Synthesized representation of high resolution images of the real world a typical range of parameters is as follows:

Length of a Line Segment 2 - 32 pixels

Height of a Line Segment not larger than its length

Width of an edge 1 - 8 pixels

Width of a ridge 1.5 - 16 pixels

Density of the background representing points

between 1 per 4x4 pixels block to 1 per 32x32 pixels block

Color components between 1 and 256 gray levels

For Synthesized representation of synthetic images (in particular, in creation of synthetic images using Synthesized authoring tools) the range of the Synthesized parameters is not strongly limited. Wide characteristic Lines, up to tens of pixels in width, can be used. Only a few of the background representing points (APs) may be needed (or no such points at all).

2.5.1.4.3.6.2 Default Resolution

The default resolution of all the geometric parameters is 1/16 of a pixel size.

The default resolution for each color component is 256 gray levels.

On higher levels of aggregation and quantization much lower resolutions can be allowed, according to a psycho-visual significance of each parameter. Also specifications of the display and expected visual conditions are taken into account.

2.5.1.4.4 SynthesizedTexture Rendering Algorithm

2.5.1.4.4.1 Synthesized Rendering Overview

The reconstruction algorithm starts with the input parameters, described above, and computes the brightness (color) value of the synthesized image at each of its pixels. It consists of the following principal parts:

- Computing brightness of Lines

- Computing brightness of Patches

- Computing brightness of the background

- Blending the computed brightness values into the final image


In the case of a color image these computations are performed independently for each color component.

2.5.1.4.4.1.1 Computing brightness of characteristic Lines (Procedure BL below)

In order to compute the brightness values of the cross-sections (the CPs) along the characteristic line, a coordinate system (u,t) is associated to its central line. Here u is the distance from the central line (with the sign) and t is the coordinate (along the line) of the projection on the central line.(Procedure DL). Using the coordinates (u,t) the brightness of the cross-sections (computed in the Procedure CS) is interpolated between the control points to a complete neighborhood of the characteristic line.

2.5.1.4.4.1.2 Computing brightness of patches (Procedure BP below)

The brightness of a patch is computed using a certain Gaussian – type brightness function, with the basis (in the image plane) – the ellipse, defined by the input patch parameters, and the height of the vertex equal to the input brightness parameter. A specific choice of the model Gaussian – type brightness function is influenced by the same considerations as the choice of the cross-sections model shapes. In particular, it can be taken to be a paraboloid with the elliptic basis and the vertex as specified by the input parameters.

2.5.1.4.4.1.3 Computing brightness of the background (Procedure BB below)

Background brightness values are computed by an interpolation between the brightness values of closed background components, the margin brightness values of the separating characteristic Lines and the brightness values of the background Area Points. The interpolation process is performed in such a way that the interpolation does not “cross” separating Lines.

In particular, this can be achieved by a “signal expansion algorithm”, in which the background representing points (and the margins of separating lines) transmit their brightness value to the neighboring pixels, which in turn transmit it to their neighbors etc. In this expansion the signal is transmitted only to the neighboring pixels, lying on the same side of each of the separating characteristic lines. Finally the background brightness value is computed at each pixel as a weighted average of the brightness values, received by this pixel in the process of signal expansion. The weights reflect the distances to corresponding background representing points.

The range of the signal expansion from each background representing point (AC) is limited to a certain constant, reflecting the density of the background representing points.

Under a proper choice of this constant, the above algorithm is computationally very efficient, since only a few operations are performed per each pixel.

2.5.1.4.4.1.4 Blending the computed brightness values into the final image (Procedure MAIN below)

The final brightness values of a synthesized image are computed as the values of the characteristic lines, patches or the background at the interior pixels of the corresponding parts of the image. At the margin areas of the characteristic lines and of the patches the final brightness is computed by averaging their brightness values with the background brightness, with the weights compute in the Procedures WL and WP, respectively.

2.5.1.4.4.2 Detailed Description of the Reconstruction Algorithm

As it was described above, the reconstruction algorithm starts with the input parameters, as above, and computes the brightness (color) value of the synthesized image at each given point z in the image plane. It consists of the following principal parts:

- Computing brightness of characteristic lines (Procedure BL below)

- Computing brightness of patches (Procedure BP below)

- Computing brightness of the background (Procedure BB below)

- Blending the computed brightness values into the final image (The MAIN Procedure below)


In the case of a color image these computations are performed for each color component.

In the description of each Procedure below, the names of the Procedures, called in the process of computations, are stressed by bold face.

2.5.1.4.4.3 Block Diagram of the Reconstruction Algorithm

2.5.1.4.4.4 Procedure MAIN: Computing the Final Brightness Values

For any point z in the image plane the final brightness B(z) of the Synthesized image at the point z is computed according to the following formula:

(1)

Here BB(z), BL(z) and BPs(z) are the brightness functions of the background, of the characteristic lines and of the

patches, computed in the Procedures BB, BL and BP, respectively, and the sum runs over all the patches Ps.

The weight functions WL(z) and WPs(z) are computed in the Procedures WL and WP, respectively, and

WB(z) = 1 – max(WL(z), WPs(z)).

Division by the sum of all weight functions in formula (1) guarantees that their sum is identically 1 and that formula (1) is a normalized averaging.

2.5.1.4.4.5 Procedure BL: Brightness of characteristic Lines

This procedure computes for any point z on the image the brightness BL(z), as defined by the cross-sections (CP) of the characteristic lines.

BL(z) needs to be computed only for those z which are "close enough" to at least one of the characteristic lines in the texture, as expressed by the weight function WL.

**The most intuitive and natural way to define the brightness of a characteristic line is to associate to it a coordinate system (uu, tt), with uu(z) the (signed) distance of the point z to the line along the normal direction, and tt(z) the length parameter along the curve of the orthogonal projection pp(z) of z onto the line.


Then the brightness cross-sections are computed according to the coordinate uu and interpolated along the line with respect to the coordinate tt.

The corresponding algorithm can be constructed. However, it provides some serious drawbacks:

1. Actual computing of the coordinates uu and tt is mathematically complicated task

2. Even for smooth curves without corners the normal direction is correctly defined only in a small neighborhood of the curve (of the size of a couple of pixels in realistic situations). Outside this neighborhood no natural mathematical solution exist for defining the normal, the projection etc.

3. For spline curves with corners between some of their links (which usually appear in realistic situations) the normal is not defined even locally. Once more, the situation can be corrected by using the mid of the corner angles, but the global difficulties of (2) remain and algorithms become rather complex.

Consequently, we show below an algorithm, which can be considered as an approximation to the “ideal one” above. Its main advantage is that the “coordinates” uu and tt (called below u and t) can be computed independently for each Line Segment (link) of all the collection of characteristic Lines. Moreover, the computation can be ultimately rendered as rather efficient (although the description below may look somewhat complicated).**

Below for any point z, u(z) is the “distance of z to characteristic lines”, S(z) is the closest link to z (with respect to the distance u) in the collection of characteristic lines, and t(z) is the parameter, measuring the projection of z onto S(z), rescaled to the segment [0, 1]. S(z), u(z) and t(z) are computed by the Procedure DL, described below.

The Procedure BL splits according to whether the link S(z) has a “free end” (i.e. an endpoint, not belonging to any other link) or not.

1. S(z) does not have “free ends”.

Let C1 and C2 denote the equations of the two cross-sections (normalized to the unit width, as described in the Procedure CS below) at the two endpoints of the link S(z). For u(z) > 0 let W1 and W2 denote the respective right widths RW1 and RW2 of the cross-sections at these points. For u(z) < 0 let W1 and W2 denote the respective left widths LW1 and LW2 of the cross-sections at these points. Then in each case

where W(z) is the interpolated width

and the values and are computed by the procedure CS.

2. S(z) has a “free end”.


If for this “free end” the parameter t is zero, the brightness BL(z) is computed as above for t(z) > 0. For

and for

Here DE is a positive tuning parameter, defining the shape of the end of a characteristic line. BM is half of the sum of the brightness parameters LB2 and RB2 of the cross-section at the free end.

If for this “free end” the parameter t is one, t(z) is replaced by 1-t(z) in the above formula.

** The formula above provides one of possible choices of the shape of characteristic lines near their ends. It assumes that the cross-section brightness gradually descends to the “mid value” value BM inside the prescribed distance DE. Other shapes can be defined, by properly computing the width and the brightness in the neighborhood of the end point.**

2.5.1.4.4.6 Procedure CS: Color Profiles (Cross-Sections) of characteristic Lines

This procedure computes a brightness value of an edge or a ridge (unit width) cross-section CS(u) for any given cross-section “interior” brightness parameters, as described above, and for any value of u.

In the Procedure BL u is the distance u(z) to the line, normalized by the width W(z) of the line, so the width parameter W is taken into account inside the BL, and it does not appear below. Similarly, the margin brightness parameters LB1 and RB1 enter the computations in the Background brightness Procedure BB.

2.5.1.4.4.6.1 Edge Cross-Section.

Normalized edge cross-section NEC(u) is defined as follows (Figure 101):

**Thus the recommended edge cross-section is composed of two symmetric parabolic segments.**

For given brightness parameters LB2 and RB2, the value CS(u) is computed as


Figure 101 – Edge cross-section definition.

2.5.1.4.4.6.2 Ridge Cross-Section

As for edges, the width of the ridges is taken into account in the Procedure BL. Similarly, the margin brightness parameters LB1 and RB1 enter the computations in the Background brightness Procedure BB. Consequently the ridge cross-section computed in the current Procedure CS, is the same for separating and non-separating ridges, and is defined by the parameters LB2, CB and RB2, as follows (Figure 102):

for < 0, and

for

Figure 102 – Ridge cross-section definition.

** Thus the recommended ridge cross-section is composed of two edge cross-sections, properly aggregated.

In the process of the blending of these cross-sections with the background (which incorporates the margin brightness values LB1 and RB1) we get back essentially the same cross-section, as shown above (Figure 103).**


Figure 103 – Blending two cross-sections with the background.

2.5.1.4.4.7 Procedure WL: Weight Function of characteristic Lines

**This block computes the weight function WL(z), which is used in a final blending of the characteristic lines with the background. The function WL(z) is equal to one in a certain neighborhood of the characteristic lines, and is zero outside of a certain larger neighborhood.**

More accurately:

1 |u(z)| < UL2·W(z)

WL(z) = UL2·W(z) <| u(z)|< UL1·W(z)

0 |u(z)| > UL1·W(z)

The distance u(z) is computed in the Procedure DL.

UL1 and UL2 are tuning parameters; see the last section “Tuning Parameters”.

Figure 104 shows a typical cross-section and a general shape of the weight function WL(z).


Figure 104 – Typical cross-section and general shape of WL(z)

2.5.1.4.4.8 Procedure DL: Distance to characteristic Lines

This Procedure computes for any point z in the texture:

1. The point p(z) on the characteristic lines which is nearest to z, i.e. the “projection” p(z) of z onto the set of characteristic lines.

2. The distance u(z) between z and p(z).

3. The link S(z) on which p(z) resides.

4. The proportion t(z) in which p(z) divides the link S(z).

Note u(z), p(z) and t(z) are NOT exactly the Euclidean distance, the corresponding mathematical projection and proportion respectively; however, in most cases they give a reasonable approximation for these mathematical entities.

These data are computed in the following steps:

5. For each link (Line Segment, LS) Si in the texture, the corresponding pi(z), ui(z), ti(z) are computed in Procedure DDL (See the Synthesized-C figure below)

6. S(z) is defined as the link S j, for which the minimum of the absolute values |u i(z)| is attained (See the figure Synthesized-D below)

7. u(z) is defined as the function uj(z) for the link Sj = S(z)

8. t(z) is defined as tj(z) for the above link Sj


Figure 105 - Synthesized-C

Figure 106 - Synthesized-D

2.5.1.4.4.9 Procedure DDL: Distance to a Line Segment

This procedure computes for any point z its (signed) distance u(z) to a given link S (Line Segment, LS), the projection p(z) of the point z onto the link S and the parameter t(z). The Procedure is essentially represented on figure Synthesized-C above (which shows, in particular, equidistant lines for the points z1 and z4.

The straight oriented segment [a, d], joining the end points of the link S is constructed, with the orientation, induced from the orientation of the Line, containing S. l1 is the straight line, containing the segment [a, d]. l2 and l3 are the straight lines, orthogonal to l1 and passing through a and d, respectively.

Now, for any z in the image plane, the function u(z) is constructed as follows:


For z between l2 and l3, the absolute value |u(z)| is the length of the segment, joining z and S and orthogonal to l1 .

For z left to l2 , |u(z)| is defined as follows:

.

Here d(z, l1) and d(z, l2) are the distances from z to l1and

l2 respectively. D is a tuning parameter, with a typical value D = 4.

For z right to l3 , |u(z)| is defined as

.

Let ll be an oriented line, formed by the interval of the line l1 from infinity to a, then by the link S from a to d, and then by the interval of the line l1 from d to infinity.

For z right to ll (with an orientation as above) the sign of u(z) is “+”. For z left to ll, the sign of u(z) is “-“.

For z between l2 and l3, the projection p(z) is the intersection point of S and of the segment, joining z and S and orthogonal to l1 . For z left to l2 , p(z) is a, and for z right to l3 , p(z) is d.

For any z, t(z) is the proportion, in which the projection of z onto the line l1 subdivides the segment [a, d]. For example, for the point z2 and z3 on Fig Synthesized-D above. t(z2) = (b-a)/(d-a), and t(z3) = (c-a)/(d-a), respectively. For z left to l2 , t(z) < 0, and for z right to l3 , t(z) > 1.

** The special form of the function u(z) above (for z outside the strip between l2 and l3) is motivated by the following reason: when computing in the Procedure BL the brightness of the line near a sharp corner, the form of the distance function u(z) determines which link will be taken as the closest to the points in the sector stressed on the Figure below. For the distance, computed as above, with the parameter D > 1, this choice is matched with the sign of u(z), as defined above. If we would have chosen D < 1, for z in the sector stressed on the Figure below, the choice of the nearest link, together with the proposed computation of the sign of u(z), would produce a color from the incorrect side of the line. See the figure below.**


2.5.1.4.4.10 Procedure BP: Brightness of Patches

Let Cx, Cy, R1, R2, a, CB and MB be the parameters of a certain patch Ps, as described above.

Let M be the linear transformation of the plane, transforming the basis ellipse of the patch to the unit circle. M is a product of the translation by (-Cx, -Cy), the rotation matrix to the angle –a, and the rescaling 1/R1 and 1/R2 times along the x and y axes, respectively. If we put for

then the equation of the basis ellipse of the patch is given by

.

The brightness function BPs(z) of the patch is then given by

for > ,

for and

for

Here UP1 > 1 is a parameter. See the Figure below.

2.5.1.4.4.11 Procedure WP: Weight Function of Patches

The weight function WPs(z) for a patch Ps as above is defined by

for

for and


for uu(z) between UP2 and UP1,

where uu(z) denotes the square root of .

Here UP2, 1 < UP2 < UP1, is another tuning parameter. See the figure below.

2.5.1.4.4.12 Procedure BB: Brightness of Background

**This Procedure computes the brightness value of the background at any point z of the image. This value is obtained as a result of interpolation between the “global” background brightness values, the margin brightness values of the characteristic Lines and the brightness values at the background representing points (ACs – Area Color Points),. The main difficulty is that the interpolation is not allowed to cross the separating lines. To overcome this difficulty a special “distance” d between the points on the image is introduced, which is the length of the shortest pass, joining these points, and not crossing separating Lines, and which is computed in the Procedure SE below. Then averaging weights are computed through the distance d.**

This block uses as an input a certain collection of the background representing points Z i, (containing the input background representing points, as described above, and the margin representing points, produced by the block “MRP’, described below). At each point Zi the brightness value Bbi is given.

The background brightness value BB(z) is finally produces by the block BB as follows:

BB(z) is the weighted sum of the global brightness GB and of the Local brightness functions Bb i(z) over all the background representing points Zi :

Here

so the expression (2) is normalized to provide a true averaging of the corresponding partial values.

The global brightness value BG is provided by the Procedure GB below, or by any of the Gradient Nodes.


The computation of the Local brightness functions Bbi(z) is performed in the Procedure LB below.

The distance functions d(z, Zi) are computed in the Procedure SE below.

The computation of the weight functions WR(d(z, Zi)) is performed in the Procedure WB below.

The weight GW(z) of the global background value GB is defined as

GW(z) is zero at any z, where at least one of the weights of the representing points is 1, and GW(z) is one at any z where all the weights of the background representing points vanish.

2.5.1.4.4.13 Procedure GB: Global Brightness of Background

This Procedure computes the global background value GB, which appears in the expression (2) in the Procedure BB.

By definition, if the point z is inside the background region of a Sub-Texture number r, for which the global value GBr is defined, GB is equal to this global value GBr. If the point z is inside the background region of a Sub-Texture, for which the global background value is not defined, GB is equal to the default global value DGB. If DGB is not defined, GB is equal to zero. Alternatively, Color Gradients can be used.

The current procedure consists in a signal expansion that transmits to each pixel its Sub-Texture number. We describe it shortly, since it essentially belongs to a higher data representation level.

First the procedure MRP is applied, which creates margin representing points, carrying the corresponding Sub-Texture numbers. These numbers are taken from the corresponding poly-links.

Second, the Signal Expansion Procedure is applied to the margin representing points, essentially as in the block SE, with the following difference: only the marking and the number of the Sub-Texture is transmitted between the pixels on each step of signal expansion.

As this procedure is completed, each pixel in the image memorizes the number of the Sub-Texture, to which it belongs.

2.5.1.4.4.14 Procedure LB: Local Brightness of the Background

Two types of the local brightness functions Bbi(z) are used. For the first type (zero order)

Bbi(z) is identically equal to the input brightness value Bbi at the point Zi .

For the second type (first order) Bbi(z) is equal to Li(z), where Li(z) is the linear function, such that

Li(Zi) = Bbi

and Li provides the best approximation of the input brightness values at the N nearest to Z i background representing points. The choice of the type of the local brightness function is determined by the flag LBF: LBF is


zero for the zero order and LBF is one for the first order of the functions Bb i(z). Here N is an integer valued tuning parameter.

** Typical value of N is 4 or 9: usually the background representing points form a regular or an almost regular grid, and the nearest neighbors are taken at each point Zi to construct the linear function Li(z).**

2.5.1.4.4.15 Procedure WB: Weights for the Background

As implied by the form of the expression, the weights WR(d(z, Z i)) depend only on the distance d(z, Z i) from the point z to the background representing point Zi. The model function of one variable WR is specified by three tuning parameters UB1 and UB2, UB1 > UB2 > 0, and BVS (Background Weight smoothness), 0 < BVS < 1, and is defined as follows:

for

for and

for

where

See the Figure below.

2.5.1.4.4.16 Procedure SE: Signal Expansion

Let D denote the domain of the Synthesized image, with “cuts” along all the separating Lines PL i. For any two points z1, z2 in D the distance dd(z1, z2) is defined as the (Euclidean) length of the shortest path, joining z1 and z2 in D and avoiding all the cuts PLi. See the figure below.

**It might be assumed that the influence of the color at z1 to the color at z2 decreases as the distance dd(z1, z2) increases. However, a precise computation of the distance dd is a rather complicated geometric problem. Consequently, we use instead of the distance dd(z1, z2) its approximation d(z1, z2), which is computed through a “signal expansion algorithm”, as described below.**


The block SE computes the distance d(z1, z2) for any two points z1 and z2 in the image plane. The algorithm is not symmetric with respect to z1 and z2: in fact, for a fixed point z1, the distance d(z1, z2) is first computed for any pixel z2 of the image. Then an additional routine computes d(z1, z2) for any given z2 (and not necessarily a pixel).

Below the notion of a “neighboring pixel” is used. It is defined as follows: for z not a pixel, the four pixels at the corners of the pixel grid cell, containing z, are the neighbors of z. For z a pixel, its neighbors are all the pixels, whose coordinates in the pixel grid differ by at most one from the coordinates of z.

Below we assume that a certain data structure is organized, in which to any pixel p on the image plane a substructure is associated, allowing to mark this pixel with certain flags and to store some information, concerning this pixel, obtained in the process of computation. We do not specify here this data structure, using instead expressions like “the pixel p is marked”, “the pixel p memorizes…” etc.

Now for z1 and z2 given, the distance d(z1, z2) is computed in the following steps:

For any pixel p the distance u(p) to the separating Line PLi is computed and stored at this pixel. The computation of u(p) is performed by the procedure DL, described above, applied only to separating poly-links PLi .

Those pixels p, for which u(p) < FU, are marked as “forbidden” pixels. The forbidden pixels are excluded from all the rest of computations, and those pixels that are not forbidden, are called below “free” ones. Here FU is a tuning parameter.

Now the proper “signal expansion” starts. In the first step each of the free neighbor pixels of z1 is marked, and this pixel memorizes its Euclidean distance from z1 as the auxiliary distance dd, to be computed. Generally, in the k-th step, any free unmarked pixel p, at least one of whose free neighboring pixels was marked in the previous steps, is marked. This pixel memorizes as its auxiliary distance dd(p) from z1, the minimum of dd at the neighboring free pixels plus the Euclidean distance of p to the neighboring pixel, at which the minimum is attained. This process is continued the number of steps, equal to the maximal dimension of the image (in pixels). After it is completed, each free pixel p on the image plane memorizes its auxiliary distance dd(p) from z1.

Now for any given point z2 on the image plane, its distance d(z1, z2) from z1 is computed as maximum of D1 and D2, where D1 is the Euclidean distance of z2 to z1 , and D2 is the minimum over the free neighboring to z2 pixels p, of dd(p) + the Euclidean distance of z2 to p.

This completes the computation of the distance d(z1, z2).

See the Figure below.


** The tuning parameter FU determines the size of a neighborhood of the separating Line, where all the pixels are marked as forbidden. Taking any value of FU, larger than 0.8, excludes a possibility of signal expansion crossing separating lines. Indeed, for any two neighboring pixels, which are on different sides of a separating line, at least one is closer to the line than 0.8 and hence is marked as forbidden. To provide stability of finite accuracy computations a bigger value of U may be taken. However, in this case signal expansion will not pass a “bottle-neck” between two separating lines, which are closer to one another than 2FU. Normally such regions will be covered by the cross-sections of these lines. However, a sub-pixel grid can be used to guarantee that signal expansion passes thin “bottle-necks”

Various implementation issues:

Computation of the distance d(z1, z2) and its usage inside the background grid interpolation form one of the central parts of the Synthesized image reconstruction. Consequently, the efficiency of the implementation of these blocks is crucial for an overall efficiency of the reconstruction.

A straightforward implementation of the signal expansion algorithm, as described above, is not optimal. However, a number of rather natural and simple modifications bring the efficiency of the signal expansion to only few operations per pixel.

9. Multi-scale implementation. The image is subdivided into blocks in 2 – 3 scales (for example, blocks of 16x16, 8x8 and 4x4 pixels). First signal expansion is performed between the highest scale blocks (say, 16x16), exactly as described above. Forbidden are the blocks, crossed by separating characteristic lines. In the second stage the forbidden blocks are subdivided into 8x8 sub-blocks, and the expansion is performed for them. The new forbidden sub-blocks are subdivided into the 4x4 ones, and the expansion is repeated. In the last stage the expansion is completed on the pixels level.

10. For an application to the background grid interpolation, the distance d(z1, z2) has to be computed only till it riches the threshold UB1 (since for larger distances the weight functions vanish). This fact allows one to restrict the number of steps in signal expansion to UB1 + 1.

11. Signal expansion and memorization of the distances at the free pixels can be implemented for all the background representing points at once (especially since the above distance restriction usually makes for any pixel only the information relevant, concerning a few neighboring background grid points).


12. In the process of signal expansion, all the mathematical data required in the interpolation block (like Euclidean distances and weight functions) can be computed incrementally in a very efficient way.**

2.5.1.4.4.17 Procedure MRP: Margin Representing Points

This Procedure constructs a grid of representing points on the margins of all the characteristic Lines together with the background brightness values (APs) at these points. Later the constructed margin points are used (together with the original background representing points) in the interpolation process in the block BB.

The margin representing points Mzj are produced in the following steps:

13. On each Line, the points wk are built with the distance UM1 from one another, starting with one of the ends (the distance is measured along the Line). The last constructed point on each Line may be closer to the end Terminal Point of this Line, than UM1.

14. At each wk the line lk orthogonal to the Line and intersecting it at wk is drown. If wk turns out to be a vertex of the Line with a nonzero angle between the adjacent Line Segments LS (links), or a crossing, lk is taken to be the bissectrice of the corresponding angle.

15. On each line lk two points (one point in the case of the bissectrice of the crossing joint angle) are chosen at the distance from the intersection point wk of lk with the Line (from the crossing wk, respectively). All the chosen points, in a certain chosen order, form the output margin representing points Mz j.

16. At each margin representing point Mzj constructed, as described above, the corresponding margin background brightness value Bbj is computed by

,

where A and B are the margin values (LB1 or RB1, respectively) of the cross- sections at the ends of the Line Segment S(Mzj), nearest to the point Mzj, and t = t(Mzj).

S(Mzj) and t(Mzj) are computed by the Procedure DL.

In the current Procedure UM1 and UM2 are tuning parameters (the first one absolute and the second relative to the width), satisfying UM1 < UB1, 1 < UM2 < UL1.

See the figure below.


The four figures below illustrate influence of different elements of the background for different values of tuning parameters.

Figure 107 shows an extrapolation of the margin values of the characteristic Line.

Figure 108 illustrates the influence of the background representing points (APs).

Figure 109 shows representing points with a larger parameter UB1.

Figure 110 shows a patch.

Figure 107 - Synthesized-AA Figure 108 - Synthesized-BB

Figure 109 - Synthesized-CC Figure 110 - Synthesized-DD2.5.1.4.5 SynthesizedTexture Rendering - Tuning Parameters

All the tuning parameters, listed below, are not changed in the process of sending and playing of images and animations. However, their tuning for specific visual displays (and for specific classes of images) may improve the visual quality and the player efficiency.

UL1 – the distance from the characteristic Lines (as a proportion to the width of the line), within which the CP (cross-section) brightness BL(z) is computed. It coincides with the exterior size of the lines averaging region. Used in the Proceduress BL and WL.

UL2 – the interior size of the lines averaging region (as a proportion to the width of the line). Used in the Procedure WL

UP1 and UP2 are the corresponding parameters for patches (referring to the relative distance with respect to the patch size). Used in the Procedures BP and WP.


UB1 and UB2 are the (absolute) size parameters for the weight functions of the background representing points, APs. The parameter BVS characterize the smoothness of these weight functions. Used in the Procedure WB.

The flag LBF specifies the order (0 or 1) of the local brightness representation. Used in the Procedure LB.

The integer parameter N is the number of neighboring background representing points taken to define the local linear approximations of the brightness. Used in the Procedure LB.

FU is the distance of the separating lines at which pixels are marked as “forbidden”.

Used in the Procedure SE.

UM1 and UM2 are the absolute and the relative to the width parameters in construction of margin representing point. Used in the Procedure MRP.

D > 1 is the parameter, defining the “asymmetry” of the distance to the Line Segment, computed in its side areas. Used in the Procedure DDL.

DE > 0 is a parameter, defining the length of the end area of characteristic Lines. Used in the Procedure BL

2.6 Modeling tools

2.6.1 Introduction

This section is related to geometric tools but the following tools don’t define a geometry per-se, they merely deform it. These tools are purely computer graphics tools and are typically used in authoring software to locally modify a shape. These deformations are independent of the representation i.e. in practice; they can be applied to both the polygonal and the parametric representation. The first represents a generalization of the standard modeling operations facilitating nonlinear global deformations of objects [17]. The second is a powerful approach that deforms objects by deforming the space in which the object is embedded [76,24]. Finally, curve-based interpolators define piecewise curvilinear animation paths.

2.6.2 Nonlinear global deformation

2.6.2.1 Overview

In a standard modeling operation such as scaling, the operation, usually in the form of a transformation matrix, is applied to each vertex (or control point) in turn. Throughout this procedure, the transformation matrix does not change. Alan Barr [17] proposed to alter the transformation while it is being applied to the object. The transformation itself becomes a function of the position at which it is applied. The three basic transformations he proposed are: tapering, twisting, and bending. Transformations that are more complex can be obtained by combining these transformations.

Tapering

To taper an object long the z-axis, x- and y-axes are just scales as a function of z:

)(and),,(),,( zfrzryrxZYX

where )(zf specifies the rate of scale per unit length along the z-axis and can be a linear or nonlinear tapering profile or function.

Twisting


To rotate an object through an angle about the z-axis:

)(and),cossin,sincos(),,( zfzyxyxZYX

where )(zf specifies the rate of twist per unit length along the z-axis.

Bending

A global linear bend along an axis is a composite transformation comprising a bent region and a region outside the bent region where the deformation is a rotation and a translation. Barr defines a bend region along the y-axis as:

maxmin yyy . The radius of curvature of the bend is 1k and the center of the bend is at 0yy . The bending

angle is: )( 0yyk , where

maxmax

maxmin

minmin

ififif

yyyyyyy

yyyy

The deformation is given by

maxmax11

minmin11

maxmin11

maxmax01

minmin01

maxmin01

)(sin)(cos)(sin)(cos

)(cos

)(cos)(sin)(cos)(sin

)(sin

yyyykkzyyyykkz

yyykkzZ

yyyyykzyyyyykz

yyyykzY

xX


We propose the NonLinearDeformer node:

NonLinearDeformer {exposedField SFInt32 typeexposedField SFVec3f axis 0 0 1exposedField SFFloat paramexposedField MFFloat extendexposedField SFNode node

}

where type is the desired deformation (0: tapering, 1:twisting, 2:bending). axis is the axis along which the deformation is performed, param the parameter of the transformation, extend its bounds, and node the geometry node on which is the deformation is performed or another NonLinearDeformer node so to chain the transformations.


Type Param Extend

0 tapering Radius { relative position, relative radius }*

1 twisting Angle Angle min, angle max

2 bending Curvature Curvature min, curvature max, y min, y max

For tapering, extend consists of a serie of 2 values: the first is the position at which the radius should be. This way a profile can be defined. The relative position along the axis of the transformation in object space: 0% at the beginning, and 100% at the end. The radius is relative to the param and is given in percentage.

2.6.3 Free-Form Deformations (FFD)

2.6.3.1 Overview

Space-warps deformations are modeling tools that act locally on an object and possibly on a set of objects. The most commonly used may be the Free-Form Deformation tool [24,76], which encloses a set of 3D points (not necessarily belonging to a single surface) by a simple mesh of control points. By moving the points of this mesh, enclosed points are attracted and moved. Therefore, complex local deformations of meshes may be specified with few parameters.


The purpose of the FFD node is to deform a sub-space in the 2D/3D scene. Our sub-scene is specified in children using the DEF/USE mechanism. In addition, our construction enables nested free-form deformations.

FFD {eventIn MFNode addChildreneventIn MFNode removeChildrenexposedField MFNode children []field SFInt32 uDimension 0 # [0, )field SFInt32 vDimension 0 # [0, )field SFInt32 wDimension 0 # [0, )field MFFloat uKnot [] # (-,)field MFFloat vKnot [] # (-,)field MFFloat wKnot [] # (-,)field SFInt32 uOrder 2 # [2, 32)field SFInt32 vOrder 2 # [2, 32)field SFInt32 wOrder 2 # [2, 32)exposedField MFVec4f controlPoint []

}

A FFD node acts only on a scene on the same level in the transform hierarchy. This apparent restriction is because a FFD applies only on vertices of shapes. If an object is made of many shapes, there are nested Transform nodes. If we pass solely the DEF of these nodes, then we have no notion of what the transforms applied to the nodes are. By passing the DEF of a grouping node, which encapsulates the scene to be deformed, we can effectively calculate the transformation applied on a node.


Mikaël Bourges-Sévenier, 03/01/-1,

text TBR

This functionality is very useful in modeling to create animations involving deformations of multiple shapes, as well as complex non-rigid deformations of a single shape. As very few control points need to be moved, the animation stream would require few bits.

Example:

# The control points of a FFD are animated. The FFD encloses two shapes, which are deformed

# as the control points move.

DEF TS TimeSensor {}

DEF PI CoordinateInterpolator4D {

key [ … ]

keyValue [ … ]

}

DEF BoxGroup Group {

children [ Shape { geometry Box {} } ]

}

DEF SkeletonGroup Group {

children [

…# describe here a full skeleton

]

}

DEF FFDNode FFD {

…# specify NURBS deformation volume

children [

USE BoxGroup

USE SkeletonGroup

]

}


ROUTE TS.fraction_changed TO PI.set_fraction

ROUTE PI.value_changed TO FFDNode.controlPoint

2.7 Animation tools

2.7.1 Animators

2.7.1.1 Overview

An animated object follows an animation path over a period of time. The path is defined in the local coordinate system of the animated object. At frame t (or at time t), the object is at position and has orientation represented by a unit quaternion . A keyframe animation is specified by pairs , where is

called a keyframe, is the key value i.e. the position the object must be at time and the object’s orientation at time . An animation is completely specified by

Eq. 3 – Animation model.

where is the animation path with parameter , the orientation path, and is the timeline. The animation path is often a succession of curves of different characteristics that join at key values. Let be the

characteristic of curve segment i and its orientation.

In the simplest case, like with VRML [5] and BIFS [1] interpolators nodes (PositionInterpolator, ScalarInterpolator, and so on), the content creator specifies . The animation is piecewise linear with

Eq. 4 – Piecewise linear animation model in VRML and BIFS.

with the number of keyframes. The equation for is known as spherical linear interpolator or slerp.

Often, it is needed to have more control over the timeline to create more realistic effects or even special effects. The animation framework proposes different types of timelines and path characteristics. Even if this specification was developed independently, we aligned it for convenience with similar concepts found in SMIL [10] and SVG [9] recommendations. However, this specification extends animation concepts in 1/2/3D with more flexibility for specifying animation paths and timelines.

Animation support in BIFS is borrowed from VRML 2.0 specification. It can be done in two ways:

1. using piecewise linear interpolators


2. programmatically using VRMLScript (in Script node) or Java language (via EAI, MPEG-J, or JSI in Script node)

While the second way is the most flexible, it requires usage of external interfaces and interpreted languages that imply a relative performance slow down. The first way is built in the browser and allows faster performance. The model is as follows: a timer (a TimeSensor node) generates clock ticks over a normalized timeline; a clock tick is in [0, 1] where 1 corresponds to the duration of the animation. Each clock tick is sent to an interpolator node (PositionInterpolator, ColorInterpolator, and so on) that generates a value that can be sent to any field of the same type, hence producing animation. An interpolator node specifies piecewise linear segments against time (Figure111).

TimeSensor

Interpolator

A node

fraction_changed

set_fraction

value_changed

<a field>

Figure 111 – Animation in VRML/BIFS. A timer sends events to an interpolator that generates values that can be sent to anode.Current interpolator nodes are very low-level animation tools. As for all low-level tools, an object must be sampled. The more curved the object, the finer the sampling; resulting in increased size of the file or stream. For animations, Figure 112 exhibits the same issue: the piecewise curve path is approximated with line segments. The sharper the curvature, the more lines are needed.

x

y

Figure 112 – Example of path. The black line is a cubic B-spline curve (7 control points, 11 knots). The dashed like is an approximation made of 29 linear segments (30 points).

Other issues arise with piecewise linear interpolators:

Issues with animation path


o Once an animation path is approximated by linear segments, it is not possible to recover the intent of the content creator. For example, what was the curvature of the path?

o If a point in the path is modified, how does it affect the overall path? Is it just local or does it change the animation path?

o Are the points parts of the animation path or are they part of the approximated curve?

o The animation path definition uses many unnecessary extra values

o The animation path does not reflect the real intent of the content creator

o The approximation may be coarse and effects will be visible if viewer close enough

o When an object follows a path, it is often necessary to have it aligned with the tangent to the path in the direction of the path parameter (or +). To provide this effect, two interpolators are needed: one for the path, one for the rotation. Once again, using two interpolators cancels the intent of the content creator and double the number of data.

o When a path is repeated, there is no possibility to start from where it ends. For example, assume one wants to follow a helix path. Only the path for 360° is needed and each new turn restarts from the end of the previous turn. Using interpolators, the whole path needs to be described i.e. for 5 turns; the helix path is reproduced 5 times!

Issues with timeline

o Figure 112 shows each value against time. It also shows that the speed of an object traveling along this path is non-uniform: it depends on the distance and the time between two consecutive values. If a content creator wishes to maintain a constant speed, he must approximate the path using a reparametrization by arclength (the distance traveled along the path) i.e. the arclength must be proportional to time.

o The current specification does not allow any velocity curve. Even if an object follows the same path, it may travel at different speeds. For example, when a car starts, it goes from 0 to a certain speed, remains at this speed, and then decelerates to rest, regardless of the path traveled.

Despite their limitations, interpolators are very important tools because they enable the content creator to specify keyframes and values at a fixed frame rate. However, in practice, the frame rate is often different than what the content creator wanted: it depends on multiple factors such as rendering frame rate, CPU speed and load, and so on. As a result, a keyframe specified by an author is often missed; making the approximation looks even coarser.

When a content creator creates an animation, he positions and orients objects in space to create an animation path. Then, he adjusts the speed at which he wants objects to travel and their orientations at few keyframes.

This specification proposes a flexible way to overcome all these issues with an extensible way to specify timelines and path. The proposed Animator nodes extend interpolators capabilities and are backward compatible with them: the default behavior of the node makes it work like an interpolator and, even by default; it may result in better compression.


The overview on NURBS curves and surfaces (section 2.3.1) reveals the key concepts used in this specification. These curves are used in path and velocity curve definitions as well as for orientation interpolation. Here, a review of orientation is made followed by the Animator nodes.

2.7.1.2 Animating object orientation

VRML and BIFS proposed the OrientationInteprolator node that interpolates pairs between key frames, where is a unit vector and the angle around the axis. Between two frames, the shortest path on the unit sphere is computed, which can be translated by applying spherical linear interpolation in quaternion space.

First let’s review quaternion mathematics and then a technique for C2 interpolation in quaternion space is exposed.

The quaternion is a point on the unit sphere in 4-space and is short-hand for with

.

A quaternion is often expressed as a scalar component and a vector component with and . Using previous notation, the multiplication of two quaternions is

A unit quaternion satisfies .

A point is rotated by a unit quaternion to the point , which is a quaternion with a scalar part of 0, so it can be interpreted as a 3-space point.

In particular the unit quaternion represents the orientation of an object where the object in its

canonical orientation has been rotated by radians about the axis .

Mathematically, an orientation can be represented by either of two antipodal quaternions (by flipping ). For motion control, only one quaternion is appropriate: the quaternion with smaller angular gap on to the previous

quaternion i.e. if , then use . This makes the orientation path always perform the shortest rotation between the orientation key frames.

Spherical linear interpolation (also called slerping technique) is equivalent to linear interpolation between two key orientations and, as for linear interpolation, produces jerky, sharply changing motion across the keys [99]. This is particularly important for camera flythrough where sharp changes in angular acceleration have to be avoided.

What is needed is a higher order method with C2 continuity. Many methods have been proposed in the literature but leads to no closed form algebraic solution and, when based on slerping, generate nonrational curves, may become expensive to calculate, and may not provide C2 continuity.


The technique proposed by Johnstone and Williams [51] produces very good results in practice and obeys derivative continuity requirements. It uses an invertible rational mapping between and .

The transformation from is [51,96]

and the transformation from is

The method is as follows:

1. Apply to key quaternions to obtain their resulting values in

2. Interpolate the resulting 4-vectors using any spline (in this specification we use natural cubic B-splines).

3. Convert the desired point to using .

However, if your animation passes close to the pole (1, 0, 0, 0), numerical instability will happen in . Johnstone [51] define “too close to the pole” if the angle between the control quaternions and (1, 0, 0, 0) is smaller than 30 degrees. His solution is to multiply all control quaternions by a quaternion that is not within 30 degrees of any other control quaternion and use to rotate all quaternions to a “safe” region. After interpolation, the resulting

quaternion is multiplied by . This can be done in a preprocessing step i.e. before the animation starts.

2.7.1.3 Animator node

The model of the node follows a content creator way of designing animation. Animator nodes specify the path as well as timelines. The Animator nodes also have flags for cumulative path when it is repeated as well as normal vector to the path for specifying the orientation at the beginning of the animation.

PositionAnimator { #%NDT = SF3Dnode# inputeventIn SFFloat set_fraction # clock tick

# timeline


exposedField SFVec2f fromTo 0 1exposedField MFFloat key [] # exposedField SFInt32 keyType []exposedField MFVec2f keySpline [0 0, 1 1 ] # 2 2D-pts

# path characteristicexposedField MFVec3f keyValue [] # exposedField MFRotation keyOrientation [] # exposedField MFFloat weight [] # exposedField SFInt32 keyValueType 0 # line

# cumulexposedField SFVec3f offset 0 0 0

# outputseventOut SFVec3f value_changedeventOut SFRotationrotation_changedeventOut SFVec3f endValue

}

The timeline is specified by key, keyType, and keySpline fields. The path is specified with keyValue, keyValueType, and keyValueSpline fields. The orientation at each key position is specified in keyOrientation.

Timeline specification

fromTo specifies when the Animator is active. By default, an animator is active for the whole duration of the timeline: from t=0 to t=1. fromTo can be used when one wants to produce animations with interruptions (Figure111).

from 0

to 0

from 1

to 1

t

Figure 113 - fromTo field usage. The same timer can be used to trigger two Animator nodes starting end ending at different time.

keyType defines the timeline and five types of predefined timelines are possible (Figure 114):


0 1

0 1

Discrete Linear

Paced Velocity spline

t

value

1

10

0

Figure 114 – Predefined types of timelines: discrete, linear, paced (constant speed), and velocity spline. The first type (keyType 0) is not shown and is user-sepcifed. Note how the animation paths are identical but the timelines so different.

Let’s denote

the time fraction received in set_fraction field,

i is the curve segment defined by two keyValues i and i+1,

.

.

.

ti and ti+1 the time corresponding to and ,

n is the number of keyValues.

keyType Description

0 User defined, like interpolators. The time is specified for each keyValue. If there are n keyValues, there must be exactly n keys.

1 Discrete. The timeline is divided into n equal intervals. The value remains constant on interval i to :


2 Linear. The timeline is divided into (n-1) equal intervals.

For a line segment , the position is .

3 Paced. The speed remains constant over the animation path. A reparametrization by arclength (the distance traveled) is needed. The arclength is defined as

where is the first order derivative of the animation path by the animation parameter u, In general, is not integrable (especially for splines) and it is left to the implementation to decide which quadrature formula to use. Whatever the approximation used, the speed should remain constant, the exact position of keyframes is not mandatory.

For piecewise linear paths, the arclength is:

,

Let be the distance traveled up to , the total length of the animation path,

and the time spent to travel , then

For orientation, is used to interpolate between and .

4 Spline. Defines a cubic Bézier velocity curve. Such a curve is defined by 4 points


: end points are and and the two

intermediate points are defined in keySpline, see Figure 115. The equation of the velocity curve is:

1

10

1

10

1

10

1

10

keySpline [ 0 0, 1 1 ] keySpline [ 0 0.75, 0.25 1 ]

keySpline [ 0.5 0, 0.5 1 ] keySpline [ 1 0, 0.25 0.25 ]

t

uu

u u

t

t t

Figure 115 – Timeline control using cubic Bézier spline.

Algorithmically, is first solved for the parameter u using a simple bisection algorithm given in section , which is appropriate since the function is strictly monotonically increasing. is determined for u. t’ is then used to

determine the path segment it belongs to determine the position value and orientation

key must contain the keyframes for each keyValue and keyOrientation.

Path specification


keyValueType specifies the type of curve characteristics. Apart from 1 (line), a curve is represented as a NURBS curve (type 3) and types 1 (quadratic Bézier) and 2 (cubic Bézier) are special cases. For NURBS and interpolating curves, the number of points N used must follow the type i.e. like [ … 3 N …]. Type 0 allows cusps in the animation path; this enables jumping from one path to another without in the same Animator node.

keyValueType

Description Number of keyValue

0 Line 1

1 Quadratic Bézier 3

2 Cubic Bézier 4

3 NURBS curve <N>

4 Interpolating cubic spline curve

<N>

Table 13 – This table gives for each keyValueType, the number of keyValue that must be in the keyValue field.

Notes:

1. For type 0, multiple line segments can be defined: the polyline passes though each point in keyValue field (like any interpolator). For other types, only one type of curve is possible (for example, if a path needs to have two cubic Bézier curves, they can be represented by a single NURBS curve and continuity constraints resolved in the knot vector).

2. For other type, an Animator node specifies only one type of curve characteristic. In general, any continuous path can be represented by a NURBS. Also, using cumulating (offset and endValue fields, see below), two Animator can be chained.

keyValue specifies the points the animation path should pass through (line and interpolating types) or the control points for curves (quadratic and cubic Bézier, and NURBS).

weight specifies the weights for each keyValue. If weight is empty then is assumed. Else, there must be as many weights as control points.

Note that type 1 and 2 are special cases of NURBS curves. They are separate for convenience and better compression due to their important use in animation:

For quadratic Bézier curves (keyValueType 2), and . 3 control points (from keyValue) are needed.

For cubic Bézier curves (keyValueType 3), and . 4 control points (from keyValue) are needed.


Non-rational B-spline curves are obtained for NURBS curve (keyValueType 4) with and a knot vector.

For NURBS curves (keyValueType 3), key contains the knot vector . While this specification allows unclamped knot-vectors (first and last knot not repeated p times), this would produce curves not passing through end points and hence produce cusps in the animation path.

An interpolating cubic spline curves (keyType 4) passes through all <N> points specified just after the type in keyValueType field. Chord length parametrization with averaging knots [99, 75] shall be used:

With this method the knots reflect the distribution of the . Furthermore, this results in a system of equation totally positive and banded with a bandwidth less than , which can be solved by Gaussian elimination without pivoting. Note that the resulting animation would be similar to a paced timeline [44, 75] if linear timeline is used with this type of path. A cubic spline was chosen but other degrees are possible. However, choosing a cubic spline provides C 2

continuity, which is desirable in animation.

keyOrientation specifies the orientation at each key value as a set of SFRotation (axis, angle). The algorithm in section 2.7.1.2 is used except for keyType 0 (discrete) where no interpolation is done.

Cumulating specification

offset specifies an offset in position to be added to value_changed (Figure 116). By default no offset is added (offset is (0,0,0)). To cumulate two animations, animation 1 routes its endValue to animation 2’s offset. If animation 2 starts at (0,0,0), the two animations will follow one after the other seamlessly (see example 2.7.1.4.2).


offset

Figure 116 – Left: two animations without offset. Right: two animations chained by an offset.

Outputs

value_changed. orientation_changed, and endValue events are output.:

value_changed is of Eq. 3.

orientation_changed is the frame orientation at of Eq. 3.

endValue is emitted when the end of the animation is reached (i.e. fromTo[1]).

2.7.1.3.1 1D and 2D Animators

For 1D and 2D, the following nodes are defined based on PositionAnimator restricted to these spaces:

PositionAnimator2D { #%NDT = SF2Dnode# inputeventIn SFFloat set_fraction # clock tick

# timeline (same as PositionAnimator)exposedField SFVec2f fromTo 0 1exposedField MFFloat key [] # exposedField SFInt32 keyType []exposedField MFVec2f keySpline [0 0, 1 1 ] # 2 2D-pts

# path characteristicexposedField MFVec2f keyValue [] # exposedField SFInt32 keyOrientation 0exposedField MFFloat weight [] # exposedField SFInt32 keyValueType 0 # line

# cumulexposedField SFVec2f offset 0 0 0

# outputseventOut SFVec2f value_changedeventOut SFFloat rotation_changedeventOut SFVec2f endValue

}

The semantic is the same as for PositionAnimator except that input and output values are 2D positions and keyOrientation has the following semantic.


If keyOrientation is 0, no angle is generated.

If keyOrientation is 1, the object is oriented normal to the curve in the direction of the curve parameter. The angle is output in rotation_changed.

If keyOrientation is 2, the object is oriented normal to the curve in the opposite direction of the curve parameter. The angle is output in rotation_changed.

ScalarAnimator { #%NDT = SF3Dnode,SF2Dnode# inputeventIn SFFloat set_fraction # clock tick

# timeline (same as PositionAnimator)exposedField SFVec2f fromTo 0 1exposedField MFFloat key [] # exposedField SFInt32 keyType []exposedField MFVec2f keySpline [0 0, 1 1 ] # 2 2D-pts

# path characteristicexposedField MFFloat keyValue [] # exposedField MFFloat weight [] # exposedField SFInt32 keyValueType 0 # line

# cumulexposedField SFFloat offset 0 0 0

# outputseventOut SFFloat value_changedeventOut SFFloat endValue

}

Same semantic as for PositionAnimator except input and output key values are scalars (and no orientation is generated).

2.7.1.3.2 Default behavior

If keyType is empty,

If key is empty, linear timeline model is assumed (keyType 1),

If key is non-empty, user-specified timeline model (keyType 0) is assumed and there must be as many keys as values in keyValue field.

In these cases, the default behavior is similar to VRML/BIFS interpolators.

weights need not be specified and is assumed by default.


2.7.1.4 Examples

2.7.1.4.1 Curve paths

This example shows how to define a cubic B-Spline.

DEF TS TimeSensor { stopTime -1 cycleInterval 5 loop TRUE}

# linear animatorDEF PA1 PositionAnimator { fromTo 0.2 0.8 key [ 0 0 0 0 0.25 0.5 0.75 1 1 1 1 ] keyValue [-2 0 0, -3 2 0, 2 2 0, 0 0 0, -1 -1 0, 3 -2 0, 2.5 0 0 ] keyValueType 3 keyType 1}

DEF XF1 Transform { children [ Shape {

appearance Appearance {material Material { diffuseColor 1 0 0 }

}geometry Cone { height 1 bottomRadius 0.5 }

} ]}

ROUTE TS.fraction_changed TO PA1.set_fractionROUTE PA1.value_changed TO XF1.translationROUTE PA1.rotation_changed TO XF1.rotation

Figure 117 – Curve path (cubic B-spline), control points, and a cone moving along.

2.7.1.4.2 Cumulating animations

Two animations are chained together but both use the same timer and output their position and orientation valued to the same node.


PA1 PA2endValue offset

Timer

Transform

fraction_changedset_fractionset_fraction

position

Figure 118 – Schematic of the events while cumulating two Animators.

DEF TIMER TimeSensor { stopTime -1 cycleInterval 5 loop TRUE}

# linear animatorDEF PA1 PositionAnimator { fromTo 0 0.5 key [ 0 0 0 0 0.25 0.5 0.75 1 1 1 1 ] keyValue [-2 0 0, -3 2 0, 2 2 0, 0 0 0, -1 -1 0, 3 -2 0, 2.5 0 0 ] keyValueType 3 keyType 1}

DEF PA2 PositionAnimator { fromTo 0.5 1 key [ 0 0 0 0 1 1 1 1 ] keyValue [ 0 0 0, 1 3 1,3 3 -1, 4 0 2 ] keyValueType 3 keyType 1}

ROUTE PA1.endValue TO PA2.offset # cumul

DEF XF1 Transform { children [ Shape {

appearance Appearance { material Material { diffuseColor 1 0 0 } } geometry Cone { height 1 bottomRadius 0.5 }

} ]}ROUTE TIMER.fraction_changed TO PA1.set_fractionROUTE TIMER.fraction_changed TO PA2.set_fractionROUTE PA1.value_changed TO XF1.translationROUTE PA1.rotation_changed TO XF1.rotationROUTE PA2.value_changed TO XF1.translationROUTE PA2.rotation_changed TO XF1.rotation


Figure 119 –Cumulating two Animators.

2.7.1.5 Bisection algorithm for spline timelines

Given the time t, we wish to determine the parameter u of a cubic Bézier spline. As the curve is strictly monotonically increasing, a simple bisection algorithm is appropriate: The equation of the path is

At t, , we use a bisection algorithm to find [99]:

= 10-4;float alongArc(float t) {

float uL=0, uR=1;float us, ts;

do {us = (uL + uR) / 2;ts = us * (c0 + us * (b0 + us * a0)) + d0;if (t < ts+) uR=us;else uL = us;

} while (t > ts+ || t < ts-)return (us * (c1 + us * (b1 + us * a1)) + d1 )

}2.8 Biomechanical tools

2.8.1 Introduction

Biomechanical models are based on a rigid skeleton made of bones. The skeleton can not be deformed i.e. the bones remains the same as it would be for human. MPEG-4 Body Animation, already part of Version 2 of MPEG-4 standard, defines such a skeleton hierarchy for virtual human models. The AFX specification deals with an extension of FBA by means of accepting the definition of any kind of skeleton (animals, robots and so on). A skeleton maintains geometrical cohesion of its constituents.


Attached to a skeleton, a skin is defined. As a result, when the rigid skeleton is deformed, the skin smoothly follows the movement.

Using inverse kinematics, it is possible to specify only the position of an end effector and the system will find the correct positions of the bones.

2.8.2 Bone-based animation

2.8.2.1 Overview

The animation of articulated figures is quite popular in 3D related applications, especially because of the desire to use human beings as avatars in virtual or hybrid environments. An articulated figure, also called kinematics linkage, consists of a series of rigid links that are connected at joints.

In order to define a static 3D pose of an articulated figure including geometry, color and texture attributes, the functionality addressed consists in considering the entire figure as one single 3D mesh referred to a global seamless mesh. In this case, the bone skeleton is provided together with a field of weighting vectors specifying, for each vertex of the mesh, the related influence of the bones directly altering the 3D position of the vertex. Moreover, the weighting factors can be specified implicitly and more compactly by means of two influence regions defined through a set of cross-sections attached to the bone.

Skin&Bones-based applications deals with animation of any kind of articulated model, preliminary attached to a skeleton and from which the realistic animation is required. Skin&Bones related nodes allow the definition of any kind of skeleton hierarchy, with any kind of attached geometry (based on IndexedFaceSet or higher order geometry nodes like nurbs and subdivision surfaces), and any appearance attributes. The Skin&Bones animation bitstream – simply called Bone-based Animation (BBA) – allows a compact representation of motion parameters.

The seamless mesh-based representation overcomes current limitations of MPEG-4 V2 and is able to provide a realistic figure animation without specifying deformation information. AFX specifications allow also to define seamless parts of an articulated model and to group them together in order to achieve real time animation. Two aspects are addressed by the Skin&Bones framework. The first one deals with the definition of the skinned model as a static model by means of the geometry and the appearance. Also as definition part of the model, a hierarchical skeleton is semantically attached. The second aspect deals with the animation of articulated models, and more specifically with the compressed representation of the animation parameters.

2.8.2.2 Skinned model definition parameters

2.8.2.2.1 Skinned model definition parameters: semantics

Defining a skinned model involves specifying its static attributes as well as its animation behavior. From a geometric point of view, a skinned model has the specificity that the set of vertices which belong to the “skin” of the model is defined in a single list. All the shapes which form the skin share the same list of vertices, thus avoiding seams at the skin level, during animation. To assure the possibility of defining different appearance at different levels of the skinned model, the skin is defined as collection of shapes. For each one it is possible to define its own set of color, texture and material attributes, and each one contains an geometry node field which refers to the skinned model vertices list.

The animation behavior of a skinned model is defined by means of a skeleton and its properties. The skeleton is a hierarchical structure constructed from bones. Three types of information are defined related to a bone:

a) the relative geometrical transformation of the bone in respect with its parent in the skeleton hierarchy;

b) the influence of the bone movement on the surface of the articulated model;

c) inverse kinematics related data specific to the bone.


Each bone has an influence on the skin surface. Thus, changing one bone position or orientation, some vertices from the model skin will be affected by a translation component. Here, defining the skinned model consists in specifying for each bone from the skeleton the set of affected vertices from the skin as well as a measure of affectedness. The influence region can be directly specified by the creator of the model or can be computed before performing animation. In the first case a list of affected vertices is part of the bone definition. In the second case distance-based parameters to describe specific volume influence is attached to the bone definition.

Figure 120 - Skin and bones influence region.

Figure 120 shows an example of a skeleton (in yellow) and the surface mesh representing the skin. Each bone is specified by means of an origin, a geometrical transformation and a length, and has an inner (red; radius r i) and an outer (brown; radius ro) influence regions. In the above example the bone employs 2 cross-sections, one at each end-point, to define the default influence region. Figure 121 shows a detailed view of how the influence regions are constructed. The number and positions of these cross-sections is specified by the designer.

Figure 121 - Bone influence regions.

The surface mesh vertex weights are calculated from the two influence zones as follows:

1. Vertices lying on or inside the inner region have a weight of 1.

2. Vertices on or inside the outer region but outside the inner region have a weight equal to do/(ro-ri) where do is the vertex distance to the outer region. The weight may also have a user specified fall-off applied for one of the following functions: x3, x2, x, sinx, x1/2 and x1/3 (where x is the weight), the default being x.

3. Vertices outside the outer region have a weight of 0.


Consider the case when the bone motion is unconstrained, implying it is able to translate, rotate and scale arbitrarily, a vertex which is affected by the bone will be transformed as follow:

For a skin vertex with , the new position induced by the bone transformation is computed as follow :

1) build matrix from bone translation matrix , bone rotation matrix ,bone scale matrix , bone

scaleOrientation matrix and bone center matrix

= x x x x x x

all the matrices are 4x4;

2) compute a displacement vector by multiplying the matrix with and the scale (the influence of bone j on the vertex i).

= x x

3) compute the new value of by adding

= +

Bone movement is specified by means of a transformation with respect to its initial registration in the skinned model static position. The new position of each vertex is calculated from the influence of each bone which affects the vertex and the bone transformation.

A set of adjacent bones forms a kinematics chain. For kinematics chains with a large number of elements it is more appropriate to animate them using inverse kinematics techniques and not directly specifying the transformation of each element. To support this kind of animation, the skinned model definition is enriched with inverse kinematics related data. Because of rapid evolution of hardware capabilities it is not appropriate to impose a specific inverse kinematics solver. Supporting inverse kinematics is reduced to defining the specific constrains at bone level as constrains related to the rotation on a direction or constrains related to the translation of a bone on a direction. Since a bone is just a semantic entity, the end-effector is usually defined as a 3D point on the skinned model surface.


In many cases, especially for complicated skinned models, it is possible to identify regions on the skin which are influenced by a small subset of the skeleton bones. In such a cases, to address optimized animation, the model has to be considered as a collection of skinned models. Thus, the bones from one skinned model do not deform the skin of others. With this optimization in mind, the definition of a skinned model within AFX permits addition of aother 3D objects at any level of the skeleton, including other skinned models.

The bones which compose the skeleton are rigid objects and while their transformation is possible, deformation is not. To address a realistic skinned model animation the “muscle” concept is employed. Mainly, a “muscle” is a curve with an influence region on the model skin and which can be deformed. To assure a generic representation of the “muscle” form, a curve representation based on NURBS is employed.

2.8.2.2.2 Skinned model definition parameters :node specification

Within a seamless-mesh-based representation a descriptive structure is defined for skeleton (through a SBBone node) and skin (through a SBSkinnedModel node). Moreover, it is possible to define sub-skeletons or to add in the skeleton hierarchy any kind of standalone 3d object by means of SBSegment. To address inverse kinematics or to define semantic points on the model surface the SBSite node is also defined. Finally, to allow local deformation effects based on curve deformation, the SBMuscle node is defined. From a model definition point of view, there are some common aspects between the AFX Skin&Bones related nodes and the H-Anim specifications [12]. The main distinguishing feature is that while H-Anim concentrates on humanoid animation, AFX can be used to animate any articulated model.

SBBone{exposedField SFInt32 boneID 0exposedField MFInt32 skinCoordIndex [ ]exposedField MFFloat skinCoordWeight [ ]exposedField SFVec3f endpoint 0 0 1exposedField SFInt32 falloff 1exposedField MFFloat sectionPosition [ ]exposedField MFFloat sectionInner [ ]exposedField MFFloat sectionOuter [ ]exposedField SFInt32 rotationOrder 0exposedField MFNode children [ ]exposedField SFVec3f center 0 0 0exposedField SFRotation rotation 0 0 1 0exposedField SFVec3f translation 0 0 0exposedField SFVec3f scale 0 0 0exposedField SFRotation scaleOrientation 0 0 1 0exposedField SFInt32 IkchainPosition 0exposedField MFFloat IkyawLimit [ ]exposedField MFFloat IkpitchLimit [ ]exposedField MFFloat IkrollLimit [ ]exposedField MFFloat IKTxLimit [ ]exposedField MFFloat IKTyLimit [ ]exposedField MFFloat IKTzLimit [ ]

}

SBBone node specifies data related to one bone from the skeleton.

The boneID field is a unique identifier which allows that the bone to be addressed at animation run-time.

The center field specifies a translation offset from the origin of the local coordinate system.


The translation field specifies a translation to the bone coordinate system.

The rotation field specifies a rotation of the bone coordinate system.

The scale field specifies a non-uniform scale of the bone coordinate system. scale values shall be greater than zero.

The scaleOrientation specifies a rotation of the bone coordinate system before the scale (to specify scales in arbitrary orientations). The scaleOrientation applies only to the scale operation.

The possible geometric 3D transformation consists of (in order): 1) (possibly) non-uniform scale about an arbitrary point, 2) a rotation about an arbitrary point and axis and 3) a translation.

The rotationOrder field specifies the rotation order when deals with the decomposition of the rotation in respect with system coordinate axes.

Two ways of specifying the influence region of the bone are allowed:

1) The skinCoordIndex field contains a list of indices of all skin vertices affected by the current bone. Mostly, the skin influence region of bone will contain vertices from the 3D neighborhood of the bone, but special cases of influence are also accepted.

The skinCoordWeight field contains a list of weights (one per vertex listed in skinCoordIndex) that measures the contribution of the current bone to the vertex in question. The length skinCoordIndex is equal with the length of skinCoordWeight. The sum of all skinCoordWeight related to the same vertex must be 1.

2) The endpoint field specifies the bone 3D end point and is used to compute the bone length.

The sectionInner field is a list of inner influence region radii for different sections.

The sectionOuter field is a list of outer influence region radii for different sections.

The sectionPosition field is a list of positions of all the sections defined by the designer.

The falloff field specifies the function between the amplitude affectedness and distance: -1 for x3, 0 for x2, 1 for x, 2 for sinx, 3 for x1/2 and 4 for x1/3.

The two schemes can be used independently or in combination, in which case the individual vertex weights take precedence.

The IKchainPosition field specifies the position of the bone in the kinematics chain. If the bone is the root of the IK chain then IKchainPosition=1. In this case, when applying IK scheme, only the orientation of the bone is changed. If the bone is last in the kinematics chain IKchainPosition=2. In this case, the animation stream has to include the desired position of the bone (X, Y and Z world coordinates). If IKchainPosition=3 the bone belongs to the IK chain but is not the first or the last one in the chain. In this case, position and orientation of the bone are computed by the IK procedure. Finally, if the bone does not belong to any IK chain ( IKchainPosition=0), it is necessary to transmit the bone local transformation in order to animate the bone. If an animation stream contains motion information about a bone which has IKchainPosition 1, this information will be ignored. If an animation stream contains motion information about a bone which has IKchainPosition 3, this means that the animation producer wants to ensure the orientation of the bone and the IK solver will use this value as a constrain.

The IkyawLimit field consists in a pair of min/max values which limit the bone rotation with respect to the X axis.


The IkpitchLimit field consists in a pair of min/max values which limit the bone rotation with respect to the Y axis.

The IkrollLimit field consists in a pair of min/max values which limit the bone rotation with respect to the Z axis.

The IKTxLimit field consists in a pair of min/max values which limit the bone translation in the X direction.

The IKTyLimit field consists in a pair of min/max values which limit the bone translation in the Y direction.

The IKTzLimit field consists in a pair of min/max values which limit the bone translation in the Z direction.

The SBBone node is used as a building block to describe the hierarchy of the articulated model by attaching one or more child objects. The children field has the same semantic as used in BIFS [1]; the absolute geometric transformation of any child of a bone is obtained through a composition with the bone-parent transformation.

SBSegment{eventIn MFNode addChildren eventIn MFNode removeChildren exposedField SFString name “”exposedField SFVec3f centerOfMass 0 0 0exposedField SFVec3f momentsOfInertia [ 0 0 0 0 0 0 0 0 0 ]exposedField SFFloat mass 0exposedField MFNode children [ ]

}

The name field must be present, so that the SBSegment can be identified at runtime. Each SBSegment should have a DEF name that matches the name field for that Segment, but with a distinguishing prefix in front of it.

The mass field is the total mass of the segment.

The centerOfMass field is the location within the segment of its center of mass. Note that a zero value was chosen for the mass in order to give a clear indication that no mass value is available.

The momentsOfInertia field contains the moment of inertia matrix. The first three elements are the first row of the 3x3 matrix, the next three elements are the second row, and the final three elements are the third row.

The children field can be any object attached at this level of the skeleton, including a SBSkinnedModel.

An SBSegment node is a grouping node especially introduced to address two issues:

1) The first one is to the requirement to separate different parts from the skinned model into deformation-independent parts. Between two deformation-independent parts the geometrical transformation of one of them do not imply skin deformations on the other. This is essential for run-time animation optimization. The SBSegment node may contain as a child an SBSkinnedModel node (see the SBSkinnedModel node description below). Portions of the model which are not part of the seamless mesh can be attached to the skeleton hierarchy by using an SBSegment node;

2) The second deals with the requirement to attach standalone 3D objects at different parts of the skeleton hierarchy. For example, a ring can be attached to a finger; the ring geometry and attributes are defined outside of skinned model but the ring will have the same local geometrical transformation as the attached bone.


SBSite {eventIn MFNode addChildren eventIn MFNode removeChildren exposedField SFVec3f center 0 0 0 exposedField MFNode children [ ] exposedField SFString name "" exposedField SFRotation rotation 0 0 1 0 exposedField SFVec3f scale 1 1 1 exposedField SFRotation scaleOrientation 0 0 1 0 exposedField SFVec3f translation 0 0 0

}

The center field specifies a translation offset and can be used to compute a bone length. The rotation field specifies a rotation of the coordinate system of the SBSite node.

The scale field specifies a non-uniform scale of the SBSite node coordinate system and the scale values must be greater than zero.

The scaleOrientation specifies a rotation of the coordinate system of the SBSite node before the scale thus allowing a scale at an arbitrary orientation. The scaleOrientation applies only to the scale operation.

The translation field specifies a translation of the coordinate system of the SBSite node.

The children field is used to store any object that can be attached to the SBSegment node.

The SBSite node can be used for three purposes. The first is to define an "end effector", i.e. a location which can be used by an inverse kinematics system. The second is to define an attachment point for accessories such as clothing. The third is to define a location for a virtual camera in the reference frame of a SBSegment node.

SBSite nodes are stored within the children field of an SBSegment node. The SBSite node is a specialized grouping node that defines a coordinate system for nodes in its children field that is relative to the coordinate systems of its parent node. The reason a SBSite node is considered a specialized grouping node is that it can only be defined as a child of a SBSegment node.

SBMuscle{exposedField MFInt32 skinCoordIndex [ ]exposedField MFFloat skinCoordWeight [ ]exposedField SFNode muscleCurve NULLexposedField SFInt32 radius 1exposedField SFInt32 falloff 1

}

The skinCoordIndex field consists of a list of vertex indices from the skinned model skin which are affected by the “muscle”

The skinCoordWeight field consists of a list of weights indicating in what measure a vertex in affected by the “muscle”

The muscleCurve field is a NurbCurve as defined in section 2.3.1.2.

The radius field specifies the maximum distance where the “muscle” will affect the skin


The falloff field specifies the function between the amplitude affectedness and distance: -1 for x3, 0 for x2, 1 for x, 2 for sin(x), 3 for x1/2 and 4 for x1/3

Performing deformation consists in affecting the form of the muscleCurve by 1) affecting the position of the control points of the curve, 2) affecting the weight of control points or/and 3) affecting the knot sequence.. Depending on the author, the animation stream can contain one animation mechanism or a combination of 1), 2) and 3).

At the modeling stage, each affected vertex iv from the skin is assigned a point civ from the curve, as the closest

point. During animation, the translation of civ obtained from the update values of controlPoint, weight or/and

knot fields, will induce a translation on iv :

1) skinCoordWeight field is specified for vertex iv , then:

cii *TvkeightskinCoordWTv ,

where k is the index of vertex iv in the model vertices index list;

3) radius field is specified, then c

i

cii

i *Tvradius

vvdfTv

),(

with f specified by the falloff field.

SBSkinnedModel{exposedField SFString name "" exposedField SFVec3f center 0 0 0 exposedField SFRotation rotation 0 0 1 0 exposedField SFVec3f translation 0 0 0 exposedField SFVec3f scale 1 1 1 exposedField SFRotation scaleOrientation 0 0 1 0 exposedField MFNode skin [ ] exposedField SFNode skinCoord NULL exposedField SFNode skinNormal NULL exposedField MFNode skeleton [ ] exposedField MFNode bones [ ] exposedField MFNode muscles [ ] exposedField MFNode segments [ ] exposedField MFNode sites [ ] exposedField SFNode weighsComputationSkinCoord NULL

}

The SBSkinnedModel node is the top of the hierarchy of Skin&Bones related nodes and contains the definition parameters for the entire seamless model or of a seamless part of the model.

The name field specify the name of the skinned model allowing easily identification at the animation run-time.


The center field specifies a translation offset from the origin of the local coordinate system.

The translation field specifies a translation of the coordinate system.

The rotation field specifies a rotation of the coordinate system

The scale field specifies a non-uniform scale of the coordinate system. scale values shall be greater than zero.

The scaleOrientation specifies a rotation of the coordinate system before the scale (to specify scales in arbitrary orientations). The scaleOrientation applies only to the scale operation.

The skinCoord contains the 3d coordinates of all vertices of the seamless model.

The skin consists of a collection of shapes that share the same skinCoord. This mechanism allows considering the model as a continuous mesh and, in the same time, to attach different attributes (like color, texture) to different parts of the model.

The skeleton field specifies the root of the bones hierarchy.

The bones fields consist in the lists of all bones previously defined as SBBone node.

The segments fields consist in the lists of all bones previously defined as SBSegment node.

The sites fields consist in the lists of all bones previously defined as SBSites node. The muscles fields consist in the lists of all bones previously defined as SBMuscle node.

The weighsComputationSkinCoord field describes a specific static position of the skinned model. In many cases the static position of the articulated model defined by skinCoord and skin fields is not appropriate to compute the influence region of a bone. In this case the weighsComputationSkinCoord field allows specifying the skinned model vertices in a more appropriate static posture. This posture will be used just during the initialization stage and ignored during the animation. All the skeleton transformations are related to the posture defined by skinCoord field.

2.8.2.3 Skinned model animation parameters

2.8.2.3.1 Animation principle

Animating a 3D articulated model requires knowledge of the position of each model vertex at each key-frame. Specifying such a data is an enormous and expensive task. For this reason the AFX animation system employs the bone-based modeling of articulated models which effectively attaches the model vertices to a bone hierarchy (skeleton).. This technique avoids the necessity to specify the position for each vertex, only the local transformation of each bone in the skeleton being animated. The local bone transformation components (translation, rotation, scale, scaleOrientation and center) are specified at each frame and, at the vertex level, the transformation is obtained by using the bone-vertex influence region.

To address streamed animation, the animation data is considered separately (independent of the model definition) and it will be specified for each key-frame.

Animating a skinned model is achieved through updates of the geometric transformation component of the skeleton by transforming the bones and/or to the muscle curve form.

A general transformation of a bone, as defined by the SBBone node, involves: translation in any direction, rotation with respect to any rotation axis, and scaling with respect to any direction and axis. In the SBBone node definition the rotation field is defined as a SFRotation. In order to change the orientation of a bone the rotation field must by updated. This representation allows a direct and bijective mapping with a rotation matrix or quaternion representations, highly used by 3D animation engines.


However, many motion editing systems use the decomposition of an orientation in Euler angles. Also in many cases of real objects, motion is more present on one or two directions because of limitations at the joint level. In such a cases, an Euler angle based representation is more appropriate.

To assure the bijectivity [91] of the transformation between the Euler angles notation and rotation matrix or quaternion representation the rotationOrder field has been introduced in SBBone node. A triplet of Euler angles

describes how a coordinate frame r rotates with respect to a static frame s, here, how a bone frame

rotates with respect to its parent frame. The triplet is interpreted as a rotation by around an axis , then a rotation by around an axis , and finally a rotation by around an axis , with different from both and

. The axes are restricted to the coordinate axes, , , and , giving 12 possibilities: , , , , , , , , , , , [32]. By considering the axis either in the bone frame (r) or its

parent frame (s), there are 24 possible values for rotationOrder.

The bone-base animation (BBA) of a skinned model is performed by frame update of the SBBone transformation fields (translation, rotation, scale, center and scaleOrientation) and/or by updating the SBMuscle curve control points position, weight or knot sequence. The BBA stream contains all the animation frames or just the data at the temporal key frames. In the last case the decoder will compute the intermediate frames by temporal interpolation. Linear interpolation is used for translation and scale components and linear quaternion interpolation is used for rotation and scaleOrientation components.

Each key-frame contains:

1) a natural integer KeyFrameIndex which indicates to the decoder the number of frames that have to be obtained by interpolation. If zero, the decoder interprets the received frame as a normal frame and sends it to the animation engine; if not the decoder computes KeyFrameIndex intermediate frames and sends them to the animation engine, as well as the content of the received key-frame.

2) the boneID as well as the animation mask for each animated bone. See below for the description of SBBone animation mask.

3) the muscleID as well as the animation mask for each animated muscle. See below for the description of SBMuscle animation mask.

4) the new values for each bone transformation component which need updating.

5) the new values for each muscle control point which have been translated.

Data from points 1), 2) and 3) represent the frame animation mask and data from 4) and 5) represent the frame animation values.

A BBA stream can contain the information related to a maximum number of 1024 SBBone and 1024 SBMuscle nodes belonging to one or more skinned models. The identifiers fields boneID and muscleID must be unique in the scene graph and must be in the range [0 …1023].

2.8.2.3.1.1 SBBone animation mask and animation values

To address high compression efficiency issues the bone transformation is represented inside an animation bitstream by each component:


Node representation Bitstream representation

SFVec3f translation int Tx, Ty, Yz

SFRotation rotation int RotAngleOnAxis1, RotAngleOnAxis2, RotAngleOnAxis3

SFVec3f scale int Sx, Sy, Sz

SFRotation scaleOrientation int Ax1, Ax2, Ax3, RotVal

SFVec3f center int Cx, Cy, Cz

The animation mask for one bone is a binary vector with the following syntax:

if (Is_Translation_changed=GetOneBit()){

Is_Translation_onX_changed= GetOneBit();

Is_Translation_onY_changed= GetOneBit();

Is_Translation_onZ_changed= GetOneBit();

}

if (Is_Rotation_changed=GetOneBit()){

Is_Rotation_onAxis1_changed= GetOneBit();



}

if (Is_Scale_changed=GetOneBit()){

Is_Scale_onX_changed= GetOneBit();

Is_Scale_onY_changed= GetOneBit();

Is_Scale_onZ_changed= GetOneBit();

}

if (Is_ScaleOrientation_changed=GetOneBit()){

Is_ScaleOrientation_XAxis_changed= GetOneBit();

Is_ScaleOrientation_YAxis_changed= GetOneBit();

Is_ScaleOrientation_Zaxis_changed= GetOneBit();


Is_ScaleOrientation_AngleValue_changed= GetOneBit();

}

if (Is_Center_changed=GetOneBit()){

Is_Center_onX_changed= GetOneBit();

Is_Center_onY_changed= GetOneBit();

Is_Center_onZ_changed= GetOneBit();

}

In this way, the size of animation mask for a bone can vary from 2 (two) bits, if there is motion on one single component, to 21 bits if all the components of local transformation change from the previous keyframe. Considering the usual case when dealing with a complex rotation (on three directions) the animation mask will have the form:

0 1 1 1 1 0 0 0

isTranslation isRotation

isRotation_onAxis

1

isRotation_onAxis

2

isRotation_onAxis

3

IsScale

IsScaleOrientation IsCenter

The animation values for a bone is a vector which contains the new values for all affected components. For the above example of complex rotation the animation value vector is:

[rotOnAxis1, rotOnAxis2, rotOnAxis3]

2.8.2.3.1.2 SBMuscle animation mask and animation values

An SBMuscle node can be animated by updating the control points values of the NURB curve, the points weight or/and the knot sequence.

For one animated SBMuscle, the animation mask is obtained as follows:

if (Is_controlPoint_field_changed=GetOneBit()){

for (i=0;i<controlPoint.length;i++)

Is_controlPoint_i_onX_changed= GetOneBit();

Is_controlPoint_i_onY_changed= GetOneBit();

Is_controlPoint_i_onZ_changed= GetOneBit();

}


if (Is_weight_field_changed=GetOneBit()){

for (i=0;i<weight.length;i++)

Is_weight_i_changed= GetOneBit();

}

if (Is_knot_field_changed=GetOneBit()){

for (i=0;i<knot.length;i++)

Is_knot_i_changed= GetOneBit();

}

with controlPoint.length= weight.length;

The number of control points and the number of elements from the knot sequence are integer numbers between 0 and 63 and will be encoded for each frame after the muscleID.

The animation values for a muscle consist of a vector witch contains the new values of all changed data. The order into the vector is obtain in correspondence to the animation mask.

2.8.2.3.1.3 Frame mask and frame values representation

A global animation frame allows animation of a subset of SBBone nodes or/and a subset of SBMuscle nodes from the scene graph and contains an animation mask field and an animation values field. The frame animation mask and the frame animation values are obtained through concatenation of the bones and muscles animation masks and animation values respectively.

Frame animation mask:

KFI NSBB NSBM IDB BM … IDB BM IDM NCP

NK MM … IDM NCP

NK MM

With:

KFI – key frame index – integer from 0 to 31;

NSBB – number of animated bones in current frame – integer from 0 to 1023;

NSBM – number of animated muscles in current frame integer from 0 to 1023;

IDB – boneID – integer from 0 to 1023;

BM – bone mask –binary vector from 2 to 21 bits as described in section ;


IDM – muscleID – integer from 0 to 1023

NCP – the number of control points for the current IDM – integer from 0 to 63

NK – the number of elements in the knot sequence for the current IDM – integer from 0 to 63;

MM – muscle mask – binary vector as defined in section 2.8.2.3.1.2

A frame animation value contains the data which is changed in the current frame for all the bones and muscles from the animation mask.. When data is a float (e.g. a translation of a control point), a conversion by 10e5 * realfloatValue is computed.

2.8.2.3.2 Animation stream syntax

A BBA object is formed by a temporal sequence of BBA object planes as depicted below:

BBA object:

--------------------- BBA ObjectPlane n

BBA ObjectPlane 2

BBA ObjectPlane 1

Alternatively, an BBA object can be formed by a temporal sequence of BBA object plane groups (called segments for simplicity), where each BBA object plane group itself is composed of a temporal sequence of 16 BBA object planes, as depicted in the following:

BBA object:

BBA ObjectPlane Group n

BBA ObjectPlane Group 2

BBA ObjectPlane Group 1

---------------------

BBA object plane group:

BBA ObjectPlane 16

BBA ObjectPlane 2

BBA ObjectPlane 1

--------------------

When the alternative BBA object bitstream structure is employed, the bitstream is decoded by DCT-based BBA object decoding in the same manner as for the FBA stream [2]. Otherwise, the bitstream is decoded by the frame-based BBA object decoding.

The BBA stream, is an extended form of the FBA stream [2]. A BBA Object plane contains the update values for Skin&Bones components (SBC) which can be translations in one direction, rotation angles, rotation axes, scale factors in one direction and weights factors.


2.8.2.3.2.1 bba_object

bba_object() { No. of bits Mnemonic/ Section

bba_object_start_code 32 bslbfdo {

bba_object_plane() 2.8.2.3.2.2} while (!((nextbits_bytealigned() == ‘000 0000 0000 0000

0000 0000’) &&( nextbits_bytealigned() != bba_object_plane_start_code)))

}2.8.2.3.2.2 bba object plane

bba_object_plane() { No. of bits Sectionbba_object_plane_header() 2.8.2.3.2.3bba_object_plane_data() 2.8.2.3.2.7

}2.8.2.3.2.3 bba_object_plane_header

bba_object_plane_header() { No. of bits Mnemonic/ Section

if (nextbits_bytealigned()==‘0000 0000 0000 0000 0000 0001’){

next_start_code()bba_object_plane_start_code 32 bslbf

}is_intra 1 bslbftemporal_header() 2.8.2.3.2.4

}2.8.2.3.2.4 temporal_header

temporal_header() { No. of bits Mnemonic/ Section

if (is_intra) {is_frame_rate 1 bslbfif(is_frame_rate)

decode_frame_rate() 2.8.2.3.2.5is_time_code 1 bslbfif (is_time_code)

time_code 18 bslbf}skip_frames 1 bslbfif(skip_frames)

decode_skip_frames() 2.8.2.3.2.6}

2.8.2.3.2.5 decode_frame_rate

decode_frame_rate(){ No. of bits Mnemonicframe_rate 8 uimsbfseconds 4 uimsbffrequency_offset 1 uimsbf

}


2.8.2.3.2.6 decode_skip_frames

decode_skip_frames(){ No. of bits Mnemonicdo{

number_of_frames_to_skip 4 uimsbf } while (number_of_frames_to_skip = “1111”)

}2.8.2.3.2.7 bba_object_plane_data

bba_object_plane_data() { No. of bits Section

bba_object_plane_mask() 2.8.2.3.2.8

bba_object_plane_values() 2.8.2.3.2.11

}

2.8.2.3.2.8 bba_object_plane_mask

bba_object_plane_mask() { No. of bits Mnemonic/ Section

KeyFrameIndex(KFI) 5 uimsbf

if(is_intra) {


bba_quant 5 uimsbf

NumberOfBones (NSBB) 10 uimsbf

NumberOfMuscles (NSBM) 10 uimsbf

for (bone = 1; bone<= NSBB; bone++) {

BoneIdentifier (IDB) 10 uimsbf

Bone_mask() 2.8.2.3.2.9

}

for (muscle = 1; muscle<= NSBB; muscle++) {

MuscleIdentifier (IDM) 10 uimsbf

NumberControlPoints (NCP) 6 uimsbf

NumberKnots (NK) 6 uimsbf

Muscle_mask() 2.8.2.3.2.10

}

}

2.8.2.3.2.9 Bone_mask

Bone_mask(){ No. of bits Mnemonic

IsTranslation_changed 1 bslbf

marker_bit 1 uimsbf

if (isTranslation){

isTranslationOnX_changed 1 bslbf

IsTranslationOnY_changed 1 bslbf

isTranslationOnZ_changed 1 bslbf


}

isRotation_changed 1 bslbf

marker_bit 1 uimsbf

if (isRotation){

IsRotationOnAxis1_changed 1 bslbf



}

isScale_changed 1 bslbf

marker_bit 1 uimsbf

if (isScale){

IsScaleOnX_changed 1 bslbf

IsScaleOnY_changed 1 bslbf

IsScaleOnZ_changed 1 bslbf

}

isScaleOrientation_changed 1 bslbf

marker_bit 1 uimsbf

if (isScaleOrientation){

IsScaleOrientationAxisX_changed 1 bslbf

IsScaleOrientationAxisY_changed 1 bslbf

IsScaleOrientationAxisZ_changed 1 bslbf

IsScaleOrientationValue_changed 1 bslbf

}

isCenter_changed 1 bslbf

marker_bit 1 uimsbf

if (isCenter){

IsCenterOnX_changed 1 bslbf

IsCenterOnY_changed 1 bslbf

IsCenterOnZ_changed 1 bslbf


}

2.8.2.3.2.10 Muscle_mask

Muscle_mask() { No. of bits Mnemonic

IsControlPoints_changed 1 bslbf

marker_bit 1 uimsbf

if (IsControlPoints_changed)


for (cp=1 ;cp<= NCP ;cp++){

marker_bit 1 uimsbf

IsControlPoints[cp]_onX_changed 1 bslbf

IsControlPoints[cp]_onY_changed 1 bslbf

IsControlPoints[cp]_onZ_changed 1 bslbf

}

IsWheighOfControlPoints_changed 1 bslbf

marker_bit 1 uimsbf

if (IsWheighOfControlPoints _changed)

for (cp=1 ;cp<= NCP ;cp++){

marker_bit 1 uimsbf

IsWheighOfControlPoints[cp]_changed 1 bslbf

}

IsKnot_changed 1 bslbf

marker_bit 1 uimsbf

if (IsKnot _changed)

for (k=1 ;k<= NK ;k++){

marker_bit 1 uimsbf

IsKnot [k]_changed 1 bslbf

}


2.8.2.3.2.11 bba_object_plane_values

bba_object_plane_values() { No. of bits Mnemonic/ Section

if(is_intra) { bba_object_coding_type 1 bslbfif (bba_object_coding_type == 0) {

bba_is_i_new_max 1 bslbfbba_is_i_new_min 1 bslbfbba_is_p_new_max 1 bslbfbba_is_p_new_min 1 bslbfdecode_bba_new_minmax() 2.8.2.3.2.12decode_bba_i_frame() 2.8.2.3.2.13

}else decode_bba_i_segment() 2.8.2.3.2.15

} NA.1.1.1.1.1 NA.1.1.1.1.2else{ NA.1.1.1.1.3 NA.1.1.1.1.4

if (bba_object_coding_type == 0) NA.1.1.1.1.5 NA.1.1.1.1.6decode_bba_p_frame() 2.8.2.3.2.14

else decode_bba_p_segment() 2.8.2.3.2.16

}2.8.2.3.2.12 decode_bba_new_minmax

decode_bba_new_minmax(){ No. of bits Mnemonicif (bba_is_i_new_max)

for (one_sbc=1; one_sbc <= NUM_SBCs; one_sbc ++) {marker_bit 1 uimsbfif (sbc_mask[one_sbc])

bba_i_new_max[one_sbc] 5 uimsbf}

if (bba_is_i_new_min) for (one_sbc =1; one_sbc<= NUM_SBCs; one_sbc ++) {

marker_bit 1 uimsbfif (sbc_mask[one_sbc])

bba_i_new_min[one_sbc] 5 uimsbf}

if (bba_is_p_new_max) for (one_sbc =1; one_sbc<= NUM_SBCs; one_sbc++) {

marker_bit 1 uimsbfif (sbc_mask[one_sbc])

bba_p_new_max[one_sbc] 5 uimsbf}if (bba_is_p_new_max)

for (one_sbc =1; one_sbc<= NUM_SBCs;one_sbc ++) {marker_bit 1 uimsbfif (sbc_mask[one_sbc])

bba_p_new_max[one_sbc] 5 uimsbf}

}


2.8.2.3.2.13 decode_bba_i_frame

decode_bba_i_frame(){ No. of bits Sectionfor (one_sbc =1; one_sbc <= NUM_SBCs; one_sbc ++)

if (sbc_mask[one_sbc])aa_decode(isbc_Q[one_sbc],isbc_cum_freq[one_sbc

])See [1]

}

2.8.2.3.2.14 decode_bba_p_frame

decode_bba_p_frame(){ No. of bits Section

for (one_sbc =1; one_sbc <= NUM_SBCs; one_sbc ++)

if (sbc_mask[one_sbc])

aa_decode(psbc_diff[one_sbc],psbc_cum_freq[one_sbc])

See [1]

}

2.8.2.3.2.15 decode_bba_i_segment

decode_bba_i_segment(){ No. of bits Sectionfor (one_sbc =1; one_sbc <= NUM_SBCs; one_sbc ++)

if (sbc_mask [one_sbc]){decode_i_dc(dc_Q[one_sbc]) 2.8.2.3.2.17decode_ac(ac_Q[one_sbc]) 2.8.2.3.2.19

}}

2.8.2.3.2.16 decode_bba_p_segment

decode_bba_p_segment(){ No. of bits Sectionfor (one_sbc =1; one_sbc <= NUM_SBCs; one_sbc ++)

if (sbc_mask[one_sbc]){decode_p_dc(dc_Q[one_sbc]) 2.8.2.3.2.18decode_ac(ac_Q[one_sbc]) 2.8.2.3.2.19

}}

2.8.2.3.2.17 decode_i_dc

decode_i_dc(dc_q) { No. of bits Mnemonicdc_q 16 simsbfif(dc_q == -256*128)

dc_q 31 simsbf}

2.8.2.3.2.18 decode_p_dc

decode_p_dc(dc_q_diff) { No. of bits Mnemonicdc_q_diff vlclbfdc_q_diff = dc_q_diff- 256


if(dc_q_diff == -256)dc_q_diff 16 simsbfif(dc_Q == 0-256*128)

dc_q_diff 32 simsbf}

2.8.2.3.2.19 decode_ac

decode_ac(ac_Q[i]) { No. of bits Mnemonicthis = 0next = 0while(next < 15) {

count_of_runs vlclbfif (count_of_runs == 15)

next = 16else {

next = this+1+count_of_runsfor (n=this+1; n<next; n++)

ac_q[i][n] = 0 ac_q[i][next] vlclbfif( ac_q[i][next] == 256)

decode_i_dc(ac_q[i][next])else

ac_q[i][next] = ac_q[i][next]-256 this = next

}}

}

2.8.2.3.3 BBA decoder semantics

BBA object

bba_object_start_code

the bit string ‘00000***’ in hexadecimal; initiates a BBA object

BBA object plane header

bba_object_plane_start_code

the bit string ‘00000***’ in hexadecimal; it initiates a BBA object plane.

is_intra 1-bit flag which when set to ‘1’ indicates that the BBA object is coded in intra mode. When set to ‘0’ it indicates that the BBA object is coded in predictive mode

Temporal Header

is_frame_rate 1-bit flag which when set to ‘1’ indicates that frame rate information follows this bit field. When set to ‘0’ no frame rate information follows this bit field

is_time_code 1-bit flag which when set to ‘1’ indicates that time code information follows this bit field. When set to ‘0’ no time code information follows this bit field

time_code 18-bit integer containing the following: time_code_hours, time_code_minutes, marker_bit and time_code_seconds as shown in Error: Reference source not found. The


parameters correspond to those defined in the IEC standard publication 461 for “time and control codes for video tape recorders”. The time code specifies the modulo part (i.e. the full second units) of the time base for the current object plane

skip_frames 1-bit flag which when set to ‘1’ indicates that information follows this bit field that indicates the number of skipped frames. When set to ‘0’ no such information follows this bit field

time_code range of value No. of bits Mnemonic

time_code_hours 0 - 23 5 uimsbf

time_code_minutes 0 - 59 6 uimsbf

marker_bit 1 1 bslbf

time_code_seconds 0 - 59 6 uimsbf

Decode_frame_rate

frame_rate 8 bit unsigned integer indicating the reference frame rate of the sequence

seconds 4 bit unsigned integer indicating the fractional reference frame rate. The frame rate is computed as follows frame rate = (frame_rate + seconds/16).

frequency_offset

1-bit flag which when set to ‘1’ indicates that the frame rate uses the NTSC frequency offset of 1000/1001. This bit would typically be set when frame_rate = 24, 30 or 60, in which case the resulting frame rate would be 23.97, 29.94 or 59.97 respectively. When set to ‘0’ no frequency offset is present. I.e. if (frequency_offset ==1) frame rate = (1000/1001) * (frame_rate + seconds/16).

Decode_skip_frame

number_of_frames_to_skip

4-bit unsigned integer indicating the number of frames skipped. If the number_of_frames_to skip is equal to 15 (pattern “1111”) then another 4-bit word follows allowing to skip up to 29 frames(pattern “11111110”). If the 8-bits pattern equals “11111111”, then another 4-bits word will follow and so on, and the number of frames skipped is incremented by 30. Each 4-bit pattern of ‘1111’ increments the total number of frames to skip with 15


bba_object_plane_mask

KeyFrameIndex( KFI) 5-bit unsigned integer indicating the number of frames that have to be interpolated between the current frame and the received frame. If 0 (zero) the decoder pass the decoded frame to the animation engine.

bba_quant a 5-bit unsigned integer used as the index to a bba_scale table for computing the quantisation step size of SBC values for predictive and DCT coding. If bba_object_coding_type is predictive, the value of bba_scale is specified in the following list:bba_scale [0-31]= { 0, 1, 2, 3, 5, 7, 9, 11, 14, 17, 20, 23, 27, 31, 35, 39, 43, 47, 52, 57, 62, 67, 72, 77, 82, 88, 94, 100, 106, 113, 120, 127};If the bba_object_coding_type is DCT this is a 5-bit unsigned integer used as the index to a bba_scale table for computing the quantisation step size of DCT coefficients. The value of bba_scale is specified in the following list:bba_scale [0 – 31] = { 1, 1, 2, 3, 5, 7, 8, 10, 12, 15, 18, 21, 25, 30, 35, 42, 50, 60, 72, 87, 105, 128, 156, 191, 234, 288, 355, 439, 543, 674, 836, 1039}

NumberOfBones (NSBB) a 10-bit unsigned integer indicating the number of bones animated by the current frame

NumberOfMuscles (NSBM) a 10-bit unsigned integer indicating the number of muscles animated by the current frame

BoneIdentifier (IDB) a 10-bit unsigned integer indicating a bone identifier; must correspond with a boneID field from a SBBone node from the scene graph.

MuscleIdentifier (IDM) a 10-bit unsigned integer indicating a muscle identifier; must correspond with a muscleID field from a SBMuscle node from the scene graph.

NumberControlPoints (NCP) A 6-bit unsigned integer indicating the number of control points of the muscle curve

NumberKnots (NK) A 6-bit unsigned integer indicating the number of elements in the knot sequence of muscle curve

Bone_mask

IsTranslation_changed 1 bit flag indicating, for the current bone, if at least one of the translation components is affected in the current frame

isTranslationOnX_changed 1 bit flag indicating, for the current bone, if the translation on X axis is affected in the current frame

IsTranslationOnY_changed 1 bit flag indicating, for the current bone, if the translation on Y axis is affected in the current frame

isTranslationOnZ_changed 1 bit flag indicating, for the current bone, if the translation on Z axis is affected in the current frame


isRotation_changed 1 bit flag indicating, for the current bone, if at least one of the rotation components is affected in the current frame

IsRotationOnAxis1_changed 1 bit flag indicating, for the current bone, if the rotation in respect with the first axis is affected in the current frame

IsRotationOnAxis2_changed 1 bit flag indicating, for the current bone, if the rotation in respect with the second axis is affected in the current frame

IsRotationOnAxis3_changed 1 bit flag indicating, for the current bone, if the rotation in respect with the third axis is affected in the current frame

isScale_changed 1 bit flag indicating, for the current bone, if at least one of the scale components is affected in the current frame

IsScaleOnX_changed 1 bit flag indicating, for the current bone, if the scale on X axis is affected in the current frame

IsScaleOnY_changed 1 bit flag indicating, for the current bone, if the scale on Y axis is affected in the current frame

IsScaleOnZ_changed 1 bit flag indicating, for the current bone, if the scale on Z axis is affected in the current frame

isScaleOrientation_changed 1 bit flag indicating, for the current bone, if at least one of the scaleOrientation components is affected in the current frame

IsScaleOrientationAxisX_changed 1 bit flag indicating, for the current bone, if the scaleOrientation on X axis is affected in the current frame

IsScaleOrientationAxisY_changed 1 bit flag indicating, for the current bone, if the scaleOrientation on Y axis is affected in the current frame

IsScaleOrientationAxisZ_changed 1 bit flag indicating, for the current bone, if the scaleOrientation on Z axis is affected in the current frame

IsScaleOrientationValue_changed 1 bit flag indicating, for the current bone, if the scaleOrientation angle value is affected in the current frame

isCenter_changed 1 bit flag indicating, for the current bone, if at least one of the center components is affected in the current frame

IsCenterOnX_changed 1 bit flag indicating, for the current bone, if the center on X axis is affected in the current frame

IsCenterOnY_changed 1 bit flag indicating, for the current bone, if the center on Y axis is affected in the current frame

IsCenterOnZ_changed 1 bit flag indicating, for the current bone, if the center on Z axis is affected in the current frame

According to the received mask, the decoder sets-up the so-called “ bone elementary mask” which contains 16 bits corresponding to: translation in X, translation in Y, translation in Z, rotation about first axis, rotation about second axis, rotation about third axis, scale factor along X, scale factor along Y, scale factor along Z, scaleOrientation axis X, scaleOrientation axis Y, scaleOrientation axis Z, scaleOrientation angle value, center deplacement in X, center deplacement in Y, center deplacement in Z. If one of the components is updated the elementary mask for it will be 1 (one); if not is 0 (zero). If in the SBBone node the bone is defined as the end-effector of a kinematics chain, the translation component from the animation stream is used as the desired location of the end-effector. The rest of the bone transformation from the animation stream, if it exists, is ignored.


Muscle_mask

IsControlPoints_changed 1 bit flag indicating, for the current muscle, if at least one of the curve control points position is affected in the current frame

IsControlPoints[cp]_onX_changed 1 bit flag indicating, for the current muscle, if control point index cp perform a translation on X axis in the current frame

IsControlPoints[cp]_onY_changed 1 bit flag indicating, for the current muscle, if control point index cp perform a translation on Y axis in the current frame

IsControlPoints[cp]_onZ_changed 1 bit flag indicating, for the current muscle, if control point index cp perform a translation on Z axis in the current frame

IsWheighOfControlPoints_changed 1 bit flag indicating, for the current muscle, if at least one of the curve control points weigh is affected in the current frame

IsWheighOfControlPoints[cp]_changed 1 bit flag indicating, for the current muscle, if for the control point index cp a new weigh is transmitted in the current frame

IsKnot_changed 1 bit flag indicating, for the current muscle, if at least one of the elements from the knot sequence is affected in the current frame

IsKnot [k]_changed 1 bit flag indicating, for the current muscle, if for the knot element index k a new value is transmitted in the current frame

According to the received mask, the decoder set up the so-called “ muscle elementary mask” which contains a variable number of bits corresponding to translation on X for the first control point of the curve, translation on Y for the first control point of the curve, translation on Z for the first control point of the curve, and so on for all the points, followed by the mask for control points weigh and the mask of knots. As for the bones, if a component is updated by the current frame the “muscle elementary mask” will be 1 (one); if not is 0 (zero).

bba_object_plane_values

Bba_object_coding_type 1-bit integer indicating which coding method is used. Value 0 (zero) means the coding method is “predictive coding”, value 1 (one) means that the encoding method is DCT

Bba_is_i_new_max 1-bit flag which when set to ‘1’ indicates that a new set of maximum range values for I frame follows these 4, 1-bit fields

Bba_is_i_new_min 1-bit flag which when set to ‘1’ indicates that a new set of minimum range values for I frame follows these 4, 1-bit fields

Bba_is_p_new_max 1-bit flag which when set to ‘1’ indicates that a new set of maximum range values for P frame follows these 4, 1-bit fields

Bba_is_p_new_min 1-bit flag which when set to ‘1’ indicates that a new set of minimum range values for P frame follows these 4, 1-bit fields

decode_bba_new_minmax

Bba_i_new_max[one_sbc] 5-bit unsigned integer used to scale the maximum value of the arithmetic decoder used in the I frame for the current Skin&Bones component (SBC)

Bba_i_new_min[one_sbc] 5-bit unsigned integer used to scale the minimum value of the arithmetic decoder used in the I frame for the current Skin&Bones component (SBC)

Bba_p_new_max[one_sbc]

5-bit unsigned integer used to scale the maximum value of the arithmetic decoder used in the P frame for the current Skin&Bones component (SBC)


Bba_p_new_max[one_sbc]

5-bit unsigned integer used to scale the minimum value of the arithmetic decoder used in the P frame for the current Skin&Bones component (SBC)

NUM_SBCs is the number of SBC in the current frame and is computed as :

NUM_SBCs=NSBB*16+NSBM*(NCP*3+NCP+NK)

sbc_mask contains NUM_SBCs elements and is obtained by concatenation of all the “bones elementary mask” and the “muscles elementary mask”, affected in the current frame.

3 Multi-User Worlds (MUW)

3.1 Glossary of Abbreviations

MSC MUTech Session Controller

MBK MUTech Bookkeeper

The MUTech Session Controller and the MUTeck Bookkeeper is described in N4272 Working Draft 3.0 of ISO/IEC 14496-1 / AMD4. The MUTech Session Controller takes care of the messages sent in the control plane (as defined in 5.1.1) while the MUTech Bookkeeper takes care of the messages sent in the data plane (as defined in 5.1.2). The MUTech Bookkeeper maintains a shadow of the shared scene graph, and simulates a client running at a framerate specified in the MUConfig (this in order to make sure that the scene graph always reflects the changes made by the different clients during the multiuser session).

3.2 Introduction

3.2.1 Multi-users world technologies are important

Previous multimedia representations (e.g. MPEG-1/2/4[1] current standards and on-going MPEG amendments, W3C standards like SMIL and HTML, other proprietary formats like QuickTime) were mostly designed for single user usage in broadcast or interactive client-server applications. In today’s world, more and more people are connected to the Internet and are performing multi-users activities as various as multimedia chat and conferencing, multi-user gaming or distance learning.

Vendors of multi-user world products have therefore found it necessary to define additional features to existing multimedia representation by which they accomplish their sharing goals. These features have been designed independently and are generally not interoperable. This was acceptable when multi-user activities were restricted to a niche market (e.g. internet videoconferencing) or to very specific classes of application (e.g. 3D multi-user worlds), but with the generalization of the demand, it is no longer acceptable for the user to repeatedly download and install new applications performing similar tasks and using similar technologies.

Demand is growing for an interoperable multimedia representation that is also designed for multi-user applications. The development of multi-user world technology is therefore extremely important for the adoption of the MPEG-4 standard in these application areas.

3.2.2 Approaches for multi-user world technology

Multi-user world technology relies upon sharing mechanisms. Two main approaches of sharing mechanisms currently exist:1. Protocol based (e.g H.323[5], VRTP[8]): these are based on the definition of the messages (bits on the wire)

between the multimedia objects that implement the sharing mechanisms;

2. APIs based (e.g. CORBA): definition of an object hierarchy as well as a communication APIs. The bits on the wire are left to an underlying middleware.

Both types of approaches have differing strengths and weaknesses and thus distinct markets have arisen around them. The second approach is already well covered in terms of standardization, but is not suitable in environments that have limited


bandwidth and processing power. Defining the set of available objects and defining the communication protocols between these objects could improve this, but then this would be equivalent to the first approach.

There is no standardized solution for the first approach addressing the complete set of requirements described in the Call for Proposals for MPEG-4 Multi-user technology (ISO/IEC JTC1/SC29/WG11 N3574).

3.2.3 Proposal Overview

This proposal is an attempt to cover the gap in the standardisation area described in the previous section, thus the increasing the spectrum of interoperable multi-user applications and devices. The proposal extends the current MPEG-4 architecture and defines new tools to satisfy MPEG-4 multi-user applications requirements.

These architecture and tools will benefit:1. The end-users: they will be able to navigate multi-user worlds from various vendors with the same tools. They

will be able to share information (e.g. avatars, a 2D whiteboard, 3D objects) with other users navigating the world with a different terminal;

2. The multi-user world authors: They will be able to create multi-user worlds and applications without being dependent on specific, specialized service providers;

3. The multi-user service providers: They will be able to provide tools and technology for sharing multimedia information that will benefit a broad set of authors and users.

This proposal is partly based on existing solutions that have solved or attempted to solve similar requirements to the ones described in the Call for Proposals for MPEG-4 Multi-user technology (ISO/IEC JTC1/SC29/WG11 N3574). The more important sources of inspiration have been the Living World specification [9] and H.323 [5]. Some of the authors of this proposal have participated actively in the design and implementations of these two standards and have distilled the following specification from their previous experiences and the specific design goals:

Seamless integration into the MPEG-4 architecture: The proposal has been developed to fit with the existing MPEG-4 architecture. It is an extension of the standard enabling multi-user application using MPEG-4 multimedia representation.

Re-use of existing MPEG-4 tools: The proposals uses as much as possible existing tools and provides multi-user extensions where needed. The tools that are mostly re-used are: BIFS (with new nodes extensions), DMIF (DAI extension), the OD protocol as well as all the MPEG-4 mono media representation (e.g. natural audio and video, synthetic objects).

Flexible architecture: The proposal is not the specification of a complete application but rather the definition of architecture and associated tools. The architecture has been designed to cover a broad range of applications from 2D text chat and audio-visual conferencing, to 3D virtual worlds and multi-user games.

The technical description is organised as follows: the two first sections concerns the architecture. They provide a high level description of the proposed solutions as well as an architectural walkthrough. The following sections provide the definition of the specific tools that enable this architecture: new nodes definition and behaviour, and semantics of the multi-user message protocol.


3.3 Technical Description

3.3.1 The Multiuser Nodes

The main multiuser nodes are the MUSession node and the MUZone node. The MUSession node defines the root node of the shared parts of an MPEG-4, scene and the location of the computer controling the multiuser session defined by this node. The MUZone node divides the shared scene into smaller sub-scenes, which may have different access rights and different servers controlling the syncronization and updating of the nodes in the sub-scene of each of the zones. The MUAvatar and the MUAvatar2D are the representation of a client in a multiuser application. The MUAvatar and MUAvatar2D nodes are direct children of the MUSession node. All the multiuser nodes have their own namespace (which are identified by a unique zoneID), a local PROTOlist and a list of ROUTEs.

3.3.1.1 MUSession


MUSession {EventIn MFNode addChildren

EventIn MFNode removeChildren

exposedField MFNode children []

exposedField MFString info []

exposedField SFBool enabled FALSE

exposedField MFString url []

EventOut SFBool isActive

}3.3.1.1.2 Functionality and semantics

The MUSession node is the top-level container for a shared scene.

The addChildren and the removeChildren eventIns, as well as the exposedField children, follows the functionality and semantics of the corresponding fields in the Transform node (9.4.2.109 in the MPEG-4 spec).

The info field contains metainformation about the multiuser session.

The enabled field is used for joining and leaving the MUSession (enable/disable). If the enabled field is tried changed to TRUE, a JoinSessionRequest message (defined in document M7751) is sent to the MSC. The response to the request is a JoinSessionAcknowledge message (defined in document M7751). When the request is granted, the MBK will update the fields of the MUSession node. In these updates, the content of the children field is transmitted to the local MUSession node.

NOTE – when the children field is updated after the client has joined the session, any MUZone node in the sub-scene of the MUSession node is distributed with an empty children field. The content of the children field is not distributed before the MUZone node has been enabled.

When the MUSession node is enabled, the client will receive updates from the MBK whenever another client performes changes to any field or node in the sub-scene of the MUSession node. Also, whenever the client tries to change the state of any field or node in the sub-scene of the enabled MUSession node, the client must generate a corresponding MURequestMessages to the MBK. (Also see document M7751 for details regarding when to generate MURequestMessages).

If the enabled field is tried changed from FALSE to TRUE, a LeaveSessionRequest message (defined in M7751) is sent to the MSC. The response to the request is a LeaveSessionAcknowledge message (defined in M7751). If granted, the fields of the MUZone node is updated, the main change is that the nodes in the children field are deleted.

If the initial value of the enabled field is TRUE, the client/player should automatically generate a JoinSessionRequest and send it to the MSC immediately after constructing the MUSession node.


NOTE – The behaviour of the MUSession node is dependent of the capabilities of the player. If the player does not support multiuser functionality, the MUSession node acts as a Group node. The nodes in the children field should in this case only be rendered if the enabled field is set to TRUE. The same applies for a multiuser capable player; when the client has not connected to the session, the children nodes are not rendered.

The isActive eventOut reports the status of the session. If the enabled field changes from FALSE to TRUE, an isActive eventOut = TRUE is generated. If the enabled field changes from TRUE to FALSE, an isActive eventOut = FALSE is generated.

The MUSession node contains a PROTOlist and a list of ROUTEs. (See 3. Encoding of MUNodes).

3.3.1.1.3 Specific rules that applies to the MUSession node

There can exist multiple MUSession nodes in a scene.

MUSession nodes CANNOT be nested.

The MUSession node opens up its own namespace. Within a MUCommandStream, the MUSession node is indicated by zoneID == 0. (zoneID == 0 is always reserved for the MUSession node). The MUZone, MUAvatar and MUAvatar2D nodes within the session all have unique Ids in the range <0, 2^MUConfig->zondeIDbits>. This ID is used as zoneID in the MUCommandStream to uniquely indicate nodes that exist in the subscene of a MUZone or a MUAvatar/MUAvatar2D node.

It is not allowed to DEF a node within one MUSession, and then USE it within another MUSession., MUZone , MUAvatar or MUAvatar2D node. (This is because all these nodes have their own namespaces.) This also means that sharing a MUZone between different MUSession nodes is not allowed.

It is possible to connect to multiple sessions at a time, but this is not trivial for the player to support. If a content creator will allow this, he/she must take care when designing the content so that objects from different sessions don’t intersect if they are not supposed to do so.

ROUTEs cannot be defined between nodes within different multiuser nodes (MUSession, MUZone, MUAvatar, MUAvatar2D).

3.3.1.2 MUZone


MUZone {eventIn MFNode addChildren

eventIn MFNode removeChildren

exposedField MFNode children []

exposedField SFBool pilot FALSE


exposedField SFBool enabled FALSE

exposedField SFVec3f bboxCenter 0 0 0

exposedField SFVec3f bboxSize -1 –1 –1

EventOut SFBool isActive


The MUZone is a container for nodes with sharing semantics. It is used to divide the shared space into smaller sub zones. MUZone nodes can be nested, and may divide the scene into parts with different access rights.

The addChildren and the removeChildren eventIns, as well as the exposedField children, follows the functionality and semantics of the corresponding fields in the Transform node (9.4.2.109 in the MPEG-4 spec).


The pilot field defines whether the client/player is piloting the MUZone or not. The responsibility of piloting a MUZone can be transferred to any client or to a centralized server. In the centralized server case, the piloting of the MUZones is by default left to the server (the BookKeeper instance is on the server). In order to support piloting on the client side, the player needs complete support for encoding and decoding of all messages defined in the MUCommandStream (as defined in M7751) as well as handeling of these messages.

The info field contains metainformation about the zone.

The enabled field is used for joining and leaving the MUZone (enable/disable). If the enabled field is tried changed to TRUE, a JoinZoneRequest message (defined in M7751) is sent to the MSC. The response to the request is a JoinZoneAcknowledge message (defined in M7751). When the request is granted, the MBK will update the fields of the MUZone node. In these updates, the content of the children field is transmitted to the local MUZone. After the MUZone node has been enabled, the client will receive updates from the MBK whenever another client performes changes to any field or node in the sub-scene of the MUZone node. Also, whenever the client tries to change the state of any field or node in the sub-scene of the enabled MUZone node, the client must generate a corresponding MURequestMessages to the MBK. (Also see document M7751 for details regarding when to generate MURequestMessages).

NOTE – when the children field is updated after the client has joined the zone, any MUZone node in the sub-scene of the MUZone node is distributed with an empty children field. The content of the children field is not distributed before the MUZone node has been enabled.

If the enabled field is tried changed from TRUE to FALSE, a LeaveZoneRequest message (defined in M7751) is sent to the MSC. The response to the request is a LeaveZoneAcknowledge. If granted, the fields of the MUZone node is updated, the main change is that the nodes in the children field are deleted.

If the initial value of the enabled field is TRUE, the client/player should automatically generate a JoinZoneRequest and send it to the MSC immediately after constructing the node.

NOTE – The behaviour of the MUZone node is dependent of the capabilities of the player. If the player does not support multiuser functionality, the MUZone node acts as a Group node. The nodes in the children field should in this case only be rendered if the enabled field is set to TRUE.

The MUZone node contains a PROTOlist and a list of ROUTEs. (see 3. Encoding of MUNodes).

3.3.1.2.3 Specific rules that applies to the MUZone node

There can exist multiple MUZone nodes within a MUSession or a MUZone. The last one follows from the fact that MUZone nodes can be nested.

The MUZone node opens up its own namespace. Within a MUCommandStream, the MUZone node is indicated by its nodeID, which is the same as its zoneID. The zoneID is a unique ID within the MUSession nodes namespace. The IDs are in the range <0, 2^MUConfig->zondeIDbits>.

Within a MUSession the DEF and USE instances of nodes must lie in precisely the same MUZone (i.e. not in nested sub-zones). DEF and USE instances that lie in different MUZones will produce undefined results.

ROUTEs cannot be defined between nodes that do not lie in precisely the same MUZone.

3.3.1.3 MUAvatar


MUAvatar {exposedField MFString url

exposedField MFNode model []

field SFInt32 clientID -1

exposedField SFVec3f position 0 0 0

exposedField SFRotation orientation 0 0 1 0


}


3.3.1.3.2 Functionality and semantics

The MUAvatar node is a 3D representation of a client in a multiuser application. The MUAvatar node can only be a direct child of a MUSession node. Each user can have only one representation in the multiuser session. Normally a user will not render its own avatar, but this is an application specific choice. The MUAvatar node is always managed (piloted) by the client represented by the node. This means that a client does not need to request the server in order to change the node, but generate the MUUpdateMessages directly. The MBK in this case works only as a distributor, passing on the messages directly to the other clients in the session. A very intelligent MBK may still prevent the changes to be carried out on other clients within the same multiuser session if they for some reason violate some predefined (application specifiec) rules for movement or changes. The MUAvatar node follows the rules set up by the current bound NavigationInfo and Viewpoint node.

It is the responsibility of the MSC in cooperation with the MBK to make sure that the correct MUAvatar nodes are distributed to the different clients. The rule is that if two clients share at least one MUZone (they have both joined the same zone) they should also receive updates for eachothers MUAvatar nodes. This means that the client will receive a MUUpdateMessage adding the other clients MUAvatar node to the children field of the MUSession node when entering such a zone for the first time. The client will receive updates for the other clients MUAvatar nodes as long as he/she shares at least one MUZone node. As soon as the clients no longer share any zones, a MUUpdateMessage removes the MUAvatar nodes of the other client/clients from the MUSession nodes children field.

The url field can refer to an IOD containing at least one BIFS stream for the client representation.

The clientID field contains the id of the client owning the avatar node.

The position field defines the MUAvatar node’s position relative to the MUSession node. If a bound Viewpoint node moves, the position field should be updated accordingly. This update is always sent to the MBK of the MUSession node in order for the server and the other clients to know the clients position.

The orientation field defines the MUAvatar node’s orientation relative to the MUSession node. If a bound Viewpoint node rotates, the orientation field should be updated accordingly. This update is always sent to the MBK of the MUSession node in order for the server and the other clients to know the clients position.

The model field contains the model of the avatar (the audio-visual representation of the client).

The info field can be used for communication, general info about the client, the clients name and so on.

NOTE - If a client wants to move the avatar of another client, there are two possibilities:

1. He/she could send MURequestMessages to the client piloting the avatar (who is the pilot is transparent to the user, the MBK handles this)

2. He/she could try to change the current bound Viewpoint node, and the avatar would follow as specified for the Viewpoint node in the MPEG-4 spec (9.4.2.112 Viewpoint).

3.3.1.4 MUAvatar2D


MUAvatar2D {exposedField MFURL url

exposedField MF3Dnode model []

field SFInt32 clientID -1

exposedField SFVec2f position 0 0

exposedField SFFloat orientationAngle 0



The MUAvatar2D node is a 2D representation of a client in a multiuser application. The MUAvatar2D node can only be a direct child of a MUSession node. Each user can have only one representation in the multiuser session. Normally a user will not render its own avatar, but this is an application specific choice. The MUAvatar2D node is always managed (piloted) by the client


represented by the node. This means that a client does not need to request the server in order to change the node, but generate the MUUpdateMessages directly. The MBK in this case works only as a distributor, passing on the messages directly to the other clients in the session. A very intelligent MBK may still prevent the changes to be carried out on other clients within the same multiuser session if they for some reason violate some predefined (application specifiec) rules for movement or changes.

It is the responsibility of the MSC in cooperation with the MBK to make sure that the correct MUAvatar2D nodes are distributed to the different clients. The rule is that if two clients share at least one MUZone (they have both joined the same zone) they should also receive updates for eachothers MUAvatar2D nodes. This means that the client will receive a MUUpdateMessage adding the other clients MUAvatar2D node to the children field of the MUSession node when entering such a zone for the first time. The client will receive updates for the other clients MUAvatar2D nodes as long as he/she shares at least one MUZone node. As soon as the clients no longer share any zones, a MUUpdateMessage removes the MUAvatar2D nodes of the other client/clients from the MUSession nodes children field.

The url field can refer to an IOD containing at least one BIFS stream for the client representation.

The clientID field contains the id of the client owning the avatar node.

The position field defines the MUAvatar node’s position relative to the MUSession node. If a bound Viewpoint node moves, the position field should be updated accordingly. This update is always sent to the MBK of the MUSession node in order for the server and the other clients to know the clients position.

The orientationAngle field defines the MUAvatar node’s orientation angle relative to the MUSession node.

The model field contains the model of the avatar (the audio-visual representation of the client).

The info field can be used for communication, general info about the client, the clients name and so on.

NOTE - If a client wants to move the avatar of another client, he/she could send MURequestMessages to the client piloting the avatar (who is the pilot is transparent to the user, the MBK handles this)

3.3.2 Encoding of MUNodes

The MUNodes calls for a special encoding schema, since the nodes can contain ROUTEs and a PROTOlist. In order to support this, the following is the proposed changes to SFNode (under section 9.3.7.3 SFNode in the current Systems specification [1]).

class SFNode(int nodeDataType, NodeData protoNodeData) { bit(1) isReused; if (isReused) bit(BIFSConfiguration.nodeIDbits) nodeID; else { int nodeGroup = 0; do { nodeGroup++; bit(GetNDTnbBits(nodeGroup, nodeDataType)) localNodeType; } while (localNodeType == 0);

if (nodeGroup == 2 && localNodeType == 1 ) { bit(BIFSConfiguration.PROTOIDbits) protoID; nodeDataType = PROTODataType; nodeType = GetNodeType(nodeGroup, nodeDataType, protoID); } else { nodeType = GetNodeType(nodeGroup, NodeDataType, localNodeType); }

bit(1) isUpdateable; if (isUpdateable) { bit(BIFSConfiguration.nodeIDbits) nodeID; if (USENAMES) String name; }

if (nodeGroup == 5 && (nodeType == MUSessionType || nodeType == MUZoneType|| nodeType == MUAvatar|| nodeType == MUAvatar2D))

PROTOlist protos; }


if (nodeGroup == 1 && nodeType == IndexedFaceSetType && BIFSConfiguration.use3DmeshCoding == 1) Mesh3D mnode; } else { bit(1) MaskAccess; NodeData nodeData = MakeNode(nodeGroup, nodeDataType, nodeType); nodeData.protoData = protoNodeData; if (MaskAccess) MaskNodeDescription mnode(nodeData); else ListNodeDescription lnode(nodeData); } if (nodeGroup == 5 && (nodeType == MUSessionType || nodeType == MUZoneType|| nodeType == MUAvatar|| nodeType == MUAvatar2D)) bit(1) hasROUTEs; if (hasROUTES) ROUTEs routes; } }}

3.3.3 Behaviour of existing MPEG-4 nodes when shared

3.3.3.1 Obtaining unique nodeIDs

In a multiuser session it is important that all nodeIDs are the same on all clients to ensure a common reference. It is the entity that handles the piloting of a zone that is responsible for assigning new and unique nodeIDs whenever new nodes are added to the zone (Usually this will be the MBK of the MUZone, since by default the MBK also pilots the MUZones it is responsible for). Before this happens, the client that initiates the action of adding the new nodes to the zone must use temporary nodeIDs.

To distinguish between temporary and permanent NodeIDs, the highest order bit of the temporary nodeID should be 1, while the permanent unique ID assigned by the MBK should have the highest order bit set to 0. The number of bits used for encoding nodeIDs should be specified in the MUConfig. This, together with the fact that all the MUNodes have a reserved the IDspace in the range [0, 2^MUConfig->zoneIDbits> leaves the following namespace for other shared nodes:

Unique IDs: [2^MUConfig->zoneIDbits, 2^(MUConfig->nodeIDbits – 1>

Temporary IDs: [2^(MUConfig->nodeIDbits – 1), 2^MUConfig->nodeIDbits>

Example:

MUConfig->zoneIDbits = 5

MUConfig->nodeIDbits = 7

There can be one MUSession node with zoneID == 0 (the zoneID == 0 is reserved for the MUSession node).

There can be up to 31 MUZone nodes with IDs in the range <0, 32>

There can be up to 87 DEF’ed nodes within a single MUNode with unique IDs in the range [32, 128>.

The range of temporary IDs is [128, 256>. (The rather strange result having more temporary IDs than unique IDs is a result of the fact that MUNodes do not get temporary IDs, they are always assigned by the MSC.

Using this mechanism, the pilot and drones of a node should have the same nodeID on all clients within the session. This makes the messaging easier, and also makes e.g. ID mapping superfluous.

3.3.3.2 Modifying a node within a MUZone/MUSession

In multiuser applications it is very important to minimize the amount of data transmitted between the client and the server, and between the different clients, while still maintaining synchronization of the shared content. The main method used for this, is to distinguish between deterministic and non-deterministic changes. If a change is non-deterministic, the MBK must be notified about it, and it should act accordingly. If a change is deterministic, there is no need to notify the MBK, because the MBK (and all the other clients) already knows about this change. A deterministic change is performed locally without any communication with the MBK. A non-deterministic change requires a MURequestMessage to be sent to the MBK. If the MBK allows the change, a MUUpdateMessage will be generated by the MBK and sent to all clients currently joined to the MUZone where the change should take place.


3.3.3.2.1 Deterministic changes

All changes that follows as a result of a ROUTE being triggered is deterministic, because all clients and the MBK have the same ROUTE defined, and the same ROUTE will be triggered on all clients and the MBK at the same time. Therefore no messages are needed for synchronization. The only exception here is ROUTEs defined from the fields of a sensor node. See the chapter on “Sensor nodes within a MUZone” for explanation and rules that apply.

3.3.3.2.2 Non-deterministic changes

All user driven events are non-deterministic. This means that all changes that follows from some action the user performs on nodes within a MUZone requires to be synchronized by the MBK. Examples are user movement and interaction with sensor nodes in the scene

3.3.3.3 Sensor nodes within a MUNode

When BIFS nodes are contained in a MUNode, they have different event processing semantics.

Conceptually if a client terminal tries to locally modify fields of the node, instead of changing fields directly in the scene graph, MURequestMessages are sent to the respective pilot/MBK. The goal is to synchronize the respective scene graph on the participating terminals.

Some BIFS Nodes (especially Sensor Nodes) require detailed processing rules.

All changes resulting from user input should result in a MURequestMessage:

if <any> user clicks on a shared TouchSensor the corresponding eventOut’s should be synchronized.

if <any> user drags a shared drag sensor the corresponding eventOut’s should be synchronized.

if <any> user enters a the area of a shared ProximitySensor the corresponding eventOut’s should be synchronized.

if <any> user enters a the area of a shared VisibilitySensor the corresponding eventOut’s should be synchronized.

For a TimeSensor the eventOuts are completely determined from the state of the fields startTime, stopTime, loop, and cycleInterval. If terminals share a common timebase it is sufficient to synchronize only these fields.

A field change using the MPEG-J methods should always result in a MURequestMessage.

A field is changed resulting from ROUTE from another node in the same Zone, if the source node is synchronized, the behaviour is deterministic, and so the target field change need not be synchronized.

Field changes for nodes outside a zone and hence outside the MUSession, are not synchronized. They will be synchronized if/when the node is added to a MUZone.

DEF and USE instances of nodes must lie in precisely the same MUZone (i.e. not in nested sub-zones). DEF and USE instances that lie in different zones will produce undefined results.

3.3.3.4 PROTO nodes within a MUZone

When a PROTO node exists within a MUZone, the exposedFields and eventIns defined in the interface of the PROTO is shared. Nodes and fields on the “inside” of the PROTO node are not shared. Issues regarding passing SFNode/MFNode into / out of a PROTO must be solved (This is not covered by this document).

The rules that applies to the PROTO node within a MUZone are as follows:

If the event that changes the value of one of the fields in the interface is non-deterministic, the event triggers a MURequestMessage so that the event can be synchronized. There is one more non- deterministic change that can occur when dealing with a PROTO node, and this is if there exist a ROUTE/IS from a field on the inside of the PROTO to a field in the interface. Changing the field on the inside also changes the field in the interface. This change is a non-deterministic change, and triggers a MURequestMessage to synchronize the field in the interface. BUT, not even this rule is enough. If a field in the interface changes because another field in the interface has changed, the event is already synchronized and no MURequestMessage should be generated. (Trying to explain it in other words: If changing an exposedField or an eventIn in the interface also changes another exposedField or an eventOut in the interface, this resulting change is deterministic, since the event was synchronized at an earlier stage.)

If the event that changes an exposedField or eventIn in the interface is deterministic, the event needs no synchronization.

3.3.3.5 Script nodes within a MUZone

All changes to eventOuts in Scripts are deterministic; hence the event does not generate a MURequestMessage. (The eventOuts are completely determined by the eventIn of the Script node.)


NOTE – The above is not completely true if the Script uses functions like random (). The result will be indeterministic. Proposal to use mustEvaluate == TRUE for indeterministic Scripts.

3.3.3.6 Inline nodes within a MUZone

Inline nodes have their own namespace, and the assignment of nodeIDs on the nodes in the file defined in the Inline node does not have to be the same on different clients, since the nodes inside per definition will not be shared.

The fields of an inline node are shared if the node exists within a MUNode (not the content of the file defined in the inline node).

3.3.3.7 Sharing streams

Sharing of streams can be done on two levels with in the MPEG-4 framework: on the object descriptor level or the elementary stream level.

1. If two nodes in a shared scene use the same ObjectDescriptor, then they point to the same elementary stream(s), thus the nodes will share the content of the stream and the decoding parameters specified in the OD.

2. If two nodes have different ODs but the ODs point to the same elementary stream, then the decoding parameters can differ between the two nodes sharing the stream. This is especially important, in the multiuser setting since different nodes sharing the same stream may be on different client terminals with different capabilities. This can be accommodated in option 2 since the decoding parameters can be adapted according to client requirements.

Both these options provide ways of sharing the streams between nodes and clients in the MPEG-4 framework; it is up to the content providers to decide which mechanism to use in a given situation.

Scenario:

Imagine two clients A and B have a shared zone and client A wants to share a video stream with client B. Say, this stream will come from a remote server called C.

Walkthrough:

1) A puts a respective node (MovieTexture) into the shared zone. The MovieTexture node carries a data URL. This data URL contains a string that points to the desired video stream, e.g. "x-dtcp://142.39.12.07/NameOfStream/...maybe_some_video_related_parameters"

2) The local DMIF instances of A and B send this string to C.

3) C sends back to A an OD for the requested stream. It contains all parameters that are necessary for decoding the stream, i.e. sLConfigDescr, decConfigDescr, ...

4) Now, with the received OD the DMIF instance of A opens the required channel for video delivery between A and C.

The last two steps apply to client B in exactly the same way, so A and B share the same stream.

Remarks:

1. Why are we using data URLs? Because, if we would use the conventional mechanism, we would have to use an OD that contains the above mentioned pointer string (see step 1). This OD would require an ID, but since it is shared there may be a conflict with existing IDs in the scene graphs of A and B. We could avoid this problem by requesting a central authority, which does the bookkeeping and assigns IDs in order to avoid these name conflicts. But this solution would be too expensive for simple peer-to-peer scenarios. Instead, the data URL sits directly on the MovieTexture node and doesn't require any ID.

2. The ID of the actual OD that is sent by the server C should be a unique reserved one, indicating that this is an OD of a shared stream, because otherwise the same name conflict as mentioned above could arise. This reserved ID for "ODs of shared streams" should be standardised.

3. A and C might be on the same terminal with no change of the walkthrough.

4. If the stream parameters are fine for A but are not appropriate for B, because B has different video decoding capabilities, then A and B cannot share the same stream. They can only share the same service. In this case B should modify the parameters carried in the data URL (see above). This can be done by means of an Update that comes from a central server, which is complicated, or via MPEG-J which requires an extension of the MPEG-J API.

Then C sends to B the appropriate OD and will also send to B the appropriate video stream afterwards.

3.3.4 The MUCommandStream

A new stream type is defined for carrying MU messages between the server and the different clients. The connections between the server and the different clients are two-way connections. The MUCommandStream consists of encapsulated BIFS-updates, request messages, acknowledge messages and command messages.


The following figure shows the relationship between the different messages that are sent through the MUCommandStream. Note that there is actually a one to many mapping from 1 to 2 and from 2 to 3.

1. MUCommandStream

2. MUCommandFrame

3. MUMessage

4. MUCommandMessage

5. MUControlMessag

6. MURequestMessage

7. MUUpdateMessage

8. MUControlRequest

9. MUControlAcknowledge

10. MUControlCommand

11. CommandFrame

3.3.5 Descriptors for MUCommandStreams

This section describes how elementary stream and object descriptors will be used to associate MU scene nodes with a given MU server, allowing session joining and scene sharing. MUSession and MUZone nodes have similar functionality to Inline nodes in their use of remote ODs, to insert a sub tree into an existing scene based on a remote url.

Figure 122 gives a high level overview of the relation between MU nodes, MU command streams, object descriptors and elementary stream descriptors.


Figure 122: Overview of MU nodes, MU streams and asociated descriptors

3.3.5.1 IOD for MUSession Node

The Initial Object Descriptor associated to an MUSession node must have at least 2 elementary stream descriptors: one for conveying MU commands downstream; and one for MU commands upstream (i.e. using a backchannel). The upstream channel is dependant on the downstream MU command channel, and this should be specified in the upstream channel’s ES descriptor. Thus for the upstream channel, the upStream and streamDependenceFlag flags should be set to True, and the value of dependsOn_ES_ID should match the ES_ID of the downstream channel.

Note that the objectTypeIndication is 3, and streamType is 11 (0xB) for MU Command streams, up stream and downstream.

In addition to MU command frames, the IOD associated with the MUSession node can also carry a standard Scene Description (Bifs) stream for Bifs updates and OD stream, for OD updates of the scene sub tree attached to the MUSession node.

3.3.5.2 IOD for MUZone Node

The MUZone node behaves in a similar way to an Inline node, in that the IOD simply contains an ES descriptor for scene description stream and an object descriptor stream. This allows Bifs and OD updates to be conveyed directly to the nodes in a


given MUZone and allows for each MUZone to maintain its own node and OD namespace. This also allows for different profiles to be specified for individual MUZone sub-trees (as well as entire MUSession sub-trees).

3.3.5.3 Example specification for an MUSession Node IOD

An example specification for a MUSession node IOD follows:

InitialObjectDescriptor{

ObjectDescriptorID 1esDescr [

ES_Descriptor {ES_ID 1streamDependenceFlag FALSEstreamPriority 3OCRstreamFlag FALSEdecConfigDescr {

objectTypeIndication 3 streamType 11upStream FALSEbufferSizeDB 1000maxBitrate 0avgBitrate 0decSpecificInfo DecoderSpecificInfoString {}

}slConfigDescr{

predefined 0useAccessUnitStartFlag TRUEuseAccessUnitEndFlag TRUEuseRandomAccessPointFlag FALSEhasRandomAccessUnitsOnlyFlag TRUEusePaddingFlag FALSEuseTimeStampsFlag TRUEuseIdleFlag FALSEdurationFlag FALSEtimeStampResolution 1000OCRResolution 1000timeStampLength 16OCRLength 0AU_Length 0instantBitrateLength 0degradationPriorityLength 0AU_seqNumLength 5packetSeqNumLength 0timeScale 0accessUnitDuration 0compositionUnitDuration 0startDecodingTimeStamp 0startCompositionTimeStamp 0

}}ES_Descriptor {

ES_ID2streamDependenceFlag TRUEdependsOn_ES_ID 1streamPriority 3OCRstreamFlag FALSEdecConfigDescr {

objectTypeIndication 3streamType 11upStream TRUEbufferSizeDB 1000maxBitrate 0avgBitrate 0decSpecificInfo DecoderSpecificInfoString {


}}slConfigDescr{

predefined 0useAccessUnitStartFlag TRUEuseAccessUnitEndFlag TRUEuseRandomAccessPointFlag FALSEhasRandomAccessUnitsOnlyFlag TRUEusePaddingFlag FALSEuseTimeStampsFlag TRUEuseIdleFlag FALSEdurationFlag FALSEtimeStampResolution 1000OCRResolution 1000timeStampLength 16OCRLength 0AU_Length 0instantBitrateLength 0degradationPriorityLength 0AU_seqNumLength 5packetSeqNumLength 0timeScale 0accessUnitDuration 0compositionUnitDuration 0startDecodingTimeStamp 0startCompositionTimeStamp 0

}}ES_Descriptor {

ES_ID 3OCR_ES_Id 101decConfigDescr DecoderConfigDescriptor {

streamType 3objectTypeIndication 2bufferSizeDB 2000

}slConfigDescr SLConfigDescriptor {

useAccessUnitStartFlag TRUEuseAccessUnitEndFlag TRUEuseTimeStampsFlag TRUEtimeStampResolution 100timeStampLength 14

}}

ES_Descriptor {ES_ID 4decConfigDescr DecoderConfigDescriptor {objectTypeIndication 1streamType 1upStream FALSEbufferSizeDB 30000maxBitrate 100000avgBitrate 50000

}}

]}

3.3.6 The MUCommandStream in detail

3.3.6.1 MUConfig

class MUConfig { bit(2) reserved; bit(1) USENAMES; bit(1) use3DMeshCoding; bit(1) usePredictiveMFField;


bit(5) nodeIDbits; bit(5) routeIDbits; bit(5) PROTOIDbits; bit(4) zoneIDbits; //limits the number of MUZones bit(5) clientIDbits; //limits the number of clients bit(32) maxUpdates; //maximum number of updates in millions/s bit(5) requestIDbits; bit(5) acknowledgeIDbits; bit(4) commandIDbits;}

The MUConfig contains the decoder info needed for encoding and decoding of a MUStream.The MUConfig has many of the same fields as in BIFSv2Config. Fields with names taken from BIFSv2Config has the exact same meaning as defined for BIFSv2Config. In addition the following fields are introduced:

USENAMES – this value replaces the USENAMES value parsed in Scene and applies for all nodes in the subscene of a MUSession node only

zoneIDbits - the number of bits used for encoding the ID of a MUZone node. This number must be less than nodeIDbits, since the namespace of MUZone nodes overlap the namespace of other nodes in the scene

clientIDbits – the number of bits used for encoding the client ID

requestIDbits- the number of bits used for encoding the RequestMessage type

acknowledgeIDbits- the number of bits used for encoding the AcknowledgeMessage type

commandIDbits- the number of bits used for encoding the CommandMessage type

isCommandStream – this parameter tells whether the stream is a MUCommandStream or a MUAminStream

maxUpdates – the maximum number of updates per second for a field, this is the minimum framerate the server will use while maintaining a shadow scene graph of the shared scene

3.3.6.2 MUCommandStream

The MUCommand stream is constructed similar to the BIFS Command stream. There are 5 main groups of messages that are transported over this stream.

In the data plate; scene command messages:

- MURequest messages

- MUUpdate messages

In the control plane; controller messages

- MURequest messages

- MUAcknowledge messages

- MUCommand messages

3.3.6.3 MUCommandFrame

class MUCommandFrame() { do { MUMessage() muMessage Bit(1) continue; } while (continue)}The MUCommandFrame serves the same purpose as the CommandFrame in the BIFS command stream.

3.3.6.4 MUMessage

class MUMessage() { bit(1) isMUCommand; if (isMUCommand) { MUCommandMessages muCommandMessages (); } else { MUControlMessage muControlMessage ();


}}The field isMUCommand defines whether the message contains MUCommandMessages (messages in the data-plane) or a control message (message in the control plane).

3.3.6.5 MUCommandMessages

class MUCommandMessages { bit(1) isMUUpdate; bits(MUConfig->zoneIdBits) zoneID; CommandFrame commandFrame ();}The isMUUpdate field defines whether the MUCommandMessages are MURequestMessages or MUUpdateMessages. If isMUUpdate == FALSE the messages are a MURequestMessages. If isMUUpdate == TRUE the messages are MUUpdateMessages. Note that ALL commands in a CommandFrame are corresponds to EITHER Requests OR Updates, they cannot be mixed within the same CommandFrame.

The zoneID field defines the MUZone in which the nodes defined in the CommandFrame exist

CommandFrame is as defined in 9.3.6.2 Command Frame in the MPEG-4 spec.

3.3.6.5.1 MURequestMessage

This is a command sent from a client to a BookKeeper in order to request changing the state of a shared object. The MURequestMessage itself is just an ordinary BIFS-Update, but the update is interpreted as a request if the MUCommandMessage has isMUUpdate set to FALSE. The zoneID defined in the MUCommand message helps identifying which nodes are to be changed, since each MUZone node opens its own namespace and there may be several shared nodes in the scene with the same nodeID.

3.3.6.5.2 MUUpdateMessage

This is a command sent from a BookKeeper to a client in order to change the state of a shared object. The MUUpdateMessage itself is just an ordinary BIFS-Update, and is interpreted as an update if the MUCommandMessage has isMUUpdate set to TRUE. The zoneID defined in the MUCommand message helps identifying which nodes are to be changed, since each MUZone node opens its own namespace and there may be several shared nodes in the scene with the same nodeID.

3.3.6.6 MUControlMessage

class MUControlMessage() { bit(1) reserved; bit(2) type; switch(type) { case 0: //MU_CONTROL_REQUEST { MUControlRequest request (); } break;

case 1: //MU_CONTROL_ACKNOWLEDGE { MUControlAcknowledge acknowledge (); } break;

case 2: //MU_CONTROL_COMMAND { MUControlCommand command (); } break; }}

3.3.6.6.1 MUControlRequest

class MUControlRequest(){ bit(MUConfig->requestIDbits) requestType; switch(requestType) { case 0: //JOINSESSIONREQUEST


{ JoinSessionRequest joinSessionRequest (); } break; case 1: //LEAVESESSIONREQUEST {

LeaveSessionRequest leaveSessionRequest (); } break; case 2: //JOINZONEREQUEST { JoinZoneRequest joinZoneRequest (); }

break; case 3: //LEAVEZONEREQUEST { LeaveZoneRequest leaveZoneRequest (); } break; case 4: //LEAVEALLZONESREQUEST { LeaveAllZonesRequest leaveAllZonesRequest (); } break; case 5: //CREATEZONEREQUEST { CreateZoneRequest createZoneRequest (); } break; case 6: //DELETEZONEREQUEST { DeleteZoneRequest deleteZoneRequest (); } break; case 7: //SESSIONTIMEREQUEST { SessionTimeRequest sessionTimeRequest (); } break; /* The following are priority 2 messages */ case 8: //SESSIONCAPABILITIESREQUEST { SessionCapabilitiesRequest sessionCapabilitiesRequest (); } break; case 9: //ZONEINFORMATIONREQUEST { ZoneInformationRequest zoneInformationRequest (); } break; case 10: //PARTICIPANTINFORMATIONREQUEST { ParticipantInformationRequest participantInformationRequest (); } break; case 11: //SETZONELIFETIMEREQUEST { SetZoneLifeTimeRequest setZoneLifeTimeRequest (); } break; case 12: //GETZONELIFETIMEREQUEST { GetZoneLifeTimeRequest getZoneLifeTimeRequest (); } break; /* Messages for zone access. */ case 13: //GROUPINFORMATIONREQUEST


{ GroupInformationRequest groupInformationRequest (); } break; case 14: //SETGROUPINFORMATIONREQUEST { SetGroupInformationRequest groupInformationRequest (); } break; case 15: //CREATEGROUPREQUEST { CreateGroupRequest createGroupRequest (); } break; case 16: //DELETEGROUPREQUEST { DeleteGroupRequest deleteGroupRequest (); } break; case 17: //GROUPADDCLIENTREQUEST { GroupAddClientRequest groupAddClientRequest (); } break; case 18: //GROUPREMOVECLIENTREQUEST { GroupRemoveClientRequest groupRemoveClientRequest (); } break; case 19: //SETZONEACCESSREQUEST { SetZoneAccessRequest setZoneAccessRequest (); } break; /* The following are messages for pilotship functionality */ case 20: //ZONEPILOTSHIPREQUEST { ZonePilotshipRequest zoneRequestPilotshipRequest (); } break; case 21: //ZONERELEASEPILOTSHIPREQUEST { ZoneReleasePilotshipRequest zoneReleasePilotshipRequest (); } break; case 22: //TRANSFERPILOTSHIPREQUEST { TransferPilotshipRequest transferPilotshipRequest (); } break; }}

3.3.6.6.2 MUControlAcknowledge

class MUControlAcknowledge() { bit(MUConfig->acknowledgeIDbits) acknowledgeType; switch(acknowledgeType) { case 0: //JOINSESSIONACKNOWLEDGE { JoinSessionAcknowledge joinSessionAcknowledge (); } break; case 1: //SESSIONTIMEACKNOWLEDGE { SessionTimeAcknowledge sessionTimeAcknowledge (); }


break; case 2: //JOINZONEACKNOWLEDGE { JoinZoneAcknowledge joinZoneAcknowledge (); } break; case 3: //LEAVEZONEACKNOWLEDGE { LeaveZoneAcknowledge leaveZoneAcknowledge (); } break; case 4: //LEAVEALLZONESACKNOWLEDGE { LeaveAllZonesAcknowledge leaveAllZonesAcknowledge (); } break; case 5: //CREATEZONEACKNOWLEDGE { CreateZoneAcknowledge createZoneAcknowledge (); } break; case 6: //DELETEZONEACKNOWLEDGE { DeleteZoneAcknowledge deleteZoneAcknowledge (); } break; case 7: //SESSIONCAPABILITIESACKNOWLEDGE { SessionCapabilitiesAcknowledge sessionCapabilitiesAcknowledge (); } break; case 8: //ZONEINFORMATIONACKNOWLEDGE { ZoneInformationAcknowledge ZoneInformationAcknowledge (); } break; case 9: //PARTICIPANTINFORMATIONACKNOWLEDGE { ParticipantInformationAcknowledge participantInformationAcknowledge (); } break; case 10: //SETZONELIFETIMEACKNOWLEDGE { SetZoneLifeTimeAcknowledge setZoneLifeTimeAcknowledge (); } break; case 11: //GETZONELIFETIMEACKNOWLEDGE { GetZoneLifeTimeAcknowledge getZoneLifeTimeAcknowledge (); } break; /* messages for zone access */ case 12: //GROUPINFORMATIONACKNOWLEDGE { GroupInformationAcknowledge groupInformationAcknowledge (); } break; case 13: //SETGROUPINFORMATIONACKNOWLEDGE { SetGroupInformationAcknowledge setGroupInformationAcknowledge (); } break; case 14: //CREATEGROUPACKNOWLEDGE { CreateGroupAcknowledge createGroupAcknowledge (); } break;


case 15: //DELETEGROUPACKNOWLEDGE { DeleteGroupAcknowledge deleteGroupAcknowledge (); } break; case 16: //GROUPADDCLIENTACKNOWLEDGE { GroupAddClientAcknowledge groupAddClientAcknowledge (); } break; case 17: //GROUPREMOVECLIENTACKNOWLEDGE { GroupRemoveClientAcknowledge groupRemoveClientAcknowledge (); } break; /* messages for pilotship functionality */ case 18: //ZONEPILOTSHIPACKNOWLEDGE { ZonePilotshipAcknowledge zonePilotshipAcknowledge (); } break; case 19: //ZONERELEASEPILOTSHIPACKNOWLEDGE { ZoneReleasePilotshipAcknowledge zoneReleasePilotshipAcknowledge (); } break; case 20: //TRANSFERPILOTSHIPACKNOWLEDGE { TransferPilotshipAcknowledge transferPilotshipAcknowledge (); } break; case 21: //ACCEPTZONEPILOTSHIP { AcceptZonePilotship acceptZonePilotship (); } break; }}

3.3.6.6.3 MUControlCommand

class MUControlCommand() { bit(MUConfig->commandIDbits) commandType; switch(commandType) { case 0: //JOINSESSIONCOMMAND { JoinSessionCommand joinSessionCommand (); } break; case 1: //LEAVESESSIONCOMMAND { LeaveSessionCommand leaveSessionCommand (); } break; case 2: //JOINZONECOMMAND { JoinZoneCommand joinZoneCommand (); } break; case 3: //LEAVEZONECOMMAND { LeaveZoneCommand leaveZoneCommand (); } break; case 4: //SESSIONTIMECOMMAND { SessionTimeCommand sessionTimeCommand ();


} break; }}

3.3.7 The message Encoding

3.3.7.1 MUControlRequest messages

3.3.7.1.1 JoinSessionRequest

class JoinSessionRequest { bit(MUConfig->requestIDbits) requestID; bit(128) terminalID; //0x0 is undefined, !=0 could be global unique number //0xFFFFFFFFFFFFFFFF is forbidden bit(32) reserved;};

3.3.7.1.2 LeaveSessionRequest

class LeaveSessionRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID;};

3.3.7.1.3 JoinZoneRequest

class JoinZoneRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID;};

3.3.7.1.4 LeaveZoneRequest

class LeaveZoneRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID;};

3.3.7.1.5 LeaveAllZonesRequest

class LeaveAllZonesRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID;};

3.3.7.1.6 CreateZoneRequest

class CreateZoneRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) parentZoneID; //id of parent MUZone node bit(MUConfig->nodeIDbits) nodeID; //id of parent node for the new MUZone bit(MUConfig->zoneIDbits) suggestedZoneID; //id of MUZone to be created

//The new MUZone will be assigned //this ID if available. //If not available, request failes

//If –1, any available zoneID will //be assigned if (USENAMES) { //Name of MUZone node to be created.

String zoneName; //If not available, request failes.} //”” is the default value and will

//succeed. bit(8) zoneLifeTime; //specify the zoneLifeTime to set for the new MUZone node};


3.3.7.1.7 DeleteZoneRequest

class DeleteZoneRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID; //id of MUZone to be deleted};

3.3.7.1.8 SessionTimeRequest

class SessionTimeRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID;};

3.3.7.1.9 SessionCapabilitiesRequest

class SessionCapabilitiesRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(32) informationType; //defines the type of information requested};

3.3.7.1.10 ZoneInformationRequest

class ZoneInformationRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID; bit(32) informationType; //defines the type of information requested };

3.3.7.1.11 ParticipantInformationRequest

class ParticipantInformationRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) participantID; bit(32) informationType; //defines the type of information requested};

3.3.7.1.12 SetZoneLifeTimeRequest

class SetZoneLifeTimeRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID; bit(8) zoneLifeTime; //specify the zoneLifeTime to set. // WHILEOWNERALIVE == 0 // WHILESESSIONPOPULATED == 1 // FOREVER == 2// WHILEOWNERALIVE - MUZone is persistent while the owner is part of the // session.

// If the owner leaves the session, the zone is deleted// If the owner is a Group, the MUZone is persistent // while any of the members in the group are part of the // session.// If all clients defined in the group have left the // session, the zone is destroyed.// WHILESESSIONISPOPULATED - MUZone is persistent while there are // participants in the session.// FOREVER - // Zone is persistent forever. MSCs implementing // session-to-session persistency would store the zone's // state in a database

};

3.3.7.1.13 GetZoneLifeTimeRequest

class GetZoneLifeTimeRequest{


bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID;};

3.3.7.1.14 GroupInformationRequest

class GroupInformationRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) groupID; bit(32) informationType; //defines the type of information requested};

3.3.7.1.15 SetGroupInformationRequest

class GroupInformationRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) groupID; bit(32) informationType; //defines the type of information to set bit(16) numbytes; bit(numbytes * 8) data;};

3.3.7.1.16 CreateGroupRequest

class CreateGroupRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(8) groupLifeTime; //specify the groupLifeTime to set for the new Group.};

3.3.7.1.17 DeleteGroupRequest

class DeleteGroupRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) groupID;}

3.3.7.1.18 GroupAddClientRequest

class GroupAddClientRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) groupID; bit(MUConfig->clientIDbits) addClientID; //clientID of client to add to Group};

3.3.7.1.19 GroupRemoveClientRequest

class GroupRemoveClientRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) groupID; bit(MUConfig->clientIDbits) removeClientID; //clientID of client to remove //from specified Group };

3.3.7.1.20 SetZoneAccessRequest

class SetZoneReadAccessRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID; //this could use clientIDbits instead bit(2) accessType; //NO_ACCESS == 0, READ == 1, READWRITE ==2, bit(MUConfig->clientIDbits) ID; // the ID of the client or Group to get the // specified acces rights to the zone // what about access for ALL? Should a


// Group be created of type ALL for this? };

3.3.7.1.21 ZonePilotshipRequest

class ZonePilotshipRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID;};

3.3.7.1.22 ZoneReleasePilotshipRequest

class ZoneReleasePilotshipRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID; // By default the piloting is set back to the BookKeeper upon receipt of this // message};

3.3.7.1.23 TransferPilotshipRequest

class TransferPilotshipRequest{ bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->zoneIDbits) zoneID; bit(MUConfig->clientIDbits) pilotID; //The ID of the client to take over //pilotship of the specified MUZone};


3.3.7.2 MUControlAcknowledge messages

3.3.7.2.1 JoinSessionAcknowledge

class JoinSessionAcknowledge{ bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED { bit(8) reasonForDenial; switch (reasonForDenial) case 0: { //0 used as a default, the message contains no more data. } break; case 1: //NO_ACCESS break; case 2: //SERVER_FULL break; } break; case 1: //REQUEST_SUCCEEDED { JoinSessionCommand joinSessionCommand(); } break; }};

3.3.7.2.2 SessionTimeAcknowledge

class SessionTimeAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED { bit(1) reasonForDenial; switch (reasonForDenial) case 0: { //used as a default, the message contains no more data. } break; } break; case 1: //REQUEST_SUCCEEDED { SFTime sessionTime (); //the server time (session time) } break; }};

3.3.7.2.3 JoinZoneAcknowledge

class JoinZoneAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED {


bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data } break; case 1: //NO_READ_ACCESS { } break; case 2: //NO_READ_WRITE_ACCESS { } break;

} } break; case 1: //REQUEST_SUCCEEDED { JoinZoneCommand joinZoneCommand(); } break; }};

3.3.7.2.4 LeaveZoneAcknowledge

class LeaveZoneAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data } break; } } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID; //id of the zone the client has left//Fields of the MUZone are updated by separate MUCommandMessages//The MUZone generate an isActive == FALSE eventOut } break; }};

3.3.7.2.5 LeaveAllZonesAcknowledge

class LeaveAllZonesAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: {


//0 used as a default, the message contain no more data } break; } } break; case 1: //REQUEST_SUCCEEDED { bit(1) leftZones; //just want to make sure that the client left more //than 0 zones if (leftZones) {//here we need to update the fields of all the top MUZone nodes the client //left. do { bit(MUConfig->zoneIDbits) zoneID; bit(1) hasMoreZones; } while (hasMoreZones); } } break; }};

3.3.7.2.6 CreateZoneAcknowledge

class CreateZoneAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED { bit(3) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data } break; case 1: //NO_READ_ACCESS { } break; case 2: //NO_READ_WRITE_ACCESS { } break; case 3: //NO_ZONEIDS_AVAILABLE { } break;

} } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID; //the id of the new zone to be // created. This is needed in order // for the client to know which // MUZone was created upon the // request. // The new MUZone node is inserted // using a separate MUUpdate. } break; }};


3.3.7.2.7 DeleteZoneAcknowledge

class DeleteZoneAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED { bit(3) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data } break; case 1: //NO_READ_ACCESS { } break; case 2: //NO_READ_WRITE_ACCESS { } break; case 3: //ZONE_FULL { } break;

} } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID; // id of the zone the client has // requested to delete. The actual // deletion of the node is executed // using a separate MUUpdate. } break; }};

3.3.7.2.8 SessionCapabilitiesAcknowledge

class SessionCapabilitiesAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data. } break; } } break; case 1: //REQUEST_SUCCEEDED { bit(1) reserved; if (reserved == 0) { bit(32) type; bit(16) numbytes; bit(numbytes * 8) data;


} } break; }};

3.3.7.2.9 ZoneInformationAcknowledge

class ZoneInformationAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: //==0 { //0 used as a default, the message contain no more data } break; case NO_READ_ACCESS: //==1 { } break; case NO_READ_WRITE_ACCESS: //==2 { } break; } } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID; bit(2) reserved; // ZoneInfo info (); } break; }};

3.3.7.2.10 ParticipantInformationAcknowledge

class ParticipantInformationAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: //==0 { //used as a default, the message contains no more data } break; } } break; case 1: //REQUEST_SUCCEEDED //==1 { bit(2) reserved; //ParticipantInformation participantInformation (); //This is dependent of the request type. }


break; }};

3.3.7.2.11 SetZoneLifeTimeAcknowledge

class SetZoneLifeTimeAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: //==0 { //used as a default, the message contains no more data } break; } } break; case 1: //REQUEST_SUCCEEDED //==1 { //probably don't need more information here } break; }};

3.3.7.2.12 GetZoneLifeTimeAcknowledge

class GetZoneLifeTimeAcknowledge { bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: //==0 { //used as a default, the message contains no more data } break; } } break; case 1: //REQUEST_SUCCEEDED //==1 { bit(3) zoneLifeTime;

//The possible values of zoneLifeTime must be defined! } break; } };

3.3.7.2.13 GroupInformationAcknowledge

class GroupInformationAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) {


case 0: //==0 { //used as a default, the message contains no more data } break; } } break; case 1: //REQUEST_SUCCEEDED //==1 { bit(MUConfig->clientIDbits) groupID;

bit(1) reserved;//GroupInformation groupInformation (); } break; }};

3.3.7.2.14 SetGroupInformationAcknowledge

class SetGroupInformationAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case 0: //REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //used as a default } break; } } break; case 1: //REQUEST_SUCCEEDED //==1 { } break; }}; bit(MUConfig->requestIDbits) requestID; bit(MUConfig->clientIDbits) clientID; bit(MUConfig->clientIDbits) groupID; bit(32) informationType; //defines the type of information to set bit(16) numbytes; bit(numbytes * 8) data;};

3.3.7.2.15 CreateGroupAcknowledge

class CreateGroupAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: //==0 { //0 used as a default } break; }


} break; case 1: //REQUEST_SUCCEEDED //==1 { bit(MUConfig->clientIDbits) groupID; //the id of the Group created } break; }};

3.3.7.2.16 DeleteGroupAcknowledge

class DeleteGroupAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //used as a default, the message contains no more data } break; } } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->clientIDBits) groupID; //The ID of the Group deleted } break; }};

3.3.7.2.17 GroupAddClientAcknowledge

class GroupAddClientAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data } break; } } break; case 1: //REQUEST_SUCCEEDED {bit(MUConfig->clientIDbits) clientID;bit(MUConfig->clientIDbits) groupID; } break; }};

3.3.7.2.18 GroupRemoveClientAcknowledge

class GroupRemoveClientAcknowledge{


bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { //0 used as a default, the message contain no more data } break; } } break; case 1: //REQUEST_SUCCEEDED {bit(MUConfig->clientIDbits) clientID;bit(MUConfig->clientIDbits) groupID; } break; }};

3.3.7.2.19 ZonePilotshipAcknowledge

class ZonePilotshipAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: //==0 { bit(8) reasonForDenial; bit(MUConfig.zoneIDbits) zoneID; //The zoneID for which zone pilotship was requested bit(MUConfig.clientIDbits) clientID; //the client ID of the current pilot of the zone //-1 if the zone is piloted by the MBK //The pilot field of the MUZone node in question is set to false //by a separate MUUpdate. switch (reasonForDenial) { case 0: { } break; } } break; case 1: //REQUEST_SUCCEEDED //==1 { bit(MUConfig->zoneIDbits) zoneID;//the pilot field of this node is set to TRUE by a separate MUUpdate } break; }};

3.3.7.2.20 ZoneReleasePilotshipAcknowledge

class ZoneReleasePilotshipAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: { bit(8) reasonForDenial;


switch (reasonForDenial) { case 0: { bit(MUConfig->zoneIDbits) zoneID; //0 used as a default, the message contain no more data.

//The pilot field of the MUZone node in question is set to TRUE //by a separate MUUpdate.

} break; } } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID;//a separate MUUpdate sets the pilot field of this node to FALSE. } break; }};

3.3.7.2.21 TransferPilotshipAcknowledge

class TransferPilotshipAcknowledge{ bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { bit(MUConfig->zoneIDbits) zoneID; //used as a default, the message contains no more data

//The pilot field of the MUZone node in question is set to TRUE //by a separate MUUpdate.

} break; } } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID;//the pilot field of this node is set to FALSE by a separate MUUpdate. } break; }};

3.3.7.2.22 AcceptZonePilotship

class AcceptZonePilotship { bit(MUConfig->requestIDbits) requestID; bit(1) requestStatus; switch (requestStatus) { case REQUEST_DENIED: { bit(8) reasonForDenial; switch (reasonForDenial) { case 0: { bit(MUConfig->zoneIDbits) zoneID; //used as a default, the message contains no more data }


break; } } break; case 1: //REQUEST_SUCCEEDED { bit(MUConfig->zoneIDbits) zoneID; } break; }};

3.3.7.3 MUControlCommands

These messages are commands in the control plane sent from server to client (or between two clients in the peer-to-peer case). The messages are not a response to a request as the MUAcknowledge messages, and may therefore be generated at any time independent of any request from a client.

3.3.7.3.1 JoinSessionCommand

class JoinSessionCommand{ bit(MUConfig->clientIDbits) clientID; bit(MUConfig->nodeIDbits) avatarID; SFTime sessionTime (); //the server time (session time) //the fields of the MUSession node is updated using separate MUUpdates //The MUSession node generates an isActive == TRUE eventOut};

3.3.7.3.2 LeaveSessionCommand

class LeaveSessionCommand { bit(32) reason; String message;};

3.3.7.3.3 JoinZoneCommand

class JoinZoneCommand { bit(MUConfig->zoneIDbits) zoneID; //id of the zone the client is joining //The client must receive information about where the MBK is. //One posibility is that the MBK is at the server. If so, there is no //need for sending any information (URL) about the MBK. The current //connection to the server will be used for sending updates for nodes //in the zone. bit(1) reserved; bit(1) useCurrentMUCommandStream; if (useCurrentMUCommandStream) { //fields of the MUZone node are updated with separate MUUpdateMessages. //A isActive == TRUE event is generated from the MUZone node defined } else { MFURL url; //the URL describing where the BookKeeper for this MUZone is }};

3.3.7.3.4 LeaveZoneCommand

class LeaveZoneCommand { bit(MUConfig->zoneIDbits) zoneID; bit(32) reason; }A MUUpdate follows this message to the MUZone node defined in the LeaveZoneCommand.

3.3.7.3.5 SessionTimeCommand

class SessionTimeCommand { SFTime sessionTime ();}


3.3.7.4 MURequestMessages

This is the group of messages used when requesting a change of an object in the scene. It is primarily to ensure that changes are distributed to all relevant clients. The Drone requests its pilot to change its state by sending an MURequestMessage, and the Pilot accepts the change or declines it. MURequestMessages should also be used when wanting to carry out user-driven changes of an object; this includes changing objects in the scene graph using MPEG-J and other authoring interfaces, and through different input-devices. It should also be used when sending messaged not pre-defined by routes.

The MURequestMessages are similar to a BIFS-Updates, but they have added information telling that this is a request and not an ordinary update. Another difference is that the nodes are not only specified by the nodeID but also by the zoneID of the MUZone/MUSession/MUAvatar/MUAvatar2D node where the node to receive the request exist. (This is because the nodeIDs are only uniquely specified within the different MUNodes, and there may be several nodes with the same nodeID existing in different MUNodes.)

3.3.7.5 MUUpdateMessages

This is the group of messages used when updating the value of a field or node. The main purpose of the message is to distribute changes to the drones located on client terminals. It is essentially a BIFS-Update, but it has a wrapper around it telling that this is a MUUpdateMessage. Another difference is that the nodes are not only specified by the nodeID but also by the zoneID of the MUNode node where the node to receive the update exist. (The reason is the same as for the MURequestMessages)

3.3.8 Description of entities mentioned

3.3.8.1 Bookkeeper and SessionController

The Bookkeeper and the SessionController are described in N4272 Working Draft 3.0 of ISO/IEC 14496-1 / AMD4. The Session Controller takes care of the messages sent in the control plane while the Bookkeeper takes care of the messages sent in the data plane. The Bookkeeper maintains a shadow of the shared scene graph, and simulates a client running at a framerate specified by the MUConfig (this in order to make sure that the scene graph always reflects the changes made by the different clients during the multiuser session).

3.3.9 Tables

3.3.9.1 Definition of informationType

Tag value Tag name0x00000000 Forbidden0x00000001-0x7FFFFFFF

Reserved for ISO use

0x80000000-0xFFFFFFFE

User private

0xFFFFFFFF Forbidden

3.3.9.2 Definition of zoneLifeTime

Tag value Tag name0x00 WHILE_OWNER_ALIVE0x01 WHILE_SESSION_POPULATED0x02 FOREVER0x03 - 0x7F Reserved for ISO use (extensions)0x80 -0xFE User private0xFF Forbidden

3.3.9.3 Definition of groupLifeTime

Tag value Tag name0x00 WHILE_OWNER_ALIVE0x01 WHILE_SESSION_POPULATED


0x02 FOREVER0x03 - 0x7F Reserved for ISO use (extensions)0x80 -0xFE User private0xFF Forbidden


Annex A

Multi-User concepts

A.1 High level architecture

This section describes the high-level architecture and required extensions to the existing MPEG-4 standard in order to achieve interoperable multi-user (MU) worlds. The intention is to define the framework within which both software vendors and content providers can implement their own mulit-user solutions. The interoperability should be defined on the following levels:

Shared objects: such that arbitrary objects can be declared as shared, and be uploaded and incorporated into MU environments

World: such that MU-environments created by different content providers can be viewed on different MU-enabled browsers.

Browser:such that any browser can be used to view and interact with MU-content provided by any server.

These levels of interoperability imply that the format for MU content, and the messaging protocol for conveying multi-user related data are standardised.

Pilot

Client 2

Client 1 Client 3

Drone

Networkcommunication

Figure 123: Drones and Pilots in scene graphs on different client terminals

The system outlined in the following sections specifies a low level protocol and infrastructure for sharing sub trees of scene graphs between multiple users. The proposal adresses how individual nodes can be shared and how sub trees that are shared can be protected against unwanted modifications from external sources (e.g. EAI, MPEG-J). This document does not specify how individual nodes should behave in distributed environments. Although general rules for sharing can be set up for different groups of nodes (e.g. interpolators, sensors, routes), rules for each node should be established in the standard such that implementors know how to develop software that supports MU technology.

The architecture for handling multiple users in an MPEG-4 environment, is based largely on the Drone/Pilot mechanism proposed in the Living Worlds specification [9]. Drones are local (i.e. existing in the scene graph of a client terminal) instances of nodes that contact their corresponding Pilots (master copies of nodes on clients or servers) when changes are to be distributed, seeFigure 123.

There are two possible synchronisation mechanisms:

1. Pee-to-peer scenario

In this scenario synchronization takes place at the terminal piloting a shared object.

A pilot receives messages requesting changes (RequestChange messages) from drones, accepts or declines updates, distributes update commands to all drones and itself. A pilot may also receive local requests for changes (e.g. via MPEG-J API) and handle them in the same way.


A drone replicates an object , requests changes of object fields by sending a request change to the pilot.

In this scenario mechanims must be defined such that:

-a new client can query the existence and state of objects piloted by other terminals.

-there is a gracefull detection, if a client who is piloting a shared object leaves the session in an abnormal way (e.g. network disconnection).

2. Centralized scenario using a bookkeeper

One or several bookkeeper instances are responsible for maintaining the state of shared objects.

It the bookkeeper is piloting the objects, it receives RequestChange messages, and distributes BIFS update commands.

If the bookkeeper does not pilot the objects, it routes the RequestChange messages to the client that has the pilot, and distributes BIFS update messages coming from the Pilot to the corresponding Drones.

The bookkeeper is responsible for transfer of pilotage and can provide shared object persistency.

Client 1

MUTechMessage Handler

DM

IF

DM

IFInstancer

MU

Channel M

anagaerClient 1

MUTechMessage Handler D

MIF

DM

IFIn

stan

ce

MU

Cha

nnel

Man

agae

r

MUTechSession

Controler

Sess

ion

Con

trol

Con

trol P

lane

Shar

ed Z

ones

&Sh

ared

Obj

ects

Obj

ects

&

Scen

es Dat

a Pl

ane

AV-StreamServer

MUTechBook-keeper

Key

MURequestChangemessageMUUpdate message

Media/OD channel

Control channelData channel

Drone node

Pilot node

Figure 124: Request/Update architectureA.1.1 MUTech Session Controller (MSC)

This module is part of the control plane. The MSC acts in a similar fashion as the H.323 MCU, setting up control channels, assigning clientIDs, maintaing an overview of capabilities (e.g. multi/unicast) and IP addresses. This information is critical for routing of messages. Each client should have a permanent control channel to the MSC. The MSC provides clients with the list of existing zones, as well as notification services for client enter/leave, zone add/remove. It maintains lists of which clients have joined which zones at any given time instance. This information can be requested by the clients, and is also used by the MUTech Bookkeeper. Information about client capabitity can be requested from the MSC. The MSC controls access to the zones in the session, and handles requests from clients who want to join or leave a zone. The MSC handles requests for creating and deleting zones. The MSC initiates the MUTech Bookkeeper.


A.1.2 MUTech Bookkeeper (MBK)

This module is part of the data plane. It is responsible for syncronizing the state of the shared objects. Its main responsibility is routing of pilot-drone messages. It can take on the responsibility of piloting the shared objects within a zone, but can also transfer pilotship of a zone and its content to a client upon request. The MBK must keep track of where the pilot of a node resides at all times. The MBK maintains a complete scene graph containing all zones and nodes within the session, and si

If the MBK is piloting an object:

All RequestChange messages for this object are processes by the MBK.

The MBK generates the necessary UpdateMessages and distributes them to all clients currently sharing the object.

If the MBK is not piloting an object:

If the MBK receives RequestChange messages to a node it doesn’t pilot, the MBK routes these messages to the client piloting the node.

Upon receiving UpdateMessages from the client piloting the node, the MBK distributes the messages to all other clients currently sharing the node.

A.1.3 MUTech Message Handler (MMH)

This module has these main purposes:1. To provide an interface to the client terminal for sending and receiving multi-user messages.

2. To bundle several multi-user messages that will together to form SL packet.

3. To tag the messages with the correct client id before sending them along the Request/Update channel open on the DMIF.

As shown in Figure 125, the MMH exposes an interface (MAI, Multi-user Application Interface) that other components in the application could use, such as the EAI and MPEG-J. This provides a convenient way for such modules to affect the scene, and ensure that modifications are distributed. The EAI or MPEG-J, would typically send a request, relating to the nodes to be modified. The request would be routed via the MMH to the pilot for validation.

Scene graph

Browser

DM

IF

Multi-user Application Interface(MAI)

DAI

RenderingrenderUser

interaction

MURequestChange

MPEG-J/EAI

MURequestChange

MURequestChangeMURequestChange

Pilot

Drone

OD

OD

ODUpdate


DM

IF

Inst

ance

MUUpdate

MU

Cha

nnel

Man

ager


Figure 125: MUTech Message Handler and MAIA.1.4 MUChannelManager (MCM)

This is a helper component responsible for setting up control & data streams between the client and MSC, through the DMIF interface DAI. The MUChannelManager provides a convenient interface for the MMH to access functionality in the DAI. Channel attributes will be described using Object Descriptor and Elementary Stream Descriptor syntax, which will allow for the inclusion of parameters such as direction, QoS information, intellectual property and object content information to be specified in the channel set-up.

Multi-user messages will be sent via the MCM and the DMIF through channels open on the DMIF server and client modules. Multi-user streams will be associated through an Object Descriptor to the MU stream type. Each MUSession node has an Object Descriptor attached with a URL to the associated MUSessionController.

AVStreamServer


Client 3


KeyMURequest channel

AV channel

MUSession node

ObjectDescriptor

MUUpdate channel

DM

IF

DM

IFInstance

MU

Channel M

anagaerMuTech Server

MU_SINK_DLL

MU_LIVE_DLL

MU_LIVE_DLL

MU_SINK_DLL

DMIF Streaming Server

MUTechSessionControler

MUTechBookkeeper

DM

IF

DM

IFIn

stan

ce

Cha

nnel

Man

agae

r

Client 1

Figure 126: The MUTech server, channel management and scene nodesThe OD will in turn contain two ESDs:

one ESD for describing the stream carrying MU-commands downstream (server to client) a second ESD describing the stream carrying MU-commands upstream (client to server). This second ESD must have

the streamDependenceFlag flag set to true and the dependsOn_ES_ID field set to the ES_ID of the downstream channel.

The DMIF streaming server provides realtime support for client server MU messages through LIVE and SINK APIs and DLLs. Which are for server-to-client and client-to-server communication respectively. Each client connected to the streaming server has his or her own, or shares MU_LIVE and MU_SINK dlls (how many dlls are loaded is an issue left to implementers of the MUTech server side system). For simplicity of architecture, it is envisaged the refernce MU implementation will use load 2 dlls per connected client as shown in Figure 126.


A.1.5 MUTech Instance

This a DMIF instance that allows an application to open message channels and transfer respective messages to and from the MSC. The instance complies with the DPI (DMIF Plug-in Interface), and makes communication with the MSC transparent for the application. In addition the filter may support peer-to-peer communication between clients, such that the MSC and MBK are not used for routing Request and Update messages. This works in a similar way that an H.323 DMIF filter works, allowing full MCU enabled conferencing for more than two participants, and allowing peer-to-peer networking when there are only two participants.

The MUTech Instance may be implemented by an existing DMIF instance supporting up- and downstreaming, another option is to deliver the elementary streams requested by the MUTechMessage Handler multiplexed together with main AV-Scene session.A.1.6 AV Stream Server

This module is essentially a placeholder in the architecture for a server that supplies scene nodes with MPEG-4 AV streams pointed to by object descriptors. This approach allows the use of unicast or multicast, local files etc. for distribution and sharing of streams between client terminals, without having to change the scene description (i.e. the same OD can be used between different clients).A.1.7 MUSession

The scene contains a MUSession node. Importantly the session node specifies the URL/Object Descriptor ID where the MuTech Session Controller can be contacted.

A.2 Rationale for supporting for transfer of pilotship

Transfering pilotship is a nice way of temporarily removing the network delay that clients experience when participating in a multi user session. It may be very usefull for some types of applications (e.g. design) and maybe also crucial. Combining the use of locks and moving the pilot may also have some nice benefits. (The concept of locking was removed, but it is possible to implement locking functionality as part of piloting an object).

The following are the signs used in the figures:

A.2.1 With the Pilot on the server

White circle is a shared object within a MUZone with pilot set to TRUE.

Blue circles are shared objects within MUZones with pilot set to FALSE

1. Client 1 initiates a change to a shared object. The node is a Drone.

2. The Drone sends a RequestChange message to the MBK, which passes it on to the corresponding Pilot node.


3. An UpdateMessage is sent from the Pilot to its Drones via the MBK, and the Pilot also updates itself.

The network delay for this operation is about the same as two times the network delay for both clients.

A.2.2 With the Pilot on client 1



1. Client 1 initiates a change to a shared object. The node is a Pilot.

2. The Pilot sends an UpdateMessage to the MBK, which passes it on to the corresponding Drone nodes. At the same time the Pilot also updates itself.

The delay for this operation is close to zero for client 1.

Client 2 will receive an update that is delayed approximately two times the network delay from server to client.


A.2.3 With the Pilot on client 2



1. Client 1 initiates a change to a shared object. The node is a Drone.

2. The Drone sends a RequestChange message to the MBK, which passes it on to the corresponding Pilot node (now at client 2).

3. An UpdateMessage is sent from the Pilot to its Drones via the MBK, and the Pilot also updates itself.

At client 1 the delay for this operation is about the same as four times the network delay.

Client 2 will receive the update delayed about two times the networks delay after the change was initiated on client 1.



Annex B

Node Coding TablesB.1 Node Tables

ColorProfile SFWorldNodeSFColorNode

0000011

Field name DEF id IN id OUT id DYN id [m, M] Q A

type MFInt32 000 000 000

leftWidth MFInt32 001 001 001

rightWidth MFInt32 010 010 010

leftBrightness1 MFColor 011 011 011

leftBrightness2 MFColor 100 100 100

rightBrightness1 MFColor 101 101 101

rightBrightness2 MFColor 110 110 110

centralBrightness MFColor 111 111 111

DepthImage SFWorldNodeSFGeometryNode

0000100001


position SFVec3f 000

orientation SFRotation 001

fieldOfView SFVec2f 010

nearPlane SFFloat 011

farPlane SFFloat 100

orthographic SFBool 101

diTexture SFNode 110

depthImageUrl SFString 111

Ellipse SFWorldNodeSFGeometryNode

0000110010



radius SFVec2f 0 0 0

FFD SFWorldNodeSF3DNode

0001000001


addChildren SFNode 00

removeChildren SFNode 01

children SFNode 0000 10 0

uDimension SFInt32 0001

vDimension SFInt32 0010

wDimension SFInt32 0011

uKnot MFFloat 0100

vKnot MFFloat 0101

wKnot MFFloat 0110

uOrder SFInt32 0111

vOrder SFInt32 1000

wOrder SFInt32 1001

controlPoint MFVec4f 1010 11 1

GradientLinear SFWorldNodeSFMaterialNode

00010101


startPoint SFVec2f 000 000 000

endpoint SFFloat 001 001 001

spreadMethod SFInt32 010 010 010

key MFFloat 011 011 011

keyValue MFColor 100 100 100

opacity MFFloat 101 101 101


GradientRadial SFWorldNodeSFMaterialNode

00011010


center SFVec2f 000 000 000

radius SFFloat 001 001 001

focalPoint SFVec2f 010 010 010

spreadMethod SFInt32 011 011 011


keyValue MFColor 101 101 101

opacity MFFloat 110 110 110

Implicit SFWorldNodeSFGeometryNode

0001110011


bboxSize SFVec3f 000 000 000

c MFFloat 001 001 001

solid SFBool 010 010 010

dual SFBool 011 011 011

densities MFInt32 100 100 100

LFM_Appearance SFWorldNodeSFAppearanceNode

0010001


tileList SFNode 00 00 00

lightMapList SFLightMapNode 01 01 01

blendList SFBlendListNode 10 10 10

vertexFrameList SFFrameListNode 11 11 11

LFM_BlendList SFWorldNodeSFBlendListNode

0010011



lightMapIndex MFInt32 0 0 0

blendMode MFInt32 1 1 1

LFM_FrameList SFWorldNodeSFFrameListNode

0010101


index MFInt32 0 0 0

frame MFVec3f 1 1 1

LFM_LightMap SFWorldNodeSFLightMapNode

0010111


surfaceMapList SFSurfaceMapNode 000 000 000

viewMapList SFViewMapNode 001 001 001

scaleRGB SFVec3f 010 010 010

biasRGB SFVec3f 011 011 011

priorityLevel SFInt32 100 100 100

LFM_SurfaceMapList SFWorldNodeSFSurfaceMapNode

0011001


triangleIndex MFInt32 00 00 00

tileIndex MFInt32 01 01 01

viewMapIndex MFInt32 10 10 10

triangleCoordinate SFTextureCoordinateNode 11 11 11

LFM_ViewMapList SFWorldNodeSFViewMapNode

0011011


vertexIndex MFInt32 00 00 00

tileIndex MFInt32 01 01 01


textureOrigin SFTextureCoordinateNode 10 10 10

textureSize SFTextureCoordinateNode 11 11 11

MeshGrid SFWorldNodeSF3DNode

0011100010


set_coordIndex MFInt32 0000

set_normalIndex MFInt32 0001

set_texCoordIndex MFInt32 0010

set_colorIndex MFInt32 0011

gridCoord SFCoordinateNode 0000 0100 0000

vertexOffset MFFloat 0001 0101 0001

vertexLink MFInt32 0010 0110 0010

coord SFCoordinateNode 0011 0111 0011

normal SFNode 0100 1000 0100

texCoord SFNode 0101 1001 0101

color SFNode 0110 1010 0110

displayLevel SFInt32 0111 1011 0111

hierarchicalLevel SFInt32 1000 1100 1000

nrLevels SFInt32 1001

nrSlices MFInt32 1010

nrVertices SFInt32 1011

solid SFBool 1001

coordIndex MFInt32 1010

normalIndex MFInt32 1011

texCoordIndex MFInt32 1100

colorIndex MFInt32 1101


MUAvatar SFWorldNodeSF3DNode

0011110011


url MFURL 000 000 000

model SF3DNode 001 001 001

clientID SFInt32 010

position SFVec3f 011 010 010 0 ]-, +[ 1 1

orientation SFRotation 100 011 011 1 10 10

info MFString 101 100 100

MUAvatar2D SFWorldNodeSF2DNode

010000001


url MFURL 000 000 000

model SF2DNode 001 001 001

clientID SFInt32 010

position SFVec2f 011 010 010 0 ]-, +[ 2 2

orientationAngle SFFloat 100 011 011 1 [0, 2.] 6 6


MUSession

SFWorldNodeSF3DNodeSF2DNode

0100010100010


addChildren SF3DNode 000

removeChildren SF3DNode 001

children SF3DNode 00 010 000


enabled SFBool 10 100 010


url MFURL 11 101 011

isActive SFBool 100

MUZone

SFWorldNodeSF3DNodeSF2DNode

0100100101011





pilot SFBool 001 011 001



bboxCenter SFVec3f 100 110 100 0 ]-, +[ 1 1

bboxSize SFVec3f 101 111 101 1 [-1, +[ 11 1

isActive SFBool 110

NurbsCurve SFWorldNodeSFGeometryNode

0100110100


color SFColorNode 000 00 00


knot MFFloat 010

order SFInt32 011

tessellation SFInt32 100 10 10

NurbsCurve2D SFWorldNodeSFGeometryNode

0101000101





knot MFFloat 010

order SFInt32 011

tessellation SFInt32 100 10 10

NurbsSurface SFWorldNodeSFGeometryNode

0101010110



texCoord SFTextureCoordinateNode 0001 001 001

ccw SFBool 0010


solid SFBool 0100

uDimension SFInt32 0101

vDimension SFInt32 0110

uKnot MFFloat 0111

vKnot MFFloat 1000

uOrder SFInt32 1001

vOrder SFInt32 1010

uTessellation SFInt32 1011 011 011

vTessellation SFInt32 1100 100 100

OctreeImage SFWorldNodeSF3DNode

0101100110


octreeresolution SFInt32 00

octree SFString 01

octreeimages SFNode 10

octreeUrl SFString 11


Parabola SFWorldNodeSFGeometryNode

0101110111


point MFVec3f 000 000 000

type MFInt32 001 001 001


colorIndex MFInt32 011 011 011

separatingFlags MFBool 100 100 100

contourFlags MFInt32 101 101 101

controlFlags MFInt32 110 110 110

colorPerVertex SFBool 111 111 111

Particles SFWorldNodeSF3DNode

0110000111



particleRadius SFFloat 00001 00001 00001

particleRadiusVariation SFFloat 00010 00010 00010

particleRadiusRate SFFloat 00011 00011 00011

emitterPosition SFVec3f 00100 00100 00100

emitVelocity SFVec3f 00101 00101 00101

emitVelocityVariation SFFloat 00110 00110 00110

creationRate SFFloat 00111 00111 00111

creationRateVariation SFFloat 01000 01000 01000

maxParticles SFInt32 01001 01001 01001

maxLifeTime SFTime 01010 01010 01010

maxLifeTimeVariation SFFloat 01011 01011 01011

gravity SFVec3f 01100 01100 01100


acceleration SFVec3f 01101 01101 01101

emitColor SFColor 01110 01110 01110

emitAlpha SFFloat 01111 01111 01111

emitColorVariation SFFloat 10000 10000 10000

fadeColor SFColor 10001 10001 10001

fadeAlpha SFFloat 10010 10010 10010

fadeRate SFFloat 10011 10011 10011

minRange SFFloat 10100 10100 10100

maxRange SFFloat 10101 10101 10101

primitiveType SFInt32 10110 10110 10110

primitive SF3DNode 10111 10111 10111

init SFParticleInitializerNode 11000 11000 11000

obstacle SFParticleObstacleNode 11001 11001 11001

ParticleInitBox SFWorldNodeSFParticleInitializerNode

01100101


size SFVec3f 0 0 0

falloff SFFloat 1 1 1

ParticleInitCone SFWorldNodeSFParticleInitializerNode

01101010


direction SFVec3f 00 00 00

spread SFFloat 01 01 01

falloff SFFloat 10 10 10

PlanarObstacle SFWorldNodeSFParticleObstacleNode

01101101



distance SFFloat 00 00 00

normal SFVec3f 01 01 01

reflection SFFloat 10 10 10

PointAttractor SFWorldNodeSFParticleObstacleNode

01110010


radius SFFloat 00 00 00

position SFVec3f 01 01 01

rate SFFloat 10 10 10

PointTexture SFWorldNodeSF3DNode

0111011000


width SFInt32 00

height SFInt32 01

depth MFInt32 10

color MFColor 11

PositionAnimator SFWorldNodeSF3Dnode

01111001


set_fraction SFFloat 0000

fromTo SFVec2f 0000 0001 0000

key MFFloat 0001 0010 0001

keyType SFInt32 0010 0011 0010

keySpline MFVec2f 0011 0100 0011

keyValue MFVec3f 0100 0101 0100

keyOrientation MFRotation 0101 0110 0101

weight MFFloat 0110 0111 0110


keyValueType SFInt32 0111 1000 0111

offset SFVec3f 1000 1001 1000

value_changed SFVec3f 1001

rotation_changed SFRotation 1010

endValue SFVec3f 1011

PositionAnimator2D SFWorldNodeSF2DNode

011111100




key MFFloat 0001 0010 0001



keyValue MFVec2f 0100 0101 0100

keyOrientation SFInt32 0101 0110 0101



offset SFVec2f 1000 1001 1000

value_changed SFVec2f 1001

rotation_changed SFFloat 1010

endValue SFVec2f 1011

ProceduralTexture SFWorldNodeSFTextureNode

10000001


type SFInt32 00000 00000 00000

width SFInt32 00001 00001 00001

height SFInt32 00010 00010 00010


cellWidth SFInt32 00011 00011 00011

cellHeight SFInt32 00100 00100 00100

roughness SFInt32 00101 00101 00101

distortion SFFloat 00110 00110 00110

seed SFInt32 00111 00111 00111

color MFColor 01000 01000 01000

xWarpmap MFVec2f 01001 01001 01001

yWarpmap MFVec2f 01010 01010 01010

xSmooth SFBool 01011 01011 01011

ySmooth SFBool 01100 01100 01100

aWarpmap MFVec2f 01101 01101 01101

bWarpmap MFVec2f 01110 01110 01110

aSmooth SFBool 01111 01111 01111

bSmooth SFBool 10000 10000 10000

aWeights MFFloat 10001 10001 10001

bWeights MFFloat 10010 10010 10010

Quadric SFWorldNodeSFGeometryNode

1000011000



P0 SFVec4f 0001 0001 0001

P1 SFVec4f 0010 0010 0010

P2 SFVec4f 0011 0011 0011

P3 SFVec4f 0100 0100 0100

P4 SFVec4f 0101 0101 0101

P5 SFVec4f 0110 0110 0110

solid SFBool 0111 0111 0111


dual SFBool 1000 1000 1000

densities MFInt32 1001 1001 1001

SBBone SFWorldNodeSF3DNode

1000101001


boneID SFInt32 00000 00000 00000

skinCoordIndex MFInt32 00001 00001 00001

skinCoordWeight MFFloat 00010 00010 00010

endpoint SFVec3f 00011 00011 00011

falloff SFInt32 00100 00100 00100

sectionPosition MFFloat 00101 00101 00101

sectionInner MFFloat 00110 00110 00110

sectionOuter MFFloat 00111 00111 00111

rotationOrder SFInt32 01000 01000 01000


center SFVec3f 01010 01010 01010

rotation SFRotation 01011 01011 01011

translation SFVec3f 01100 01100 01100

scale SFVec3f 01101 01101 01101

scaleOrientation SFRotation 01110 01110 01110

IkchainPosition SFInt32 01111 01111 01111

IkyawLimit MFFloat 10000 10000 10000

IkpitchLimit MFFloat 10001 10001 10001

IkrollLimit MFFloat 10010 10010 10010

IKTxLimit MFFloat 10011 10011 10011

IKTyLimit MFFloat 10100 10100 10100


IKTzLimit MFFloat 10101 10101 10101

SBMuscle SFWorldNodeSF3DNode

1000111010


skinCoordIndex MFInt32 000 000 000

skinCoordWeight MFFloat 001 001 001

muscleCurve SFGeometryNode 010 010 010

radius SFInt32 011 011 011

falloff SFInt32 100 100 100

SBSegment SFWorldNodeSF3DNode

1001001011




name SFString 000 010 000

centerOfMass SFVec3f 001 011 001

momentsOfInertia SFVec3f 010 100 010

mass SFFloat 011 101 011


SBSite SFWorldNodeSF3DNode

1001011100









scale SFVec3f 100 0110 100



SBSkinnedModel SFWorldNodeSF3DNode

1001101101






scale SFVec3f 0100 0100 0100


skin SFNode 0110 0110 0110

skinCoord SFNode 0111 0111 0111

skinNormal SFNode 1000 1000 1000

skeleton SFNode 1001 1001 1001

bones SFNode 1010 1010 1010

muscles SFNode 1011 1011 1011

segments SFNode 1100 1100 1100

sites SFNode 1101 1101 1101

weighsComputationSkinCoord SFNode 1110 1110 1110

ScalarAnimator SFWorldNodeSF3DnodeSF2DNode

10011110101








keyValue MFFloat 100 0101 0100



offset SFFloat 111 1000 0111

value_changed SFFloat 1000

endValue SFFloat 1001

SolidRep SFWorldNodeSFGeometryNode

1010001001



solidTree SFNode 01 01 01

densityList MFInt32 10 10 10

SubdivisionSurface SFWorldNodeSFGeometryNode

1010011010


set_colorIndex MFInt32 0000

set_coordIndex MFInt32 0001

set_texCoordIndex MFInt32 0010

set_creaseEdgeIndex MFInt32 0011

set_cornerVertexIndex MFInt32 0100

set_creaseVertexIndex MFInt32 0101

set_dartVertexIndex MFInt32 0110



coord SFNode 00001 1000 001

texCoord SFNode 00010 1001 010

subdivisonType SFInt32 00011 1010 011

subdivisonSubType SFInt32 00100 1011 100

invisibleEdgeIndex MFInt32 00101

subdivisonLevel SFInt32 00110 1100 101

ccw SFBool 00111

colorIndex MFInt32 01000

colorPerVertex SFBool 01001

convex SFBool 01010

coordIndex MFInt32 01011

solid SFBool 01100

texCoordIndex MFInt32 01101

creaseEdgeIndex MFInt32 01110

dartVertexIndex MFInt32 01111

creaseVertexIndex MFInt32 10000

cornerVertexIndex MFInt32 10001

sectors SFNode 10010 1101 110

SubdivSurfaceSector SFWorldNodeSFSubdivSurfaceSectorNode

1010101


faceIndex SFInt32 000

vertexIndex SFInt32 001

tag SFInt32 010 000 000

flatness SFFloat 011 001 001

theta SFFloat 100 010 010


normal SFVec3f 101 011 011

normalTension SFFloat 110 100 100

SynthesizedTexture SFWorldNodeSFTextureNode

10101110




children SFNode 000 010 000

pixelWidth SFInt32 001 011 001

pixelHeight SFInt32 010 100 010

background SFNode 011 101 011

viewport SFNode 100 110 100

WaveletSubdivisionSurface SFWorldNodeSFGeometryNode

1011001011


backChannel MFString 000

baseMesh SFBaseMeshNode 001 00 00

detailsStream MFString 010

frequency SFFloat 011 01 01

fieldOfView SFFloat 100 10 10

quality SFInt32 101 11 11

B.2 Node Definition Type Tables

SF2DNode 5 nodes 3 bits

Node name nodeID # DEF # IN # OUT # DYN

reserved 000


MUAvatar2D 001 6 3b 5 3b 5 3b 2 1b

MUSession 010 4 2b 6 3b 5 3b 0 0b

MUZone 011 6 3b 8 3b 7 3b 2 1b

PositionAnimator2D 100 9 4b 10 4b 12 4b 0 0b

ScalarAnimator 101 8 3b 9 4b 10 4b 0 0b

SF3DNode 13 nodes 4 bits


reserved 0000

FFD 0001 11 4b 4 2b 2 1b 0 0b

MeshGrid 0010 14 4b 13 4b 12 4b 0 0b

MUAvatar 0011 6 3b 5 3b 5 3b 2 1b

MUSession 0100 4 2b 6 3b 5 3b 0 0b

MUZone 0101 6 3b 8 3b 7 3b 2 1b

OctreeImage 0110 4 2b 0 0b 0 0b 0 0b

Particles 0111 26 5b 26 5b 26 5b 0 0b

PointTexture 1000 4 2b 0 0b 0 0b 0 0b

SBBone 1001 22 5b 22 5b 22 5b 0 0b

SBMuscle 1010 5 3b 5 3b 5 3b 0 0b

SBSegment 1011 5 3b 7 3b 5 3b 0 0b

SBSite 1100 7 3b 9 4b 7 3b 0 0b

SBSkinnedModel 1101 15 4b 15 4b 15 4b 0 0b

SF3Dnode 2 nodes 2 bits


reserved 00


PositionAnimator 01 9 4b 10 4b 12 4b 0 0b


SFAppearanceNode 1 node 1 bit


reserved 0

LFM_Appearance 1 4 2b 4 2b 4 2b 0 0b

SFBlendListNode 1 node 1 bit


reserved 0

LFM_BlendList 1 2 1b 2 1b 2 1b 0 0b

SFColorNode 1 node 1 bit


reserved 0

ColorProfile 1 8 3b 8 3b 8 3b 0 0b

SFFrameListNode 1 node 1 bit


reserved 0

LFM_FrameList 1 2 1b 2 1b 2 1b 0 0b

SFGeometryNode 11 nodes 4 bits


reserved 0000


DepthImage 0001 8 3b 0 0b 0 0b 0 0b

Ellipse 0010 1 1b 1 1b 1 1b 0 0b

Implicit 0011 5 3b 5 3b 5 3b 0 0b

NurbsCurve 0100 5 3b 3 2b 3 2b 0 0b

NurbsCurve2D 0101 5 3b 3 2b 3 2b 0 0b

NurbsSurface 0110 13 4b 5 3b 5 3b 0 0b

Parabola 0111 8 3b 8 3b 8 3b 0 0b

Quadric 1000 10 4b 10 4b 10 4b 0 0b

SolidRep 1001 3 2b 3 2b 3 2b 0 0b

SubdivisionSurface 1010 19 5b 14 4b 7 3b 0 0b

WaveletSubdivisionSurface 1011 6 3b 4 2b 4 2b 0 0b

SFLightMapNode 1 node 1 bit


reserved 0

LFM_LightMap 1 5 3b 5 3b 5 3b 0 0b

SFMaterialNode 2 nodes 2 bits


reserved 00

GradientLinear 01 6 3b 6 3b 6 3b 0 0b

GradientRadial 10 7 3b 7 3b 7 3b 0 0b

SFParticleInitializerNode 2 nodes 2 bits


reserved 00


ParticleInitBox 01 2 1b 2 1b 2 1b 0 0b

ParticleInitCone 10 3 2b 3 2b 3 2b 0 0b

SFParticleObstacleNode 2 nodes 2 bits


reserved 00

PlanarObstacle 01 3 2b 3 2b 3 2b 0 0b

PointAttractor 10 3 2b 3 2b 3 2b 0 0b

SFSubdivSurfaceSectorNode 1 node 1 bit


reserved 0

SubdivSurfaceSector 1 7 3b 5 3b 5 3b 0 0b

SFSurfaceMapNode 1 node 1 bit


reserved 0

LFM_SurfaceMapList 1 4 2b 4 2b 4 2b 0 0b

SFTextureNode 2 nodes 2 bits


reserved 00

ProceduralTexture 01 19 5b 19 5b 19 5b 0 0b

SynthesizedTexture 10 5 3b 7 3b 5 3b 0 0b

SFViewMapNode 1 node 1 bit



reserved 0

LFM_ViewMapList 1 4 2b 4 2b 4 2b 0 0b

SFWorldNode 44 nodes 6 bits


reserved 000000

ColorProfile 000001 8 3b 8 3b 8 3b 0 0b

DepthImage 000010 8 3b 0 0b 0 0b 0 0b

Ellipse 000011 1 1b 1 1b 1 1b 0 0b

FFD 000100 11 4b 4 2b 2 1b 0 0b

GradientLinear 000101 6 3b 6 3b 6 3b 0 0b

GradientRadial 000110 7 3b 7 3b 7 3b 0 0b

Implicit 000111 5 3b 5 3b 5 3b 0 0b

LFM_Appearance 001000 4 2b 4 2b 4 2b 0 0b

LFM_BlendList 001001 2 1b 2 1b 2 1b 0 0b

LFM_FrameList 001010 2 1b 2 1b 2 1b 0 0b

LFM_LightMap 001011 5 3b 5 3b 5 3b 0 0b

LFM_SurfaceMapList 001100 4 2b 4 2b 4 2b 0 0b

LFM_ViewMapList 001101 4 2b 4 2b 4 2b 0 0b

MeshGrid 001110 14 4b 13 4b 12 4b 0 0b

MUAvatar 001111 6 3b 5 3b 5 3b 2 1b

MUAvatar2D 010000 6 3b 5 3b 5 3b 2 1b

MUSession 010001 4 2b 6 3b 5 3b 0 0b

MUZone 010010 6 3b 8 3b 7 3b 2 1b

NurbsCurve 010011 5 3b 3 2b 3 2b 0 0b


NurbsCurve2D 010100 5 3b 3 2b 3 2b 0 0b

NurbsSurface 010101 13 4b 5 3b 5 3b 0 0b

OctreeImage 010110 4 2b 0 0b 0 0b 0 0b

Parabola 010111 8 3b 8 3b 8 3b 0 0b

Particles 011000 26 5b 26 5b 26 5b 0 0b

ParticleInitBox 011001 2 1b 2 1b 2 1b 0 0b

ParticleInitCone 011010 3 2b 3 2b 3 2b 0 0b

PlanarObstacle 011011 3 2b 3 2b 3 2b 0 0b

PointAttractor 011100 3 2b 3 2b 3 2b 0 0b

PointTexture 011101 4 2b 0 0b 0 0b 0 0b

PositionAnimator 011110 9 4b 10 4b 12 4b 0 0b

PositionAnimator2D 011111 9 4b 10 4b 12 4b 0 0b

ProceduralTexture 100000 19 5b 19 5b 19 5b 0 0b

Quadric 100001 10 4b 10 4b 10 4b 0 0b

SBBone 100010 22 5b 22 5b 22 5b 0 0b

SBMuscle 100011 5 3b 5 3b 5 3b 0 0b

SBSegment 100100 5 3b 7 3b 5 3b 0 0b

SBSite 100101 7 3b 9 4b 7 3b 0 0b

SBSkinnedModel 100110 15 4b 15 4b 15 4b 0 0b


SolidRep 101000 3 2b 3 2b 3 2b 0 0b

SubdivisionSurface 101001 19 5b 14 4b 7 3b 0 0b

SubdivSurfaceSector 101010 7 3b 5 3b 5 3b 0 0b

SynthesizedTexture 101011 5 3b 7 3b 5 3b 0 0b

WaveletSubdivisionSurface 101100 6 3b 4 2b 4 2b 0 0b


References

1. ISO/IEC 14496-1, Coding of Audio-Visual Objects: Systems.

2. ISO/IEC 14496-2, Coding of Audio-Visual Objects: Visual.

3. ISO/IEC 14496-3, Coding of Audio-Visual Objects: Audio.

4. ISO/IEC 14496-6, Coding of audio-visual objects, Part 6: Delivery Multimedia Integration Framework (DMIF)

5. ISO/IEC 14772-1, The Visual Reality Modeling Language (VRML), 1997, http://www.web3d.org/technicalinfo/specifications/vrml97/index.htm

6. ISO/IEC 15444-1 “JPEG 2000 Image Coding System,”, December 1999.7. ITU-T Rec. H.323, Packet-Based Multimedia Communications Systems, September 1999

8. Virtual Reality Transfer Protocol, http://www.web3d.org/WorkingGroups/vrtp/

9. “Scalar Vector Graphics (SVG) 1.0.” W3C Recommendation. http://www.w3.org/TR/2000/CR-SVG-20000802/

10. “Synchronized Multimedia Integration Language (SMIL) 2.0.” W3C Recommendation. . http://www.w3.org/AudioVideo/

11. Living Worlds Proposal Draft 2, http://www.vrml.org/WorkingGroups/living-worlds/draft_2/index.htm

12. Web3D Consortium, The Humanoid Animation Specification, http://www.h-anim.org

13. Blaxxun Interactive. NURBS extension for VRML97. http://www.blaxxun.com/developer/contact/3d/nurbs/overview.html. April 1999.

14. Intel 3D software toolkit, http://www.intel.com

15. Abrash Michael. “Michael Abrash’s graphics programming black book, special edition”. The coriolis group, Inc.,Scottsdale, Arizona, 1997.

16. Abrash Michael. “Quake’s lighting model: surface caching” Dr. Dobb’s sourcebook #260, November/December, 1996

17. Bajaj C., Coyle E., and Lin K.. Arbitrary Topology Shape Reconstruction from Planar Cross Sections. In Graphical Models and Image Processing, 58(6):524-543, 1996.

18. Barr A.H. “Global and Local deformations of solid primitives.” Computer Graphics, 18(3), 21-30, Proc. Siggraph ’84.

19. Beeson Curtis. “X3D Multitexture”. http://www.web3d.org/TaskGroups/x3d/quadramix/multitexture.htm. May 1999.

20. Biermann H., Levin A., Zorin D.. Piecewise-smooth subdivision surfaces. SIGGRAPH 2000 Conference Proceedings. New Orleans, Louisiana. July 23-28, 2000. pp. 113-120

21. Bishop G., Oliveira M.M., "Relief Textures" SIGGRAPH'2000 Proceedings

22. Blinn J.F. “Simulation of wrinkled surfaces” Computer Graphics, 123, pp. 286-292, 1978

23. Blinn J.F. “A generalization of algebraic surface drawing”. ACM Transactions on Graphics, 1(3):235-256, July 1982.


http://mrl.nyu.edu/publications/piecewise-smooth/

http://www.web3d.org/TaskGroups/x3d/quadramix/multitexture.htm

http://www.intel.com/

http://www.blaxxun.com/developer/contact/3d/nurbs/overview.html

http://www.vrml.org/WorkingGroups/living-worlds/draft_2/index.htm

http://www.w3.org/AudioVideo/

http://www.w3.org/TR/2000/CR-SVG-20000802/

http://www.web3d.org/technicalinfo/specifications/vrml97/index.htm

24. Bloomenthal J, Polygonization of Implicit Surfaces, Computer Aided Geometric Design, 5, 4 November 1988.

25. Bloomenthal J., “Calculation of reference frames along a space curve”, Graphic Gems, Academic Press, 1990: pp. 567-571

26. Briskin M., Elichai Y., Yomdin Y., “How can Singularity Theory help in Image Processing?”, In “Pattern Formation in Biology, Vision and Dynamics”, A. Carbone, M. Gromov and P. Prusinkiewitcz, Editors, World Scientific Publishers, 1999., pp. 392 – 423.

27. Catmull E. and Clark J. Recursively generated B-spline surfaces on arbitrary topological meshes. Computer-Aided Design, 10:350-355, September 1978.Coquillard, Sabine. Extended Free-Form Deformation: a sculpturing tool for 3D geometric modeling. INRIA, RR-1250, June 1990.

28. Chaikin G.M. “An algorithm for high speed curve generation”. Computer Graphics Image Processing, 3:346-349, 1974.29. Chen S.E. Quicktime VR - Image-Based Approach to Virtual Environment Navigation, Proceedings of

SIGGRAPH 95, Computer Graphics Proceedings, Annual Conference Series, pp. 29-38 (August 1995, Los Angeles, California). Addison Wesley. Edited by Robert Cook. ISBN 0-201-84776-0.

30. Chen W-C., Grzeszczuk R., Bouguet J-Y. Light Field Mapping: Hardware-Accelerated Visualization of Surface Light Fields. Part of “Acquisition and Visualization of Surface Light Fields,'' SIGGRAPH 2001 Course Notes for Course #46.

31. Coquillard, Sabine. Extended Free-Form Deformation: a sculpturing tool for 3D geometric modeling. INRIA, RR-1250, June 1990.

32. Craig John J.. Introduction to Robotics: Mechanics and Control, 2nd edition. Addison-Wesley, Reading, MA, 1989.

33. Debevec P., “Introduction to Image-Based Modeling, Rendering, and Lighting”. Siggraph'99 courses, 1999 ( http://www.debevec.org/IBMR99/)

34. Debevec P.E., Taylor C.J. and Malik J. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach, Proceedings of SIGGRAPH 96, Computer Graphics Proceedings, Annual Conference Series, pp. 11-20 (August 1996, New Orleans, Louisiana). Addison Wesley. Edited by Holly Rushmeier. ISBN 0-201-94800-1.

35. Debevec P.E., Yu Y. and Borshukov G.D. Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping, Eurographics Rendering Workshop 1998, pp. 105-116 (June 1998, Vienna, Austria). Eurographics. Edited by George Drettakis and Nelson Max. ISBN 3-211-83213-0.

36. Demers Joe. “X3D ParticleSet”. http://www.web3d.org/TaskGroups/x3d/quadramix/x3dparticelset.htm. June 1999.

37. DeRose T., Kass M. and Truong T. Subdivision surfaces in character animation. In SIGGRAPH’98 Conference Proceedings, ACM, Addison Wesley, pp. 85-94, July 1998.

38. Diehl Stephen, “VRML++: a language for object-oriented virtual-reality models”, Proc. 24th International Conference on Technology of Object-Oriented Languages and Systems, Beijing, 1997

39. Doo D. and Sabin M. “Behaviour of recursive division surfaces near extraordinary points”. Computer-Aided Design, 10(6):356-360, November 1978.

40. Dyn N., Levin D. and Gregory J. A. “A four-point interpolatory subdivision scheme for curve design”, Computer-Aided Geometric Design, 4:257-268, 1987.

41. Dyn N., Levin D. and Gregory J. A. “A Butterfly Subdivision Scheme for Surface Interpolation with Tension Control”, ACM Transactions on Graphics, 9(2):160-169, April 1990.

42. Dyn N., Levin D. and Liu D. “Interpolatory convexity-preserving subdivision schemes for curves and surfaces”, Computer-Aided Design, 24(4):211-216, April 1992.


http://www.web3d.org/TaskGroups/x3d/quadramix/multitexture.htm

http://www.debevec.org/IBMR99/

http://www.intel.com/research/mrl/research/lfm/course_notes_lfm.pdf

http://www.intel.com/research/mrl/research/lfm/course_notes_lfm.pdf

43. Eberly, D.H. 3D game engine design, Academic Press, 2001

44. Farin G., Curves and surfaces for computer aided design, 2nd edition, Academic Press, Boston 1990.

45. Fuchs H., Kedem Z. M., and Uselton S. P.. Optimal Surface Reconstruction from Planar Contours. In Communicarions of the ACM, 20(10):693-702, October 1977.

46. Funge. J.D. “AI for Computer Games and Animation: A Cognitive Modeling Approach”. A K Peters Ltd, August 1999.

47. Gersho A., Gray R. M. Vector Quantization and Signal Compression. Kluwer Academic Publishers, 1992.

48. Grahn Holger. “Cell and Portal proposal” blaxxun interactive. http://www.blaxxun.com/developer/contact/3d/vrml/samples/features/cell_portal/celllportal.html.

49. Grahn Holger "Particles", blaxxun interactive http://www.blaxxun.com/developer/contact/3d/vrml/samples/features/particles/particles.html

50. Hoppe Hugues, DeRose Tony, Duchamp Tom, Halstead Mark, Jin Huber, McDonald John, Schweitzer Jean, and Stuetzle Werner. Piecewise smooth surface reconsruction. In Computer Graphics Proceedings, Annual Conference Series, pages 295–302. ACM Siggraph, 1994.

51. Johnstone J.K. and Williams J.P. A rational quaternion spline of arbitrary continuity. Tech. Report.http://www.cis.uab.edu/info/faculty/jj/cos.html, 1999

52. Keppel E. Approximating complex surfaces by triangulation of contour lines. In IBM Journal of Research and Development, 19(1):2-11, January 1975.

53. Kleene S.C. Logique Mathématique. Éditions Jacques Gabay, Paris, 1967.

54. Kleene S.C. Introduction to Metamathematics. Wolters-Noordhoff Publishing, Groningen, 1971.

55. Kobbelt L. “ 3 -Subdivision”. SIGGRAPH’00 Conference Proceedings, pp. 103-112, July 2000.

56. Kochanek D.H.U. “Interpolating splines with local tension, continuity, and bias control”. Computer Graphics, 18(3): 33-41, Proc. Siggraph ’84.

57. Khodakovsky A., Schröder P., and Sweldens W.. Progressive Geometry Compression. SIGGRAPH 2000 proceedings

58. Lachaud J. O.. Extraction de surfaces à partir d’images tridimensionnelles: approches discrete et approche par modèle deformable. Ph.D. Thesis. July 1998.

59. Lamport Leslie, “Time, Clocks and the Ordering of Events in a Distributed System,” Communications of the ACM, 21(7), July 1978, pages 558-565, QA75.A68.

60. Levoy M., Rusinkiewicz S., "QSplat: A Multiresolution Point Rendering system for Large Meshes". SIGGRAPH'2000 Proceedings

61. Loop C. Smooth subdivision surfaces based on triangles. Master’s thesis, Department of Mathematics, University of Utah, August 1987.

62. Lorensen W. E. and Cline H. E.. Marching Cubes: A high resolution 3D surface construction algorithm. In M. C. Stone, editor, Computer Graphics (SIGGRAPH ’87 Proceedings), volume 21, pages 163-169, 1987

63. Lounsbery M., DeRose T., and Warren J.. Multiresolution Analysis for Surfaces of Arbitrary Topological Type. ACM Trans. on Graphics, 16(1), Jan. 1997.

64. Luebke David and Georges Chris Department of Computer Science University of North Carolina at Chapel Hill . "Portals and Mirrors: Simple, Fast Evaluation of Potentially Visible Sets" http://www.cs.virginia.edu/~luebke/publications/portals.html

65. Mallat S., "A Theory for Multiresolution Signal Decomposition: The Wavelet Representation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 7, pp. 674-693, July 1989.


http://www.cs.virginia.edu/~luebke/publications/portals.html

http://www.blaxxun.com/developer/contact/3d/

http://www.blaxxun.com/developer/contact/3d/vrml/samples/features/cell_portal/celllportal.html

66. Mignot A., Bourges-Sevenier M., Rotgé J.-F., -Solid Model Representation, AFX proposal m7729, ISO/IEC/JTC1/SC29WG11 MPEG2001, Pattaya, December 2001

67. Moller Tomas and Haines Eric, “Real-time rendering”. A K Peters Ltd., June 1999.

68. Morán F. and García N. Hierarchical Coding of 3D Models with Subdivision Surfaces. IEEE ICIP 2000 proceedings.

69. Munteanu A., Cornelis J., and Cristea P., “Wavelet-Based Lossless Compression of Coronary Angiographic Images,” IEEE Transactions on Medical Imaging, 18(3): 272-281, 1999.

70. Munteanu A., Cornelis J., Van der Auwera G., and Cristea P., “A wavelet based lossless compression scheme with progressive transmission capability”, Int. J. of Imaging Systems and Technology, John Wiley&Sons, 10(1): 76-85, 1999.

71. Munteanu A., Cornelis J., Van der Auwera G., and Cristea P., "Wavelet Image Compression - The Quadtree Coding Approach," IEEE Transactions on Information Technology in Biomedicine, vol. 3, no. 3, pp. 176-185, September 1999.

72. Nishimura H. and al. “Object modeling by distribution function and a method of image generation”. Trans. IECE Japan, Part D, J68-D(4):718-725, 1985.

73. Nishino K., Sato Y. and Ikeuchi K. Eigen-Texture Method: Appearance Compression based on 3D Model. IEEE Computer Vision and Pattern Recognition, pp. I:618-624.

74. Perlin K. and Hoffert E.M.. “Hypertextures”. Conference proceedings on Computer graphics, 1989, pp. 253 -262

75. Piegl and Tiller. The NURBS book. Springer-Verlag. 1997

76. Poston Tim, Nguyen H. T., Heng Pheng-Ann, and Wong Tien-Tsin. Skeleton Climbing: Fast Isosurfaces with Fewer Triangles. In Proceedings of Pacific Graphics'97, pages 117-126, Seoul, Korea, October, 1997.

77. Poston Tim, Wong Tien-Tsin and Heng Pheng-Ann. Multiresolution Isosurface Extraction with Adaptive Skeleton Climbing. In Computer Graphics Forum, 17(3):137-148, September 1998.

78. Press W.H., Teukolsky S.A., Vetterling W.T., Flannery B.P. “Numerical Recipes in C”, Cambridge University Press, 2nd Edition, 1992.

79. Roehl Bernie. “Regions” http://ece.uwaterloo.ca/~broehl/vrml/regions.html

80. Rotgé, J.-Fr. L’arithmétique des formes : une introduction à la logique de l’espace, PhD thesis, Université de Montréal, Montréal, 1997.

81. Rotgé, J.-F., Principles of solid geometry design logic,p 233-254, in Proceedings of the CSG 96 Conference, Winchester, UK, 17-19 April 1996.

82. Rotgé, J.-Fr. Geometry Manual. SGDL Systems Inc., Montreal, 2001.

83. Said A. and Pearlman W.A.. Image compression using the spatial-orientation tree. Proc. IEEE Int. Symp. Circuits and Syst., Chicago, IL, pages 279-- 282, May 1993.

84. Said A. and Pearlman W., "A New Fast and Efficient Image Codec Based on Set Partitioning in Hierarchical Trees," IEEE Transactions on Circuits and Systems for Video Technology, vol. 6, no. 3, pp. 243-250, 1996.

85. Samet H. The Design and Analysis of Spatial Data Structures. Addison-Wesley Publishing Company, Reading, Massachusetts, 1990.

86. Schweitzer J. E. “Analysis and Application of Subdivision Surfaces”, PhD thesis, University of Washington, August 1996.

87. Sederberg and Parry. Free-Form Deformation of solid geometric models. SIGGRAPH’96, vol 20, pp. 151-160. ACM, August 1996.

88. SGDL Systems Inc, Reference Manual, SGDL Systems Inc., Montreal, 2001

89. Shade J., Gortler S., Hey L., Szeliski R., "Layered Depth Images". SIGGRAPH'97 Proceedings


http://ece.uwaterloo.ca/~broehl/vrml/regions.html

90. Shapiro J.M., "Embedded Image Coding Using Zerotrees of Wavelet Coefficients," IEEE Transactions on Signal Processing, vol. 41, no. 12, pp. 3445-3462, 1993.

91. Shoemake Ken, ”Euler Angle Conversion“, "Gramphics Gems IV", Academic Press Professional, Toronto, 1994

92. Signoroni A. and Leonardi R., "Progressive ROI Coding and Diagnostic Quality for Medical Image Compression," Proceedings of SPIE, Vol. 3309, pp. 674-685, 1997.

93. Singhal S.K. and Cheridon D.R. “Exploiting position history for efficient remote rendering in networked VR”. In Presence: Teleoperators and Ves, 4(2):169-199.

94. Ström J. and Cosman P. C., "Medical image compression with lossless regions of interest," Signal Processing, vol. 59, no., pp. 155-171, 1997.

95. Taubin G. “A Signal Processing Approach to Fair Surface Design”. SIGGRAPH’95 Conference Proceedings, pp. 351-358, August 1995.

96. Vlachos A., Isidoro J. "Smooth C2 Quaternion-Based Flythrough Paths", Game Programming Gems 2, Charles River Media, 2001

97. Wallin A.. Constructing isosurfaces from CT data. In IEEE Computer Graphics and Applications, 11(6)28-33, November 1991.

98. Warren J. “Subdivision Methods for Geometric Design”. Technical Report (110 pages), Rice University, November 1995.

99. Watt A. and Watt M., Advanced animation and rendering techniques. ACM Press, 1992.

100.Williams L. “Pyramidal parametrics”, Computer Graphics, 17(3):1-11, Proc. Siggraph ’83.

101.Wood D. N., Azuma D. I., Aldinger K., Curless B., Duchamp T., Salesin D. H., Stuetzle W. Surface Light Fields for 3D Photography, Proceedings of SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, pp. 287-296 (July 2000). ACM Press / ACM SIGGRAPH / Addison Wesley Longman. Edited by Kurt Akeley. ISBN 1-58113-208-5.

102.Wyvill Brian and Kees van Overveld. "Metamorphosis of Boolean Compound Soft Objects". Implicit Surfaces '96, Eurographics, 1996.

103.Yu T., Lin N., Liu S. J., and Chan A. K., "A Region-of-Interest Based Transmission Protocol for Wavelet-Compressed Medical Images," Proceedings of SPIE Conference on Wavelet Applications IV, Vol. 3078, pp. 56-64, April 1997.

104.Zonenschein L. et al.,Texturing Composite Deformable Implicit Objects, Proceedings of SIBGRAPI ’98, http://www.visgraf.impa.br/RefBib/Data/PS_PDF/texture-sib98/SLE085.pdf

105.Zorin D., Schröder P. and Sweldens W. “Interpolating Subdivision for Meshes with Arbitrary Topology”, SIGGRAPH’96 Conference Proceedings, pp. 189-192, August 1996.

106.Zorin D. “Subdivision and Multiresolution Surface Representations”, PhD thesis, California Institute of Technology, January 1998 (submitted in September 1997).

107.Zorin D., Schröder P. et al. “Subdivision for Modeling and Animation”, SIGGRAPH’00 Conference Course Notes, July 2000.


Documents

ISO/IEC TC JTC 1/SC 29 N - IPSJ/ITSCJ · Web viewFor a convex corner , , where is the angle between the two crease edges spanning the sector, for concave corners,. For concave corners,