Multimodal Location Based ServicesSemantic 3D City Data as Virtual and

Augmented Reality

Jochen Wendel Alexander Simons

Andreas Koch

José Miguel Santana Agustín Trujillo

José Pablo Suárez

Collaboration

2

Outline• Project Goals

• Urban and energy-related data models

• Tools: The Glob3 Mobile framework

• Representing a CityGML model in 3D

• Multimodal approach

• Virtual Globe View

• Virtual Reality Mono

• Virtual Reality Stereo

• Augmented Reality

• Use case: Visualising solar irradiation

• Conclusions

3

Project Goals• Prototype an interactive and mobile system for exploring:

• Semantic building / urban representational models (e.g. CityGML)

• Energy related data sets.

• Explore the possibilities of multi-LoD maps, Virtual Reality and Augmented Reality for the visualisation of 3D urban models and temporal data-series.

• Create a iOS app, that serves as demonstration of the service.

4

Urban and energy-related data models• CityGML: Information model for urban objects representation.

• Extension of GML (from OGC)

• Classes, hierarchies, aggregations, relations, and properties.

• Used on 3D virtual cities, analysis, simulation…

• Extendable via Application Domain Extensions (ADE)

• Buildings represented in different levels of detail.

5

• Karlsruhe Model: CityGML LoD 2

• CityStats, Heat D./L.:Energy-related data per building

• SolarB: Temporal data-series representing energy simulations

6

Urban and energy-related data models

Urban and energy-related data models• Karlsruhe model

• +30.000 buildings [LoD2] (from a total set of 80.000).

• +230.000 surfaces [Grounds, Walls and Rooftops]

• Building parameters:

• Heat demand, Volume, GHG Emissions, solar gains, 50 demographic variables…

• High resolution textured models of emblematic buildings and monuments.

7

Glob3 Mobile• Open-source framework for Map & Virtual Globe Applications.

• Native implementations for iOS, Android and Web.

• Developed by CTIM (ULPGC) and IGO Software.

• Available: https://github.com/glob3mobile/g3m

• Automatic Multiplatform Porting

8

G3mFeatures

Manykindsofscenario

9

G3mFeatures2D/2.5D/3DInteractiveSymbology

10

CityGMLto3D

• OpenGL / WebGL only understand triangles!

• The data received via CityGML needs to be translated into a triangulated mesh model to be displayed.

• Model storage can be local or on a remote server.

• Karlsruhe model ~= 400 MB.

• Fetching, parsing and tessellation are executed on parallel threads.

11

12

CityGML Building

Surfaces

Energy DataBounding Vol. and

Render Info

IDs

• SAX Parsing of the CityGML documents:

• CityGMLBuilding class model:

CityGMLto3D:Parsing

STATICDATA

XML SAX Parsing

JSON Parsing

CITYGMLMODEL

CityGML Building

Surfaces

Energy DataBounding Vol. and

Render Info

IDs

• Creating a single triangular mesh of buildings.

• For a high number of models, batched drawing is essential [x49]

• Gouraud shading was used as illumination model.

13

VerticesTessellation

Semantical colouring

Visibility TestIndices

Normal Extraction Vertex Normals

Vertex ColorColour Palette Legend

CityGMLto3D:Tessellation

City Model Mesh

• Visibility test, eliminating shared walls.

• Proximity based limiting the O(n!) complexity.

• EPSG:4326 to Cartesian World Coordinates

• Addition of DEM displacement

• Ear Clipping Algorithm for wall tessellation

• Computationally expensive algorithm [O(n2)]

• 91% of tessellated surfaces are 4 edged.

14http://www.sunshine2k.de/coding/java/Polygon/Kong/

Kong.html

CityGMLto3D:Tessellation

• Classic GIS 3D map approach.

• Multitouch navigation.

• Buildings are selectable showing their associated data.

• OpenStreet Map as base layer. [Mapnik]

15

MultimodalApp:VirtualGlobe

MultimodalApp:VirtualGlobe

16

VIDEO INTERACTING WITH THE MAP

SELECTING BUILDINGAND CHANGING CITY

COLOURING

MultimodalApp:VirtualReality

• Sensors output of iOS, Android and Javascript.

• Control the UI rotations.

• Device location given by GPS.

17

GeolocalizedReferenceSystem

Multiplatformdevice-attituderepresentations

MultimodalApp:VirtualReality

VIDEO OF VR MONO AT WORK

18

MultimodalApp:VRStereo• Stereorenderingofthescene

• Low-levelOpenGLcodingfromscratch.• Stereoparallelprojection.• Usinga1:1scalefortheinteraxial.• Stereofrustumcullingtoimproveperformance.

19

MultimodalApp:VRStereo

• UseofCardboard-likeVRheadsets.

20

MultimodalApp:AugmentedReality

• DeviceAttitudeTracking+RealImageMerging• Phone’scamerahastobemathematicallyrepresented.• IntrinsicparametersobtainedwithOpenCV.

21

IntrinsicC.Matrix ExtrinsicCameraParameters

MultimodalApp:AugmentedReality

22

MULTIFRAMEBUFFERRENDERING

Usecase:SolarB–Solarirradiancemodel

• Numericalsimulationoftheamountofsunlightcapturedbythebuildingsurfaces.• BasedonŠúri,M.andHofierka,J.(2004)

• Finalresult:TemporaldataseriesofSolarradiationpertimeintervalforsurfacesandbuildings[kWh]• Surfacesrepresentedaspointsetswitharesolutionof1p/m2.

23

a)Buildingpart classification b)Pointgrid generation c)Shadowing analysis d)Aggregationof results

Usecase:SolarB–Solarirradiancemodel

• Example:TheEIFERofficesinKarlsruhe(7stories)• 14surfaces[3071points]• 1yearofsolarirradiationdata[8760timestamps]

24

VIDEO SELECTING BUILDING, ACTIVATING POINT CLOUD AND ZOOMING AROUND ON MAP MODE

Conclusions

• Theprototypedemonstratesfeasibletoapplymobilecomputergraphicstechnologiestothestudyofmassiveurbandatasets.

• Theappisabletodirectlyretrievesemanticbuildingdatafromenergysimulations/modelsinastandardisedway(OGC,CityGMLstructure).

• VRandARareusefultechniquesforfieldexpertsonenergyresearchandinfrastructuremaintenance.

• Formediumsizedcities,multi-level-of-detailisnotnecessary.• GPS&CompassaccuracyforVRandspeciallyforARisstillunreliableonmanyurbansituations.• Alternativepositioningtechniquescouldachievebetterresults.

25

Thanksforyourattention

26

Jochen Wendel Alexander Simons

Andreas Koch

José Miguel Santana Agustín Trujillo

José Pablo Suarez