three.js& leap motion

An introduction & overview of 3D graphics in the browser

• 3D primer• What is three.js• First steps with three.js• What is a Leap Motion Controller• Leap demo• Washington Post special project review

Overview

This is not a tutorial.3D graphics in and of itself is a very complicated subject. This presentation is not a tutorial but does attempt to cover (very briefly) the basics and help you understand where to get started with three.js & how things fit together. With that said- the documentation around three.js is very good and it’s fairly easy to find tutorials online.

A simple scene consisting of a

light, cube, plane and camera.

An Example 3D Scene

3D Primer• The lights, cameras & objects are collectively called

a scene. In three.js it is a hierarchy & the scene is the root of the tree.

• Objects or geometry such as cubes and planes (also called primitives) are created by points in space called vertices.

• A group of vertices and their edges create polygons.• A group of polygons (objects) along with a material

create a mesh.• The camera captures the picture and in the example

to the left we are looking through the camera.• Lights allow us to cast shadows and render

highlights.

First Stepswith

three.js

• Follow the three.js “Getting Started” section in the docs.https://threejs.org/docs/index.html#Manual/Introduction/Creating_a_scene

• Load a custom model instead of using geometry.• Add a light & cast shadows.

three.jsexample

. ├── index.html ├── js │ ├── ext │ │ ├── three.js │ │ └── three.modules.js │ └── main.js └── run.py

directory structure

1 console.info("main.js loaded"); 2 3 var scene = new THREE.Scene(); 4 var camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000); 5 console.info("scene & camera created"); 6 7 var renderer = new THREE.WebGLRenderer(); 8 renderer.setSize(window.innerWidth, window.innerHeight); 9 console.info("renderer created"); 10 11 document.body.appendChild(renderer.domElement); 12 console.info("DOM modified"); 13 14 var geometry = new THREE.BoxGeometry(1, 1, 1); 15 var material = new THREE.MeshBasicMaterial({color: 0x00ff00}); 16 var cube = new THREE.Mesh(geometry, material); 17 scene.add(cube); 18 console.info("Cube created"); 19 20 camera.position.z = 5; 21 console.info("Camera repositioned"); 22 23 function render() { 24 // Animate the cube 25 cube.rotation.x += 0.01; 26 cube.rotation.y += 0.01; 27 requestAnimationFrame(render); 28 renderer.render(scene, camera); 29 } 30 console.info("Starting render") 31 render();

demo

Free “digger” model from TurboSquid.

loadingcustommodels

Model geometry must be triangulated.

Note how there are now diagonal lines and all of the surfaces have triangles.

Sidebar: Use a 3D graphics program to adjust your model’s geometry.

Any 3D program can adjust geometry. Blender is free and can easily do this for us.

Loading JSON

• Use the free utility from the three.js project to convert from OBJ to JSON.https://github.com/alteredq/three.js/blob/master/utils/exporters/convert_obj_three.py

• Use the built-in JSONLoader to load the object.• Watch file sizes! If you are using a “real” web server

enable gzip compression!

1 var greenMat = new THREE.MeshBasicMaterial({color: 0x00ff00}); 2 3 // Create a JSON loader 4 var loader = new THREE.JSONLoader(); 5 6 // Load triangulated model as JSON 7 loader.load( 8 'models/digger_scaled_tri.json', 9 // onLoad callback 10 function (geometry) { 11 // Add geometry to a new mesh combining a material 12 var object = new THREE.Mesh(geometry, greenMat); 13 // Add our object to the scene 14 scene.add(object); 15 } 16 );

demo

lighting &shadows

three.js has many types of lights but we’ll focus on the directional light.

• Directional lights shine from a specific direction not a specific location.

• The documentation describes this as a light that acts like the sun.

• Shadows are only calculated for objects that exist within the light’s shadow camera’s field of view.

• The shadow camera for a directional light is orthographic (think 2D, e.g. top down).

• In the picture to the left the orange lines denote the shadow camera’s field of view.

three.js uses shadow mapping with the WebGL renderer.

• Shadow mapping was introduced by Lance Williams in 1978.

• Computation takes place on the GPU with WebGL.

• Think of a shadow map like a texture map. They are applied (mapped) to the surface of the geometry during rendering.

• The size of the map directly affects the shadow detail because the size of the map is the size of the buffer and a larger map/buffer can hold more information about the shadow.

• Shadows aren’t necessarily hard but they require some tweaking and experimenting to get right.

demo

The Leap Motion Controller lets you use your computer in a whole new way. Reach out and swipe, grab, pinch, or punch your way through the digital world.

• Uses two wide angle IR cameras to detect hands in 3D space above the device.• Tracks hands, fingers and wrists / arms and translates motion from real world

space to 3D environment coordinates.• For developers it is plug & play and very accessible.• It is a bit finicky and detecting / supporting custom gestures can be laborious.

Leap Motion Controller

demo

In 2015 I was contracted to help build a 3D interactive experience exploring the restoration of the U.S. Capitol Dome.

The interactive experience took place at the White House Correspondents Dinner. The scope was narrowed down to build a kiosk where visitors could approach one of two large displays and interact with a 3D model of the dome using a Leap Motion controller.

Washington PostSpecial Project

3D Model

three.js enabled the rapid development & delivery of the experience within a browser.Many issues could be ignored because we were operating as a kiosk in a known environment:• Device hardware specs were known (fast enough to ignore

optimizations)• No external bandwidth requirements (all files are hosted & served

locally)• Human explanations & help to explain how to interact with the

demo.• No need to open up “heavier” tooling such as Unity.

LeapMotion

Web Browser

three.js

ear ly p ro to type

Things didn’t go as planned

3 Weeks to DeliverInteraction

Spent the majority of the time figuring out gestures and hand

controls. Tried “gimbal” controls. Settled on “joystick” controls.

“Hot Spots” & PolishingUsed ray casting for “fast enough” collision detection between hands and “hot spots”. Added dynamic “demo” mode to pique interest.

Rendering & LightingLost a good chunk of time to loading

a model of the entire capital. Reverted to just the dome. Had

trouble with shadows & model size.

demo

THANK YOU!

@leetrout