BioVJ User Guide - Amazon S3The Amon Tobin ISAM tour in 2011 showcased the future of Projection Mapping and brought the art to the attention of the American mainstream. The V Squared

BioVJ User Guide| Version 1.0

BioVJ User Guide

BioVJ by Biolumence, San Francisco, CA http://biovj.shopify.com 1

http://www.google.com/url?q=http%3A%2F%2Fexample.com&sa=D&sntz=1&usg=AFQjCNH5iRlnhCg_mk89zURn-g01qQYJEw


Table of contents

Table of contents Introductions

What is projection mapping? Terminology and Computer Graphics Basics

History What are the pieces of a projection mapping program?

User Interface Source Tree

Clip Roots Sources Mixers Filters

Content Table User Interface Source Columns Mixer Columns

Property Frame and Filters User Interface CPU Filters GPU Filters

Tap Sync and Time Signature Equalizer

Value Sources Ranges and Value Source Components

Absolute Range and Typing Dynamic Range and its Use Default Value and Looping

Types of Values Sources UI BPM MIDI OSC FFT Random Delays

Triggers UI MIDI

BioVJ by Biolumence, San Francisco, CA http://biovj.shopify.com

2



OSC Key Triggers

Preferences and Output Windows Preferences Dialog

Clip Folders Video Cache Refresh Rate Slider Latency Slider Graphics Performance Output Aspect Ratio

Output Window Dialog Managing Output Window Selecting Columns to Output Stretch

X3D and Geometry Model Projection Mapping Basics How to make a X3D model for BioVJ Model and Mapping Components

Perspective and Viewpoints Geometry Model Columns to Geometry Mapping

User Interface Geometry Trees SubChannels Mapping Generators

MIDI User Interface

Mappings Feedback: APC and Launchpad

OSC User Interface

Mappings Hosting and Clustering BioPlayer

Macros Actions

Set Value Action Set Delay Value Action Fire Trigger Action Change Face Edge Rotation Action


3



Recording Macros Delay Generators Rotation Generator

Credits FAQ


4



Introductions Projection Mapping is the art of precisely controlling projected light on a three dimensional surface. This is used to create the illusion of perspective from a specific point of view, usually an audience's point of view. BioVJ is a software application designed for VJs and visualists to create and perform realtime projection mapping performances. This document describes the BioVJ software and how to use it to create dynamic and impactful visual shows for entertainment, communication, artistic and creative purposes.

What is projection mapping? Projection Mapping usually involves one or more projectors shining light on one or more real world objects (usually called a "stage"). The light reflects from the stage to the viewer (or audience). But the audience and the projector are not exactly in the same place and might be quite far apart. And so the stage when viewed from both the audience and the projector viewpoints isn't quite the same. Faces will be slightly a different shapes, some faces might be visible from one vantage point and not from another, etc. But in projection mapping we warp the image from the projector so it appears as one coherent image from the audience's point of view. BioVJ is designed to do this for you automagically just by moving around the audience and camera viewpoints inside of a model.

Consider a simple block on the ground 4’ in front of you. The block looks different when


5



viewed from 1’ off the ground than when viewed from 4’ off the ground. Faces are different shapes, and from the audience’s point of view they may see a different part of the image on each face. Managing these differences and warping the projected image to present a coherent view from the audience’s point of view is the most difficult part in creating a projection mapped performance. BioVJ provides the tools to handle this type of warping for you so you can concentrate on the artwork of creating a projection mapping presentation or performance.

Terminology and Computer Graphics Basics There are several terms that will be used throughout this document that you will need to know. In 3D graphics, we refer to Shapes as the 3D objects that make up a 3D model or Scene. Shapes are made up of Faces (ie flat parts of the surface of the shape) which are defined by three or more Vertices (ie points). So a pyramid has 5 faces, 1 square face on the bottom and 4 triangular faces on the sides. And a pyramid has 5 vertices, one on the top and four around the base. A pyramid is one of the countless types of 3D shapes you can use in a 3D model. Geometry refers to a Shape, Face or Vertex in a 3D model. 3D Geometries can be transformed by three different types of operations: translations, rotations and scales. Translations simply move the entire geometry (eg shape, face or point) in the X, Y or Z axis. Scale simply makes the shape larger or smaller in the X, Y or Z axis. Rotations are more complex however. Rotations are defined by a center point (X, Y, Z), an axis of rotation (X, Y, Z), and an amount of rotation. So to flip something towards a viewer looking down the Z axis, the center point would be the center of the geometry and the axis of rotation would be either (1, 0, 0) or (1, 0, 0) (assuming Y is up from the viewer's POV). BioVJ exposes this rotational interface in its raw form as that creates the most power for the users but at the cost of rotations being far more complex to use. Other 3D packages often collapse rotation into an amount of rotation in each axis (X, Y, Z) but this doesn't allow for rotation around points outside of the object which can create very interesting effects.

History The first example of projection mapping according to our research was the Haunted Mansion ride at DisneyLand. There were several other smaller projects using simple mapping techniques by various independent artists, but modern projection mapping didn’t appear until academics began experimenting with projecting on 3D objects in the late 90s, mainly at UNC Chapel Hill. At that time, Projection Mapping was referred to as “Spatial Augmented Reality”. Over the 2000s, other academic projects advanced the state of the art in projection mapping. But in


6



LA, the art of Projection Mapping was being advanced by a small software company called Derivative and their package, Touch Designer. The Amon Tobin ISAM tour in 2011 showcased the future of Projection Mapping and brought the art to the attention of the American mainstream. The V Squared Labs production team created a beautiful visual show which showed the impact that visual production can have on a music tour. As beautiful and creative as the ISAM show was, it wasn’t create using a practical artistic process that can be duplicated on most tours. That show was the result of 1000s or hours of custom computer graphics production, even though it used the Touch Designer package. In 2011, we decided to start writing BioVJ because we wanted to evolve our 2D visual shows into 3D projection mapping events. However, due to the complexity of Touch Designer and the general lack of good mapping tools, this quickly became very impractical. So we started writing BioVJ to create a mapping package that allowed us to focus on the art, instead of the complexities of using TouchDesigner, D3 or other node based graphics packages aimed at projection mapping. We think you will find that BioVJ is far easier to use than other mapping packages and allows for a complex show without requiring indepth knowledge of how the graphics are actually generated. You only need to focus on the art and the synchronization of the performance and let BioVJ handle all the difficulties of perspective correction and linear algebra.

What are the pieces of a projection mapping program? A Projection Mapping software package should contain several basic pieces. First, there needs to be a way to play movies, images and other video sources. Second, there should be a way to mix those video sources. Next, there needs to be a way to map video sources to the geometry model of the stage. And finally, there needs to be a way to manipulate the model to make it appear to change shape. Some projection mapping packages don't really provide all of these necessary pieces. This leads to VJs that don't model their stages before the show, but instead must show up hours before the show, point a projector at the stage and begin to draw rectangles around each box. This is an incredibly limiting and time consuming artistic process. Clip Player The clip player allows video sources to enter the application. Clips can either be prerendered


7



entire scenes or they can be only parts to be used to make a larger scene. Clips have simple VCR like controls: start, stop, pause. In the case of video files, they also have rewind, fast forward, clip start/stop points, and scrubs (moving directly to another part of the video file). Clips (and other video sources) can be processed by filters. In BioVJ, both 1.0 and 1.5 version FreeFrame and GLSL filters are supported. Filters have values that can be changed by the user to change how to warp, blend, color or change the video feed in some desired way. The different filters have different sets of values which can be dynamically manipulated by the VJ during the show. Standard filter frameworks such as FreeFrame have been created and used by VJ programs to allow creative code to be used by multiple VJ applications. Mixers Much like audio mixers combine audio sources, VJ programs have video mixer components that allow video sources to be combined to produce a new video feed. VJ programs typically allow basic fading, chroma key and luma key functionality along with mathematical based mixing modes such as add, sub, etc. BioVJ provides a native mixer component along with the ability to run FreeFrame and GLSL mixers inside of BioVJ. Mapping to Geometry The first mapping that most VJs will do with BioVJ or any other projection mapping software is to map one entire video feed to each face on a stage. This creates a very busy mapping but shows off the plain geometry of the stage very nicely. When creating a mapped scene, you often want to stretch one image from a video source across multiple pieces of the stage. Often each flat face of the stage will be angled differently creating different warpings of the image. We overcome this by cutting the larger video feed into smaller pieces each of which represent what the audience should see on that one face from their point of view. Then that piece of the feed is mapped to that face in the model which makes a flat mapping of the image. This allows you to remove the effects of the geometry on the audience POV, allowing you to create your own effects which creates the appearance that the stage is changing its shape. BioVJ provides tools which create these complex mapping for you saving you time when creating a higher quality mappings.


8



Macro System When performing a live projection mapping shows, often you want to control dozens or hundreds of model elements at once. Macro systems allow for the scripting of actions that couldn't be performed in real time by a human using a GUI. Perhaps you want the geometry reconfigure into a different shape. This can be accomplished by recording each movement of the geometry in a macro to use during a performance. Or by creating macros that allow you to control multiple values at once using delay effects. BioVJ has a macro system that allows for delays, waves and ripple effects to be created and controlled in real time. Also generator tools to create complex macros are built directly into BioVJ.

User Interface

Source Tree On the left side of the main BioVJ window is the Source Tree. This allows the user to select video sources for use in a performance. The source tree is divided into several sections with different types of sources in each branch. The first branch of the source tree contains video and image files. The second branch contains other video sources. The third branch contains additional mixers. The fourth and final branch contains filters. When BioVJ starts or the “Refresh Source Tree” item is selected from the File menu, BioVJ scans certain folders to displays the files in those folders in the Source Tree. Specifically, BioVJ scans the Clip Roots, and the sources, mixers, and filters directories in the directory where BioVJ is installed. On Windows this is usually in C:\Program Files\BioVJ\.

Clip Roots Most VJs have directories (ie folders) that contain all the movies and images they use during a performance. We call these clip folders. When BioVJ starts for the first time, the Preferences dialog is launched. At the top of the Preferences dialog is list containing all the directories (folders) that contain video and image files. You can add or remove clip folders at any time by selecting Preferences from the File menu which launches the Preferences dialog. When BioVJ is launched, the clip roots will be explored and all directories, sub directories, video files, and image files in those folders will be displayed in the Media branch of the Source Tree. All files and directories will be displayed in alphabetical order.


9



Sources In addition to video and image files, BioVJ can get video feeds from FreeFrame generators (plugins with 0 inputs and 1 output), video capture devices, and from other applications via screen capture. When BioVJ is started, it will scan the sources directory inside of the BioVJ application directory for FreeFrame and GLSL plugins. If any are found, they are listed in the ‘Filters’ folder inside of the Sources branch of the Source Tree. When BioVJ is started, it scans for all capture devices on the system and lists any it finds under the Capture folder in the Sources branch of the Source Tree. Any of the files, scripts, plugins or capture devices in the first two branches of the Source Tree can be dragged into the source columns of the Content Table from which they can be played and saved into a BioVJ project file.

Mixers In addition to sources, BioVJ supports FreeFrame mixers (plugins with 2 or more inputs and 1 output). When BioVJ starts, it scans the mixers directory in the BioVJ application directory. Any FreeFrame or GLSL plugins it finds in that directory will be listed under the Mixers folder of the Source Tree. The


10



FreeFrame mixers can be dragged into cells in a mixer column to be used.

Filters The final directory that BioVJ scans on startup to build the Source Tree is the filters directory in the BioVJ application directory. It contains all the FreeFrame or GLSL filters (plugins with 1 input and 1 output) which can be used on video sources. FreeFrame filters come in two flavors, the CPU flavor (1.0 version of FreeFrame) and the GPU flavor (the 1.5 version of FreeFrame). The GPU filters can be used on any video source except mapping cells. However, the CPU filters can only be used on video files, image files, FreeFrame 1.0 generators, and Capture devices. GLSL sources, filters and mixers are just .frag (and possibly a matching .vert) file with GLSL code in it using similar inputs as in the ShaderToy API.

Content Table The Content Table contains all the video sources, mixers and mappings used and played during a performance. Content is arranged into columns with each Column containing only sources, only mixers or only mappings. Additional Columns and Rows can be created at any time using the Channels menu.

User Interface The Content Table is designed to allow easy access to all the pieces of a VJ's performance. Columns, Rows and Cells can be renamed by double clicking on them and editing the name. When a Cell is clicked on, the Property Frame will display specific controls for that type of source, mixer, or mapping. Columns Columns are like channels in an audio mixer. Each piece of content, each mixer and each mapping lives in its own Cell in a Column. When a piece of content is played, its output is displayed in a preview window at the top of the column. Only one piece of content from any Column can be playing at once. Columns can be collapsed and expanded using the triangle to the left of the Column's name at the top of the Content Table. Columns can be dragged back and forth to reorder them. To do this, click on the name of the Column and drag it on to the name of another Column at the top of the Content Table. Cells Cells can be dragged around inside of their Columns and from Column to Column as long as the new Column is of the same type as the old Column. In other words, source Cells can only


11



be in source columns and mixer Cells can only be in mixer Columns. When a Cell is played, it pauses or stops the Cell that was previously playing in this Column. You can always drag sources from the Source Tree into the Content Tree. However, if you drag over an existing cell, you will delete the existing cell. Keep in mind that if you drag multiple items from the Source Tree, you might delete cells several places below where you dropped the items from the Source Tree. All Cells have a name, a play trigger and a color. The names and colors make organizing your content in the Content Table easier. The play trigger allows for MIDI and OSC triggers to play a specific cell. To edit the color or trigger, click on the Cell and edit those values using the controls on the left side of the Property Frame at the bottom of the main BioVJ window. Rows Content is also arranged into Rows. Additional Rows can also be created using the Channels menu. Rows have a play button that allows them to play all the Cells in their Row at once.

Source Columns BioVJ has source channels which appear as columns in the main content area. BioVJ supports 5 different types of video sources: video files, image files, GLSL/free frame generators, screen capture and capture devices. Each movie, image, or other video source in BioVJ has its own cell in the content table in a source column. Cells can be dragged from one location to another inside of a source column. However they can't be moved into mixer or mapping columns. All cells in source columns have stretch and centering controls builtin to them. Often aspect ratio problems cause video content to be clipped at the bottom and top of the screen. The stretch and centering controls which appear to the right and bottom of the cell's control area in the Property Frame at the bottom of the BioVJ window. These controls allow for the video to be


12



stretch and moved both vertically and horizontally which allows for some interesting effects and clipping of undesirable artifacts from video content. Video Files Video files are read into BioVJ using the GStreamer library on Linux and DirectX on Windows. Any video file format and codec combination that GStreamer/DirectX can read, BioVJ can read. When the video file is dragged into the Content Table, BioVJ attempts to read and cache the video file's frames in a memory. Because video files can be very large, especially when uncompressed, BioVJ will only load the entire file if it has enough memory to do so. The amount of Frame Cache currently in use is usually displayed in the bottom left hand corner of the main BioVJ window. Sometimes BioVJ will unload other video files to free up space in the Frame Cache. Video files can to loaded manually by clicking on the video file's cell and then clicking on the Loaded toggle button on the right side of the video file's controls or simply clicking on the play button. When the loaded toggle button is active (ie depressed), the video file will be loaded into the RAM or disk caches depending on available RAM cache space. After clicking on the play button, when a video file is loaded it will play immediately, but if a video file isn't loaded it is loaded instead of played. When a very large video file is loaded, BioVJ will cache the file to disk instead of RAM and cache parts of the file to RAM on demand. Due to all of this loading and unloading of movie file, BioVJ can be overloaded, especially on older or slower hardware. To prevent this, make sure your video cache (set in the Preferences) is as large as your hardware can accommodate. We recommend 1/3 to 1/2 of your total physical RAM. If you are still having problems with slow file playback, first try reducing the quality slider in the Preferences Dialog. Then if that doesn’t work try reducing the video resolution of your video files using a 3rd party video editors. There are several components in the video file's controls. The top slider is the Scrub slider which shows where we are in the current playing of the file. When the file is playing, the scrub slider will be constantly be in motion. When the file is paused or stopped, the scrub slider won’t be moving. When the Scrub slider is moved by the user, it changes where in the file we are currently playing. Video files have two clip points which define the start and end of the playback of the video file. You can change these clip points by changing the two endpoint marks on the slider below the Scrub slider in the video file's controls. Below that are the VCR like controls for the video file playback. Reverse sets the file's current playback to the first clip point (beginning of the video by default). Pause/Play switch between


13



a paused and a playing state. Stop stops video file playback entirely and resets to the first clip point. The looping chooser selects between forward only and forward/backward playback modes. The BPM toggle only works when a tempo has been indicated via the tap sync at the top of the BioVJ window. When a tempo has been set, the BPM button causes playback to "surge" with the music, which makes playback appear to be in sync with the tempo. Finally, the playback rate slider is at the right side of the video file's controls. It allows the VJ to set the rate at which to play the video. Note this can be set to a negative number which means play in reverse. Video files can have any FreeFrame filters on them. Images Standard image files like PNG, GIF and JPEG are supported. Images larger than 4096x4096 are not supported. Images can have any FreeFrame filters on them. FreeFrame Generators FreeFrame generators are rare but provide a powerful, but hard to use mechanism for getting any content into BioVJ. A FreeFrame generator is just a FreeFrame plugin with 0 inputs and 1 output. That output can be a frame buffer (ie raw video frames) or OpenGL textures. FreeFrame 1.0 plugins will always be outputting frame buffers and FreeFrame 1.5 plugins will always be outputting OpenGL textures. FreeFrame 1.0 generators can have any FreeFrame filters on them. FreeFrame 1.5 generators can have GPU FreeFrame (1.5) filters on them only. GLSL Vertex Shader sources are also supported in BioVJ. The shaders are stored in .vert files which is the standard extension for OpenGL Vertex Shaders. Some basic uniform variables from the Shader Toy API are supported and the number of texture unit inputs from the .frag files is used to compute the type of plugin the vertex shader represents: source, filter, or mixer. GLSL Shaders from the shader toy site can be dropped into BioVJ in the source folder (for shaders with no texture inputs), filters folder (for shaders with 1 texture uniform variable usually named iChannel0), or mixer folder (for shaders with more than 1 texture uniform variable). Screen Capture Screen capture is the process of scraping the raw bytes of an application's GUI directly from the OS’s frame buffer. It allows for a slow but universal interface between other applications and BioVJ. To create a Screen Capture cell, simply double click on an empty cell in any source column. The top left coordinates (X, Y) and the size (width, height) of the window to


14



be scraped can be configured for each screen capture cell in the Property Frame at the bottom of the BioVJ window after selecting the screen capture cell. Screen Captures can have only GPU FreeFrame filters on them currently. Capture Devices Video capture hardware, sometimes called video rippers or capture cards allow computers to read external video feeds. Often for VJs, this means one VJ's laptop sending a video signal to another VJ's laptop which might be controlling the actual mapping. When BioVJ starts up (or when the Source Tree is refreshed), it scans the computer hardware for all connected devices. When you want to get video from a capture device, simply drag it from the Source/Capture folder of the Source Tree into the Content Table and play the new Cell. Capture Devices can have any FreeFrame filters on them.

Mixer Columns Mixer columns allow VJs to combine video feeds from other columns to create a new video feed. Mixer Cells each have a specific configuration of input columns and mixing parameters. Once a Cell in a mixer Column is playing, one and only one mixer Cell in a mixer Column is always playing from then on out. Mixer Cells can't be paused or stopped, mixer cells can only be played. Native Mixer There is a native two channel mixer built directly into BioVJ. To create a native mixer cell, simply double click on an empty cell in a mixer Column. This will create a new native mixer defaulting to a simple fade combining the first two Source Columns equally. The most basic control is the crossfader. The crossfader only affects how channels are mixed in Fader, Chroma and Luma mixing modes. There is also the left and right flashers which when depressed show only the left or


15



right source. Below the crossfader is the Column selection area which allows the user to select which columns to use as input to this mixer. The swap button between the two choosers switches the left and right input of the mixer. The main mode of the native mixer is set by a set of button spanning the bottom of the mixer's Property Frame (shown in the image above). These buttons are labeled Fader, Chroma, Luma, Add, Sub, etc. Which mixer mode is selected determines how the input columns are mixed. Here is a summary of all the mixer modes and how they work.

Fader Uses the crossfader and alpha blending to combine the two column's video feed. Moving the crossfader to the left shows more of the first column's video; moving the crossfader to the right shows more of the second column's video.

Chroma Key Does a generic chroma key blend. The chroma color is selected using the color picker button just to the left of the Chroma button. A chroma key finds pixels in the first video feed that are similar to the chroma color and replaces them with the pixels from the second video feed. The value of the crossfader determines how similar pixels color values need to be before it's replaced by pixels from the second feed.

Luma Key Does a generic luma key blend. A luma key finds pixels in the first video feed that are similar in luminance (how bright the pixel is) to the luma value and replaces them with pixels from the second video feed. The luma value is set by the vertical slider on the right side of the Luma button. When the luma value is large (near or equal to 1.0) then only bright pixels are substituted; when the luma value is small (near or equal to 0.0) then only dark pixels are substituted. The crossfader's value determines how similar in luminance pixels in the first video feed need to be before being replaced by pixels from the second feed.

Add Simply adds the R, G, B and A values from each feed together. Sub Subtracts the R, G, B and A values of the first feed from the R, G, B, and A

values of the second feed. If the R, G, B, or A value from the first feed is less than the value from the second feed, 0 is used instead.

Mult Multiplies the R, G, B, and A values from each feed together. Div Divides the R, G, B, and A values from the second feed by the first feed. When

the value from the first feed is 0, 0 is returned. More Uses the larger of the two R, G, B, and A values from each feed. Less Uses the smaller of the two R, G, B, and A values from each feed. Absdiff Subtracts the R, G, B and A values of the first feed from the R, G, B, and A

values of the second feed. If the R, G, B, or A value from the first feed is less than the value from the second feed, the absolute value of the difference is used.

Avg Uses the average of the two R, G, B, and A values from each feed. Addm Simply adds the R, G, B and A values modulo 256 from each feed together. Sub Subtracts the R, G, B and A values of the first feed from the R, G, B, and A


16



values of the second feed modulo 256. Modulo Divides the R, G, B, and A values from the second feed by the first feed and

returns the remainder (modulo). Bitwise Uses the AND of the two R, G, B, and A values from each feed.

FreeFrame Mixer FreeFrame mixers have different configuration parameter depending on which plugin is doing the mixing. Due to the small number of FreeFrame mixers on the net, this feature is still experimental. It should work but has only been tested minimally. BioVJ also support GLSL Vertex Shader mixers that have 2 or more texture uniform variables declared and used. Those shaders should be stored in .vert files in the mixer folder (usually C:\Program Files\BioVJ\mixers\). The vertex shader’s uniform variable are extracted by BioVJ and sliders controlling their values are in the properties frame when the GLSL mixer’s cell is selected. The input textures are bound to the output textures of the channels being mixed together. Example GLSL Vertex Shaders are provided in the source, filter and mixer folders. Other GLSL Vertex Shaders can be downloaded from the shader toy site.

Property Frame and Filters The Property Frame is at the bottom right of the main BioVJ window. When you click on a Cell (or Mapping or Model element), the Property Frame displays the correct controls to allow you to view and edit the Cell.

User Interface When a cell is selected, on the left side of the Property Frame are the controls that edit values that are present on all types of Cells. These properties are the Cell's name, its color in the Content Table and its play trigger. The play trigger is a way to play cells from external MIDI or OSC hardware such as Midi controllers or an iPad running TouchOSC. Many actions in BioVJ can be triggered in this way, see section 3.3 below for more information on triggers.

CPU Filters In BioVJ, there are two types of FreeFrame filters. The old 1.0 version FreeFree frame filters which do all their work on the CPU. So they are called CPU filters throughout this document. CPU filters can only work on sources which provide BioVJ with a stream of video frames. These include video files, image files, and Capture Devices. Mixer cells can't have CPU filters on them. Multiple filters can be applied to a Cell. This means that the output of one filter will be the


17



input to the next filter. This means that filters have an “order”. Filters can be dragged around to change their order with one rule. CPU filters must always be before GPU filters.

GPU Filters The newer 1.5 version FreeFrame plugins do all their work on the GPU (ie graphics card) and so they are called GPU filters. They provide BioVJ with a stream of changing OpenGL Textures. FreeFrame 1.5 filters can be applied to any source or mixer Cell. To the VJ however, GPU and CPU filters will look and act the same. However, all CPU filters on a source must be processed before any GPU filters. This means that if a Cell has a CPU filter on it, GPU filters must processes the output of the CPU filter and NOT the source of the Cell directly. BioVJ will simply not let a VJ put CPU filters “after” GPU or GLSL filters.

GLSL Filters GLSL filters are Vertex Shaders with 1 and only 1 texture uniform variable declared and used. That texture variable is bound to the output texture of the source being filtered. The output of the vertex shader is then rendered as the output of the video signal being filtered.

Tap Sync and Time Signature Because VJs are often performing live with a DJ or live musician, the music's time signature and tempo are important. BioVJ comes with a builtin tap sync in the middle of the toolbar at the top of the main BioVJ window. By tapping on that button a few times in rhythm with the music, the VJ can indicate the current tempo of the music and allow Biovj to use the rhythm of the music as input. BioVJ also allows for different types of musical signatures so that settings in BioVJ that refer to whole bars and quarter notes match the music's notion of quarter notes and bars.

Equalizer The graphic equalizer appears in the bottom left hand corner of the main BioVJ window. It displays an audio equalizer of the sound coming into the laptop's microphone. The values from the equalizer can be feed into BioVJ using a FFT (Fast Fourier Transform) value source which is explained further below.


18



Value Sources In BioVJ, there are many different objects for the VJ to interact with to manipulate the visuals. For instance, every mixer cell has a crossfader value and a luma key value. These values can simply be changed in the GUI by a VJ during a set. But sometimes the VJ wants the value to change on its own based upon some external value. A VJ might want to map a slider on their MIDI controller to the mixer's crossfader, or a VJ might want to have the crossfader move back in forth in rhythm with the music's tempo. A VJ can rightclick on almost any slider in BioVJ and a menu will popup displaying various options for possible value sources for that slider. The first option in the menu is always “UI” which means only the UI changes that value. This can also be thought of as an “off” setting. There are many other value sources listed in this menu and their meanings are described below.

Ranges and Value Source Components A value in BioVJ is usually represented by a slider in the BioVJ GUI. But each slider also has several other values embedded in it that are mostly invisible to the user but important to understand. A value has an “Absolute Range”, a “Dynamic Range”, a default value, and obviously its current value. Also, sometimes a value has a looping state specifying a forward/forwardbackward/backward state of the slider.

Absolute Range and Typing Consider the case of the crossfader's slider and the luma's slider in a native mixer Cell. The crossfader's slider goes from 1 to 1. The luma's slider only goes from 0 to 1. These are the “Absolute Range” of these values. A Range has a minimum value below which the value of the slider never goes and a maximum value above which the value of the slider never goes. Each type of slider in BioVJ has an Absolute Range embedded in it that never changes.

Dynamic Range and its Use However, almost all sliders in BioVJ also have another slider just below or to the slide of it. On these sliders are two marks that can be moved to indicate a minimum or maximum value for the slider. This is the value's “Dynamic Range”. The minimum value of the Absolute Range is always less than or equal to the minimum value of the Dynamic Range. The maximum value of the Absolute Range is always greater than or equal to the maximum value of the Dynamic Range.


19



The Dynamic Range of a value defines the maximum value that the slider can be set to by an external source such as the tempo or a MIDI controller's slider. So if the Dynamic Range of the crossfader was set to 0 to 0.5, when a MIDI control mapped to it is at its maximum setting the crossfader's value is 0.5. When the MIDI control is moved to its minimum setting, the crossfader's value will be 0.0 (instead of 1.0 as it might normally be). This same idea applies to all other types of external values, such as BPM value sources (tempo synced) and FFT value sources (from the audio equalizer).

Default Value and Looping Values in BioVJ will sometimes also have a looping value. The looping value can be forward, forwardbackward, or backward. Sometimes, only forward and forwardbackward options are offered when appropriate. The looping value affects how the tempo is translated into a value when a BPM value source is used. When a value moves with the rhythm of the music, it will reach one end of the Dynamic Range and either skip to the other end of the Range or begin moving back towards the other end of the Range. When the looping chooser is set to forward and the value reaches the Dynamic Range's maximum value, the value skips around to the Dynamic Range's minimum value. When the looping chooser is set to forwardbackward and the value reaches the Dynamic Range's maximum value, it changes direction and begins moving back towards the Dynamic Range's minimum value. When it reaches the Dynamic Range's minimum value, it changes directions again and heads back towards the Dynamic Range's maximum value. When the looping chooser is set to backwards, it moves down from the Dynamic Range's maximum value to the minimum value. At the time the value reaches the minimum value, it skips back to the Dynamic Range's maximum value. On properties of model elements, there is also a toggle button labeled ‘Important’. This button when toggled on puts this property in the current mapping cell’s property view. This allows you to setup lists of relevant properties to be controlled for a specific mapping in one properties view.

Types of Values Sources BioVJ supports an ever growing range of value sources. Currently, there are seven different types of value sources in BioVJ.

UI The UI value source is the default. It can be thought of as “off” meaning that the only way to change this value is by moving the slider in BioVJ with the mouse. Any value can be moved in the UI at any time by moving a slider, but when a value source is set to UI, that is the only way to do it. When another value source is specified, the external source might “fight” with the values being set by the UI which can produce some weird effects.


20



BPM The BPM value source is the way to use tempo to change a value automatically in rhythm with music. The value will move between the Dynamic Range's minimum value and maximum value. A BPM value source is specified by an amount of time in musical notation (quarter note, half note, 1 bar, etc). During that amount of time, the values will cycle through all its values. How it does this exactly is specified by the looping state of the value.

MIDI The MIDI value source is the way to use external MIDI devices to change values in BioVJ. Each MIDI slider or knob on a MIDI controller can be registered as a MIDI mapping inside of BioVJ. Then a value source can be set to take its values from that MIDI mapping. When the MIDI controller is set to its maximum value, the value in BioVJ will be set to the Dynamic Range's maximum value. When the MIDI controller is set to its minimum value, the value in BioVJ will be set to the Dynamic Range's minimum value.

OSC The OSC value source is the way to use external OSC inputs to change values in BioVJ. Each part of the OSC interface can be mapped to a specific OSC URL inside of BioVJ. Then a value source can be set to take its values from that OSC mapping. When the OSC input is set to its maximum value, the value in BioVJ will be set to the Dynamic Range's maximum value. When the OSC input is set to its minimum value, the value in BioVJ will be set to the Dynamic Range's minimum value.

FFT Sometimes, VJs want to capture audio from a microphone or a line input from the mixer and use this in their set. In BioVJ, this can be done using a FFT value source. FFT stands for Fast Fourier Transform which is the usual way that audio hardware converts audio into a waveform usually seen in graphic equalizers and other visualization components in audio system. BioVJ does the same thing to produce a visualization of an audio feed in the Graphic Equalizer component in the bottom left hand corner of the main BioVJ window. The FFT in BioVJ produces 10 bands representing the amount of audio signal in each of the bands of the audio spectrum. The lower bands of spectrum represent the “lows” or the bass lines. The higher bands of the spectrum represent the “highs” or the higher frequencies usually created by vocals or high pitched instruments. Any of these bands can be used to supply a value much the same was as the MIDI or OSC value sources do. The Dynamic Range is used to scale the value into the desired range.


21



Important note: sound travels quite slowly when compared to light, so FFT value sources often have some latency built into them. Lines directly from a preamp mixer will have the lowest latency and despite the difficulty of running them before shows, the work often pays off with latencies you couldn't get when using a microphone.

Random Random value sources drift randomly inside of the Dynamic Range of a value.

Delays The final type of value source is the most difficult to use but has the highest payoff in terms of the quality of the effects that it can be used to produce. The Delay Value Source takes a value from another value source elsewhere in BioVJ. And its delayed by a certain number of frames. Delay Value Sources are usually only created by macros as they are difficult to setup. Usually, in BioVJ sets, there are 1000s or 10000s of delay value sources in the macros and only a handful of all the other types of value sources. Currently delay value sources can only be created inside of the macro system. This should change in the near future.

Triggers Just like sliders in BioVJ, buttons have an associated model for interacting with them. Instead of values, we say the buttons have triggers which specify which external source will launch/flip the trigger. Many buttons in BioVJ can be rightclicked upon to popup a menu that specifies how that button can be triggered by an external source such as a MIDI or OSC trigger. However, triggers are much simpler than values and only three types of triggers are currently supported in BioVJ.

UI The UI trigger is the default. It can be thought of as “off” meaning that the only way to launch this trigger is by pressing its button in the BioVJ User Interface (UI). Any trigger can be launched by the UI at any time, but when a value source is set to UI, that is the only way to do it.

MIDI The MIDI trigger is the way to use external MIDI devices to trigger actions in BioVJ. Each MIDI trigger can be registered in a MIDI mapping inside of BioVJ. Then a trigger can be set to


22



launch when that MIDI mapping is triggered.

OSC The OSC trigger is the way to use OSC inputs to trigger actions in BioVJ. Each OSC trigger can be registered in a OSC mapping inside of BioVJ. Then a trigger can be set to launch when that OSC mapping is triggered.

Key Triggers A key trigger allows a keyboard key press to trigger an event inside of BioVJ much like a hotkey or accelerator. Currently, there are only 9 allowed keys: a, w, s, d, f, j, i, k, l

Preferences and Output Windows Like most applications, BioVJ has a set of configuration preferences that allow VJs to tailor the application to their preferences and the hardware on which it is running. These include things like locations of video and image files, the speed of the graphics hardware and the aspect ratio of the resulting output windows.

Preferences Dialog When BioVJ starts for the first time, the Preferences dialog is launched. This allows the VJ to tell BioVJ where their clip libraries are located on their computer. These setting will be saved (to a file called biovj.conf) and used every time BioVJ is started in the future. Note: changing some of these options requires restarting BioVJ before they take effect.


23



Clip Folders Most VJs have directories (ie folders) that contain all their clip library they use during shows. In BioVJ, these are called clip folders. At the top of the Preferences dialog is list containing all the directories (folders) that the VJ has indicated contain video and image files. You can add or remove clip folders at any time by selecting Preferences from the File menu which launches the Preferences dialog. When BioVJ starts, it scans all the directories and subdirectories of the clip root folders. It puts all the directories, images and video files in the source tree under the Media folder. These video and images files can then be dragged into the Content Table to loaded them into the current project and use them in a performance.

Video Cache When BioVJ plays video files, it first must decode and load the frames of the video file into its cache. When the video file is very large, BioVJ will load only parts of the file at a once and load in new frames when needed. During a performance, VJs often want to play video files in “trick” modes such as playing forward or back very fast, or many scrubs or frequently changing play rates. To accomplish this, VJ applications have to preload and decode video files into large internal caches so frames can be access quickly. However, these caches often take up large amounts of RAM and if a cache is larger than the amount of physical RAM the computer has, the application will run very slowly. So BioVJ allows you to configure the size of this cache. We recommend setting the cache to half the size of the physical RAM in the laptop. So if you have 8GB of RAM, the cache size should be set to 4096MB (4GB). The default setting is 2GB as the minimum RAM that BioVJ can run in is 4GB. The amount of the cache being currently used is often displayed in the bottom left corner of BioVJ under the Graphic Equalizer.

Refresh Rate Slider The refresh rate controls how many new frames of video are displayed each second on the monitors windows inside of BioVJ. The monitor windows are in the main BioVJ window at the top of each column in the Content Table. They are used to monitor each channel’s video feed and don’t necessarily need to be refreshing very quickly during a performance. The monitor windows are not the same as the output windows which have their own top level window and are usually used to control the output on one projector or other type of video output device. The output window’s refresh rates are controlled by the graphics performance slider


24



discussed below.

Latency Slider Latency refers to the amount of time it takes for a video frame to be displayed on screen from the moment that BioVJ decides to display that frame till the moment when the frame is actually on the screen. Latency is often measured in milliseconds (1/1000th of a second) and humans can detect latency issues down to 10 milliseconds. However, in BioVJ latency is measured in microseconds (1/1000000th of a second) to allow the maximum precision in tuning for graphics hardware. Throughput refers to the number of frames that can be displayed on screen in a specific amount of time. For instance, 60 frames/sec is a measure of throughput. When BioVJ is playing a video file in BPM sync mode (with the BPM button pressed in the video Cell's controls) BioVJ has to start trying to display video frames a certain amount of time before the frame is displayed on the screen. This amount of time is the frame latency and is usually in the 7 to 16 millisecond range. To compensate for this, BioVJ allows VJs to tune the amount of latency by using the BPM latency slider. To tune this value, turn on some background music and tap sync to the beat. Then play a video file and press the BPM button in that video file's controls. Watch the video in the preview window in the Content Table. Notice how the playback rate speeds up and slows down with the beat. If the video file seems to change its playback rate at the wrong time, change the BPM latency value until the file seems to play with the beat.

Graphics Performance Modern 3D graphics hardware is quite complex and provides a bewildering array of options for how to render scenes. These include Antialiasing options, types of lighting algorithms, and quality settings for sub and super sampling used in a variety of algorithms. In addition, BioVJ also has internal texture buffers whose sizes and configuration impacts video quality and rendering speed (ie frame throughput). All of these configuration options combine to control the quality and speed of BioVJ’s video render. However, BioVJ's aim is to be easy to use for nontechnical VJs and so exposing all of these options here wouldn't be desirable to VJs who don't have experience with complex 3D computer graphics. So we collapse all of those complex options into one simple slider. Moving the slider all the way to the right will improve video quality at the cost of a lower frame rate. Moving the slider all the way to the left will improve frame rate but at the cost of poorer quality graphics.


25



When in maximum quality mode assuming 4:3 aspect ratio, internal textures will be rendered in buffers of size 4096x3072 and output windows will render @ 55hz. In the lowest, the buffers will be 800x600 @ 25hz. The aspect ratio changes the width, not the height.

Output Aspect Ratio Aspect ratio refers to the ration between a resolution's width and height. Analog movies (films) and most older computer monitors have an aspect ratio of 4:3. Newer HDMI signals have aspect ratios of 16:9 or less commonly 16:10. Setting the aspect ratio here determines the choices for output window resolutions. However, an output window's size can always be changed to any resolution at any time. However, often the video frames will be stretched to match the new aspect ratio. Setting the correct aspect ratio here in the Preferences dialog reduces the amount of stretching BioVJ has to do to video frames. Its recommended that you set the aspect ratio to 4:3 for VGA or other analog formats and 16:9 for HDMI.

Output Window Dialog The Output Window dialog can be launched by selecting the Output menu item in the File menu in the main BioVJ window. It allows VJs to create and configure any number of output windows for displaying output on external displays (eg projectors).

Managing Output Window Output windows have four basic properties: a name, a resolution, a stretch and a source column. The name is simply for convenience. The resolution is the size of the output window and can by changed either in the Output Window dialog by changing the dimension column or by simply resizing the output window. When the output window is set to fullscreen, its resolution is set to the native resolution of the screen it is on.


26



Selecting Columns to Output The third property of Output Windows is the source column. VJs can set which column's video is displayed in a specific Output Window. In the main BioVJ window, on the right side is the Master Area. This contains controls for setting the source column of active output windows. Output windows in other applications can be configured here as well. Recording controls are also displayed in the output area and will be explained below. If you click on an output window itself, you can use <ctrl>+f to make an output window fullscreen and <esc> to take it out of fullscreen mode.

Stretch The final property is the stretch allows multiple output columns to be displayed in one Output Window. This is to support output hardware like triple heads that take in one very wide video feed and split the signal into multiple outputs. Recording Output Sometimes VJs will want to record their sets. BioVJ allows for any Output Window to be recorded to disk. Below the source column chooser for an Output Window (in the Master Area) are the controls for recording the video in that window. To record output, set the name of the file into which the video will be saved and click on the record button to the left. If multiple Output Windows are present, the master record button in the toolbar at the top of the main BioVJ window must also be pressed. This allows multiple video feeds to start recording at once. When only one Output Window is present, the main record button does nothing.

X3D and Geometry Model The core of projection mapping is the model. BioVJ loads models from X3D files which is an open format for expressing 3D Geometry. If your 3D model editor program (ie Blender, Maya, C4D, etc) can't export X3D, then export it to something Blender can read and use Blender to add the final export to X3D. The reason for this is that camera viewpoints need to be placed into the model. X3D allows for multiple camera and audience positions to be specified which allows for all the information BioVJ needs to correct the audience's perspective. Formats like OBJ don't have camera positions in them and other 3D geometry formats were either closed (undocumented) or incomplete. This is why X3D is used to model the scenes in BioVJ. We recommend Blender if you don't already use a 3D editor.


27



Projection Mapping Basics In projection mapping you model the stage's 3D geometry in an editing program, including the projectors, lights, audience's viewpoint(s), and virtual objects you wish to add to the stage. Then you export this model into your mapping program (ie BioVJ) and map video sources to parts of the model. This mapping program will allow you to place video on various parts of the model, but it will do so always from the audience's perspective. Then when the projector's position is calibrated onsite, the mapping software will preserve the audience's viewpoint no matter where the projectors are placed as long as light it getting on the stage. Obviously pointing straight on, as close as possible to the audience's viewpoint is the best position for the projector but sometimes the nature of the venue forces you to put the projector somewhere else.

How to make a X3D model for BioVJ Always start a projection mapping projection by modeling the stage and scene in a 3D editor like Blender. Include all the details needed by the builder to create the stage and make sure they can build the physical stage as accurately as possible. Then add in lights, projectors and the audience's viewpoints. Each projector needs its own viewpoint/camera and each audience vantage point (usually only one of these unless you are doing a 180 degree or wider scene) also needs its own viewpoint/camera. Often you will want to include lights; both spotlights and area lights are recommended. Also, you can include other virtual objects you wish to use throughout your show. Don't worry about making too many virtual objects in scene as you can always disable parts of the geometry in BioVJ during a show.

Model and Mapping Components 3D models are made up of 4 basic types of components: Geometry, Transformations, Lights, and Viewpoints. Geometry is a virtual object which represents something in the real world. It has a shape and a form in 3D space. Usually geometry is composed of faces, edges and vertices. Transformations are groupings of geometry which allow the entire group to be manipulated in 3D space by either translating (moving the location), rotating, or scaling (changing the size of the geometry). Lights are like light sources in the real world. They determine which parts of the model will be brighter and which parts will be darker and help cast shadows over the model during rendering. Finally viewpoints are the most important components as they determine the projector and audience's locations in the model which should match the real world location where the projection mapping performance is to take place.


28



Using the Mapping menu in the main BioVJ window, you can add transforms, lights and viewpoints to the model. If you add a light (BioVJ supports point, spot or directional lights) it is added to the selected transform node in the model, if the selected node is a transform node, otherwise it is added to the first parent of the selected node that’s a transform node. The same is true for viewpoints. Transform nodes take the selected node(s) from the model as their new children and add themselves as a child of the parent. Nodes can be renamed and renaming all the transforms, faces and points in the model is recommended to make creating macros easier. We usually add a transform around the entire scene too to allow the entire stage to be moved at once easily for calibration purposes. When a X3D model is loaded, a mapping column is created with one empty mapping cell. Each cell in the mapping channel is one mapping configuration each with their own subchannels (pieces of a channel) and mappings between channels or subchannel and geometry objects. Each mapping can be selected by playing its cell. There are three ways to generate parts of the mapping.

1) By drawing by hand each texture from source channels and then dragging each channel and created subchannel to faces and other geometry objects.

2) By generating a grid of subchannels and dragging them manually to each face or geometry object.

3) By generating a mapping using the mapping generator dialog which can be launched by the 'Generate Map' button.

All three methods are described at the end of this section.

Perspective and Viewpoints Viewpoints serve two purposes in BioVJ: projectors and audience viewpoints. Projector viewpoints represent the projector in the real world are moved only during calibration onsite. Once they are calibrated, they don't move during the show. Audience viewpoints are only moved when the model is first loaded to adjust for minor differences in how the models are rendered by different programs. The Audience viewpoints are used to generate the mappings so that models rendered from the audience’s viewpoint in other programs appear correct from the audience's point of view no matter where the projector is physically located. The calibration button in the viewpoint's properties allows for a calibration to be locked in during a show. However, this produces a mapping that is warped and if models have tessellated faces(ie faces which break up larger planes) the mapping will look distorted. So when designing a show, its best to decide beforehand if you want to use the calibration feature when deciding if to tessellate faces in the model or not. If you believe the projector will be close to the audience’s viewpoint then you can break up larger faces into smaller faces, if not, its best not to break up larger faces.


29



Geometry Model The geometry model contains all the parts of the stage as well as virtual geometry which doesn't actually exist but you wish to make appear during a performance. All geometry objects have a visible flag which allows users to turn on/off each piece of the model. And like all other flags in BioVJ, this can be triggered from MIDI and OSC devices as well as key presses. All geometry objects also have a name which sometimes is dynamically generated. It is strongly advised for you to rename all the faces and points in the model to human readable names just after importing an X3D when building a projection mapping performance. This will make the rest of the process much easier. Geometry Nodes Geometry nodes are groups of faces grouped together by the 3D program that generated the model. They only exist to provide groups of faces and points that define one 3D shape. Faces Faces are planes defined by 3 points or more points but usually by 4 points. Faces can flipped forward, backwards, left, right, spun left or spun right. Then can also be transformed, rotated or scaled. And faces can be rotated around a specific edge.

Points Points can be translated only. They can either be used in macros or to do minor correctors to a model that is not quite the same as designed. Do not do both in the same set as you will


30



lose your calibration when using macros that move points. Transforms Transforms allow for multiple geometry nodes or other transforms to be grouped and manipulated together. When building your model, it's often good to put a blank transform node around each shape in the model so that each shape has a handle to be manipulated by. Viewpoints Viewpoints each have a couple of special values in addition to their basic values of location. The first is the field of view which is important to get right during calibration. The second is the calibration button. This button should only be used when the project and viewpoint are far apart at the location of the projection mapping presentation. If they are not far apart, leave the calibration button unselected. You select the audience viewpoint in the projector's property view to lock in calibration settings. Another odd thing about a viewpoint is that each one creates their own video feed. These can be selected when telling an output window what channel to display. Lights Lights allow for geometry manipulations to appear much more real than they would otherwise. BioVJ supports two different global settings for each mapping that control how lighting affects a scene. The first is the lighting button and the second is the shadow button. Like most other buttons in BioVJ, users can right click upon these button to map them to MIDI or OSC mappings or hot keys. There are three types of lights in BioVJ: Directional lights, Point lights, and Spotlight lights. Directional lights emulate light sources that are very far away. Spotlights give a cone of light like their real world counterparts. Point lights are more like an exposed light bulb throwing light in all directions.

Columns to Geometry Mapping Mappings tell BioVJ what piece of what video source to display on what face in the model. Mappings can be from either channels (ie nonmapping columns) or subchannels which are pieces of a channel. Each subchannel has a set of coordinates which define part of a video frame. The edges of the subchannel can be edited by selecting the subchannel and moving the yellow boxes in the subchannels properties view. Mappings can point to any type of geometry object except for a point. SubChannels Subchannels use 3 or 4 coordinates to control the 4 corners of a shape defining the pixels


31



from a specific source channel to use to paint light on faces in the model. Subchannels can be rotated and scaled to allow for adjustments to the model for faces that have their vertices

listed in a weird order. These can also be used to create some interesting effects. However, remember that subchannels are only used when their mapping cell objects are playing. So if another mapping cell is being used, then changing the values of a

subchannel will have no effect until its mapping cell is played again. Mappings Mappings have their own value sources which allow them to be rotated or scaled. Mappings are created by dragging a channel or subchannel onto a model element. When a channel or subchannel is dragged onto a model element that groups other geometry together (ie a transform or a shape node), then all geometry under the element where the channel was dropped are also mapped using the same mapping object.

User Interface When a X3D model is imported into BioVJ, a mapping channel is created and a new part of the GUI is enabled inside of the main window. This component called the Mapping Frame displays all the values for one set of mapping defined by a mapping cell. On the left side of the Mapping Frame are buttons for creating new mapping cells, controlling lighting conditions, controlling face culling and launching mapping generator. Below that is a list of the channels and subchannels in this mapping cell. On the right side of the Mapping Frame is the model expressed as a tree. Selecting any of the channels, subchannels, or model elements displays all of its values in the Property Viewer at the bottom of the main BioVJ window.

Geometry Trees The geometry tree displays all the elements in the imported X3D file. BioVJ supports a small


32



subset of Level1 of X3D. Mappings are also displayed in the geometry tree. When a channel or subchannel is dragged to a geometry node, a new child node is created in the geometry tree. Selecting this node shows the value sources of that mapping, allowing for rotation and scaling of the mapping. Mappings can be dragged to different faces as well.

SubChannels When you select a subchannel, its properties are displayed in the property view at the bottom of the BioVJ window. The first thing you will notice in the property's view is the display window showing which part of the video frame is being mapped. You can edit the subchannel by clicking and dragging around the corners of the yellow boxes in the display window. The display window displays all the other subchannels in the same channel on the display window in a lighter yellow. Clicking inside of one of their boxes selects that subchannel. There are also controls for zooming and panning around the channel to edit the subchannel's clips more accurately.

Mapping Cells Mappings and subchannels are grouped together into mapping cells. Each mapping cell are just cells shown in the mapping column in the main BioVJ window. New mapping cells can be created by double clicking in an empty cell in the mapping column or by using the ‘new mapping cell’ choice in the ‘Mapping’ menu. When a mapping cell is selected, its subchannels and mappings are shown and can be edited in the geometry tree on the right side of the main window. When a mapping cell is played, its subchannels and face mappings are used when rendering the geometry. This way, several different types of mappings can be created and used and quickly switched between during a performance. Mapping cells have several other parameters that control the rendering of geometry. The first two are Cull Face and CCW (CounterClockWise) which control how the model is rendered. Traditionally, vertices are ordered in counterclockwise relative to the audience’s viewpoint. So turning on Cull Face and turning off CCW will render only the back faces of the model. All other settings will render the front faces and render the back faces if Cull Face is off. Flat Lighting causes the normals of each vertex in the model to be calculated from the center of each face instead of from the vertex. When enabled, flat lighting is a bit brighter and renders with a flat, cartoonish effect. Depth check turns on and off the Zbuffer checking of the geometry model. Lighting turns on and off the phong lighting calculations and the Shadow parameter turns on and off the shadow map calculations. Mapping cells also can use video input to dynamically manipulate the geometry of the model. The ‘Geo Morpher’ allows the user to select another video channel to use in the fragment shader when calculating the location of vertices. The red, green and blue values of the


33



nearest pixel of the morpher channel to the rendered location of a vertex are used to calculate changes to the X, Y and Z values of the vertex. Initially the red changes the X, green changes the Y and blue changes the Z but a 3x3 matrix is used to control the exact amount each vertex location should change based upon the pixel’s RGB values.

Mapping Generators There are two basic ways that mappings can be automatically generated in BioVJ. One is to use the grid generator in the channel's property view. You simply select the number of columns and rows in the grid and a perfect set of subchannels are created. You must then drag subchannels by hand onto the geometry model to create the mapping. When models are generated, sometimes hundreds of geometry objects are created. Doing this by hand would take days and so BioVJ provides a generator tool for painting a channel's entire frame onto a specific part of a model. To access the mapping generator, click on the 'Auto Generate Mapping' menu item in the ‘Mapping’ menu. When you click on this button, a popup dialog appears allowing you to select a type of face culling, allowing you to get just the front faces, just the back faces, or both front and back faces if you have set up your model

correctly (all face vertices listed in counterclockwise order). This is known as the 'right hand rule' and is a standard convention in computer graphics. The second thing you can choose in the Generate Map tool is the channel to import video from. The third thing to choose is

the viewpoint to use as the perspective and finally a display window to allow the user to edit the bounds of the generated mapping. After setting up all of these values, just click on the generate button and a mapping is created. You can repeat this process to map and overwrite the mappings of several types of previously generated mappings.


34



MIDI MIDI (Musical Instrument Device Interface) is a protocol for communicating events from control surfaces (ie often incorrectly called MIDI controllers) to any other type of MIDI capable hardware. Often MIDI devices can be plugged directly into computers via USB or other standard ports. BioVJ can be configured to use these devices to control almost any aspect of a performance.

User Interface To use MIDI devices you must first “teach” BioVJ to know what MIDI messages to listen for and what to call each one. To do this, open the MIDI Mapping dialog by selecting the Edit > MIDI menu in the main window. This should launch the MIDI mapping editor which allows users to create MIDI mappings. To create mappings, select your desired MIDI device in the top left corner of the MIDI Mapping dialog. Then click on the listen button and begin pressing buttons and moving sliders on your MIDI device. One mapping will be created for each button or slider. Then double click to the right of each of the mappings and name each one. This is important as these names will be used in the rest of the BioVJ GUI to uniquely identify each slider or button on your MIDI device.

Mappings Mappings are used throughout the BioVJ GUI to supply values and events to value sources and triggers. So to use a MIDI device to control some aspect of the model, for instance the X, Y and Z positions of a model element, you find the slider in the BioVJ GUI and right click on it. A popup menu will be displayed allowing the user to select the source of this value. In the MIDI sub menu of this popup, will be a list of the MIDI devices currently plugged into the computer. Inside these MIDI devices are the mappings for each device. So to control the location of a model element, you select the desired mapping from this popup menu and then when the MIDI slider is moved, the model element's values change. One mapping can provide values for any number of values in BioVJ.


35



Feedback: APC and Launchpad BioVJ is designed to recognize APC and Launchpad MIDI devices to light up LEDs in the MIDI devices indicating the cells in the content table that contain movie clips and other content sources. Adding a cell to the content table lights up the LED button corresponding to that cell. Pressing that button on the APC or Launchpad causes the cell to play.

OSC OSC (Open Sound Control) is another way for devices to communicate, but with OSC that happens over an internet connection. Much like with MIDI, you can create mappings which allow remote hardware to control values and trigger events in BioVJ. However, because of how OSC works, it can do much more including allowing multiple BioVJ instances to work as one unit, controlling one scene.

User Interface We will first start with the mappings, unlike MIDI mappings they are not specific to one device. To edit OSC mappings, launch the OSC Editor dialog by selecting the Edit > OSC menu in the main BioVJ window. The OSC editor dialog will open displaying the existing OSC mappings. To create new ones, setup your OSC devices to send messages to 225.121.121.121 (for multicast, might require additional configuration of your router) or to your current IP address of your laptop. The OSC port on BioVJ is always 7777. Then click on the 'listen' button in the OSC editor. Then move a slider to press a button in your OSC device. The paths should show up in the list in the OSC editor. Give each path a name.

Mappings Just like MIDI mappings, OSC mappings can be used to control parts of BioVJ. However, since OSC mappings are not device specific, they are all just listed together in the OSC submenu in the BioVJ popup menus. And they are listed regardless of if the OSC device is connected or not. To set a specific value to be controlled by a specific OSC mapping, find the slider that controls that value in the BioVJ GUI. Then right click on the slider and a popup menu will be displayed. In the OSC submenu will be all the OSC mappings. Just select one of the mappings. Then move the slider or press the button in the OSC device to change the value in BioVJ.


36



Hosting and Clustering OSC can also be used to allow multiple BioVJ programs on separate networked computers. There is also a program provided for free called BioPlayer which allows extra output windows on remote machines to be controlled by BioVJ running on another computer. Running BioVJ and BioPlayer on different machines on the same local network will automatically communicate with each other allowing each to control the output windows of the others.

BioPlayer BioPlayer is a free application bundled with BioVJ that allows for creation of output windows on other machines controlled by one BioVJ on one laptop. Simply start BioPlayer and a single output window will appear. If a BioVJ isn’t running somewhere on the same subnet, start one. In the Master Panel to the right of the Monitor Windows in the Content Table are a list of the Output Windows in BioVJ. Below that will be new chooser allowing you to select a specific channel or viewpoint to display in the output window of BioPlayer. Make sure to start BioPlayer first so it gets all the files to load from BioVJ. It will also need access to all the same video clips and other effects as the main BioVJ application. If you need to synchronize a BioPlayer with BioVJ, use the Sync button in the OSC Dialog. Capture Device and Screen Capture sources work with BioPlayer by streaming H.264 video over UDP from BioVJ to BioPlayer. This is very bandwidth intensive and requires good networking gear (gigabit ethernet is recommended). You can use <ctrl>+o to popup an Output Window Dialog to configure the amount of output windows this BioPlayer has. You can use <ctrl>+f to make an output window fullscreen and <esc> to take it out of fullscreen mode.

Macros Macros give the ability to set large numbers of values at once. In a complex stage, hundreds or thousands of faces might be present meaning that an entire stage has tens of thousands of values which can be modified to create an effect during a show. Managing them all by hand would be impossible which is why macros and macro generators are a big part of BioVJ. Macros allow for large numbers of values to be set at once, to have their values changed in a specific way, or to have their values to be delayed values from other sources creating a “chase” of values. This allows for ripples moving around the model all controlled by a specific value or set of values. Macros are part of the saved BioVJ project model.


37



There are four ways to create macros: by hand, using the recording tool, using the delay generator, or using the rotation generator. We will describe how to create macros in each of these four ways in that order.

Actions Actions are basic components of a macro in BioVJ. There are several different types of actions: actions that change a value in a specific way (from one value to another), actions that set specific value sources, actions that set values to be delayed from another value source, actions that trigger events (ie playing cells, make geometry objects visible, etc), and actions that change which edge around which to rotate a face.

To edit macros, just click on the macro button in the right side of the toolbar at the top of the main BioVJ window. This will launch the macro editor dialog. To create an action, you can simply click on the 'Add Action' button in the top left of the macro editor dialog. This will create an empty action. Start editing the action from left to right. Begin by choosing the type of action you wish. Columns will change in response to changes in other values to the left.


38



For instance, when a set value source action is selected, only value sources are displayed in the dropdown lists to the right. You can select multiple actions and macros by holding down the <ctrl> key and clicking in the first and last element to select. You can select multiple actions individually using the <shift> key and clicking on actions you wish to select. Then those macro actions can then be copied, dragged to other macros, or deleted Change Value Action Changes in values are created or recorded and have their start and end positions fixed. The start position is the value of the value source when the action is created. The end position is determined by the amount of the change in the value. Only the amount of change can be edited. Several types of value changes exist in BioVJ. The first type of action changes the value constantly over the course of the action. The second type is the burst type which changes the value fast at first and slows down as the action approaches its end. The third type is the slide action which is changes slowly at first and then changes more quickly towards the end of the action. The slide and burst actions are inverses of each other. Random Action The random action changes a value in a random fashion each time its activated and it lasts until a new value source is set in its place.

Set Value Action The set value action allows the user to set any type of value source with any range in any value in the object. Usually set value source actions come in large groups allows for multiple values to be changed by a single source (eg BPM, MIDI slider, OSC slider, or audio input).

Set Delay Value Action A set delay value action allows one value source to be a previous value from another value source in the model. Delay actions are arranged in trees in the macro editor. When you create a delay action, it can't be changed into another type of action. You can add other delay actions to a tree by changing a child's action type to 'Set Value as Delay'. This will create a new child action to set. The amount of delay between the old and new action is set in the speed column. Selecting the object and property sets the specific values in the model that will be changed or provide a values changes to the next specific value in the model.


39



Fire Trigger Action Fire trigger actions can trigger events in BioVJ like playing a cell or turning on shadows, making an object in the model visible, flipping a face or other events triggered by buttons in BioVJ.

Change Face Edge Rotation Action Each face selects one edge around which to rotate. The set face rotation edge action sets which edge around which a face can rotate.

Recording Macros Macros can be recorded by simply selecting a macro in the macro editor and clicking on the record button in the top middle of the macro editor. Then all changes to the values in BioVJ will be recorded as macro actions until the recording button is clicked again. Delays are inserted into the macro by adding change value actions with a change set to 0. Setting the speed of those actions will change the time when all other changes to that value source will be made when the action is being played.

Delay Generators The delay generator allows you to create “chases” of values by creating “set delayed value” macro actions. First select the macro to which you want to add delay macro actions. The delay generator is launched by pressing the Delay Generator button at the top of the macro editor dialog. Then Simply select elements from the model on the right of the delay generator dialog and drag them to the left. New elements on the right can be dragged to be children of the element tree you are building on the left. Once you build up a tree of model elements on the left, choose the types of value sources for which to create delays. Do this by clicking on the checkboxes on the bottom of the delay generator dialog. You must have at least one element in the left pane for the checkboxes to be shown. After you are happy with all of your choices, click the generate button to create the actions. You can edit the generated delay trees in the macro editor.

Rotation Generator The rotation generator is the most powerful tool for generating macros in BioVJ. It allows for delay trees and rotations to be generated to allow for complex ripple and wave effects to be easily created in seconds. It is launched by clicking on the Rotation Generator button in the macro editor.


40



There are several options to set in the rotation generator. The first controls how to select geometry to include in this set of generated macro actions. You can choose to either select all geometry inside of a sphere (ie within a certain distance of a specific point), or to select all the geometry inside of a rectangular prism. You change these values by changing the X, Y, and Z values for the location and edge points. The edges of the selection area are drawn in the display area in the right side of the rotation generator in yellow. The Locus option controls how to generate rotations, either all rotations facing towards one central point, or all rotations in the same orientation. The first type is called 'Point', the second type is called 'Line'. When using point type, only one set of X, Y, and Z coordinates is set and it controls the location of the center of rotation. Its drawn as an orange dot in the display area in the right side of the rotation generator. When using 'Line' type, two sets of X, Y, and Z points representing the start and end of a line which is drawn in orange in the display area. All rotations are around the axis defined by that line.

The Generation Type option chooses what to generate. Delay trees of different types of transforms like translations, rotations and scales and settings rotation values can all be generated. This option allows you to generates just useful subsets of these generated macro actions. Rotations simply set the Rotational axis of geometry objects so changes to their rotational angle are also needed to rotate the objects in the model. So a delay on the rotational angles is usually needed as well. However, since we are generating delays, we can


41



also generate delays that change other things like a geometry's location (ie translation) and size (ie scale). The Delay Direction option chooses if the delays should be generated to start from the outside to the inside or from the inside to the outside. The Delay Speed option chooses how many frames to spread out the complete delay propagation over. So its the amount of video frames displayed between when a value is used by the first value in the generated action and when a value is last used by a value in the generated action. The Rotational Amount slider is simply for testing the generated rotations before you click on generate. After you are happy with the values, click on the generate button and the macro actions will be generated into the selected macro.

Credits BioVJ is written by Biolumence, a San Francisco Bay area VJ for the last nine years. He has worked with DJs such as Carl Cox and Paul Oakenfeld but mainly works in the SF Burning Man community. He is known for his complex, stylized projection mapping shows.

FAQ Why X3D and not 3DS or OBJ? 3DS is ancient and poorly documented. Also, OBJ files don't have viewpoints. Other formats are not public specified (the companies they belong to don’t tell others how they work).


42


Documents

BioVJ User Guide - Amazon S3The Amon Tobin ISAM tour in 2011 showcased the future of Projection Mapping and brought the art to the attention of the American mainstream. The V Squared