Eyesweb Siempre User Manual

EyesWeb XMI 5.3.0 – SIEMPRE Library

December 22, 2012

Part I

Description

1

The EyesWeb SIEMPRE Library includes a collection of software modules and patcheswhich have been specifically designed and developed to support the requirements from theEU ICT Siempre project. In particular, the SIEMPRE Library integrates the EyesWebplatform with modules which are required for a distributed multimodal data recording,playback, analysis.

• The SMPTE decoder block received an audio signal in inputs and decodes the SMPTEtimecode contained in the signal. See SMPTE time code and the references therin fora description of the SMPTE encoding schema.

• The SMPTE encoder block generates an audio signal with an encoded time code.

• The Wave File Writer block writes timeseries in the binary format chosen for theSIEMPRE recordings. The chosen binary format uses a small subset of the Broadcast

Wave Format specifications.

• The DeckLink input block add support for the DeckLink family of framegrabbers,which are used in the Casa Paganini setup (see Acquisition setup at Casa Paganini)for synchronized audio/video input from High Definition (HD) videocameras.

• The Qualisys SMPTE decoder block, which decodes the SMPTE timecode as packedin 32 bits by the Qualisys Track Manager software. This block, toghether with theOSC support already available in EyesWeb, allows integration between EyesWeb andthe Qualisys system. This provides the possibility to analyze, in real-time, the datatracked by the Qualisys Motion Tracking system. For the aim of the SIEMPREproject, the integration is particularly useful to be able to synchronize with the Qual-isys Software to visualize the recorded data.

• The Qualisys SMPTE encoder block, which encodes an SMPTE timecode to a packed32 bits integer number with the same coding schema as the Qualisys Track Manager

software.

• The Timecode generator block, which generates a timecode locked to an audio inputsignal; it may be useful for data playback.

• The Qualisys RT Protocol Decoder block, which get the complete data packet, assent by the Qualisys Track Manager software, and decodes all of the data in it. It letyou to get all the data in a compact way; e.g., all the tracked markers are received inan unique labelled set, instead of being received by separate OSC blocks.

• The Qualisys TSV Reader block, which reads the data exported by the Qualisys

Track Manager software in the TSV (Tab Separated Values) format.

In the following chapters, some examples are described, which show the overall infras-tructure used for many of the recordings which happens in the EU ICT Siempre project

and the related EyesWeb patches.

2

http://www.infomus.org/siempre

http://en.wikipedia.org/wiki/SMPTE_time_code

http://en.wikipedia.org/wiki/Broadcast_Wave_Format

http://en.wikipedia.org/wiki/Broadcast_Wave_Format

http://www.blackmagic-design.com/products/decklink/

http://www.qualisys.com/





http://www.infomus.org/siempre

Chapter 1

Example 1: String Quartet recording

The aim is to record audio, video, motion capture, and biometric-sensors data of perfor-mances of a String Quartet. The recording setup includes a Qualisys Motion Trackingsystem, one or more HD videocameras, ambient microphones, instrument microphones,wearable biometric sensors. The recording happens on distinct PCs, and the synchro-nization of the data on the computers is guaranteed by timestamping each sample with areference SMPTE clock which is received by all computers (see Figure 1.1). In the depictedexample, three computers are used, however, it is straighforward to integrate new computerand devices in the recording infrastructure.

The three computers which are visible in the setup perform different operations:

• The black computer in the middle (A) records the Motion Tracking data. The samecomputer also operates as the master clock generator, i.e., it generates the SMPTEsignal which is propoagated to the other computers.

• The lower-left computer (B) records audio/video data coming from the HD camera(JVC GY-HD-251 in this case).

• The lower-right computer (C) records audio data from the instruments.

EyesWeb and the SIEMPRE Library are used for several purposes in this setup.In computer (A) a patch to generate the SMPTE signal on multiple audio channels is

used. The patch si visible in Figure 1.2.In computer (B) a patch to save audio/video data from the HD camera is used. The

patch si visible in Figure 1.3.In computer (C) a patch to save audio data from the microphones on the instruments

is used. The patch si visible in Figure 1.4.

3

Figure 1.1: The acquisition setup installed at Casa Paganini, InfoMus Lab, Genova, Italy.

4

Figure 1.2: The smpte.eywx patch generates an Smpte signal.

Figure 1.3: A patch to save the HD audio/video signal.

Figure 1.4: A patch to save the multichannel audio signal from the microphones on theinstruments.

5

Chapter 2

Qualisys integration

The SIEMPRE project is based on the use of the Qualisys Motion Tracking system toanalyze the movements of the performers. EyesWeb has been extended in order to integratewith Qualisys and receive the tracking data in real-time. Figure 2.1 shows an example of anEyesWeb patch receiving motion tracking data in real-time. The Z coordinate of a markerplaced on the top of a bow is received by the EyesWeb patch and plotted (see the graph inthe lower part of the patch). Moreover, the patch also receives the SMPTE timecode fromQualisys; this timecode allows for synchronization of the processed data.

Another interesting integration example is given in Figure 2.2. In this example, theSmpte audio signal is generated by EyesWeb basing on the value received, via OSC, fromthe Qualisys RTM software. This mechanism may be used, for instance, to listen to therecorded audio tracks in sync with the playback of the recorded motion-tracking data.

As a matter of facts, many audio editors1 allow for using the Smpte value as the referenceto which all the audio data is synchronized. When the audio data is recorded, it is times-tamped with the current Smpte value; when the same Smpte value is received in input, thecorresponding audio data is played. Thus, by regenerating the Smpte signal, EyesWeb cancontrol the playback of the recorded data in synch with the playback of the motion trackingdata in the Qualisys RTM software. To use this patch, open a QTM file2 and process itwith the command Run real time processing on file..., which enables the Qualisys real timeprotocol (i.e., data streaming via OSC). Then, run the patch; you should see the sameSmpte value in both the status bar of the Qualisys software and in the EyesWeb displayin the top of the patch (the one with a black background and red digits). The maximumallowed difference between these values can be tuned by means of the slider. However, notethat imposing a small threshold may cause the generated smpte signal to be resynched tothe value streamed by Qualisys too many times. This can cause degradation of the qualityof the played audio signal as the generated Smpte signal contains discontinuities.

The example in Figure 2.3 shows how the data received by the QTM software can bevisualized in EyesWeb by means of the 3d display. In this patch, the data is receivedby the OSC block using the Raw Mode; i.e., te data is not actually decoded and eachreceived packet is sent, as is, to the Qualisys RT Protocol Decoder block. This blockprovides, as output, two labelled sets, one for the idenditified markers and another one forthe unidentified markers. The identified markers are sent to the BoneLines3DMaker block,

1The setup at Casa Paganini uses Adobe Audition as the audio editor2Of course, the file must have been recorded with Smpte enabled

6

Figure 2.1: The qualisys rt.eywx patch shows the integration between EyesWeb and theQualisys RTM software.

7

Figure 2.2: The qualisys controlled smpte.eywx patch regenerates an audio Smpte time-code based on the Smpte valued received by Qualisys via OSC.

8

which adds bones among markers to form some skeleton structures. These structures arefinally added to an OpenGl 3d world and visualized to the user. Note the difference with thepatch shown in Figure 2.1: in that patch an OSC receiver is used to get a single coordinate(the Z) of a single marker. That aproach makes sense when you need to show a smallnumber of markers and you know their labels in advance (as the OSC receiver needs thename of the marker as the OSC tag; however, when the number of marker increases, as itis in the String Quartet case, that approach is unfeasible due to the large number of blockswhich would be required. To overcame this problem, the Qualisys Visualization patch usesthe RT Protocol decoder block: this block receives a raw packet from the OSC block, anddecodes all the data in it (the packet is composed of many OSC bundles). The data is sentto the output pins; in particular two output pins provide the labelled set of identified andunidentified markers. Thus, the data can be easily managed with standard EyesWeb blocksworking with geometric labelled sets, or it can be visualized to the user as it happens inthis patch.

Figure 2.3: The qualisys visualization.eywx patch shows the visualization, in EyesWeb,of the 3D mocap data.

9

Chapter 3

Mocap data playback and analysis

Once the mocap data is recorded with the Qualisys software (or with any other mocapsystem), the data can be playback and analyzed in EyesWeb. This chaptershows someexample patches, from simple ones which simply playback the mocap data, to more complexones, where data from different sources is playback in synch in EyesWeb. Note that in thischapter we are not going to use the Qualisys software anymore: we suppose the data hasbeen recorder, may be labelled, and exported in some readile format. Most examples willuse the TSV format (Tab Separated Values) or the C3D format. Thus, you do not needto have the Qualisys software to test these patches; you can use the example files providedwith EyesWeb.

This first simple example, visible in Figure 3.1, demonstrates the playback of a TSV(tab Separated Values) file as exported by Qualisys. The TSV player can be controlled byfeeding it with an external clock signal. In this case the clock is extracted from an audiosignal.

The example in Figure 3.2 shows a simple processing of the data coming from a QualisysTSV file. In this case the data of each single performer is extracted, and, for each subsetof markers, we compute the bounding parallelepiped and draw it. Moreover, we extractthe three markers of the head, compute their baricenter, and then we connect the obtainedfour head baricenters through bones. The resulting structure is visualized in the 3d display,together with the performers skeletons and bounding parallelepipeds.

10

Figure 3.1: The qualisys tsv player.eywx patch shows the visualization, in EyesWeb, ofthe 3D mocap data exported by Qualisys in TSV format.

11

Figure 3.2: The qualisys tsv processing.eywx patch shows a simple processing of datacoming from a Qualisys TSV file.

12

Part II

Reference

13

Chapter 4

SIEMPRE Catalog

4.1 Blocks

4.1.1 Qualisys6DTSVReader

bitmap

class name Qualisys6DTSVReadercatalog name SIEMPREcatalog id SIEMPREclass id Qualisys6DTSVReader

Read TSV file from Qualisys. The file must contains at least markers name. In the TSVexport settings window check: - Include TSV header - Export time data for every frame -Export column headers - Include TSV header

Outputs

OutputDoubleLabelledSetid output labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*

The labelled set of 3d points double, containing points from qualisys.

14

OutputUnidentifiedDoubleLabelledSetid output unidentify labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*

The labelled set of 3d points double, containing unidentified points from qualisys.

OutputFrameNumberid output frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*

Extracted data’s frame number.

SMPTEid output smptetype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

Extracted data’s SMPTE.

Parameters

FileNameid file nametype Kernel, String datatype (Kernel Catalog).type id kernel, string

layout

Filename,MustExist=true,SaveMode=false,OverwritePrompt=true,Filter=”TSV files (*.tsv)—*.tsv——”

File name

15

ActivationModeid read modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:BangLineTimePolling

domain [ 0, 4 )Activation mode:Bang: external clock connected to the input, is used to activate the block.On line number: external integer value connected to the input (which is interpreted as

the line number of the file) is used to activate the block.On time: external time value connected to the input (which is interpreted as one of the

time columns of the file) is used to activate the block.Polling: an internal signal is used to activate the block. If the file has at least one time

column the signal is generated according to the offset of consecutive times. If there is notime, Eyesweb automatically activates the block.

LoopModeid eof modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:LoopStopPauseNoOutput

domain [ 0, 4 )Loop mode.It works only if ActivationMode is set to Polling mode. Loop: activates circular loop

mode.Stop: not implemented yet.Pause: generates the current value of file. If the input value of line or time is out of

bound it generates first or last value of file.NoOutput: output is not generated.

UseSMPTEid use SMPTEtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

It specifies if the block use SMPTE or time column to read the file.

16

4.1.2 QualisysC3DReader

bitmap

class name QualisysC3DReadercatalog name SIEMPREcatalog id SIEMPREclass id QualisysC3DReader

Read C3D file format. The file follows C3D specifications(www.C3D.org) Actually theblock doesn’t read analod data.

Outputs





17

Parameters


layout

Filename,MustExist=true,SaveMode=false,OverwritePrompt=true,Filter=”C3D files (*.C3D)—*.C3D——”

File name


layout

Combo Box:BangLineTimePolling






layout


domain [ 0, 4 )Loop mode.Loop: activates circular loop mode.Stop: not implemented yet.

18

Pause: generates the current value of file. If the input value of line or time is out ofbound it generates first or last value of file.

NoOutput: output is not generated.

19

4.1.3 QualisysRTProtocolDecoder

bitmap

class name QualisysRTProtocolDecodercatalog name SIEMPREcatalog id SIEMPREclass id qualisys rtprotocol decoderauthors Paolo Coletta

Decodes the Raw Data received from the Qualisys QTM software via the OSC RealTimeprotocol.

Details The OSC Realtime protocol of QTM can also be decoded by using ”standard”EyesWeb OSC blocks. However, receiving data about all the markers is not strainghforward,as the number unidentified QTM trajectories is time-varying. This block gets all such dataas once, building labelled-sets as output

Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012

Inputs

RawDataid inputtype Base, RawDatatype id base, raw data

requiredrequired for initializationrequired for execution

read only/read write read onlyreferred as inplace *no*referred as inherited *no*

The Raw Data received from the Qualisys QTM software via the RT protocol

Outputs

Timestampid timestamp outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

The time elapsed since the ecordng was started

20

SMTE timecodeid smpte outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

The value of SMPTE as received by QTM

Frame numberid frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*

The frame number QTM

Unidentified markersid unidentified markerstype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*

The set of markers which were not recognized by QTM as belonging to an AIM model

Identified markersid identified markerstype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*

The set of markers which were recognized by QTM as belonging to an AIM model

Parameters

TimeCode Framerateid frame ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:CustomATSC24/film (24fps)NSTC (29.97fps)PAL (25fps)ATSC30 (30fps)

domain [ 0, 5 )

21

The framerate of the SMPTE timecode, i.e., how many timecode words are available ina second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected

Custom Framerateid custom frame ratetype Kernel, Double datatype (Kernel Catalog).type id kernel, double

The custom framerate of the SMPTE timecode; this parameter is used only when Time-Code Framerate is set to Custom

Sampling rateid subframe sampling ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )

The sampling rate of the data in the source system. Since the SMPTE is expressed alsousing frames and subframes, the sampling rate of the date is required to correctly interpretthe frame and subframe fields.

22

4.1.4 QualisysSmpteDecoder

bitmap

class name QualisysSmpteDecodercatalog name SIEMPREcatalog id SIEMPREclass id qualisys smpte decoderauthors Paolo Coletta

Decodes an SMPTE timecode from an integer number. The SMPTE is assumed to beencoded with the same format used in the Qualisys RT protocol (see details).

Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: MinutesBit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit


Inputs

Encoded SMPTE timecodeid inputtype Kernel, Int datatype (Kernel Catalog).type id kernel, int



The encoded SMPTE timecode as provided by the Qualisys RT protocolo. Usuallyobtained by means of the OSC block

Outputs

Decoded timecodeid outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

The decoded timecode

23

Parameters


layout


domain [ 0, 5 )The framerate of the SMPTE timecode, i.e., how many timecode words are available in

a second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected





24

4.1.5 QualisysSmpteEncoder

bitmap

class name QualisysSmpteEncodercatalog name SIEMPREcatalog id SIEMPREclass id qualisys smpte encoderauthors Paolo Coletta

Encodes an SMPTE timecode to an integer number, to make it suitable, for instance, tobe sent via OSC. The encoding protocol is the one used by the Qualisys RT communicationprotocol (see details).

Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: MinutesBit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit


Inputs

Timecodeid inputtype Kernel, Time datatype (Kernel Catalog).type id kernel, time



Timecode to be encoded to a packed SMPTE integer

Outputs

Encoded timecodeid outputtype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*

The encoded timecode

25

Parameters


layout








26

4.1.6 QualisysTSVReader

bitmap

class name QualisysTSVReadercatalog name SIEMPREcatalog id SIEMPREclass id QualisysTSVReader

Read TSV file from Qualisys. The file must contains at least markers name. In the TSVexport settings window check: - Include TSV header - Export time data for every frame -Export column headers - Include TSV header

Outputs





OutputFrameNumberid output frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*

Extracted data’s frame number.

27

SMPTEid output smptetype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

Extracted data’s SMPTE.

Parameters


layout

Filename,MustExist=true,SaveMode=false,OverwritePrompt=true,Filter=”TSV files (*.tsv)—*.tsv——”

File name


layout

Combo Box:BangLineTimePollingFrame





On frame number: external integer value connected to the input (which is interpretedas the frame number of the file) is used to activate the block.

28

MediaTimeModeid media timetype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

It specifies if the block set time column of the file as media time. If not, interpolatedtime is used to set the value. If the input is not a time , this parameter is ignored and theblock take the column value in the file ad default.


layout


domain [ 0, 4 )Loop mode.It works only if ActivationMode is set to Polling mode. Loop: activates circular loop

mode.Stop: not implemented yet.Pause: generates the current value of file. If the input value of line or time is out of

bound it generates first or last value of file.NoOutput: output is not generated.

UseSMPTEid use SMPTEtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

It specifies if the block use SMPTE or time column to read the file.

29

4.1.7 QualisysTSVWriter

bitmap

class name QualisysTSVWritercatalog name SIEMPREcatalog id SIEMPREclass id QualisysTSVWriter

Writes to file a geometric labelled set of 3d points. Points coordinates are written astab-separated values, using the same format of the Qualisys software.

Notes By default, the block writes the data in a Qualisys compatible format. Tothis aim, an header is written in the file which is needed by Qualisys to correctly read thedata; the header is needed even if you plan to read the generated file through the EyesWebQualisysTsvreader block. However, the user can decide to omit such header; this makesthe resulting file not compatible with the Qualisys software, but easier to be imported inthird parties software (e.g., Calc, Excel, Matlab). In this case, since the column names arenot written: it is up to the user to remember the meaning of each column

’Labels mode’ parameter: the column number of the generated file must uniquely iden-tify a marker (label), otherwise reading back the file will be impossible. However, the labelsavailable in the input dataset might change during time (e.g., missing markers when captur-ing, new markers available, etc), thus, the problem arises to decide which labels (and hencewhich columns) will be written to file. The user can select among six possible solutions,through the ’Labels mode’ parameter: Custom: the user can specify the number of labelsand, for each label, its value. Note that the set of user-specified labels is furtherly processedby the block to get rid of empty or duplicate labels

Inputs

InputGeometricLabelledSet3DDoubleid input labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d double



30

Input Geometric Labelled Set of 3D double points. This must be an homogeneous listof 3d geometric points.

Parameters

Filenameid file nametype Kernel, String datatype (Kernel Catalog).type id kernel, string

layout

Filename,MustExist=true,SaveMode=true,OverwritePrompt=true,Filter=”TSV files (*.tsv)—*.tsv—All files (*.*)—*.*——”

The name of the output file

Input typeid input typetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layoutCombo Box:DoubleInt

domain [ 0, 2 )Specifies if the input geometric labelled set is double or int.

Write time columnid write timetype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

Specifies if the block writes the time column on the output file. The value of the timecolumn is extracted from the media time of the input data

Write smpte columnid write smptetype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

Specifies if the block writes the smpte column on the output file. If set to true, anadditional input will be available; the value of the smpte is received through this input

Write frame number columnid write frame numbertype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

Specifies whether to block writes the frame number column on the output file

31

First frame numberid first frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )

Specifies the value of the first frame number. By default, frames start at 1, but adifferent value can be specified.

Write headerid write headertype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

Specifies whether the block writes the file header or not. See the block notes for details

NumOfCamerasid num of cameratype Kernel, Int datatype (Kernel Catalog).type id kernel, int

Number of cameras. It’s in the Qualisys TSV header.

Frequencyid frequencytype Kernel, Int datatype (Kernel Catalog).type id kernel, int

Frequency. It’s in the Qualisys TSV header.

Number of analog channelsid num of analogtype Kernel, Int datatype (Kernel Catalog).type id kernel, int

Number of analog channels. This value is not used actually by EyesWeb when readingthe TSV, however, it is used by the Qualisys software and must match the number ofcolumns in the corresponding * a.tsv file.

Analog frequencyid analog freqtype Kernel, Int datatype (Kernel Catalog).type id kernel, int

Analog frequency. This value is not actually used by EyesWeb when reading the TSV,however, it is used by the Qualisys software and refers to the sampling frequency of theanalog data which is available in the corresponding * a.tsv file.

Descriptionid descriptiontype Kernel, String datatype (Kernel Catalog).type id kernel, string

32

A free-text description to be wrote in the TSV header

TimeStampid time stamptype Kernel, String datatype (Kernel Catalog).type id kernel, string

A string representing the timestamp (date and time) of the recording session (e.g.,Friday, December 21st, 2012

Labels modeid labels modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:CustomFirst non-empty frameFirst frameKinect Microsoft SDKKinect OpenNIFile

domain [ 0, 6 )Specifies how the block decides which labels are written to file. See the block notes for

details

33

4.1.8 SmpteDecoder

bitmap

class name SmpteDecodercatalog name SIEMPREcatalog id SIEMPREclass id smpte decoderauthors Paolo Coletta

Decodes an SMPTE timecode from the audio signal.

Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), withminor changes to build under Win32. The derived source code is available at https://svn.infomus.org/public/libltcsmpte/trunk


Inputs

Audio Streamid inputtype Base, PCMAudioBuffertype id base, pcm audio buffer



The input audio stream which should contain the SMPTE timecode track

Outputs

Timecodeid outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

The decoded timecode

34

Parameters

TimeCode Trackid time code tracktype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 0, +infinity )

The zero-based index of the timecode track. If it is greater than the number of availablechannels, the last channel will be used.

Output correction modeid output correction modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:NoneOutput offsetBuffer start

domain [ 0, 3 )In general the SMPTE timecode might be not aligned with the audio buffer. This

parameter specifies how to cope with this issue.None: the error caused by this misalignemnt is neglectedOutput offset: an addition output is added to the block. The output specifies where

the SMPTE timecode started with respecte to first sample of the audio buffer. The offsetmight be negative

Buffer start: the output timeoeis adjusted such that it can be interpreted as the timecodeassigned to the first sample of the buffer

Enable Output Lockedid enable output lockedtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

Enable/disable an output providing info about whether the SMPTE signale has beedlocked


layout


domain [ 0, 5 )

35

The framerate of the SMPTE timecode, i.e., how many timecode words are available ina second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected



36

4.1.9 SmpteEncoder

bitmap

class name SmpteEncodercatalog name SIEMPREcatalog id SIEMPREclass id smpte encoderauthors Paolo Coletta

Encodes an SMPTE timecode to an audio signal.

Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), withminor changes to build under Win32. The derived source code is available at https://svn.infomus.org/public/libltcsmpte/trunk


Inputs

Audio clockid inputtype Base, Audio clocktype id base, audio clock



The input audio clock

Outputs

Outputid outputtype Base, PCMAudioBuffertype id base, pcm audio bufferinplace id *no*inherited id *no*

The encoded timecode

37

Parameters

TimeCodeid time codetype Kernel, Time datatype (Kernel Catalog).type id kernel, time

The initial timeocde.

Resetid resettype Kernel, Trigger datatype (Kernel Catalog).type id kernel, trigger

Reset the encoder to the initial timecode value


layout






Enable timecode outputid enable timecode outputtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool

If this parameter is true, an additional output is added to the block which providesthe interpolated timecode of the begining of the audiobuffer (i.e., the timecode virtuallyassigned to sample 0 of the buffer)

38

4.1.10 TimeCodeGenerator

bitmap

class name TimeCodeGeneratorcatalog name SIEMPREcatalog id SIEMPREclass id timecode generatorauthors Paolo Coletta

Generatae a timecode starting locked to the input audio clock signal.


Inputs

Audio clockid inputtype Base, Audio clocktype id base, audio clock



The input audio clock

Outputs

Timecode outputid timecode outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*

The timecode virtually assigned to the sample 0 of the output buffer

Parameters

TimeCodeid time codetype Kernel, Time datatype (Kernel Catalog).type id kernel, time

39

The initial timeocde.

Resetid resettype Kernel, Trigger datatype (Kernel Catalog).type id kernel, trigger

Reset the encoder to the initial timecode value

40

4.1.11 WaveFileWriter

bitmap

class name WaveFileWritercatalog name SIEMPREcatalog id SIEMPREclass id wave file writerauthors Paolo Coletta

Write the input time series as a wav file.


Inputs

TimeSeriesid inputtype Kernel, Generic datatypetype id kernel, generic datatype


read only/read write read writereferred as inplace *no*referred as inherited *no*

Time series to be saved to file

Required interfacesKernel, StaticTimeSeries

Parameters

Filenameid param filenametype Kernel, String datatype (Kernel Catalog).type id kernel, string

layout

Filename,MustExist=true,SaveMode=true,OverwritePrompt=true,Filter=”Wave files (*wav)—*.wav—All files (*.*)—*.*—All files (*.*)—*.*——”

Name of the wav file. Use an empty name to stop the recording

41

Formatid param formattype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:32bit float (float)64bit float (Double)Normalized 32bit float (float)Normalized 64bit float (Double)

domain [ 0, 4 )Defines the format used to write the sample into the file : ’float’ each sample is writed

as 32bit floating value; ’double’ each sample is writed as 32bit floating value;

Titleid param titletype Kernel, String datatype (Kernel Catalog).type id kernel, string

Title tag. It will be mapped to the ’INAM’ Exif tag

Datetimeid param datetimetype Kernel, String datatype (Kernel Catalog).type id kernel, string

Datetime tag. It will be mapped to the ’ICRD’ Exif tag

Commentid param commenttype Kernel, String datatype (Kernel Catalog).type id kernel, string

Comment tag. Free text that will be mapped to the ’ICMT’ Exif tag

Timecodeid param timecodetype Kernel, String datatype (Kernel Catalog).type id kernel, string

Timecode tag. It will be mapped to the ’ISMP’ Exif tag.

AddMetaDataid add metadata infotype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:No metadataUsing InfoChunkUsing iXml

domain [ 0, 3 )

42

Defines the format used to write metadata into the file.

43

4.1.12 DeckLinkInput

bitmap

class name DeckLinkInputcatalog name DeckLinkcatalog id decklinkclass id decklink inputauthors Paolo Coletta

Acquire synchronized audio/video from a DeckLink card.

Notes With the partial support of the EU ICT Project 250026 - SIEMPRE (SocialInteraction and Entrainment using Music PeRformance Experimentation), 2010-2012

Outputs

Outputid video outputtype Base, Imagetype id base, imageinplace id *no*inherited id *no*

AudioOutputid audio outputtype Base, PCMAudioBuffertype id base, pcm audio bufferinplace id *no*inherited id *no*

Parameters

Device Indexid device indextype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 0, +infinity )

The zero-base index of the Decklink card, in the case that more than one is installed inthe system

44

InputModeid input modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layoutCombo Box:Audio/VideoVideo only

domain [ 0, 2 )Specifies whether to capture audio, or video, or both

Video Modeid video modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layout

Combo Box:NTSC 720x486 30/1.001fps interlacedNTSC 720x486 30/1.001fps interlaced (3:2 pulldown)PAL 720x576 25fps interlacedNTSC 720x486 30/1.001fps progressivePAL 720x576 25fps progressiveHD 1920x1080 24/1.001fps interlacedHD 1920x1080 24fps interlacedHD 1920x1080 25fps progressiveHD 1920x1080 30/1.001fps progressiveHD 1920x1080 30fps progressiveHD 1920x1080 25fps interlacedHD 1920x1080 30/1.001fps interlacedHD 1920x1080 30/1.001fps interlacedHD 1920x1080 50fps progressiveHD 1920x1080 60/1.001fps progressiveHD 1920x1080 60fps progressiveHD 1280x720 50fps progressiveHD 1280x720 60/1.001fps progressiveHD 1280x720 24/1.001fps progressive2K 2048x1556 60fps interlaced2K 2048x1556 60fps interlaced2K 2048x1556 60fps interlaced

domain [ 0, 22 )Select the resolution and framerate of the video stream

45

Pixel Formatid pixel formattype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layoutCombo Box:UYVY 4:2:2 packedBGRA 4:4:4 raw

domain [ 0, 2 )Select the pixel format of the video stream

Num Audio Channelsid num channelstype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )

Specifies the number of audio channels to capture

Audio Sample Formatid sample formattype Kernel, Int datatype (Kernel Catalog).type id kernel, int

layoutCombo Box:16bits signed32bits signed

domain [ 0, 2 )Specifies the format of each sample of audio

46

Part III

Appendices

47

Appendix A

SIEMPRE Binary Format

A.1 SIEMPRE – Binary file format

Monodimensional timeseries will be saved or converted to time-aligned Broadcast WaveFormat compatible files (i.e., extended WAVE files).

The SIEMPRE project will exploit subset of the Broadcast Wave Format standard. Inparticular it will fix the number of channels to 1 (monodimensional data), the format toWAVE FORMAT IEEE FLOAT (0x03), and the bits per sample to 32.

For a general description of Broadcast Wave Format and Wave files you may refer tothe Microsoft documentation or start from the following links:

• Audio File Format Specifications

• SonicSpot

A.1.1 Format description

The WAVE (sub-)format used for SIEMPRE has the following structure:

Field Length ContentChunkID 4 “RIFF”

ChunkSize 4Size, in bytes, of the RIFF chunk. Should beequal to the length, in bytes, of the file minus 8.

WaveID 4 “WAVE”

Wave chunks . . .

Some chunks. Each chunk is identified by afourCC (four characters identifier) followed bythe size, in bytes, of the chunk (the size doesnot include the FourCC and the size itself). Twomandatory chunks are the “fmt” chunk and the“data” chunk, which contains the characteristicsof the file (sample rate, num channels, samplesize, etc.) and the sample data respectively.

48

http://www-mmsp.ece.mcgill.ca/documents/audioformats/wave/WAVE.html

http://www.sonicspot.com/guide/wavefiles.html

The fmt chunk and the data chunks have the following structure:

Fmt ChunkField Length Content

ChunkID 4 “fmt ”

ChunkSize 4For the SIEMPRE case the value is 16, a genericreader should be prepared to face different sizestoo.

FormatTag 2For the SIEMPRE case the value isWAVE FORMAT IEEE FLOAT, i.e., 0x03.

NumChannels 2For the SIEMPRE case the value is fixed to 1,i.e., monodimensional channels

NumSamplesPerSeq 4Sampling rate (e.g., somthing around 250.0 forkinematical sensors)

AvgBytesPerSec 4Data rate, in bytes, of the file. Can be computedas NumSamplesPerSeq * BlockAlignment

BlockAlignment 2

Size of a block (a sample for each channel). Canbe computed as NumChannels * (BitsPerSample/ 8). For the SIEMPRE case NumChannels isfixed to 1, BitsPerSample is fixed to 32, thusBlockAlignement is fixed to 4.

BitsPerSample 2Number of bits for each sample. For the SIEM-PRE case this is fixed to 32 (single precisionfloating point numbers)

Data ChunkField Length Content

ChunkID 4 “data”

ChunkSize 4For the SIEMPRE case the value is 4 (size ofsample) multiplied by the number of samples

Data . . . This is the actal data: sequence of float numbersin the SIEMPRE case

Padding bytes 0 or 1In the SIEMPRE case this is not needed as datais aligned. In the general case this is only neededif data is not aligned to a 16bits boundary

49

A.1.2 Application compatibility

The files produced by the Wave File Writer block have been tested to load in the followingapplications:

• Players

– VLC media player

– Windows Media Player

– Quick Time player

• Music Software

– Audacity

– Sony Sound Force

– Sony Vegas Pro

• Computing environments

– Matlab: does not load additional info about the file. This is a general limitationfor the WAVE format support in Matlab.

50

Documents

Eyesweb Siempre User Manual