01/02/04 1

NEESGrid Data Turbine Video as Data

These notes are based on a lot of work from people from the NEES sites and the NEESGrid SI

Data/Deployment team. Comments to Charles Severance (csev@umich.edu)

01/02/04 2

Overview

• This is a design document to build common understanding and consensus - it will change over time

• These new requirements and designs come out of the Experiment Based Deployment efforts at Minnesota and UC Davis

• They represent significant new work in the area of data, video, images, and telepresense

• The work is being undertaken jointly by the SI team and the sites involved as part of the EBD

• Much of this is triggered by the availability of Data Turbine and making use of its features and rethinking how we approach solving problems with Data Turbine as a framework

01/02/04 3

DT Main System

PTZ/USB

StillCapture

DT Client

Capturing

CameraControl

DAQData

CaptureDT Client

AudioAudio

EncoderDT Client

BT848 rbnbjcap

DT Client

Still Capture - Minnesota / Paul HubbardVideo capture - From CreareAudio capture - From Creare (TBD)Data Capture - From sites (upwards compatible)NEES NSDS Driver - Paul HubbardCamera Control Plugin - UM / Minnesota

NEESNSDSDriver

Each still capture produces two channels - Small 1-5fps stream + large single images when picture is taken

ControlPlugin

01/02/04 4

DT Main System

User Views / Still Camera

Quicktime Storage System

QuicktimeSlicingTool

Still Image / Camera Control

~

< >^

^

< >

ControlPlugin

Data Viewer

Thumbnail + Audio + Data

< > +

ThumbnailProcess

Thumbnail - UBuffalo / UmichiganThumbnail viewer - Creare v1 / Michigan v2Camera Control Applet - Minnesota / MichiganQuicktime Slicing tool - Michigan (lower priority)Stored Data Viewer (existing) - MichiganJPEG Viewer - CreareQuickime Viewer - Apple

01/02/04 5

DT System

Robotic Control

RoboticControl

Robotic Control Applet

< >^

^

BT848Video

Frames

DT Client

ControlPlugin

Robotic Control: UCDavisRBNB Control Plugin: Michigan

01/02/04 6

Macintosh(es)DT Main System

RBNB to Streaming Quicktime

QT -> MP4Encoding

DT

Clie

nt

Quicktime Storage System

Quicktime Streaming System

QuicktimeStreaming

Server

Live Viewers

QT Encoding - MichiganQT Viewer - Apple :)

This effort is lower priority than the other efforts

01/02/04 7

DT Main System

RBNB and the Axis Format

OutboundAxis

DT

Clie

nt

InboundAxis

DT

Clie

nt

Texas Telepresense

This effort is lower priority than the basic video as data and still image capture, but is not expected to be too difficult.

Inbound Axis - Written by ORSTOutbound Axis - Written by ORST and uBuffalo (separate versions)Texas Telepresense - Chris StantonNEES Telepresense - ANL

This should interoperate with the standard Creare viewer

Web Browser

NEES / ANLTelepresense

01/02/04 8

DT Main System

RBNB and RTP Video

OutboundRTP

DT

Clie

nt

JMStudioRTP Viewer

Access Grid

AudioAudio

EncoderDT Client

BT848 rbnbjcap

DT Client

This effort has very low priority and the design is included for architectural completeness and to illustrate possibilities.

01/02/04 9

Software: JPEG Frame Capture

• Retrieves and compresses frames into JPEG and places them in a designated DT channel

• This is part of Creare distribution

01/02/04 10

Software: Audio Encoder

• Creare had this in the previous release based on a modified version of JMStudio but in the next release they rewrote the code as the JPEG encoder and did not port the audio code

• We are working with Creare to get this feature added to the JPEG Capture

• Written in JAVA on XP

01/02/04 11

Software: Still Image Capture

• This will send out a 320x240 thumbnail channel of JPEGS at a rate of 1-5 per second (or as quickly as we can send the thumbnails)

• When high resolution pictures are taken, they are also placed in a Data Turbine channel as JPEGs

• The control of the camera will be done using NTCP with a control gateway which is accessed through a Data Turbine plug-in.

• We are researching which camera to use and whether or not to use the PTZ (standard) interface to the camera or to use a proprietary interface - as of now, it looks as though we will use the a proprietary interface as it is the only way to get both streaming low-res for control feedback and still take high-res at the same time.

• Written in: ??? Using the SDK on XP ???• This will be developed by Minnesota and ANL• The camera control will be jointly developed by Michigan and Minnesota

01/02/04 12

Software: Data Capture

• This will develop a Data Turbine data injector which will comply to the DAQ -> NSDS Driver Protocol so existing DAQ code can communicate (and have the data injected into DT) with no changes to existing DAQ code.

• Responsibility: ANL

01/02/04 13

Software: Robotic Control

• This is a very specialized robotic control system which is being co-developed with the custom robotic system

• It will use an NTCP control protocol for control

• Responsibility: UC Davis (Stephanie)

01/02/04 14

Software: QT - MPEG 4 Encoding

• This software will subscribe to the JPEG video channels and the audio channel from Data Turbine and compress these into MPEG-4

• The MP4 data will be both sent to the QuickTime Streaming Server and into permanent storage at 640x480

• Users may see 8-10 second lag from real-time when watching using Quicktime Streaming

• Written in: Java on either Mac OS/X or XP

• This is lower priority than stills and JPEG streaming

• Responsibility: Michigan (Chuck)

01/02/04 15

Software: Control Gateway

• This software will act as a gateway between the applets which are providing the control interface and the internal control protocol (NTCP)

• This software will also be responsible for enforcing security so that only the properly authorized users will be able to perform the control operations.

• Written in: JAVA on Linux• Responsibility: Michigan (Terry)

01/02/04 16

Software: Still Image/Camera Control

• This software will provide the user interface for the control of the camera including pan, tilt, zoom, and “take picture”

• This software will subscribe to the thumbnail channel for the camera and display the images to allow the operator to get some feedback as they move the camera.

• Applet written in JAVA with some support in CHEF• Responsibility: Michigan (Lars) and Minnesota

(Chen)

01/02/04 17

Software: Thumbnail + Audio + Data Viewer

• This software allows the simultaneous display of an audio channel and some number of data and thumbnail channels.

• Thumbnail channels can either be still camera or video or video camera video.

• The user can launch the still image viewer from one of the thumbnails.

• The user can also launch the stored data viewer for a portion of the time scale with one or more of the video cameras and data channels.

• This software only displays sensor data in a numeric format.

• This software can view live data or view previous data.

01/02/04 18

Software: Thumbnail + Audio + Data (cont)

• The timeline can be scrubbed forwards and backwards. This will depend heavily on Data Turbine’s capability to move forward and backwards through data while maintaining data synchronization - while scrubbing, the images may not flip instantly - but within a second after letting go of the scrubber, the data and images should appear.

• The user can request a subset of the timeline to be extracted and viewed using the stored viewer - this may take a minute or so.

• Applet written in JAVA with some support in CHEF

• Responsibility: Michigan (Lars, Chuck)

01/02/04 19

Software: Data Viewer

• This is an updated version of the current Stored data viewer

• Applet written in JAVA with some support in CHEF

• Responsibility: Michigan (Lars)

01/02/04 20

Software: Robotic Control Applet

• This software will control the UCDavis Robot

• It will make use of the functionality used in the Camera Control applet

• Applet written in JAVA with some support in CHEF

• Responsibility: UC Davis (Stephanie) Michigan (Terry)

01/02/04 21

Software: Launch Desktop Application

• This is a relatively simple new feature where we use the repository capability of NEESgrid to map data out of the repository and launch it with a particular (configurable) MIME type.

• This will allow direct launching of the UCDavis 3D viewer from information stored in the repository.

• Responsibility: Michigan (Chuck) UC Davis

01/02/04 22

Other Important Details

• Streaming DT channels (audio, video, data) into the repository

• Metadata issues / project browser issues for automatic configuration and access control

01/02/04 23

Miscellaneous

• Michigan is writing a “fake video injector” so we can develop applications without needing a real live video source

01/02/04 24

Conclusion

• There is a lot of work here

• The designs have and will continue to change as we gain experience with Data Turbine