Music Visualizer 2.0
A better music visualizerRafi Khan, Mike Levine, Jacob Metrick, and Hari GanesanConceptStreaming music and visualizer to mobile deviceInteractive visualizer with touchesSync to multiple phonesTwo user functions:New: a user (master) creates a sessionJoin: a user joins an existing session
What Is It Good For?Audience: teens, young adults, other bored peopleFun group activityParties! Bar Mitzvahs! Weddings!
Technical ChallengesStreaming images to each phoneCombining touch events from various phonesIntegrating touches into visualizer quicklyNetwork OptionsTCP/HTTPUDPSome combination of both
Technical ChallengesThe Visualizer!Need to do audio parsing for volume, frequency detectionFFT, amplitude, etc.Map touch events to changes in volume, or frequency, or something elseMake it fastMake it cool
The Client SideListening to touch eventsMixing eventsDifferentiating between touches and swipes
Rendering the bitmap images successfully
The Server SideGetting packets and transferring to input of visualizerSo far: one threadGetting data via UDP from client
The Server SideUsing Processing (processing.org)Making visualizer based on inputsSo we need three more threads:One for doing the actual rendering One for receiving video data from processingOne for sending video data to client(s)Total threads: 4! Wow!Workflow (creating session)Phone posts to /newServer responds with a port number (eg. 10001)Server starts sending visualization data (compressed as JPG) on 10001Phone opens an HTTP connection on 10001, and renders the incoming JPG dataWhenever the phone detects a touch event, it sends the event via UDP
Workflow (joining session)Phone posts to /join with the port numberServer responds to confirm.Exact same workflow as with creating!
Future ConsiderationsMaking the full-scale appLimiting number of people/sessionViewers/Enablers?Private/Public sessions?More optionsMultiple visualizersDifferent touch functionalities