Upload
vohanh
View
214
Download
0
Embed Size (px)
Citation preview
DesigningInteractiveAudiovisualSystemsforImprovisingEnsembles
WilliamHsu
DepartmentofComputerScienceSanFranciscoStateUniversity
SanFranciscoCA94132USA
Abstract.Since2009,Ihavebeenworkingonreal-timeaudiovisualsystems,usedinmanyperformanceswithimprovisingmusicians.Real-timeaudiofromaperformanceisanalysed;theaudiodescriptors(alongwithdatafromgesturalcontrollers)influenceabstractanimationsthatarebasedongenerativesystemsandphysics-basedsimulations.Themusiciansareinturninfluencedbythevisuals,essentiallyinafeedbackloop.Iwilldiscusstechnicalandaestheticdesignconsiderationsforsuchsystemsandtheirintegrationintothepracticesofimprovisingensembles,andsharesomeexperiencesfrommusiciansandaudiencemembers.
Keywords:Interactiondesign,improvisation,audiodescriptors,generativesystems,physics-basedsimulations
Introduction
ForyearsIhavebeeninvolvedinfreeimprovisationwithbothacousticinstrumentsandliveelectronics.Asaperformer,softwaredesignerandlistener,Ihavebeenveryinterestedintheassociationsmadebymyself(andothers)betweensonicevents/gesturesandvisual/physicalphenomena.IamattractedtovisualsystemsthatexhibittheoppositionsandtensionsthatIenjoyinimprovisedmusic.
Since2009,Ihavebeenworkingonreal-timeaudiovisualsystemsfornon-idiomaticfreeimprovisation.Ihaveusedthesesystemsinover50performanceswithimprovisingmusicians,includingChrisBurns,JohnButcher,JamesFei,GinoRobair,BirgitUlher,andmanyothers;manysucheventswereevening-longconcertsinvolvinguptosixdifferentsystems/pieces.Performanceshavebeenhostedatavarietyofvenues,suchasZKM(Karlsruhe),STEIM(Amsterdam),CNMAT(Berkeley),theSonglinesseriesatMillsCollege,theSanFranciscoElectronicMusicFestival,andtheNIMEandSMCconferences.
Figure1showsatypicalblockdiagramofmysystems.Inperformance,twoormorechannelsofaudioenterthesystemfrommicrophonesorthevenue’ssoundsystem.TheaudioisanalyzedbyaMax/MSPpatchthatextractsestimatesofloudnessandtempo,andsometimbralfeatures.AudiodescriptorsaresentviaOpenSoundControlmessagestoananimationenvironment,usuallyimplementedinProcessingorOpenFrameworks.Theseinteractiveanimationsareinfluencedbythereal-timeaudiodescriptorsfromthemusicians’performance,andbyphysicalgesturesfromcontrollers.
Videogeneratedbytheanimationenvironmentsisusuallyprojectedbehindorabovethemusicians.Figure2showsatypicalstagesetup,fromasetbyEKG(KyleBruckmannandErnstKarel)atthe2013SanFranciscoElectronicMusicFestival,withvideogeneratedbymysystemTes.Theanimationsarevisibletothemusiciansandinfluencetheirperformance,thusformingafeedbackloop.EachsystemhasbothcomponentsthatIcontrol,andautonomouscomponentswiththeirownbehaviouralrules.Performingwithoneofmysystemsinvolvesongoingnegotiationbetweenthecontrollablecomponentsandtheautonomousmodules.
Figure1.BlockDiagramofInteractiveAudiovisualPerformanceSystem
DesignGoalsandPracticalConsiderationsThesearetheinitialgoalsformysystems:
• Eachsystemwillbeprimarilyusedinthecontextofabstractfreeimprovisation.• Therewillbeminimaluseoflooping/pre-sequencedmaterials;eachsystemwillbehavelikeacomponentinan
improvisingensemble.• Eachsystemwillbea“playable”visualinstrument;systembehaviourcanbeinfluencedbyphysicalcontrollers.• Eachsystemhasautonomouscomponentsthatguidesitsbehaviour;thisbehaviourmaybeinfluencedbyreal-time
audio.• Eachsystemshouldevokethetactile,nuancedandtimbrallyrichgesturescommoninfreeimprovisation.• Thereshouldbecross-referencesbetweenaudioandvisualevents.However,overlyobviousmappingsshouldbe
avoided.
Figure2.PerformancebyEKGwithvideobyBillHsu,2013SanFranciscoElectronicMusicFestival.Photo:PeterBKaars.com
Non-idiomaticfreeimprovisationisasocialmusic-makingpractice,producingopensonicconversationsthattendnottobeeasilyinterpretedoranalysedinnon-musicalterms.Gaveretal.(2003)discusstheroleofambiguityininteractivedesign;someoftheircommentsareusefulforunderstandingthepracticeandexperienceoffreeimprovisation,andhowonemightapproachaudiovisualdesignforconcertsofimprovisedmusic.Forexample,theauthorscouldeasilyhavebeenreferringtosonicgesturesinimprovisation,which“maygiverisetomultipleinterpretationsdependingontheirprecision,consistency,andaccuracyontheonehand,andtheidentity,motivations,andexpectationsofaninterpreterontheother.”Anothercommentwouldfindresonancewithmanyfansofimprovisedmusic:“…theworkofmakinganambiguoussituationcomprehensiblebelongstotheperson,andthiscanbebothinherentlypleasurableandleadtoadeepconceptualappropriationoftheartefact.”
Topreservetheessentialopennessandambiguityofafreeimprovisation,Ifeelthatthevisualsshouldnotover-determinethenarrativeoftheperformance.Hence,mostofmyworkutilizesgenerativeabstractvisualcomponentsthatexhibitarangeofbehaviours.Mypreferenceisforunstable,evolvingformsthatfacilitatesettinguptensionsbetweenabstractandreferentialelements,forarichervisualexperience.Forexample,aparticleswarmorsmokesimulationmaymoveincomplex,pseudo-randomconfigurations,orself-organizeintorecognizablestructures.ThesetransitionsevokeFriedrichHayek’sconceptof“sensoryorder”,whereinobserversorganizerawchaoticstimuliintoperceptuallyrecognizableobjects(Hayek1999).
Eachofmysystemsisprimarilybasedonasinglecomplexprocess,usuallyagenerativesystemoraphysics-basedsimulation.Themovementandevolutionofthevisualcomponentsineachsystemfollowtherulesoftheunderlyingprocess.Audiodescriptorsfromthereal-timeperformanceaudioaffectmostlyhigh-levelparametersofthebaseprocess;theydonotmapdirectlytolowleveldetailsofthevisualcomponents.Frommyexperimentsandobservations,allowingtheunderlyingprocesstodeterminethelowleveldetailsofthevisualsresultsinmore“organic”andaestheticallyconsistentmovementsandtransitions.
Eachsystemreceivesseveralinputstreams,representingeventsoccurringat(often)widelydifferentrates.Forexample,anaudiodescriptorrepresentingawell-definedpercussiveonsetmayoccurextremelyinfrequently,whileonethatrepresentsloudnessoracontinuoustimbralcharacteristicmaybepresentedatregularintervalsof(say)100-200milliseconds.Inaddition,eventsintheinteractiveanimationsubsystemalsohavearangeoftemporaldistributions.Arapidonsetvisualevent,suchasthesudden“birth”ofasmallclusteroftinyobjectsthatexpandsintovisibility,maytakeafewhundredmillisecondsfromtheinitialtriggertoitsfinal,relativelystablestate.Ontheotherhand,abroadsweepinggesture,representing(forexample)atidalflowinafluidsystem,maytakeseveralsecondstocomplete,withitseffectsbeingvisiblelongafterthegestureinitiation.Hence,caremustbetakenwitheachsystemtomanageeacheventtypebasedontypicalratesofoccurrence.Forexample,itmaybeintuitiveforapercussiveaudioonseteventtotriggertheformationofasmallobjectclusterintheanimation.However,iftheliveperformerisabusypercussionist,thevisualenvironmentmayquicklybecomeclutteredandoverwhelmedwithobjects.Acertainamountofthresholdingisalwaysnecessaryformanagingtheratesofautomaticevents;thesystemoperatorshouldalsohavetheoptiontoadjusttheresponsivenessoftheanimationsystemtoselectedeventtypes.
Thereisacoreofexperiencedmusicalcollaboratorswhohaveworkedwithmysystemsregularly.Whenworkingwithregulars,itiseasyforustoquicklyconvergeona“setlist”ofpiecesforanevening’sconcert.Abriefminuteorsobeforetheperformancewitheachpieceisusuallysufficientformusicianstore-familiarizethemselveswiththesystem.Ialsocollaboratewithmusicianswhohaveneverworkedwithaudiovisualperformancesoftware,withminimaltimeforrehearsalsbeforeanevening’sconcert.Hence,interactionmodalitieshavetobeintuitiveandeasytoexplain;withonlyafewminutes’preparation,amusicianshouldunderstandasystem’sbehavioursufficientlytoexploreandimprovisewithit,ontopofthecognitivedemandsofnegotiatinganimprovisation.
Collaboratingmusicianshavetakenseveralapproachestoworkingwithmyaudiovisualsystems.Afewhavechosennottolookatthevideoatall;theyfeltthattheywantedtofocusonsound.Somehavementionedthetemptationtotryto“push”theanimationsintospecificoutcomes,viatheirsonicgestures;thistemptationmayobviouslydistractfrommusic-making.Mostofmyfellowperformershavepreferredsomeconversationsaboutthechosensystemsbeforeaperformance,andactivelyengagewiththevideowhenplaying.
RelatedWorkAudiovisualperformanceiswidespreadintheclubmusiccommunity.TheVJLaborwebsite(http://vjlabor.blogspot.com),forexample,showcasessuchwork.Themusicisoftenbeat/loop-based,withrelativelystabletemposandeventrates,andlittlevariationinspaceoruseofsilence.Thevisualsoftenincorporateloopsandsimplecycles,andworkwithpre-recordedfootagethatmaybemanipulatedinreal-time.
Interactivevideoisalsoacomponentinmanycompositions.Forexample,violinistBarbaraLueneburg’sDVDWeaponofChoice(http://www.ahornfelder.de/releases/weapon_of_choice/index.php)includesasamplingofcomposedpieceswithliveorstaticvideo,byAlexanderSchubert,YannisKyriakides,DaiFujikura,andothers.Someofthepiecesincorporateliveaudioorsensorinput.
Inmyexperience,interactivevisualstendtobesignificantlylesscommoninfreeimprovisation.ThestrategiesthatworkwellintheVJandcomposercommunities,inmyopinion,oftendonotmapwelltonon-idiomaticfreeimprovisation,witheventdensitiesthatmayvarywidelyinshorttimewindows,useofspaceandsilence,andoverallconversationsthatdevelopfrommomenttomoment,withlittleornopre-arrangedcompositionalstructure.Thetechnologiesandinternaldetailsofthesesystemsalsotendtobepoorlydocumented.
AmajorearlyaudiovisualperformanceprojectinvolvingimprovisersisLevinandLieberman’sMessadiVoce(LevinandLieberman,2004).TheprojectappearstobeprimarilydesignedaroundJoanLaBarbaraandJaapBlonk,twovocalistswhoarerenownedfortheirexplorationofextendedvocaltechniquesinperformance.Real-timecameraandaudioinputfromtheperformersdriveanarrayofgenerativeprocesses,includingparticlesystemsandfluids.Messacomprisestwelveshort,theatricallyeffectivesectionsoveratotalof30-40minutes;myownworktendstowardlongersectionsof8-10minuteseach,witheachsectionbasedonadistinctgenerativeprocess,formoreextended“conversations”.InMessa,LaBarbaraandBlonkarealwaysthecentersofvisualattentiononstage;camera-basedtrackingoftheirbodiesisasignificantcomponentoftheliveinteractions.Withmysystems,mycollaboratorstendtofocusonworkingwithabstractsound;whilethemusiciansarevisibleonstage,theirmovementsarenottracked,andthevisualattentionoftheaudiencetendstobeprimarilyonthelivevideo.Fromtheonlinedocumentation,itisnotclearifMessadiVocehasbeenperformedwithimprovisersotherthanLaBarbaraandBlonk,orwhetherithasbeenrevivedsincethe2004/5performancesandinstallations.
Morerecentprojectsincludetheperformanceduoklippav,focusingonliveaudiovisualcutup/splicing(CollinsandOlofsson,2006);trombonistAndyStrain’saudiovisualpiecesforschoolchildren(http://andystrain.com);BillyRoisz’slivevideoworkwhichincorporatesrelativelyminimalisttransformationsofstillimages,foundfootageandanalogartifacts(http://billyroisz.klingt.org/video-works);andWilliamThibault(http://www.vjlove.com),whooftenmanipulatesdensedatanetworkvisualizationsinperformancewithfreeimprovisers.
ExampleSystemsTodate,Ihavebuiltoveradozendistinctaudiovisualperformancesystems,somewithnumerousvariants.Mostofthesesystemshaveparticipatedinnumerousperformanceswithimprovisingmusicians.ThefirstwastheparticlesystemInterstices,introducedin(Hsu2009).Iwillfocusmostlyonfourofthelatersystems:FlowForms,Flue,Fluke,andLeishmania.
FlowFormsisbasedontheGray-Scottdiffusion-reactionalgorithm(Pearson1993),whichhasbeenwidelyusedinthegenerativeartcommunity.Twosimulated“chemicals”interactanddiffuseina2Dgrid,accordingtosimpleequations.Parametersthatcontroltheconcentrationsofthesimulatedchemicalsaremodifiedbyandtrackactivityfromthereal-timeaudio.Sonicallyactivesectionsresultinmorerobustvisuals;longperiodsofsilencewillresultfragmentationofthepatterns,eventuallytoleaveadarkscreen.Inaddition,hiddenmasksrepresentingshapesorimagescanbeintroducedtoguidetheformationofvisiblepatterns.Figure3showsanexampleofasimulatedchemicalflowingintoapre-loadedimagemask.
Figure3.Gray-ScottDiffusionReactionProcesswithTransitionintoHiddenImageMask
Flueisasmokesimulation,basedonaportofJosStam’sstablefluidscode(Stam1999).Twosmokesourcesmovethroughspace,eachactivatedandpushedbyareal-timeaudiostream.Again,activitylevelsinthevisualsimulationtracksroughlyactivitylevelsintheaudio,buttherearenosimplemappingsoflowlevelbehaviors.Hiddenmaskscanbeintroducedtoconstrainthemovementofthesmoke.Figure4showssimulatedsmokecoalescingintotheshapeofaskull,thendispersing.
FlukeisbasedonStephanRafler’sextensionofConway’sGameofLife(Rafler2011).Thealgorithmisverycompute-intensive;IadaptedTimHutton’sOpenCLimplementationthatrunsontheGPU.Real-timeaudioactivitytriggerstheformationofstructuresina2Dspace;algorithmparametersareconstantlymodulatedforthevisualactivityleveltotrackactivitylevelsintheperformanceaudio.
Figure4.SmokeSimulationwithSmokeSourcesfillingHiddenSkullMask
Leishmaniaisaninteractiveanimationenvironmentthatvisuallyresemblescoloniesofsingle-cellorganismsinafluidsubstrate.Eachcell-likecomponenthashiddenconnectionstoandrelationshipswithothercomponentsintheenvironment.Thecoloniesevolveand“swim”throughthesubstrate,basedonacombinationofcolonialstructureandinter-relationships,andflowsinthefluidsubstratethatmightbeinitiatedbygesturalinput.LeishmaniahasbeenusedextensivelyinperformancewithChristopherBurns’Xenoglossiainteractivemusicgenerationsystem.Thesetwosystemscommunicatewithoneanotherinavarietyofways.Theanimationisinfluencedbythereal-timeanalysisofaudiofromXenoglossia.Inaddition,thetwosystemsexchangeOSCnetworkmessages,informingeachotherofeventsthataredifficulttoextractfromautomaticanalysis,suchaspendingsectionchanges,structuredrepetitionswithvariations,andtheconfigurationofanimationcomponents.
Reactions
Sofar,Ihavemostlyfocusedonbuildingcomplexaudiovisualsystemsforliveperformances“inthewild”;littletimehasbeenspentsettinguplaboratory-likesituationsformoreformalevaluations.Evaluationproceduressuchasthosedescribedin(HsuandSosnick,2009),targetinginteractivemusicsystemsfromthepointsofviewofboththeperformingmusiciansandaudiencemembers,mightbeadaptedforinteractiveaudiovisualsystems;thepresenceofcomplexvisualcomponents,withdisparatebehaviouraltypes,significantlycomplicatesuchevaluations.Instead,wewillsummarizesomegenerallypositivefeedbackonperformances,frommusiciansandaudiencemembers;theyappeartosupportsomeofouroriginaldesigngoals.
FrequentOakland-basedcollaboratorGinoRobair(percussion/electronics):“WhatIfindfascinatingishowtheinteractivepieceschallengethemusicianswhoplaythem.Oneimmediatelywantstofigureouthowto"game"thesystem,andcontrolitwithwhatweareplaying.Yetthealgorithmsconfoundthat,forcingthemusicianstotreatthecomputersystemasarealduetpartnerwhois"listening"butnotnecessarilyrespondinginapredictableway(Robair,personalcommunication).”
Anotherfrequentcollaborator,Hamburg-basedBirgitUlher(trumpet/electronics):“WhatIespeciallylikeabout[Bill’sanimations]isthestrongconnectiontothemusicwithoutbeingtooobviousorillustrative.WorkingwithBill'sanimationsopensupalotofnewlevelsofcommunication,theaudioinputofthemusiciansaretransformedintovisualswhichcanbeseenonascreenwhileplaying,soitisakindofseesawofinfluences.Thevisualsinfluencetheplayersandviceversa,alsothecommunicationbetweenthemusicianshasanadditionallevelsincetheirinteractionisseenonthescreenaswell(Ulher,personalcommunication).”
London-basedJohnButcher(saxophones),whowithRobairandmyselfcomprisetheaudiovisualtrioPhospheme:“Thesystems'responsesmaybesubtleorstriking,butalwaysseemorganic.There'sacreative,two-wayencounterwherethemusicianisonlypartlydrivingtheprocessandtheinspirationtheytakefromtheevolvingvisualreactionsopensspaceforsomeofthemoreunpredictableconsequencesonehopesforwithhumaninteractions.Fromastructuralpointofviewit'sinterestingthatthiscanhappenwithintheconsistent"flavour"ofaparticularvisualidea,givingeachimprovisationavaluablecoherence(Butcher,personalcommunication).”
Robair,UlherandButcherhaveallworkedwitharangeofmyaudiovisualpieces.Chicago-basedChrisBurns(electronics)hasmostlyworkedwithLeishmania,onanumberofoccasions;heshareshisexperiences:“[Leishmania]offersarichvarietyofvisualbehaviorswithinafocussed,consistent,admirablystarkandunapologeticallydigitalaesthetic.Thatvarietymakesitasuitableaccompanimentforawiderangeofmusicalchoices;italsomakesthesystemunpredictable,withmusicalstimulileadingtoanumberofpossiblevisualreactions.Leishmaniaprovidesstimulusaswellasresponse.Theanimationsareevocativewithoutbeingcommanding-thevisualsinspireformalandgesturalideasinmyperformance,buttheyneverfeelconstraining,limiting,ordemandingofaparticulartypeofmusicalresponse.Inshort,Leishmaniafeelslikeaverysophisticated,capable,andprovocativeduopartner-whichhaseverythingtodowithduopartnerwhoconstructedit(Burns,personalcommunication).”
ReviewerStephenSmoliarwroteaboutaperformancein2014withJamesFei,GinoRobair,andOferBymel:“…Itseemedclearthat[Hsu]was“playing”hisinteractiveanimationsoftwarefollowingthesamelogicandrhetoricoffreeimprovisation…Whatwasmoststriking…washowHsucouldusevisualcharacteristicssuchasflowandpulsetoachieveanimationsthatprobablywouldhavebeenjustasmusicalhadtheybeendisplayedinsilence.”(Smoliar2014)
SummaryIhavebeenluckytoworkwithmanyinspiringimprovisingmusiciansonaudiovisualprojects.Theircreativeandfascinatingapproachestoperformancesituations,frommanagingtexturalandgesturalmaterials,toworkingwithspaceandpulseinthecontextofnon-referentialabstractmaterials,haveinformedmyaudiovisualsystemdesignsatmanydifferentlevels.I’dliketothankmycollaboratorswhohavebeengenerouswiththeirtimeandfeedback.Ihaveshared
somenotesandexperienceshere,buthighlyencourageviewingoneoftheperformancesonvideoor(especially)live;freeimprovisation,withorwithoutvisuals,isverymuchasocialmusic-makingpracticethatisbestexperiencedinliveperformances.Aselectionofvideodocumentationisavailableonline:
SetwithJamesFei(reeds)andGinoRobair(percussion)atOutsoundSummitFestival2014,SanFrancisco:https://www.youtube.com/watch?v=NLFj26zfqsI
PerformancewithChrisBurns(electronics)ofXenoglossia/LeishmaniainComputerMusicJournalonlineanthology(requireslogin):http://www.mitpressjournals.org/doi/abs/10.1162/COMJ_x_00276#.VJtz6LiALw
ShortdemoofFluke(musicbyBirgitUlherandGinoRobair):https://vimeo.com/106125702
ShortdemoofFlowForms(musicbyChrisBurns):https://vimeo.com/78739548
ReferencesCollins,NickandOlofsson,Fredrik.2006.Klippav:livealgorithmicsplicingandaudiovisualeventcapture.InComputerMusicJournal,Vol.30,No.2(Summer2006),pp.8-18.
Gaver,William,Beaver,JacobandBenford,Steve.2003.AmbiguityasaResourceforDesign.InProceedingsofCHI’03.
Hayek,Friedrich.1999.TheSensoryOrder:AnInquiryintotheFoundationsofTheoreticalPsychology.UniversityofChicagoPress,Chicago,IL.
Hsu,William.2009.Somethoughtsonvisualizingimprovisations/improvisingvisualizations.InProceedingsof6thSoundandMusicComputingConference(SMC'09).
Hsu,WilliamandSosnick,Marc.2009.EvaluatingInteractiveMusicSystems:AnHCIApproach.InProceedingsofInternationalConferenceonNewInterfacesforMusicalExpression(NIME2009)
Levin,GolanandLieberman,Zachary.2004.In-situspeechvisualizationinreal-timeinteractiveinstallationandperformance.InNPAR2004Jun7(Vol.4,pp.7-14).
Pearson,J..1993.Complexpatternsinasimplesystem.Science261,189-192.
Rafler,Stephan.2011.GeneralizationofConway’s“GameofLife”toacontinuousdomain---SmoothLife.2011.arXivpreprintarXiv:1111.1567.
Smoliar,Stephen.2014.BillHsubringsmusicalrhetorictohisimprovisedinteractiveanimations.2014.RetrievedJanuary10,2016fromhttp://exm.nr/1tzVSlp.
Stam,Jos.1999.Stablefluids.InProceedingsof26thannualconferenceonComputerGraphicsandInteractiveTechniques(SIGGRAPH’99).