Mul$mediaTechniques* inAndroid · MediaAPI COMP 355 (Muppala) Multimedia 3 Audio/Video* Playback* Record* MediaPlayer* SoundPool* MediaRecorder*

Mul$media Techniques in Android

Some of the informa$on in this sec$on is adapted from WiseAndroid.com

Mul$media Support

•  Android provides comprehensive mul$media func$onality: –  Audio: all standard formats including MP3, Ogg, Midi, … –  Video: MPEG-‐4, H.263, H.264, –  Images: PNG (preferred), JPEG, GIF (not recommended)

•  Several different media formats and codecs are supported •  You can play audio or video from media files stored in the

applica$on's resources (raw resources), from standalone files in the filesystem, or from a data stream arriving over a network connec$on.

COMP 355 (Muppala) Multimedia 2

Media API


Audio/Video

Playback Record

MediaPlayer SoundPool MediaRecorder

Media Playback and Recording

•  Standard method to manipulate audio/video: –  Media playback supported through the MediaPlayer and SoundPool

classes –  Media recording supported through the MediaRecorder class –  Creates its own thread for processing –  Requires audio as file or stream based data

•  Direct access to raw audio: –  AudioTrack and AudioRecorder classes –  Useful to manipulate audio in memory, manipula$ng the audio buffer,

or any other usage not requiring file or stream data –  Does not create its own thread


Media Playback

•  MediaPlayer is designed for longer sound files or streams (larger files) –  The files will be loaded from disk each $me create is called, this will

save on memory space but introduce a small delay (not really no$ceable)

•  SoundPool –  is designed for short clips which can be kept in memory decompressed

for quick access –  suited for sound effects in apps or games –  loads lots of “medium” sized sounds into the memory and you may

exceed your limit (16Mb) and get an OutOfMemoryExcep$on.


MediaPlayer Methods

•  Crea$ng a media player: MediaPlayer player = new MediaPlayer();

•  Specifying the source of the media: –  If file in the resource directory raw:

player = MediaPlayer.create(context, R.raw.music_file); –  If file or stream:

player.setDataSource(path); player.prepare();

•  Start playback player.start();

•  Pause playback player.pause();

•  Stop playback player.stop(); player.release();


MediaPlayer State Diagram


SoundPool Methods

•  Crea$ng a media player: SoundPool soundPool= new SoundPool(2, AudioManager.STREAM_MUSIC, 100);

•  Specifying the source of the media: –  If file in the resource directory raw:

SoundPoolMap soundPoolMap = new HashMap<Integer, Integer>(); soundPoolMap.put(streamID, soundPool.load(this.getBaseContext(), R.raw.birdsound, 1));

–  If file or stream: soundPool.load(String path, int priority);

•  Start playback –  soundPool.play(streamID, streamVolume, streamVolume, 1, 0, 1f);

•  Pause playback –  soundPool.pause(streamID);

•  Stop playback –  soundPool.stop(streamID); –  soundPool.unload(int soundID);


Video Playback and Recording •  Video playback and recording also uses the MediaPlayer and

MediaRecorder classes with similar steps as in audio •  Use VideoView widget in the ac$vity UI screen to host video:

<VideoView android:id="@+id/video" android:layout_width="fill_parent" android:layout_height="fill_parent"> </VideoView>

•  Get a handle to the video view in your code: •  videoView = (VideoView) this.findViewById(R.id.video);

•  Set the path to the video file: •  videoView.setVideoURI(Uri.parse(“\sdcard\video.mp4”));

•  Set the focus: •  videoView.setFocus();

•  You can use a MediaController to control the video •  MediaController mc = new MediaController(this); •  videoView.setMediaController(mc);


MediaRecorder Methods •  Create a new instance of Media recorder:

–  MediaRecorder recorder = new MediaRecorder(); •  Set the audio source:

–  recorder.setAudioSource(MediaRecorder.AudioSource.MIC); •  Set output file format:

–  recorder.setOutputFileFormat(MediaRecorder.OutputFormat.THREE_GPP); •  Set output file name:

–  recorder.setOutputFile(PATH); •  Set the audio encoder:

–  recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB); •  Prepare the media recorder:

–  recorder.prepare(); •  Start capture

–  recorder.start(); •  Stop capture

–  recorder.stop(); •  Finish recording

–  recorder.release();


MediaPlayer and MediaRecorder

•  Rich API designed for long playing media streams, such as voice audio recordings, music and videos –  seeking opera$ons supported –  Source buffering

•  Heavyweight resource-‐wise •  Slow to ini$alize •  Not suitable for low-‐latency scenarios like playing short audio

samples for sound effects, such as in games. •  Designed for situa$ons with no more than one or two

MediaPlayers working at the same $me


SoundPool Class

•  The SoundPool class manages and plays audio resources for applica$ons.

•  A SoundPool is a collec$on of samples that can be loaded into memory from a resource inside the APK or from a file in the file system

•  The SoundPool library uses the MediaPlayer service to decode the audio into a raw 16-‐bit PCM mono or stereo stream

•  This allows applica$ons to ship with compressed streams without having to suffer the CPU load and latency of decompressing during playback

•  Typical usage scenario includes games with sound where several sounds are used in a level and may have to be played with overlap some$mes


SoundPool Class Features

•  Senng the maximum number of sounds to play at the same $me

•  Priori$zing the sounds so that the low-‐priority ones will be dropped when the maximum threshold is reached

•  Pausing and stopping sounds before they finish playing •  Looping sounds •  Changing the playback rate (effec$vely, the pitch of each

sound) •  Senng stereo volume (separate for lep and right channels)


AudioTrack Class

•  The AudioTrack class manages and plays a single audio resource for Java applica$ons

•  It allows to stream PCM audio buffers to the audio hardware for playback. This is achieved by "pushing" the data to the AudioTrack object one of the write(byte[], int, int) and write(short[], int, int) methods.

•  An AudioTrack instance can operate under two modes: sta$c or streaming.

•  In Streaming mode, the applica$on writes a con$nuous stream of data to the AudioTrack, using one of the write() methods.

•  The streaming mode is most useful when playing blocks of audio data that for instance are: –  too big to fit in memory because of the dura$on of the sound to play, –  too big to fit in memory because of the characteris$cs of the audio data (high

sampling rate, bits per sample ...) –  received or generated while previously queued audio is playing.


AudioTrack Class

•  The sta$c mode is to be chosen when dealing with short sounds that fit in memory and that need to be played with the smallest latency possible

•  AudioTrack instances in sta$c mode can play the sound without the need to transfer the audio data from Java to na$ve layer each $me the sound is to be played

•  The sta$c mode will therefore be preferred for UI and game sounds that are played open, and with the smallest overhead possible.

•  Upon crea$on, an AudioTrack object ini$alizes its associated audio buffer •  The size of this buffer, specified during the construc$on, determines how

long an AudioTrack can play before running out of data •  For an AudioTrack using the sta$c mode, this size is the maximum size of

the sound that can be played from it •  For the streaming mode, data will be writen to the hardware in chunks of

sizes inferior to the total buffer size.


AudioTrack Class

•  AudioTrack class allows us the flexibility of: –  Decoding audio from any format that is not supported by the plavorm –  Modifying or enhancing audio on the fly (such as applying distor$on,

vocoder, reverb effects – but you have to code them yourself!) –  Loading audio from custom sources to play it on the fly (but you have

to deal with buffering and stability yourself – be sure not to develop another MediaPlayer!)

•  The price is that you are on your own regarding buffering, latency and correctness of the sound you produce with AudioTrack.


Android Audio Comparison Audio Requirements Audio Class Choice

Require low latency, such as in games or sound effects

SoundPool/ AudioTrack (for more flexibility)

Need to play video that has an audio track MediaPlayer

Play a set of short sounds many $mes SoundPool

Stream audio from an external source, e.g. HTTP or TCP stream

MediaPlayer/ AudioTrack (if MediaPlayer not suitable)

Generate audio from scratch, such as by using math formulas / frequency modula$on

AudioTrack

Play background music MediaPlayer

Modify the audio on the fly AudioTrack


Documents

Mul$mediaTechniques* inAndroid · MediaAPI COMP 355 (Muppala) Multimedia 3 Audio/Video* Playback* Record* MediaPlayer* SoundPool* MediaRecorder*