1

Enhance Qualcomm® Snapdragon™ audio using Android Audio APIs

Harikishan Desineni, Director of Engineering Qualcomm Innovation Center, Inc.

Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.

3

Agenda

Session overview

1 How Qualcomm

Technologies incorporated the

digital signal processor (DSP)

2 Snapdragon Audio

SDK

3 Open DSP program

4 Demo

5

4

Includes the following sound effects:

− Equalizer (EQ) – Adjusts the sound level of certain frequency ranges by use of audio filters to improve the quality of overall perceived sounds

− Virtualizer – Virtualizes audio channels to compensate for the mismatch between the loudspeaker setup used when producing the audio, and the audio output device used when the audio is played back by the end user

− Reverb – Simulates the reverberation that results from reflections from surrounding walls and objects in an environment to make the sound more live and real

− BassBoost – Dynamically boosts low-frequency content of an audio signal to achieve better listening experience on bass for a headset device

How Qualcomm® Snapdragon™ Audio+ uses the DSP

Qualcomm Snapdragon Audio+ is a product of Qualcomm Technologies, Inc.

5

Media playback block diagram

Surface Flinger Audio Flinger

MDP/MDDI

Media Player

Media Player Service

StageFright

User space Java

User space Native

Kernel space

DSP

Video OMX Component

Audio OMX Component

Video HW

PCM-audio

Audio HAL

Audio HW

Media Server

Audio Drivers Display Drivers Video Drivers

6

Audio framework: effects offloaded to DSP

Media Player

Media Player Service

Awesome Player

Audio Track AudioFlinger

Qualcomm® Hexagon™

AudioEffects

Software AudioEffect

BassBosst

Virtualizer

ReverB

X

Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.

7

Snapdragon Audio SDK

8

Snapdragon Audio SDK Fluence™ Pro EC NS detection Snapdragon SDK for Android: Device Info

The Device Info feature set is designed to allow an application to detect if the device it is running on has a Snapdragon processor. This insight lets you make informed decisions about how to take full advantage of features enabled by Snapdragon processors.

Using Device Info is designed to let you tap into the following hardware features to help improve the performance of your applications:

− Is Snapdragon This feature is designed to allow the application to detect at runtime if the device has a Snapdragon processor

− Echo Cancellation/Noise Suppression

Echo Cancellation/Noise Suppression is designed to foster better audio communication quality at lower power by allowing full duplex mode during VOIP calls. These features are specifically tuned to each device, and use technology available on normal mobile voice calls so you can offload CPU/software-based echo cancellation and noise suppression.

Fluence Pro is a product of Qualcomm Technologies, Inc.

9

Snapdragon Audio SDK com.qualcomm.snapdragon.sdk.deviceinfo

Class Description

DeviceInfo The DeviceInfo API provides a mechanism to query a device about the presence of certain functionalities.

Class Description

DeviceInfo.Features The list of features that can be detected with the function isFeatureSupported(Feature, Context)

Sample code is subject to SDK license agreement. Source: http://developer.qualcomm.com

10

Snapdragon Audio SDK

public static final DeviceInfo.Features FEATURE_FLUENCE

Noise Cancellation Technology

− If isFeatureSupported returns true for this feature, then it can be assumed that echo cancellation is supported on the voice path.

− If isFeatureSupported returns true for this feature, and the standard Android API android.media.audiofx.AcousticEchoCanceler.isAvailable() also returns true, then it can be assumed that echo cancellation is also available on the audio path.

11

Snapdragon Audio SDK

FEATURE_SURROUND_SOUND_RECORDING

public static final DeviceInfo.Features FEATURE_SURROUND_SOUND_RECORDINGSurround Sound Recording is designed to allow for 5.1 surround sound input using existing microphones on the device

12

OpenDSP program

13

Audio Modules – Common Interface

Common interface for audio modules that are defined to ease integration with the Elite Framework

Audio Postprocessor Interface (APPI) for audio processing algorithms

Common Audio Processor Interface (CAPI) for audio codec modules

Processing

Algorithm Wrapper with common interface (APPI)

Codec Decoder / Encoder

Wrapper with common interface (CAPI)

Note: The terms algorithm, module, and library are used interchangeably in this presentation.

14

Audio Modules – Static and Dynamic

Two types of modules, based on loading: − Static modules (*.lib)

− Linked to the aDSP image at compile time

− Symbol are resolved at compile time

− aDSP image must be recompiled when a module changes

− Dynamic modules (*.so) − Loaded at run-time into aDSP heap memory

− Symbols are resolved when the module is loaded into memory

− aDSP image does not need to be recompiled when a module changes

Each module is identified by unique global module ID

Audio static modules are added to the AMDB at boot time

Audio dynamic modules are added to the AMDB at run time

APPI mod-n (*.so)

CAPI mod-3 (*.so)

CAPI mod-2 (*.lib)

APPI mod-1 (*.lib)

15

Per Object PostProcessing (POPP)

Per Object PreProcessing (POPreP )

Processing chain per audio stream

Default POPP topologies: − Playback session

− Record session

DSP audio mixer

Downmix if needed

Resampler Audio Encoder Volume

DSP audio mixer

Volume Audio Decoder Resampler

POPP Topologies

16

Common Object PostProcessing (COPP)

Common Object PreProcessing (COPreP)

Processing chain per device

Default COPP topologies: − Playback session

− Record session

AFE Audio

pregain

AFE CodecRX Gain

DSP Audio Mixer

Audio Volume

DSP Audio Mixer

COPP Topologies

17

Custom Audio Topology – Static and Dynamic Two types of audio topologies: − Static topologies

− Created at compile time

− aDSP image must be recompiled if the topology changes

− Dynamic topology − Created at run time

− aDSP image does not need to be recompiled if the topology changes

− APIs are defined for adding custom dynamic topologies from the applications processor:

Custom topology is designed to use any module defined in the AMDB by using Module IDs

Customized topology with an OEM algorithm

ASM_CMD_ADD_TOPOLOGIES // for POPP and POPREP ADM_CMD_ADD_TOPOLOGIES_V5 // for COPP and COPREP

Audio Mixer Volume OEM

Algorithm Audio Decoder Resampler

18

Audio Dynamic Topology How are dynamic topologies added at run time?

APR

ADM

Topo DB

ASM_CMD_ADD_TOPOLOGIES T2 (-M1-M4-), T3 (-M1-M2)

topo_db_add_topologies()

T1

T2

T3

aDSP

AMDB

M1

M2

M3

M4

New audio sessions can be created using T2 and T3 topologies after adding them to the topology database

Client Processor

ASM

19

ISV Module1 ISV Module2

Application

HAL

RTAC Kernel Driver

APR Handler DSP

Payload

ASM_CMD_SET_PP_PARAMS/ AMD_CMD_SET_PP_PARAMS

RTAC Userspace module Send_rtac()

ACDB Loader

Acdb_store()

Acdb_send()

Acdb_get()

acdb file

Get/setParameter(key”,value)

Software Architecture – Block Diagram

20

Software Architecture – Android APIs Existing set parameter android API extended to have new key value pairs

Setparameter() with persistence − set_parameter(key=”ISV_PERSIST_CFG”, value=” DataBinary, *”);

− Data Binary – Entire calibration data binary that is stored in calibration database. Even if user changes single parameter through UI, the app is responsible for packaging the entire data binary for persistence

Setparameter() without persistence − set_parameter(key=”ISV_NOPERSIST_CFG”, value, “List_MID_PID, *”);

− List_MID_PID – List of dynamically modified calibration to be applied on ISV modules in ADSP. This could be an array of Module ID, Parameter ID and data, or could be only one set of ModuleId, Parameter ID and values. Here, the data is not persisted in audio calibration database. The app is responsible for packaging the modified modules and parameters.

21

Demo

22

developer.qualcomm.com

Augmented reality

Context awareness

Peer to peer

Wireless health

Graphics & gaming

Computer vision

Android applications

App development SDKs & tools

Marketing opportunities

Case Studies & Tutorials

News & Information

Hardware development platforms

Forums & technical support

Tools & resources to help developers build, integrate, and optimize

Qualcomm Developer Network

23

For more information on Qualcomm, visit us at: www.qualcomm.com & www.qualcomm.com/blog

Thank you

FOLLOW US ON:

© 2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All rights reserved. Qualcomm, Snapdragon, Hexagon and Fluence are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Uplinq is a trademark of Qualcomm Incorporated. All trademarks of Qualcomm Incorporated are used with permission. Other product and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.