True Brew Test Guide

TRUE BREW® Test Guide

QUALCOMM Proprietary

QUALCOMM Incorporated 5775 Morehouse Drive

San Diego, CA. 92121-1714 U.S.A.

This manual may not be copied, except as otherwise provided in your software license or as expressly permitted in writing by QUALCOMM Incorporated.

Copyright © 2003 QUALCOMM Incorporated All Rights Reserved

Printed in the United States of America.

All data and information contained in or disclosed by this document are confidential and proprietary information of QUALCOMM Incorporated, and all rights therein are expressly reserved. By accepting this material, the recipient agrees that this material and the information contained therein are held in confidence and in trust and will not be used, copied, reproduced in whole or in part, nor its contents revealed in any manner to others without the express written permission of QUALCOMM Incorporated.

Export of this technology may be controlled by the United States Government. Diversion contrary to U.S. law prohibited.

BREW, BREW SDK, and MobileShop are trademarks of QUALCOMM Incorporated.

QUALCOMM, Binary Runtime Environment for Wireless, TRUE BREW, and The Grinder are registered trademarks of QUALCOMM Incorporated.

All trademarks and registered trademarks referenced herein are the property of their respective owners.

QUALCOMM Proprietary

TRUE BREW® Test Guide

80-D4187-1 Rev. F

December 16, 2003

Contents

1 Introduction 4

1.1 Related documents 4 1.2 Application testing overview 5 1.3 Abbreviations and acronyms 5 1.4 Terminology 6 1.5 What’s new in this guide 7

2 Testing Overview 8

2.1 Overview of test requirements 8 2.2 Evaluation of the test results 9

2.2.1 Errors that cause an application to fail TBT 10 2.2.2 Observations that may be documented as issues 11 2.2.3 What TBT does not evaluate 12

2.3 Overview of testing procedures 12 2.3.1 Test materials 12 2.3.2 Basic procedures 14

3 TRUE BREW Test Requirements 17

3.1 Entrance Criteria 17 3.2 Exploratory Testing 18 3.3 Handset/AUT Interaction 18 3.4 BREW Features 19 3.5 BREW User Interface 20 3.6 Adversarial 20 3.7 Application Download 21

Appendix: Notes 22

iii QUALCOMM Proprietary

1 Introduction

The TRUE BREW® Test Guide documents the test requirements that are used by TRUE BREW Testing (TBT) to evaluate a Binary Runtime Environment for Wireless® (BREW) application. A companion document, TRUE BREW Test Procedures, contains test procedures that can be used by developers to evaluate their applications before submission to TBT. This document is organized as follows.

1 Introduction Describes this test guide as well as other TBT documents and web pages. Defines abbreviations, acronyms, and terms. Tells what’s new in this guide.

2 Testing Overview Provides an overview of TBT, including a high-level look at the requirements, pass/fail criteria, and test procedures.

3 TRUE BREW Test Requirements

Presents detailed requirements in seven categories that are used by TBT when evaluating an application: Entrance Criteria, Exploratory Testing, Handset/Application Under Test (AUT) Interaction, BREW Features, BREW User Interface, Adversarial, and Application Download.

Appendix: Notes Contains miscellaneous notes and clarifications that should be considered when testing an application.

1.1 Related documents

The following documents are related to TRUE BREW Test Guide:

TRUE BREW Test Procedures (80-D4187-3 Rev. A)

Provides a spreadsheet that presents the test requirements along with procedures for testing applications against the requirements and columns to note test results, issues, and comments.

TRUE BREW Application Specification Template (80-D4187-2 Rev. A)

Provides the base for creation for the Application Specification (App. Spec.) document that is required when submitting an application to TBT.

TRUE BREW Process Overview (80-D4299-1)

Describes the overall TBT process and supporting information.

TRUE BREW Testing: Developers Guide to TRUE BREW “Passed With Notes” Process (80-D4527-2)

Describes the TRUE BREW passed-with-notes test status.

4 QUALCOMM Proprietary

Introduction

1.2 Application testing overview

For an overview of application testing, see the following:

TRUE BREW process overview document:

http://www.qualcomm.com/brew/developer/developing/docs/80-D4299-1_D.pdf

The TRUE BREW Process Overview provides a high-level overview of the TBT process. You should review this document before reading the requirements and procedures in this test guide.

TBT web page:

http://www.qualcomm.com/brew/developer/developing/apptesting.html

This web site also contains information about BREW application and extension testing. If you are developing a BREW extension, see this web page. In particular, it explains the process of creating the sample application used to test the extension.

BREW central document repository:

http://www.qualcomm.com/brew/developer/developing/brewdocs.html

This web site contains all BREW documents made available to developers for application development and testing.

1.3 Abbreviations and acronyms

Following is a list of abbreviations and acronyms used in this document.

ADS Application Download Server.

API Application Programming Interface.

AUT Application under test.

BAR BREW Applet Resource file.

BDS BREW Distribution System.

BREW™ Binary Runtime Environment for Wireless®. BREW is an application development environment that is layered upon public services offered by mobile ASICs.

BREW SDK™ BREW Software Development Kit. This allows application developers to quickly write and test applets and modules in a familiar Windows desktop environment.

BSE Base station emulator


http://www.qualcomm.com/brew/developer/developing/docs/80-D4299-1_D.pdf

http://www.qualcomm.com/brew/developer/developing/apptesting.html

http://www.qualcomm.com/brew/developer/developing/brewdocs.html

Introduction

CDMA Code Division Multiple Access.

EFS Embedded File System.

EULA End user license agreement.

GPRS General Packet Radio System.

MIF Module Information File. The MIF Editor generates this binary file, which contains information regarding the list of classes and applets supported by the modules.

MOD Dynamically loaded module file.

OEM Original equipment manufacturer.

OTA Over the air. Implies that an operation is performed on wireless network that is transmitting and receiving (whether a commercial system or test system).

QIS QUALCOMM Internet Services.

TBT TRUE BREW Testing

1.4 Terminology

The following terms are used in this document.

Application Specification

A document that describes an application submitted for TBT. The developer creates an App. Spec. according to the guidelines published on the documents page of the BREW Developer Extranet (Developer Extranet). The App. Spec. assists testers in the testing of an application.

BREW Application Manager

The Application Manager is a BREW application that enables handset users to start and manage BREW applications.

BREW Emulator A simulation environment for developing and testing BREW applications.

Cancelled The test status for an application that does not meet the TBT Entrance Criteria.

Device data sheet A document containing BREW specifications for a particular handset. It is available on the Developer Extranet.

Dormancy A data network state in CDMA2000 1X in which the RF network resources associated with a data connection are released while the upper-layer networking connections (for example, PPP) are not released.

Entrance Criteria A set of tests that are used to evaluate whether full testing will be performed on an application.

Error An application behavior or condition that violates one or more TBT requirements (for example, a handset resets when an application is suspended).

Exit Occurs by using menu options, the End key, the Clear key on the first application screen, or any other OEM-specific method. A normal exit is also referred to as “exit.”

Fail The test status for an application that does not meet one or more TBT requirements. A failed application has one or more errors.


Introduction

Issue An observation about an application behavior or condition that does not violate the TBT requirements (for example, "word misspelled" or "error message confusing"). Carriers may not accept applications with issues.

Linger time-out value

The number of seconds (specified in the device data sheet) that a data call will stay up after an application has terminated execution (more precisely, after a network connection has been terminated).

MobileShop™ A BREW application that allows handset users to browse, purchase, and download BREW applications from the Application Download Server (ADS).

Pass The test status for an application that meets all TBT requirements.

Pass-with- notes The test status for an application that does not meet one or more TBT requirements but is still conferred the passed-with-notes TRUE BREW Tested status because the errors detected during testing have a low impact on the end-user or are very unlikely to occur in real-world conditions. Carriers may not accept applications that are passed-with-notes.

Time-based events Any type of action that is generated because a certain date or time has occurred (or a duration of time has elapsed).

Wireless services This indicates that the handset has service with a carrier. This is used generically to indicate either 800 megahertz Code Division Multiple Access (CDMA) cellular, 1900 megahertz CDMA PCS, 800 megahertz AMPS, or any other type of service, such as GSM/GPRS, UMTS, cdmaOne, and CDMA2000 1x/1xEV/1xEV-DO.

1.5 What’s new in this guide

This version of the test guide (80-D4187-1 Rev. F) differs from the previous release (80-D4187-1 Rev. E) in that it:

Adds camera and position location tests.

Separates test requirements from test procedures, placing the procedures in a separate document, TRUE BREW Test Procedures (80-D4187-3 Rev. A).

Clarifies the types of errors that will cause an application to be failed or passed-with-notes.

Clarifies the types of issues that will be documented in a test report and may be considered by carriers when evaluating an application (even though the issues will not cause the application to fail TBT or to be passed-with-notes).


2 Testing Overview

This section provides an overview of TBT, including a high-level look at the requirements, pass/fail criteria, and test procedures.

2.1 Overview of test requirements

This document specifies requirements that are used to evaluate a BREW application. The requirements fall into the following major areas:

Entrance Criteria: test requirements that determine whether full testing of an application should be performed. Applications that do not meet these requirements are cancelled. These tests are primarily performed through inspection.

Exploratory Testing: Using the Application Specification (App. Spec.) as a guide, the tester explores and exercises the application features and functionality, similar to the way the application would be used by an end-user.

Handset/AUT Interaction: These test requirements focus on the interaction of the application with core handset features—including incoming calls, SMS, and alerts. The focus is to ensure that execution and termination of the application do not interfere with native OEM voice and data services.

BREW Features: These test requirements highlight application features and functionality that, in most cases, can be directly associated with a BREW Application Programming Interface (API) or functionality. It is expected that these features are also evaluated during exploratory testing.

BREW User Interface: These test requirements focus on several user interface requirements (for example, use of the Clear key and the Send key) that must be met by all BREW applications.

Adversarial: These test requirements focus on operating the application under extraordinary circumstances, including but not limited to stability testing (for example, running the application for a long period of time), stress testing (for


Testing Overview

example, rapid key inputs), boundary tests (for example, evaluating the behavior of application when memory resources have been depleted), and loss of service

Application Download: These test requirements focus on the behavior of the application with the BREW Distribution System (BDS); for example, downloading, disabling, and recalling an application.

For each of the above test areas, this test guide documents requirements that must be met by the application. In addition, a set of test procedures is provided in spreadsheet in a separate document, TRUE BREW Test Procedures (80-D4187-3 Rev. A).

2.2 Evaluation of the test results

After the testing of an application is completed, the test results—in particular, the errors detected during testing—are evaluated. Based upon this analysis, an application will fall into one of the following three categories:

Pass: The application meets all TBT requirements. No errors are detected.

Pass-with-notes: The application meets all TBT requirements with minor errors that are noted. These errors that are passed-with-notes have a low probability of occurrence and/or have minimal impact on the end-user experience. For more information, see TRUE BREW Testing: Developers Guide to the TRUE BREW “Passed With Notes” Process, which can be found at: http://www.qualcomm.com/brew/developer/developing/docs/80_D4527-2_A.PDF

Fail: The application does not meet the TBT requirements. See section 2.2.1 Errors that cause an application to fail TBT for more information on criteria that will fail an application.

Cancel: The application does not meet the TRUE BREW Entrance Criteria requirements.

In addition to test results (errors) that cause an application to fail or pass-with-notes, the testing of an application may produce observations, made by the tester, that are not part of the TRUE BREW pass/fail criteria. These observations are referred to as issues. Though not part of the pass/fail criteria, these issues may be documented in the test report. Operators may consider these issues when evaluating an application. It is important to note that the identification and documentation of these issues is not the goal of TBT. They are documented and identified as time permits.


http://www.qualcomm.com/brew/developer/developing/docs/80_D4527-2_A.PDF

Testing Overview

2.2.1 Errors that cause an application to fail TBT

The following errors are in this category:

Failure of a specific test requirement as specified in the test guide.

Interference with handset functionality or performance: applications that interfere with the ability of the handset to function properly, including the reliability, performance, and availability of handset functions. For example, an application will fail if it interferes with:

– Incoming/outgoing phone calls

– Incoming/outgoing SMS messages

NOTE: A networking application, while in a data call, may prevent incoming voice calls, depending on the OEM and operator. This condition will not fail an application.

Interference with the wireless network: applications that interfere with the performance of the wireless network, including the performance of other handsets on the network.

Missing or incorrect functionality: An application will fail if functionality is not present or does not meet functional requirements (as specified in the App. Spec.).

Undocumented functionality or behaviors: An application can be failed if it contains undocumented functionality or behaviors that are not within the scope of the functionality and behaviors specified in the App. Spec.

Instabilities or poor performance: An application can be failed if it is unstable or performs poorly, including the following:

– The handset crashes or hangs because of the application.

– The application crashes, unexpectedly terminates, or hangs.

– The application performs poorly, making it difficult or impossible to use.

– The application responses to user requests take longer than 3 seconds when not accompanied by a comfort display (graphic or textual notification that the application is waiting, working, and so forth).


Testing Overview

2.2.2 Observations that may be documented as issues

In addition to the above criteria for failing an application, testers also make observations about the application that will be documented in the test report. These observations are referred to as issues. Issues are not considered when determining whether an application passes or fails TBT (errors are evaluated to pass or fail an application). But carriers may not accept an application with issues. Issues will be reported if the application deviates from the following ideal behaviors:

Drawing screens: Each screen (such as splash, help, pop-up boxes, text entry) repaints correctly and with the correct contents. Repaints occur correctly after dismissing a screen that overlays another one.

Screen contents: Screen contents are clearly displayed; the user can easily view them. Directions and information can be read and understood. Error messages displayed by the application under adversarial conditions (for example, loss of service) are not confusing.

Keypad input: Keypad functions correctly navigate through the application. Keypad functions are handled correctly on each screen (such as splash, help, pop-up boxes, and text entry).

Text entry: Keypad entry fields handle input text correctly. Keypad entry is handled correctly on each screen (such as splash, help, pop-up boxes, and text entry), and the application does not accept invalid input values.

Scroll bars: Scroll bars work correctly on each screen (such as splash, help, pop-up boxes, and text entry).

In addition, testers will always note the following:

Resource usage: The number and size of files created by application in application directory and shared directory.

Deviations from the above behaviors may be classified as errors. For example, if a screen does not repaint correctly, making it difficult or impossible to use the application, the behavior will be reported as an error; this error will fail the application. If the repaint problem is noticeable but does not affect usability, the deviation will be reported as an error that will then be passed-with-notes. And if the repainting problem is barely noticeable, it will likely be reported as an issue (and the application will pass if there are no errors).


Testing Overview

2.2.3 What TBT does not evaluate

In addition to the above errors and issues that are evaluated as part of testing, it is important to emphasize that TBT does not evaluate or comment on the following:

The quality of the application in a subjective sense. For example, TBT does not comment that an application is "fun" to play.

The marketability of, or business case for, an application. For example, TBT does not comment that an application might be popular with a certain user community.

The appropriateness of content. For example, TBT does not consider or comment on the suitability of application content for a particular audience or market segment.

2.3 Overview of testing procedures

This subsection provides a high-level description of the techniques, procedures, and tools that can be used to test an application. In addition to this high-level overview, more detailed procedures are provided in a separate document, TRUE BREW Test Procedures.

Note that the focus of TBT is to evaluate the application against the TBT requirements. The procedures can and should be modified and optimized on an ongoing basis in order to better evaluate the applications. It is also expected that developers will use alternate procedures in some cases, for example:

Developers do not have access to a download server. Therefore, disable/restore testing must be performed by manually removing and replacing the dynamically loaded module (MOD) and BREW Applet Resource (BAR) files. Details for this procedure are available in a separate document, TRUE BREW Test Procedures. A).

It is not possible to create all error conditions (for example, memory allocation failure). Developers should therefore review the source code to ensure that the application is properly handling error conditions.

2.3.1 Test materials

The following tools, products, and materials can be used to support testing:


Testing Overview

Mobile device: the handset, PDA, or similar computing unit on which the testing is conducted. For readability, the specific term handset is used in this document rather than the more general term mobile device. A data cable is also required to connect a PC to the handset. The mobile handset requires either carrier service or some type of test equipment, such as a BSE to provide the signal. The handset requires data services for network-level privilege levels.

Device data sheet: This information is available on the Developer Extranet at https://brewx.qualcomm.com/developer/support/phonedetails.jsp. It provides detailed specifications for each supported handset.

BREW Application Loader (AppLoader): This tool is available on the Developer Extranet at: https://brewx.qualcomm.com/developer/support/devtools.jsp?page=dx/commbrewtools. This tool is part of the BREW Testing and Commercialization tools suite. Use it to move an application from a PC to the handset’s Embedded File System (EFS) when the handset is cabled to the PC.

BREW MIF Editor: This tool is available as part of the BREW Software Development Kit (BREW SDK™). It can be downloaded from the Developer Extranet at the following location: https://brewx.qualcomm.com/brew/sdk/download.jsp. Use it to review the contents of a Module Information File (MIF) or modify the usage-based license limits (available functionality in SDK version 2.0.0.x).

Max File Count: a BREW application used in testing that creates the maximum number of files on a handset. It can be downloaded from the Developer Extranet at https://brewx.qualcomm.com/developer/support/devtools.jsp?page=dx/commtoolsmisc.

The Grinder®: This tool is available on the Developer Extranet as part of the BREW Testing and Commercialization tools suite. It sends a continual stream of events to the AUT to allow stress and stability testing.

The Shaker: This tool is available on the Developer Extranet as part of the BREW Testing and Commercialization tools suite. It works in conjunction with The Grinder by manipulating the handset’s actual environment. For example, it can fill the handset’s file system and fill the handset’s available memory.

Microsoft Paint (or equivalent): Provides the size (in pixels) and color depth (bits per pixel) of a graphical image that has been extracted from the MIF. For BREW Compressed Image (BCI) images, use the BCI Editor.


https://brewx.qualcomm.com/developer/support/phonedetails.jsp

https://brewx.qualcomm.com/brew/sdk/download.jsp

Testing Overview

Timepiece: a stopwatch or other accurate timepiece to validate the duration of timers and alarms used by an application.

Hoffman Box: an electromagnetically insulated enclosure in which a handset will not acquire service or will lose service. Note that a Hoffman Box can be constructed with a box that has been insulated with aluminum foil on the outside and inside.

2.3.2 Basic procedures

2.3.2.1 Entrance Criteria

Entrance Criteria testing is largely performed by inspection of the package submitted by the developer, including the MIF (privileges, notifications, and so forth), directory structure and files in the submission, and the App. Spec. There is one test requirement evaluated by the TBT lab—downloading the application from an ADS and starting it up—that cannot be performed by the developer. If the package meets all other BREW requirements, and the developer has started the application on the handset by cable loading, the application should download from an ADS and start up with no issue.

2.3.2.2 Exploratory Testing

Exploratory testing is performed manually with the application on the handset.

A good introduction to exploratory testing can be found in the document General Functionality and Stability Test Procedure (James Bach, Satisfice, Incorporated): http://www.satisfice.com/tools/procedure.pdf.

2.3.2.3 Handset/AUT Interaction

This testing is performed with the application on the handset.

This phase focuses on evaluating the behavior of the application when suspended and the ability of the application to resume activity at or near the point of suspension. These test requirements can be tested using a combination of incoming calls, SMS messages, and alerts. The application behavior should be independent of the particular type of suspension (for example, incoming call, SMS, and alert). The suspend/resume behavior should be evaluated on all screens and all state transitions (for example, setting up data calls, tearing down data calls, and transitioning from one display to another).


http://www.satisfice.com/tools/procedure.pdf

Testing Overview

2.3.2.4 BREW Features

This testing is performed manually with the application on the handset.

These tests are performed by evaluating specific capabilities or functions of an application (for example, saving data, downloading ring tones, and setting up a network connection). All of these capabilities would have been evaluated to some degree during exploratory testing. The purpose of these tests is to focus in on significant handset and/or BREW features.

In many cases, these features interact with features available in the native OEM interface. For example, an address book application may provide an interface to create address book entries that are stored on the handset. It may be possible to manipulate these entries through the OEM interface as well. It is important, during testing, to ensure that there are no interaction problems with this feature (for example, entries created by the application do not appear in the OEM interface).

2.3.2.5 BREW User Interface

This testing is performed manually with the application on the handset.

These tests focus on the use of specific inputs; in particular, the Clear key and the Send key.

2.3.2.6 Adversarial

This testing is performed:

Manually with the application on the handset: for example, placing the handset and application in the Hoffman Box, in particular network applications, to ensure that the application handles loss of service.

Manually with the application on the handset in conjunction with the MaxFileCount and Shaker tools: for example, (1) creating the maximum number of files on the handset (MaxFileCount) to ensure that the application handles file creation failures; and (2) using up memory on the handset (Shaker) to ensure that the application handles memory allocation failures. Note that developer can also evaluate these behaviors by reviewing the source code.

Automatically using The Grinder on the handset.

Automatically using The Grinder on the Emulator.


Testing Overview

Manually on the Emulator; for example, performing limited exploratory testing with the application running on the Emulator and then ensuring that all memory allocated by the application is de-allocated.

2.3.2.7 Application Download

This testing is performed manually by the TBT lab through interaction between the handset and the ADS. It is not possible for the developer to interact with an ADS. See TRUE BREW Test Procedures for suggestions on how developers can perform these tests through other means.


3 TRUE BREW Test Requirements

This section presents the requirements used by TBT when evaluating an application.

3.1 Entrance Criteria

3.1.1 Developer Signature: The MOD and MIF files shall be signed. All signed files must be signed with valid developer signatures. Files that are modified on the handset must be unsigned or signed-modifiable.

3.1.2 MIF Graphics: MIF graphics (image, thumbnail, and icon) are the correct sizes. It must be possible to visually determine that a graphic has been selected in the Application Manager. The sizes are as follows: thumbnail 16 (w) x 16 (h) pixels; icon image 26 (w) x 26 (h) pixels; image icon (for BREW 2.0 and beyond) 65 (w) x 42 (h) pixels or smaller. For earlier BREW versions, the image icon is less than or equal to the maximum size specified in the device data sheet.

3.1.3 MIF Settings: The AUT’s ClassID is not displayed as a dependency. The MIF does not define both an application and an extension. The MIFs do not have the "hidden" flag checked for any applet. All notifications, privileges, and protected classes used by the AUT are documented in the App. Spec.. The Screensaver box is selected only for screen-saver AUTs.

3.1.4 Nested Directories in Package: The AUT package does not contain any subdirectories.

3.1.5 Zero Byte Files in Package: The AUT ARM directory does not contain any empty files (size 0 bytes).

3.1.6 Download, Startup, Delete: The AUT can be downloaded from the ADS. The AUT appears in the Application Manager. The AUT can be started from the Application Manager. The AUT can be deleted from the Application Manager. All modules created on the handset during application download are removed when the application is removed, including extensions (with the exception of extensions referenced by other applications).


TRUE BREW Test Requirements

3.1.7 Version Number: The App. Spec. documents the software version of the AUT. The version in the App. Spec. matches the version that is either (a) contained in the MIF or (b) embedded/compiled into the AUT (as specified in the App. Spec.). These version numbers must match exactly. It must be possible to unambiguously associated an application that is on the handset with the particular submission and the App. Spec. The software version number must increase with each new submission of the AUT.

3.1.8 Developer Test Results: The application submission includes a filled out copy of the test cases as documented in TRUE BREW Test Procedures. For each requirement, the developer has specified whether the application meets the requirements (P), does not meet the requirement (F), or the requirement does not apply (NA). Notes are included as appropriate.

3.2 Exploratory Testing

3.2.1 Primary Functionality: The AUT functions as specified in the App. Spec. The AUT does not contain functionality that does not fall within the scope of the functionality described in the App. Spec.

3.2.2 User Interface: The AUT primary functionality can be operated through the user interface provided by the AUT. Issues will be noted in the following areas: drawing screens, screen contents, keypad input, text entry, and scroll bars.

3.2.3 Supported Languages: The AUT functions as specified in the App. Spec. for each language supported by the AUT.

3.2.4 Content Server: The AUT functions properly on a sampling of data that resides on the content server.

3.3 Handset/AUT Interaction

3.3.1 Suspend/Resume: The AUT suspends, maintaining significant AUT state (for example, it doesn't lose data entered by the user), and it resumes at or near the point of suspension. Suspend can occur because of incoming voice calls, incoming SMS, and alerts (including the startup of a native or BREW screen saver). The AUT does not interfere with the ability to read the OEM Caller ID display.

3.3.2 Handset Core Features Unaffected by AUT: Operation of the AUT does not interfere with core handset functionality. After executing and terminating an application, the following handset features are functioning: (1) incoming voice call;



(2) incoming SMS; (3) outgoing voice call; (4) outgoing SMS; (5) data calls end within linger time-out value (networking applications only).

3.4 BREW Features

3.4.1 Licensing: For an AUT supporting a usage-based License Type (Price Type either Purchase or Demo), ensure that the remaining number of uses is decremented according to the rules specified in the App. Spec. For an AUT that restricts application functionality for the Demo Price Type, ensure that the restricted functionality cannot be invoked with Demo Price Type.

3.4.2 Networking: The AUT establishes, maintains, and tears down a data call for each network protocol supported by the handset (for example, IS-95B, CDMA2000 1X, and GPRS). After exiting the application (through any means), the data call ends within the linger time-out value (per the device data sheet). This is applicable only to applications supporting this capability.

3.4.3 Voice Calls: The AUT sets up and tears down a voice call. This is applicable only to AUTs supporting this capability.

3.4.4 Application-Directed SMS: The AUT handles or ignores (as appropriate) application-directed SMS messages, both while the AUT is running and when the AUT is not running. This is evaluated for all applications.

3.4.5 Outgoing SMS (Mobile-Originated): The AUT sends mobile-originated SMS messages from within the AUT (as specified in the App. Spec.). This is applicable only to AUTs supporting this capability.

3.4.6 Timers and Alarms: The AUT sets and responds to timers and alarms. This is applicable only to AUTs supporting this capability.

3.4.7 Address Book: The AUT correctly handles the interface and operations on the OEM’s address book. This is applicable only to AUTs supporting this capability.

3.4.8 Data Storage: The AUT is able to retrieve saved data. Data may be stored locally on the handset or on the content server. This is applicable only to AUTs supporting this capability.

3.4.9 Ringers and Wallpaper: The AUT downloads, saves, and plays ring tones and/or wallpaper (as specified in the App. Spec.). Downloaded tones or images can be accessed through the native UI. This is applicable only to AUTs supporting this capability.



3.4.10 Camera: The AUT is able to take, store, and send pictures (as specified in the App. Spec.). This is applicable only to AUTs supporting this capability.

3.4.11 Position Location, Non-Tracking Applications: The AUT supporting non-tracking position location services is able to retrieve position fixes and use these fixes as specified in the App. Spec. This is applicable only to AUTs supporting this capability.

3.4.12 Position Location, Tracking Applications: The AUT supporting tracking position location services is able to retrieve position fixes and use these fixes as specified in the App. Spec. This is applicable only to AUTs supporting this capability.

3.4.13 Screen Saver: A screen-saver AUT can be (1) selected from the Application Manager, (2) initiated as a screen saver by the handset, and (3) terminated in response to key events. This is applicable only to AUTs supporting this capability.

3.4.14 Sound Control: The AUT provides control over sounds as specified in the App. Spec. Native sound settings (for example, volume) prior to application startup are restored on normal application termination,with exceptions noted in the App. Spec. (for example, the application allows the selection of particular ring tones that should persist after application termination). This is applicable only to AUTs supporting this capability.

3.5 BREW User Interface

3.5.1 Clear Key: The Clear key moves the user up (back) one level in the AUT menu hierarchy. Use of the Clear key at the AUT’s root menu terminates the application.

3.5.2 Send Key: The AUT correctly handles the use of the Send key. This key is not used except for predefined operations associated with external over-the-air communications (voice calls, data calls, or mobile-originated SMS).

3.5.3 Response Time: The AUT informs the user with visual feedback when there are delays greater than 3 seconds in duration.

3.6 Adversarial

3.6.1 Loss of Service: The AUT gracefully handles no-service and loss-of-service conditions. A no-service condition will result in connection failure when the



AUT attempts to establish a connection. A loss-of-service condition will affect the AUT after a connection has been established. The requirement applies to applications using data services or voice services.

3.6.2 Dynamic Displays: Dynamic AUT displays (for example, displays that change continuously without user input) are stable for long periods of time (for example, 5 minutes).

3.6.3 Continual Keypad Entry: The AUT is robust in the face of a rapid succession of random keypad inputs.

3.6.4 AUT Stability: The AUT is stable when running over a long period of time.

3.6.5 Low Flash Memory: The AUT must be stable (for example, does not crash handset and gracefully terminates execution) when faced with low EFS conditions (for example, in the face of file-create and -write failures when attempting to write to flash memory).

3.6.6 Low RAM Memory: The AUT must be stable (for example, it does not crash the handset and gracefully terminates execution) when faced with low RAM memory conditions (for example, in the face of memory allocation failures).

3.6.7 Exit AUT: The AUT terminates execution in response to a Clear key (when invoked from the root application menu), any means described in the App. Spec. (for example, menu selection), the End key, and other OEM-specific means (for example, closing the clamshell). The AUT can be started and operated after termination.

3.7 Application Download

3.7.1 Disable and Restore: The AUT can be disabled and restored after it is loaded on a mobile handset. Persistent state (stored data) results are not lost.

3.7.2 Application Recall: The AUT can be recalled by an ADS.


Appendix: Notes

This section contains miscellaneous notes and clarifications that should be considered when testing an application:

OTA versus cable loading of an application: Developers do not have access to a download server. Therefore, the AUT must be loaded onto the handset by cable. The handset should still be activated in order to perform testing on the live network; for example, suspend/resume testing and accessing content servers for networking applications. Testers (for example, TBT or carriers) with access to a download server should not OTA download and cable-load without fully removing the application between each of the above operations (for example, cable-load, then delete before OTA download).

Viewing the application version: The version number of an application may be embedded within the application itself (for example, within the MOD file). The developer must specify in the App. Spec. how to view this embedded application version. The developer may also choose to store a version number in the MIF as part of the Copyright String or separately using the Module Version field (located within the General tab).

Distinction between EFS and RAM memory Adversarial tests: EFS tests evaluate the response of the application to file-create or file-write failures when storing data on persistent handset storage (for example, flash memory). RAM memory tests evaluate the response of the application to a memory allocation failure when attempting to dynamically allocate memory.

Distinction between EFS-create and EFS-write Adversarial tests: There are two primary adversarial (error) conditions that an application may encounter when interfacing to the EFS: inability to create a file (because the fixed handset file directory space is full) and inability to write to a file (the file has already been created, but EFS has no additional memory for the application to write data). Handsets frequently have a fixed number of files that can be created, either on the handset overall or within BREW. After this limit is reached, additional file-creates fail. In addition, the handset has limited memory for storing data on the EFS. After this limit is reached, file-writes fail. Both conditions can be created on the handset to observe the behavior of the application in response to these error conditions.


Appendix: Notes

The directory space can be filled with the MaxFileCount utility. The free filespace can be filled with the Shaker utility.

Difficulty of Adversarial memory tests: Starving the application of memory resources on the handset (whether RAM or EFS) is difficult. The interfaces to these resources can be distributed throughout the application. Creating a condition in which one access succeeds (for example, a memory allocation) and the next fails is difficult or impossible on the handset. Therefore, the developers should inspect source code for all interfaces to memory resources to ensure that the application detects and responds to error conditions (for example, reports the error to the user).

MaxFileCount test: An application that creates N files must have appropriate error-handling for each of these file-creates. To test this, the MaxFileCount utility allows the tester to fill the handset with the maximum number of files (that are allowed within BREW). The application can then be operated to observe its behavior when it cannot create any files. With this accomplished, the tester then deletes one file (using the MaxFileCount utility). The application (when operating again) will now succeed in one file create but will fail on the next. This procedure should continue (deleting files with MaxFileCount) until N files have been deleted in total. At this point, the application can successfully create all necessary files. The MaxFileCount utility is available on the Developer Extranet.

Noteworthy stages of a network connection: A network (data) connection passes through the following primary stages that should be considered when testing an application: (1) setup, the time period during which the connection resources are allocated and authentication may be performed; (2) active, the time period during which an application is receiving or transmitting data across a connection; (3) idle, the time period during which an application is not using a connection to transfer data; (4) dormant, in CDMA 1X, a time period during which an idle connection's underlying network resources (for example, the RF channel) have been de-allocated while the upper layer protocols (for example, PPP) remain in place; (5) shutdown, the time period during which an established connection is terminated.

Linger-timer: The linger time is a value (specified in the device data sheet) that is used by the handset to determine how long (in seconds) to maintain a data call (for example, the physical network resources) after an application has closed the socket associated with the data call. After an application closes a socket, the handset will keep the data call up for the linger timer (in anticipation that the same or a different application may open a new connection). After the linger timer expires, the data connection is closed. In addition, the linger timer is used by CDMA 1X systems to release the physical network resources of a connection that


Appendix: Notes

has been idle while still keeping upper layer protocols (for example, PPP) virtually connected. This is referred to as dormancy.

Adapting to network state: Applications should not rely on hard-coded time-out values to track the state of a network connection; for example, the time required to establish a data connection may vary from network to network or even vary over time on a particular network. Applications must rely on the network state information that is provided by BREW in order to determine whether it is appropriate to read or write from a connection, and whether a connection previously established has been disrupted (for example, because of loss of service).

Loss-of-service (LOS) testing: In the wireless environment, it is possible for a handset to lose service because of poor coverage. Applications that rely on the network (data networking applications, voice applications, or SMS) must respond to error conditions that are encountered when service is unavailable. A networking application, for example, may time out when establishing a connection (setup); while transferring or receiving data (active); while initiating a transfer or reception of data after a period of inactivity (idle or dormant); or while closing a connection (shutdown). A Hoffman box (electromagnetically isolated container) can be used to produce LOS conditions. The following are the most important conditions to focus on:

– The application times out when establishing a connection. Put the handset in a Hoffman box for 20 seconds. Pull it out (should indicate no service). Initiate a command on the AUT to initiate a data transfer (for example, download a ring tone). Place the handset back into the Hoffman box. Wait approximately 1 minute. Pull it out. View the AUT on the handset. The AUT should have responded to the error condition.

– During a data transfer when the connection is active, initiate a data transfer. Place the handset in a Hoffman box for approximately 1 minute. View the AUT on the handset. The AUT should have responded to the error condition.

– During initiation of a data transfer after an idle/dormancy period, initiate a data transfer. Let it complete. Let the handset sit idle for linger timer (from the device data sheet) for approximately 20 seconds. Place the handset in Hoffman box for about 20 seconds. Pull it out. Initiate an AUT data transfer. Place the handset back into box for approximately 1 minute. Pull it out. View the AUT on the handset. The AUT should have responded to the error condition.

Handset variations when responding to SMS: When receiving an SMS message while an application is in a data call, some handsets will suspend the


Appendix: Notes

application and tear down the data call. Other handsets will note the reception of an SMS (sound and/or annunciator) but will not suspend the application nor terminate the data call.

Incoming call during dormant data connection: A handset on a CDMA 1X network may receive an incoming voice call when a network (data) connection is dormant. Therefore, the application can be suspended while it has a connection open. This cannot occur on an IS-95 network. Developers should ensure that applications can handle suspend events when a data connection is open.

Supported language testing: An application may display a particular language by checking the language setting of the handset. Or the application may itself provide a language setting. After an application is thoroughly tested in one language, the remaining languages must be investigated (explored) by changing the language setting of the handset and/or the language setting of the application (as appropriate for a particular handset and application combination).

Suspend/resume testing: Handsets generate suspend/resume events to the application in many ways; for example, incoming voice call, SMS, low-battery, volume-change, and other means. To the application, these suspend and resume events, although originating from different underlying causes, appear the same. Therefore, TBT focuses on the SMS suspend event, which can be easily automated on an OTA commercial system. Note that the TBT suspend/resume test does perform some minimal evaluation using suspend events other than SMS, although this is a basic sanity check (handset testing should ensure these different events appear identical to the application).

Multi-mod files, including applications using extensions: Applications that use extensions (private or public) or are composed of multi-modules are tested no differently than a single-module application. Testing of the extensions themselves is specified on the Application and Extension Testing page of the Developer Extranet.

Resource utilization: TBT does not fail applications for inefficient use of resources; in particular, file system (EFS) resources and network resources (for example, data connection air time). TBT does note an application's use of EFS resources (number and size of files created). Some carriers may reject an application if it uses resources inefficiently (without justification). For network utilization, developers should consider that an operator may want an application to minimize connection time, whereas other operators (for example, using a 1X network) may want the application to keep a data connection open and allow dormancy to free up network resources. In addition, some carriers may charge for data by the byte, whereas others charge for connection time. The developer


Appendix: Notes

should consider specific carrier requirements or guidelines when creating a networking application.

Packaging: Applications must be packaged properly before submission to TBT. Packaging requirements are checked automatically by the submissions web site. To ensure that an application is packaged properly, see Instructions on how to package an application for submittal on the Application and Extension Testing page of the Developer Extranet at the following location:

https://brewx.qualcomm.com/developer/operations/submitapphowto.jsp

Signing of files: TBT requires that the MOD and MIF files of an application are signed before submission for testing. In addition, the developer may choose to sign other files, such as the BAR file or other data files. Developers should either (a) not sign files that are modified on the handset or (b) sign these files with the "Gets Modified" flag. Otherwise, the application will not start up on the handset after the file has been modified. The purpose of the "Gets Modified" flag is to ensure the integrity of the files between the developer and TBT. These files are not included in the signature process that takes place on the handset (only those marked "Included in Signing" are included in the signature process on the handset). See the AppSigner help manual for more information (available in the BREW Tools Suite).


https://brewx.qualcomm.com/developer/operations/submitapphowto.jsp

Documents

True Brew Test Guide