Andrea Superbeam Array Microphone Speech Recognition

Andrea Audio Test Labs White Paper

Andrea Superbeam® Array Microphone Speech Recognition Performance

Using Microsoft Office XP

July 12, 2002

Revision 3A

Andrea Electronics Corporation 45 Melville Park Rd Melville NY, 11747

(800) 442-7787 www.andreaelectronics.com

Prepared by: Mark Schmidt

VP Engineering

Copyright © 2002 Andrea Electronics Corporation All rights reserved.

Andrea Audio Test Labs 2 Superbeam® Array Microphone Speech Recognition Performance

1.0 Goals We compare the Andrea Superbeam® Array Microphone’s Speech Recognition performance to other microphones. Tests were performed under the same conditions to determine, on a relative basis, how the Superbeam microphone compares to a close-talking USB headset and a single element OEM microphone. The speech engine utilized in the tests was Microsoft Office XP’s. The intention is not to determine absolute Speech Recognition accuracy. With this in mind, we are using only a single male voice for speech input. Minimum training is performed. Additional training will improve absolute accuracy. The accuracies herein are for comparison purposes only and do not reflect actual user experiences. A serious user will train more than the minimum, and the speech engine will improve even without training as a user corrects mistakes. The questions we answer are:

• How does the Andrea Superbeam Array Microphone compare to a Noise Canceling USB Headset?

• How does the Andrea Superbeam Array Microphone with Andrea’s DSDA® Beam forming and

PureAudio® Noise reduction software compare to other solutions?

• Is accuracy affected by the acoustics of the room or the area where the microphone is used?

• Is an analog, noise canceling, close-talking headset affected by ambient noise?

• Can PureAudio® alone (as included with Analog Devices SoundMAX® CadenzaTM) improve speech recognition accuracy of an analog, noise canceling, close-talking headset?

• Can PureAudio® alone (as included with Analog Devices SoundMAX® CadenzaTM) improve

speech recognition accuracy of low cost PC OEM microphones?

2.0 Results Summary • The Andrea Superbeam Array Microphone offers outstanding, consistent performance under a

wide range of conditions. The results indicate the Superbeam Array Microphone performance is comparable to or better than a close-talking USB headset under a variety of conditions. Other solutions lack consistent performance under varying noise environments.

• Andrea’s PureAudio Noise Reduction Algorithm, as found in Analog Devices SoundMAX

Cadenza improves the speech recognition accuracy of low cost OEM microphones, but does not improve performance enough in noisy and reverberant environments. In those situations, the best solution is the Andrea Superbeam Array Microphone with its combination of DSDA Beam Forming and PureAudio noise reduction.

• Even close-talking noise canceling headsets are affected by ambient noise. An analog, noise

canceling headset can benefit from Andrea’s PureAudio noise reduction software.


3.0 Detailed Results

3.1 Data Presentation and Analysis

3.1.1 How does the Andrea Superbeam Array Microphone compare to a Noise Canceling USB Headset?

3.1.1.1 Superbeam vs. USB Headset Speech Recognition Accuracy With the MS Office XP Speech Engine the Superbeam Array outperforms the USB Headset in every test environment with and without noise.

70

75

80

85

90

OfficeQuiet

OfficeNoise

CubeQuiet

CubeNoise

CubeVoice

ConfRm

Quiet

ConfRm

OfficeNoise

Chart 1 - Superbeam vs USB Headset - MS Office XP Speech Recognition Accuracy - %

SuperbeamUSB Headset

Table 1 – Superbeam vs. USB Headset - MS Office XP Accuracy % Small

Office Quiet

Small Office with Office Noise

Open Cube Quiet

Open Cube with Office Noise

Open Cube with Voice Noise

Conf Rm Quiet

Conf Rm with Office Noise

Superbeam 85 82 87 82 82 85 78 USB Headset 79 80 78 80 80 78 75

Note: The USB Headset is a Plantronics DSP-300


3.1.2 How does the Andrea Superbeam Array Microphone with Andrea’s DSDA® Beam-forming and PureAudio® noise reduction software compare to other solutions?

3.1.2.1 “Open Cube” – Speech Recognition Accuracy The Superbeam Array outperforms both a USB Headset and an OEM PC microphone in the “open cube” environment. Office noise degrades the OEM PC Microphone performance making it unusable for speech recognition.

50

60

70

80

90

Quiet Office Noise Voice Noise

Chart 2 - Open Cube - MS Office XP SpeechRecognition Accuracy - %

SuperbeamUSB HeadsetOEM PC Mic

Table 2 - Open Cube - MS Office XP Accuracy %

Quiet Office Noise Voice Noise Superbeam 87 82 82 USB Headset 78 80 80 OEM PC Mic 77 52 74

Note: OEM PC Mic is a Telex M-30


3.1.2.2 “Small Office” –Speech Recognition Accuracy With MS Office XP the Superbeam Array outperforms both a USB Headset and an OEM PC microphone. The OEM PC Mic has trouble even in quiet conditions because of room acoustics, and things get much worse with the addition of office noise. Again, the OEM PC Microphone performance is unusable for speech recognition.

50

60

70

80

90

Quiet Office Noise

Chart 3 - Small Office - MS Office XP SpeechRecognition Accuracy - %


Table 3 - Small Office - MS Office XP Accuracy %

Quiet Office Noise Superbeam 85 82 USB Headset 79 80 OEM PC Mic 69 51



3.1.2.3 “Conference Room” – Speech Recognition Accuracy With MS Office XP the Superbeam Array again outperforms both the OEM PC Mic and the USB Headset in the conference room environment. The room acoustics are a problem for the OEM PC Mic. Again, the OEM PC Microphone performance is unusable for speech recognition.

405060708090

Quiet Office Noise

Chart 4 - Conference Room - MS Office XP SpeechRecognition Accuracy - %


Table 4 – Conf Room - MS Office XP Accuracy %

Quiet Office Noise Superbeam 85 78 USB Headset 78 75 OEM PC Mic 71 45



3.1.3 Is accuracy affected by the acoustics of the room or the area where the microphone is used?

Yes, room acoustics do affect speech recognition accuracy. Users of speech recognition software complain that sometimes it works well, other times it performs poorly. Room acoustics play a part in this. A major problem is reflections off walls, floors and ceilings. Also, room size determines when the reflections (echo) will return to the microphone. The Superbeam Array Microphone uses advanced de-reverberation techniques to reduce echo and minimize its effect on the speech engine.

3.1.4 Is a close-talking, noise canceling headset affected by ambient noise? The surprising answer is: Yes. Most high quality headset microphones utilize a “Noise Canceling” microphone element. At low frequencies, (e.g. 100 Hz) the mic element will reduce ambient noise by about 25 dB relative to the speaker. The noise cancellation effectiveness is reduced as frequency goes up until there is no attenuation of ambient noise at about 3 kHz. Any noise above 3 kHz will be picked up by the microphone and sent on to the speech recognition software. Figure 1 is a typical Noise Canceling microphone element frequency response (bare element, not mounted in a headset). The close-talking response, given by the curve at L=1 cm shows a flat response from 100 Hz to 10 kHz. The far-field response, given by the curve at L=50 cm shows the attenuation (Noise Canceling). In fact, Andrea’s PureAudio can improve the accuracy of an analog close-talking headset under some noise conditions. PureAudio reduces “stationary” noise sources across the frequency range with 10-15 dB in addition to the reduction built in to the microphone element.

Figure 1 – Panasonic Noise Canceling Electret Microphone Element Frequency Response

The estimated frequency response needed by speech engines is 100 Hz to 6 KHz. PureAudio can further reduce noise in the 1 KHz and up range that is not canceled by the Noise Canceling Microphone. Note: A USB Headset will not be aided by PureAudio as part of Analog Devices SoundMAX Cadenza. USB Headsets use external audio Codecs and utilize their own drivers and noise reduction software. SoundMAX Cadenza is available for the built in Codec of the computer only.


3.1.5 Can PureAudio® alone (as included with Analog Devices SoundMAX® CadenzaTM) improve speech recognition accuracy of an analog, noise canceling, close-talking headset? Recognition accuracy of an analog close-talking headset was improved by adding Andrea’s PureAudio as found in Analog Devices SoundMAX Cadenza. The improvement was evident in a “Small Office” environment with its difficult acoustics. There was no improvement in the “Open Cube” environment because it is acoustically less challenging.

70

75

80

85

Quiet Office Noise

Chart 5 - Small Office - MS Office XP SpeechRecognition Accuracy - %

Andrea Headset withPureAudioAndrea Headset

Table 5 – Small Office - MS Office XP Accuracy %

Quiet Office Noise Andrea Headset with SoundMAX PureAudio

83 84

Andrea Headset 76 76 Note: The Andrea Headset is an NC-61


3.1.6 Can PureAudio® alone (as included with Analog Devices SoundMAX® CadenzaTM) improve speech recognition accuracy of low cost PC OEM microphones? Andrea’s PureAudio will improve significantly the speech recognition accuracy of a low cost, single element PC OEM microphone. With the MS Office XP Speech Engine the PureAudio significantly improves recognition accuracy, BUT we can see noise reduces the accuracy to an unacceptable level. We cannot recommend this solution for all environments. We recommend the Superbeam array with its consistent, excellent performance that exceeds the accuracy of a close-talking headset microphone.

405060708090

OfficeQuiet

OfficeNoise

CubeQuiet

CubeNoise

CubeVoice

ConfRm

Quiet

ConfRm

OfficeNoise

Chart 6 - PureAudio improvement of OEM PC Mic - MS Office XP Speech Recognition Accuracy - %

OEM PC Mic with PureAudioOEM PC Mic

Table 6 –PureAudio improvement of OEM PC Mic - MS Office XP Accuracy % Small

Office Quiet

Small Office with Office Noise

Open Cube Quiet

Open Cube with Office Noise

Open Cube with Voice Noise

Conf Rm Quiet

Conf Rm with Office Noise

OEM PC Mic with PureAudio

80 66 84 83 83 80 64

OEM PC Mic 69 51 77 52 74 71 45 Note: The OEM PC Mic is a Telex M30


4.0 Test Methodology Notes

4.1 Enrollment Enrollment is the process of training the speech recognition software to your voice so that it can improve its accuracy. Since each microphone has its own tonal qualities, signal to noise, and sensitivity, each microphone requires separate enrollment. To minimize test time, and to accentuate any differences between microphones, the minimum enrollment is performed for each test microphone. Accuracy will most certainly improve if additional training is performed.

4.2 Test Script Our test script has a word count of 466. Punctuation is counted as words. A recording of a “trained voice” 45 yr old male is used. The text of the script is given in the Appendix. The script is direct dictation into Microsoft Word 2002 (Part of Office XP) using the Microsoft Speech Engine.

4.3 Scoring Scoring is done using an Andrea Audio Test Labs File Comparison Utility developed for speech test scoring. The utility compares the reference script to the speech-recognized script of the microphone under test. The scoring method counts substitutions, which are words, or commands that the computer misinterprets, and deletions, which are words or commands that the computer misses. The Speech Recognition Accuracy score is calculated as follows:

Accuracy Score (%) = ((Total words – Substitutions – Deletions) / Total words) x 100 Each test is repeated at least three consecutive times, and the average score is calculated.

4.3.1 Consider relative scores only Because of the lack of an industry standard for “speech recognition accuracy” and the differing scoring approaches, all accuracy data should be made on a relative basis only.

4.3.2 Alternate scoring approach considering insertions There are different approaches on how to score continuous speech recognition accuracy. The Andrea scoring assumes that only substitutions or deletions are considered errors. Another approach is to consider insertions as relevant.

Alternate Accuracy Score with insertions (%) = ((Total words – Substitutions – Deletions – Insertions) / Total words) x 100

Example:

Original Speech: “The age of automation…” Speech program interpretation: “The age of water nation…”

The Andrea Accuracy Score here is: 4 (total words) – 1 (“automation” substituted by “water nation”) / 4 = 75%

The alternate scoring considering insertions is:

4 (total words) – 1(“automation” substituted by “water”) – 1(insertion of “nation”) /4 = 50%


The accuracy scoring is usually lower when insertions are considered as errors. For accuracies of 80% and above the scoring will be about 1-5% lower when considering insertions. For accuracies below 80% the scoring can be 5-20% lower when considering insertions.

4.4 Results may vary Because of the many variables involved with speech recognition testing, recreation of our test environments may result in somewhat different results. The relative results however should be the same. We have made every attempt to minimize test variations, and after hundreds of tests our results have been very consistent.


Appendix

1.0 Test Environments Testing was performed under the following controlled, but real life conditions:

• A small office • An open cubicle • A moderate sized conference room

1.1 Test Setup All setups used a computer (PC-1) as the source of the speech files that are played through a Brüel & Kjær artificial head and torso simulator. Noise files, when needed, are played from PC-1 through an amplifier then through an amplified off axis speaker. To ensure repeatable results the Speech and noise files are combined into a single stereo file that is played left channel to the head and torso simulator and right channel to the off axis speaker. The Device Under Test (DUT) is connected to a second computer that is configured for speech recognition. The input for all tests (except the USB Headset) is via an Analog Devices Communication Network Riser (CNR) card with an 1885 Codec, Stereo Mic input and Mono Mic input.

PC-1 Source Audio .wav files

Audio Amp

Artificial Head & Torso

Amplified Speaker

PC-2 Win XP and Mic Input

Microphone DUT

A source of variability is sound pressure level measurements. There are two main “weightings” for Sound Pressure Level (SPL): “A” weighting and “C” weighting. “A” weighting compensates for the non-linear response of the human ear, while “C” weighting is a flatter response. The response of the human ear to sound is worse at low and high frequencies. The two weightings will give different results depending on the frequency distribution of the sound. In an office environment, significant content can be missed by “A” weighting that we cannot hear, but the microphone will pick-up. The microphone and speech engine will be affected by this inaudible noise so we have considered it. Low frequency blower and air exchanger noise is of particular concern, as it is difficult to hear. All Andrea SPL readings use “C” weightings. We believe “C” weighting reflects more accurately the range of sounds the microphone picks up.


1.1.1 Small Office Testing in a 10’3” x 11’ x 9’ (high) closed office with sheetrock walls, dropped ceiling acoustic 2’ x 4’ tiles and carpeting is the “Small Office” data. “Quiet” conditions include noise from the system under test’s electrical circuitry and acoustic noise from the fans in the computer enclosure. The test computer has three internal fans: one on the Pentium, one on the power supply and one on the rear of the case. This simulates what might be experienced with the door to an office closed. “Office noise” conditions include the “Quiet” noise and add a looped playback of Track 6 (Office) of “The New CBS Audio-File Sound Effects Library CD 2”. This “office noise” track includes voices, phones, doors and HVAC background as recorded in a typical office setting. The SPL of the sound track is adjusted for 65dBC SPL as measured at the microphone under test’s location. The noise speaker is 90° off-axis, 57” from the microphone under test (when mounted on the computer monitor), and is in the left of the picture below. This simulates what might be experienced with the door to an office open.


1.1.2 Open Cubicle Testing in a modern office environment with sound deadening soft partitions, 9’ suspended ceiling of acoustic 2’ x 4’ tiles and carpeting is the “Open Cube” data. The test cubicle is in a larger area with a total of 8 comparable sized cubicles. It is an interior partition with cubicles on three sides. “Quiet” conditions include noise from the system under test’s electrical circuitry and acoustic noise from the fans and hard drive in the computer enclosure. The test computer has three internal fans: one on the Pentium, one on the power supply and one on the rear of the case. This simulates what might be experienced working alone in an open office after normal work hours. “Office Noise” conditions include the “Quiet” noise and adds looped playback of Track 6 (Office) of “The New CBS Audio-File Sound Effects Library CD 2”. This “office noise” track includes voices, phones, doors and HVAC background as recorded in a typical office setting. The SPL of the sound track is adjusted for 65dBC SPL as measured at the microphone under test’s location. The noise speaker is 60° off-axis, 5 feet from the microphone, and is in the left of the picture below. This simulates what might be expected in a normal office during peak work hours. “Voice Noise” conditions include the “Quiet” noise and add an off-axis looped playback of “The age of automation” by a “trained voice” 30-year-old female. The SPL is adjusted for 94 dBC @ 1” from the speaker. This 60° off axis voice is 5 feet from the microphone, resulting in approximately 65 dBC at the microphone position. This simulates what might be experienced with someone talking close to your cubicle.


1.1.3 Moderate Sized Conference Room Testing in a 23’ x 13.5’ x 9’ (high) Conference room with sheetrock walls, suspending ceiling of 2’ x 4’ acoustic ceiling tiles and carpeting is the “Conf Rm” data. During all tests both doors to the room were closed. “Quiet” conditions include noise from the system under test’s electrical circuitry and acoustic noise from the fans and hard drive in the computer enclosure. The test computer has three internal fans: one on the Pentium, one on the power supply and one on the rear of the case. This simulates what might be experienced working alone in a conference room with the doors closed. “Office Noise” conditions include the “Quiet” noise and adds looped playback of Track 6 (Office) of “The New CBS Audio-File Sound Effects Library CD 2”. This “office noise” track includes voices, phones, doors and HVAC background as recorded in a typical office setting. The SPL of the sound track is adjusted for 65dBC SPL as measured at the microphone under test’s location. The noise speaker is 60° off-axis, 5 feet from the microphone, and is in the left of the picture below. This simulates what might be experienced working in a conference room with the doors open.


2.0 Test Microphones – Summary of Characteristics Microphone Type Computer

Interface Used

Comments

Andrea Superbeam Array Microphone

Far-field two-element array with unidirectional elements. Separate channels for each microphone.

Analog stereo Mic level input

Requires Analog Device Codec with Stereo Mic input (and SoundMAX Cadenza software). Retail $24.95

Andrea NC-61

Close-talking, headset microphone with acoustic noise canceling element

Analog Mic level mono input

“Gold Standard” for speech recognition accuracy. Selected by IBM for inclusion with ViaVoiceTM from 1998 to 2002. Retail $29.95

Plantronics DSP300

Close-talking, headset microphone with acoustic noise canceling element and DSP processing for noise reduction and signal enhancement

USB •Recommended by Microsoft for use with Office XP •Provided with a version of IBM ViaVoiceTM (Release 9 Professional). Est. Retail $79 to $109.95

Telex M-30 OEM PC Mic

Far-field single element with a unidirectional element • AATL Lab testing showed –14dB

@ 180º @1 KHz

Analog Mic level mono input

Supplied with some Dell computers. Est. Retail $9.99 - $14.99


3.0 Test Equipment & Test Software ITEM QTY Reference Type (or equivalent)

Equipment Computer PC-1 1 Dell Dimension XPS T450, 450 MHz PIII 128MB RAM

Creative SoundBlaster Live! Platinum soundcard. Computer PC-2 1 Intel motherboard D845WN with Analog Devices AD1885 Codec,

Intel Pentium 4 1.6 GHz with Intel cooling fan, 256 MB PC133 RAM, IBM 20.57GB 7200 RPM Hard Drive, #IC35L020AVER07; Mid Tower ATX case with Power Man ATX300 Watt Power Supply Model FSP-300-60BTV; Video card Matrox Millennium G450, 32 MB; Audio via Analog Devices CNR card with Stereo Mic input, Mic input using 1885 Codec. Three fans in PC: one rear exhaust, one Power Supply, one on Pentium heatsink

Artificial Head & Torso 1 B &K (Brüel & Kjær) type 4128 Head and Torso Simulator Audio Amplifier 1 Radio Shack Model MPA-46 Cat No. 320-2042 Amplified Speaker 1 Fostex Model 6301B3E (33W) Sound Level Meter (Digital) 1 Extech Model 407750 (±1.5 dB accuracy) Sound Level Meter (Analog)

1 Radio Shack Cat No. 33-2050

Speaker Stand 1 Fidek FS-102 Cables 1 Audio interconnect cables as required

Software PC-1 Windows 98 SE 1 Microsoft PC-2 Windows XP Professional

1 Microsoft

Office XP Professional 1 Microsoft Persono Version 1.9.211a 1 Plantronics software for DSP-300 Office Noise 1 Track 6 (Office) of “The New CBS Audio-File Sound Effects

Library CD 2 Audio Setup wav file 1 “Office XP Setup” script recorded by 45-year-old male (RS)

“trained voice”. Enrollment Script wav files 1 “Office XP enrollment” script recorded by 45-year-old male (RS)

“trained voice”. Recognition Script Wav file 1 Andrea “The Age Of Automation” script recorded by 45-year-old

male (RS) “trained voice”. Recognition Script Wav file 1 Andrea “The Age Of Automation” script recorded by 30-year-old

female (CB) “trained voice”. Recognition Script .txt file 1 Reference text for the wav file Scoring Utility 1 Andrea Digital Scoring Utility


4.0 Test Script The text of the “Age of Automation”* test script follows. The age of automation, which was supposed to free American business from the burden of paperwork under which it had so long struggled, has instead only made things worse. Computers don't create less paperwork; what they do is make it possible to create more and better looking paperwork faster and more efficiently. Now that every office has fax capability in addition to computers, the mountain of paperwork is getting taller at a much faster rate. And what a mountain it is! A few years back one division of a large corporation we were working with removed 10 T of paper during a cleanup campaign. A California personnel company surveyed over 900 personnel directors about the portions of their jobs that was committed to routine paperwork. More than half of them said they spend between one and three hours each day, which works out to as much as roughly 94 work days a year. I can think of some fields in which the paperwork total would be far higher. Copies of reports and memoranda are circulated widely, with no one questioning whether they are needed. Top management is bombarded with documents from several levels down, and lower level managers find themselves submerged in cascades of paperwork from above, which seems to gather authority and urgency at each successive level. The possibility of "Computer crashes" dictates that files be backed up on disks as well as on "Hard" paper copies so now we have disc files to manage, but we have not gotten rid of our paper files. I can't count the number of people I know who have vowed to put all the vital paper information into their computers and then still kept all the paper files as well as back up disks. What do we do with all this paper? We push it around for a while, moving it from one stack to another, leave it in the in basket for a few days hoping it will disappear, maybe, and eventually file it - where it will probably never be looked at again. Less than 5 percent of filed documents are ever referred to after the first year, but we store them for years, "Just in case. " The solution? Devise ways to screen it more effectively, to handle faster the papers that do get through to you, and review skeptically the ones you are accustomed to creating. Be willing to try some unconventional techniques: Stop sending a report to see if anyone misses it! Setup system and stick to them. Purchase large wastebaskets and investigate possibilities for recycling. * From “The Time Trap” Third Edition, by Alec Mackenzie

Andrea Audio Test Labs 19 Superbeam Array Microphone Speech Recognition Performance

5.0 Test Data

5.1 Microsoft Office XP Speech Engine Complete Test Results

Small Office Open Cube Conf Room Quiet Office Noise Quiet Office Noise Voice Noise Quiet Office Noise

Microphone Setup Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy (%) (%) (%) (%) (%) (%)(%)

Close-talking microphones @ 1" Andrea NC-61 Close-talking Noise canceling headset 75.6 76.0 76.2 75.9 - 78 77 NC-61 with PureAudio 83.3 83.6 78.7 75.6 - 79.7 80.4 Plantronics DSP300 USB NC headset 78.9 79.9 77.8 79.7 - 78 74.7

Far-field Microphones @ 18" Andrea Superbeam Array Microphone (Array2-S) 84.6 81.6 87.2 82 81.8 84.5 78 Telex M-30 69.2 51 77 51.9 73.9 71.4 45.1 Telex M-30 with PureAudio 79.9 66.3 84.12 83.3 82.9 79.9 63.9 Notes to Office XP test data 1: All test percentages are the average of at least three consecutive runs. Tests performed 4/19/2002 to 4/29/2002 2: Enrollment is repeated for each setup and each location 3: SoundMAX® CadenzaTM V3.2 Drivers composed of: AEAudio.sys driver version 3.0.2.17 and Smwdm.sys version 5.12.1.3046 4: Computer P4 1.6 GHz, running Windows XP

Documents

Andrea Superbeam Array Microphone Speech Recognition