Joseph Picone, PhDHuman and Systems Engineering

Professor, Electrical and Computer Engineering

Bridging the Gap in Human and Machine Performance

HUMAN AND SYSTEMS ENGINEERING:

Evolution For Better Infrastructure

• 10 years at MS State

• Public Domain Speech Recognition

• Jumpstarted in 1997 by a DoD grant

• Center for Advanced Vehicular Systems

• State funded to support Nissan

• Three Complementary Thrusts

• Extension center colocated with Nissan in Canton, Mississippi

• Statewide economic development

• Assist first-tier suppliers

Introduction to Human and Systems Engineering Page 2 of 7

A Virtual Tour of CAVS at Mississippi State University

Page 3 of 7

Intelligent Electronic Systems At A Glance

Computer Networking:• Wireless Communications• Intelligent Sensors• Collaborative Vehicles

Intelligent Systems:• Speech Processing• Machine Learning• Dialog Systems• Human Factors and Ergonomics

Integrative Activities:• Challenge X• Capstone Design Experiences

Introduction to Human and Systems Engineering Page 4 of 7

Phase I Testbed: Campus Bus Networking

• Instrument the campus bus system to collect real-time data

• Modular architecture to support a variety of sensors and high speed data communications

Introduction to Human and Systems Engineering Page 5 of 7

Dialog Systems Applications in Automotive

• Noise robustness in both environments to improve recognition performance

• Advanced statistical models and machine learning technology

• In-vehicle dialog systems improve information access.

• Advanced user interfaces enhance workforce training and increase manufacturing efficiency.

Introduction to Human and Systems Engineering Page 6 of 7

Speaker Verification Via Metadata Extraction

• Recognition of emotion, stress, fatigue, and other voice qualities are possible from enhanced descriptions of the speech signal

• Fundamentally the same statistical modeling problem as other speech applications

• Fatigue analysis from voice under development under an SBIR (from Shriberg, et al., IEEE Spectrum, April 2003)

Introduction to Human and Systems Engineering Page 7 of 7

The Challenge X Program

• Competition created by automotive industry, government, and academic partners

• Challenges university-level engineering students to decrease total energy consumption and emissions

• Maintain or exceed vehicle utility and performance

• Cooperative venture between industry and universities

• Faculty Advisor:G. Marshall Molen

Introduction to Human and Systems Engineering Page 8 of 7

APPLICATIONS OF RISK MINIMIZATIONTO SPEECH RECOGNITION

Jon Hamaker, Aravind Ganapathiraju and Joseph PiconeIntelligent Electronic Systems

Human and Systems Engineering

URL: http://www.isip.msstate.edu/publications/seminars/external/2004/dod/

ABSTRACT: Statistical techniques based on Hidden Markov models (HMMs) with Gaussian emission densities have dominated the signal processing and pattern recognition literature for the past 20 years. However, HMMs suffer from an inability to learn discriminative information and are prone to overfitting and over‑parameterization. In this presentation, we will review our attempts to apply notions of risk minimization into pattern recognition problems such as speech recognition. New approaches based on probabilistic Bayesian learning are shown to provide an order of magnitude reduction in complexity over comparable approaches based on HMMs and Support Vector Machines.

BIOGRAPHY: Joseph Picone is currently a Professor in the Department of Electrical and Computer Engineering at Mississippi State University and an Academic Thrust Leader at the Center for Advanced Vehicular Systems. For the past 15 years he has been promoting open source speech technology. He has previously been employed by Texas Instruments and AT&T Bell Laboratories. Dr. Picone received his Ph.D. in Electrical Engineering from Illinois Institute of Technology in 1983. He is a Senior Member of the IEEE and a registered Professional Engineer.

Abstract and Biography

Applications of Risk Minimization Page 1 of 10

• Optimal decision surface is obviously a line

• Introduce two more data points

• How much can we trust isolated data points?

• Can we integrate prior knowledge about data, confidence, or willingness to take risk?

Generalization and Risk

• Optimal decision surface is still a line (good generalizaton)

• Optimal decision surface changes abruptly

Applications of Risk Minimization Page 2 of 10

• Deterding Vowel Data: 11 vowels spoken in “h*d” context; 10 log area parameters; 528 train, 462 SI test

Approach % Error # Parameters

SVM: Polynomial Kernels 49%

K-Nearest Neighbor 44%

Gaussian Node Network 44%

SVM: RBF Kernels 35% 83 SVs

Separable Mixture Models 30%

RVM: RBF Kernels 30% 13 RVs

Static Pattern Classification With SVMs

Applications of Risk Minimization Page 3 of 10

Applications of SVMs to Conversational Speech

Information Source HMM Hybrid

Transcription Segmentation AD SWB AD SWB

N-Best Hypothesis 11.9 41.6 11.0 40.6

N-Best N-Best 12.0 42.3 11.8 42.1

N-Best + Ref. Reference 6.6 — 3.3 5.8

N-Best + Ref. N-Best + Ref. 11.9 38.6 9.1 38.1

Notes:

• SVMs not exposed to alternative segmentations during training (closed-loop)

• SVM performance is high when there is no mismatch between the training and evaluation conditions

• Complexity (parameter count) approaches HMMs

Applications of Risk Minimization Page 4 of 10

• A kernel-based learning machine

• Incorporates an automatic relevance determination (ARD) prior over each weight (MacKay)

• A flat (non-informative) prior over completes the Bayesian specification

)),(|w(N)|w(PN

i iii

0

10

N

iii )x,x(Kww)w;x(y

10

)w;ix(yi

e)w;x|t(P

1

11

Relevance Vector Machines

Applications of Risk Minimization Page 5 of 10

• The goal in training becomes finding:

• Estimation of the “sparsity” parameters is inherent in the optimization – no need for a held-out set!

• A closed-form solution to this maximization problem is not available. Iteratively reestimate

)X|t(p

)X|,w(p)X,,w|t(p),w(p

where)X,t|,w(p,wmaxargˆ,w

ˆ andw

Iterative Reestimation of Hyperparameters

Applications of Risk Minimization Page 6 of 10

• Deterding Vowel Data: 11 vowels spoken in “h*d” context; 10 log area parameters; 528 train, 462 SI test

Approach % Error # Parameters

SVM: Polynomial Kernels 49%

K-Nearest Neighbor 44%

Gaussian Node Network 44%

SVM: RBF Kernels 35% 83 SVs

Separable Mixture Models 30%

RVM: RBF Kernels 30% 13 RVs

RVM and SVM Comparison — Static Patterns

Applications of Risk Minimization Page 7 of 10

• RVMs yield a large reduction in the parameter count while attaining superior performance

• Computational costs mainly in training for RVMs but is still prohibitive for larger sets – O(N3) vs. O(N2) for SVMs and O(N) for HMMs

Approach Error

Rate

Avg. # Parameters

Training Time

Testing Time

SVM 16.4% 257 0.5 hours 30 mins

RVM 16.2% 12 30 days 1 min

RVM and SVM Comparison — Alphadigits

Applications of Risk Minimization Page 8 of 10

ApproachError

RateAvg. #

ParametersTraining

TimeTesting

Time

SVM 15.5% 994 3 hours 1.5 hoursRVM

Constructive 14.8% 72 5 days 5 mins

RVMReduction 14.8% 74 6 days 5 mins

• Data increased to 10,000 training vectors

• Reduction method has been trained up to 100k vectors (on toy task). Not possible for Constructive method

Preliminary Results on Learning

Applications of Risk Minimization Page 9 of 10

• Reduction of complexity at the same level of performance is interesting:

• Results hold across tasks

• RVMs have been trained on 100,000 vectors

• Results suggest integrated training is critical

• Risk minimization provides a family of solutions:

• Is there a better solution than minimum risk?

• What is the impact on complexity and robustness?

• Applications to other problems?

• Speech/Non-speech classification?

• Speaker adaptation?

• Language modeling?

Summary — Practical Risk Minimization?

Applications of Risk Minimization Page 10 of 10

Applications to Speech Recognition:

1. J. Hamaker and J. Picone, “Advances in Speech Recognition Using Sparse Bayesian Methods,” submitted to the IEEE Transactions on Speech and Audio Processing, January 2003 (in revision).

2. A. Ganapathiraju, J. Hamaker and J. Picone, “Applications of Risk Minimization to Speech Recognition,” to appear in the IEEE Transactions on Signal Processing, August 2004.

3. J. Hamaker, J. Picone, and A. Ganapathiraju, “A Sparse Modeling Approach to Speech Recognition Based on Relevance Vector Machines,” Proceedings of the International Conference of Spoken Language Processing, vol. 2, pp. 1001-1004, Denver, Colorado, USA, September 2002.

4. J. Hamaker, Sparse Bayesian Methods for Continuous Speech Recognition, Ph.D. Dissertation, Department of Electrical and Computer Engineering, Mississippi State University, December 2003.

5. A. Ganapathiraju, Support Vector Machines for Speech Recognition, Ph.D. Dissertation, Department of Electrical and Computer Engineering, Mississippi State University, January 2002.

Influential work:

6. M. Tipping, “Sparse Bayesian Learning and the Relevance Vector Machine,” Journal of Machine Learning, vol. 1, pp. 211-244, June 2001.

7. D. J. C. MacKay, “Probable networks and plausible predictions --- a review of practical Bayesian methods for supervised neural networks,” Network: Computation in Neural Systems, 6, pp. 469-505, 1995.

8. D. J. C. MacKay, Bayesian Methods for Adaptive Models, Ph. D. thesis, California Institute of Technology, Pasadena, California, USA, 1991.

9. E. T. Jaynes, “Bayesian Methods: General Background,” Maximum Entropy and Bayesian Methods in Applied Statistics, J. H. Justice, ed., pp. 1-25, Cambridge Univ. Press, Cambridge, UK, 1986.

10. V.N. Vapnik, Statistical Learning Theory, John Wiley, New York, NY, USA, 1998.

11. V.N. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, New York, NY, USA, 1995.

12. C.J.C. Burges, “A Tutorial on Support Vector Machines for Pattern Recognition,” AT&T Bell Laboratories, November 1999.

Applications of Risk Minimization

Brief Bibliography