VIENTATION PAGE

AD-A264 754 .....I illl l l l I~t!ll l lllil . ..: . . " . .. . .. ' -. *.. . . . ...-

REPORT DATE i FRPORT TYPE AN ) DAT.I 0o,

r 1l1 AL, 3(1 SEP P'

4. TITLE AND SUBTITLE • 5 A. -',c- -

THE 1"91 NEURNI. INFORLATION PROCESS 5r.'FC

,SY.;TEMS--NAFIJL & SYNTHFT [C -IU')

6. AUTHOR(S)

Professor john loodv D T C. FSF S- .IAV rl PrTIC"7. PERFORMING ORGANIZATION NAME(S) A RS U ,r" ' " 7 -"

- ,'MAY 14 1993, REPORT NUV8B-;

,CcInpllt er Sc [enCle

NEW Ha Ver: c 66520-21.58 ,"-

9 SPONSORING,, MONITORING AGENCY NAME(S) AND ADORESS(ES) 10 SPCNSOR;NG VC%'0- h'

AFOSRi'NM AGENCY R[POwT N ',"5F.

110) DUNCAN, AVE. SUTE B115 AFOSR--

BOLL INC .FE Y 2(' 332-" '"1

11. SUPPLEMENTARY NOTES

12a DISTRIBUTION AVAILABILITY STATEMENT 1 DIST 2, ,

APPROVED FOR PUBLIC RELEASE: DISTP[BUTIO\ IS UNLMIT 'ED L'L

13. ABSTRACT WMaxmum 200 words)

The 0•91 "Neural Information ProcessIng Systems-Natural and Synthetic" (NIPS) was

heLd in Denver Colorado. from 2-5 1)ecember 1991. Since its inception in 1987. -,,

NIPS conferenCe has attracted researchers from many disciplines who are applying

their expertise to problems in the f Lel ( of neural networks. The conference andthe? following two day workshop have become a forum for presenting the latest:

research rEsuLts and for leading researcher's to gather and exchanre ideas. The

[991 conference maintained the high Le-vel of excitenent of its predecessors.

[mportant new theoreticzal resIlts were presented concerning the capabiliitv and

Vene rali zat Lon performance of networks.

B5- 2 93-10670Ii~I11 III• UF II!i •I liI 1114 SUB.;'T TERMS *• ,..,. . •- -

2 2

16 PRICE COOE

1? SECURITY CLASSIFICATION 18 SECURITY CLASSIFICATION i 19 SECURITY CASS, - A .IN CN 'AA

OrF REPORT OF TH GEI OF

UINCLASS IFI I ED OFCASS{E'[EI) A 'Hft SS1F TED SAR(SAME AS REFP.

FINAL TECHNICAL REPORT

FOR

YALE UNIVERSITY

AFOSR-91-0438

30 SEP 91 / 9 SP 92

,Acct>I,.y Fe' - , ..

NTIS C q&j

ByDtibulionr

Availability Codes

0 I A',hui a,'id/or

1 1~i~ jttaL_

D,'partment ot Comiputer cien~c and U ngincering

OREGON GRADUATE INSTITUTE

SCIENCE & TECHNOLOG\ .

February -1. 1993

Capt. Steven Suddarth, Ph.D.AFOSR/NE, Bldg. .110Boiling Air Force BaseWashington, DC 20332

Dear Dr. Suddarth:

This letter and the attached materials constitute the final report forS .---- AFOSR Grant 91-0'-38 which provided S5,000 for student travel grants

for the 1991 Neural Information Processing Systems Conference. Themoney was used to help 20 students as indicated in the attached list.

Also attached is a copy of the front matter of the proceedings whichresulted from NIPS ý9l. As is evident, many of the students we wereable to help made substantial contributions to the conference program.We are very grateful for your generous support and hope that you willbe able to continue to support NIPS conferences in the future.

ASinc

rely,

hn MoodyAssociate ProfessorNIPS*91 General Chairman

Department of Computer Sclerice ind I-n~ginerinit

OR EGO\ GRADLATIN 'ST!TLTE

SCIE\CE & TECIH\O~l)Q ~

1* $250 Quanfeng Wu qw0w~andrew.cmu edu

2.* $250 David Plaut dcpc~cs.cmii.edu3* $250 John Hampshire hamps~speech2.cs.cmu.edu4* $250 Frederick Waugh waugh~curly.harvard.edu

6.* $250. Lori Pratt pratt~paul.rutgers.edu7.* $250. Barak Pearimutter barak~james~psycb.yale

8.* $250. Alexandre Pouget alex~cajal.edec.edu9* $250 Steve Fisher fisherc~helmholtz.sdsc.edu

10.* $250. Sowmya Raxuachandran sowmya~cs.utexas.edu

11.* $250 Andrew Moore awm~mit.ai.edu12.* $250. Zoubin Ghahramani zoubint~psyche.mit.edu13.* $250. Ying Zhao yzhaocDai.mit.edu14.* $250 David Cohni pablo~Dcs.washington.edu

1S.* $250 Steve Nciwlan nowlan~helmholtz.sdsc~edu

16.* $250 Sherif Botros smb~ai.mit.edu17.* $250 Gary Scott scott~chewi.che.wisc.edu

18.* $250 Tony Bell tony~helmholtz.sdsc.edu20.* $250. Elizabeth Thomas cmjv465t0hermes.chpc.utexas.edu22.* $250 Tony Zador zadort~cs.yale.edu23.* $250 Antonette Logar N0801r0ttacsl.ttu.edu

ADVANCES IN

NEURALINFORMATION

PROCESSINGSYSTEMS 4

OTHER TITLES OF INTERESTFROM MORGAN KAUFMANN PUBLISHERS

NIPS-3-Advances in Neural Information Proceuing SystemsProceedingr of tie 1990 ConferenceEdited by Richard P. Lippmann. John E. Moody, and David S. Toureazky

NIPS-2.Advances in Neural lntfrmation Processing SystemsProceedings of the 1989 ConferenceEdited by David S. Touretzky

"ANIS- I-

Advances in Neural Informaion Processing SystemsProceedings of the 1988 ConferenceEdited by David S. Touretzky

Computer Systems That Learn: Classificatin and Prediction Methods from Statstics, Neural Nets,Machine Learning, and Zipert Systems

By Sholom M. Weiss and Casimir A. Kulikowski

Connectionist Models Summer School Proceedings1990 Edited by David S. Touretzky, Jeffrey L. Elman, Terrence J. Sejnowski,

and Geoffrey E. Hinton1988 Edited by David S. Touretzky, Geoffrey E. Hinton, and Terrence J. Seinowski

Learning MachinesBy Nils J. Nilsson, with an Introduction by Terrence J. Scjnowski and Halbert White

Foundations of Genertic AlgorithmsEdited by Gregory J.E. Rawlins

Genetic Algorithms: Proceedings of the Fourth !nwrnarwnal ConferenceEdited by Rick Belew and Lashon Booker

Genetic Algorithms: Proceedings of the Third International ConferenceEdited by David Schaffer

COLT-Proceedings of the Annual Workshops on Computational Learning Theory1990 Edited by Mark Fulk and John Case1989 Edited by Ron Rivest, Manfred Warmuth, and David Haussler1988 Edited by David Haussler and Leonard Pitt

ADVANCES IN

NEURALINFORMATION

PROCESSINGSYSTEMS 4

EDITED BY

JOHN E. MOODY

YALE UNIVERSITY

STEVE J. HANSONSIEMENS RESEARCH CENTER

RICHARD P LIPPMANNMIT LINCOLN LABORATORY

MORGAN KAUFMANN PUBLISHERS

2929 CAMPUS DRIVESUITE 260

SAN MATEO CALIFORNIA 94403

Editor Bnrce M Spaz

Production Manager Yone Ourmn

Project Management Profirnonal Book CenterComposition PfJ~sziona/ Book Centr

Cover Design Jo Jatksen

MORGAN KAUFMANN PUBLISHERS, INC.Editorial Office:2929 Campus Drive, Suite 260San Marco, CA 94403(415)578-9911

Copy.ight 0 1992 by Morgan Kaufmann Publishers, Inc.

All rights reservedPrinted in the United States of America

No part of this publication may be reproduced, stored in a retrieval system, or trans-mitted in any form or by any means-electronic, mechanical, photocopying, recording,or otherwise-without the prior written permission of the publisher.

96 95 93 94 92 5 4 3 2 1

library of Conges Cataloging in Publication Datais available for this title.

ISSN 1049-5258ISBN 1-55860-222-4

CONTENTS

Preface xv

Part I NEUROBIOLOGY

Models Wanrtyd: Must Fit Dimensions of Sleep and Dreaming ............. 3J. Allan Hobson, Adam N. Mamelak, andJeffrry P Sunon

Stationariry of Synaptic Coupling Strength Between Neurons with NonstanionaryDischarge Properties ........... ...................... II

Mark Sydorenko and Ert D. YoungPerturbing Hebbian Rules .............................. 19

Peter Dayan and Geoffrey Goodhill

Statistical Reliability of a Blowfly Movement-Sensitive Neuron ............ 27Rob de Ruyter van Steweninck and William Bwlek

The Clusteron: Toward a Simple Abstraction for a Complex Neuron ........ .35Bartletr W. ,Mel

Network activity determines spatio-temporal integration in single cells ....... .43Ojvind Bernandep, Christof Koch, and Rodney I Douglas

Nonlinear Pattern Separation in Single Hippocampal Neuronswith Active Dendritic Membrane ...... .......................... 51

Anthony M, Z7dor, Brendaf Claiborne. and Thomas H. BrowmSelf-organisation in real neurons: Anti-Hebb in 'Channel Space'? ............ 59

Anthonyj. BellSingle Neuron Model: Response to Weak Modulation in the Presence of Noise . . . 67

A.R. Bulsara, E. W Jacobs, and F MossDual Inhibitory Mechanisms for Definition of Receptive Field Characteristicsin a Cat Striate Cortex ...... ................................ 75

A.B. Bonds

V

A comparison between a neural netwok model for the formationof brain maps and experimental data .............................. 83

K Obermayer, K Schuiten. and G. G Bludel

Retinogeniculate Development: The Role of Competitionand Correlated Retinal Activity ........................ .... 91

Ran Keesing, David G. Stork. and Carla . Shatz

Part II NEURO-DYNAMICS

Locomotion in a Lower Vertebrate: Studies of the Cellular Basisof Rhythmogenesis and Oscillator Coupling ........................ 101

James T Buchanan

Adaptive Synchronization of Neural and Physical Oscillators .............. 109Kenji Doya and Shuji Yoshizawa

5u.z Synchronization without Frequency Locking in a CompletelySolvable Network Model .................................... 117

Heinz Schuster and C/rirof Koch

Oscillatory Model of Short Term Memory .......................... 125David Horn and Marius Usher

Part III SPEECH

Multi-State Time Delay Neural Networks for Continuous Speech Recognition 135Parick Haffner and Alex Waibel

Modeling Applications with the Focused Gamma Net .................. 143Jose C. Principe, Bert de Vries, Jyh Ming Kuo, Pedro Guides de Oliveira

Time-Warping Network: A Hybrid Framework for Speech Recognition ....... 151Esther Levin, Roberto Pienaceini. and Enrico Bocchiei

Improved Hidden Markov Model Speech Recogi.ition Using RadialBasis Function Networks .................................... 159

Elliot Singer and Richard P Lippmann

Connectionist Optimisation of Tied Mixture Hidden Markov Models ....... 167Steve Renak, Nelson Morgan. Hervi BourlaranHoracio France, and Michael Cohen

Neural Network-Gaussian Mixture Hybrid for Speech Recognitionor Density Estimation ....... ............................... 175

Yoshba Bengio, Renato De Mori. Giovanni Flammia. Ralf Kompe

JANUS: Speech-to-Speech Translation Using Connectionistand Non-Connectionist Techniques ..... ........................ 183

Alx Wi444 Ajay N. Jain. Arthru McNair, Joe Tebeiskis, Lovise OswrlteozHireaki Saiso. On# Schmidbauer, Ito Sloboda, and Monika Wosecryna

vi

Forward Dynamics Modeling of Speech Motor ControlUsing Physiological Data ......... ..................... 191

Makoto Hrayama. Eric Vankiotii-Bateson,Mieuo Kawato. and Michael L Jordan

English Alphabet Recognition with Telephone Speech ............... 199Mark Fanr)t Ronald A. Cole, aid Krut Rogmski

Part IV LANGUAGE

Generalization Performance in PARSEC-A Strucrtred ConnectionistParsing Architecture ......... ................................ 209

Ajay N. Jain

Constructing Proofs in Symmetric Networks ..... ................... 21-Gadi Pinkas

A Connectionist Learning Approach to Analyzing Linguistic Stress ......... 225Prahlad Gupw and Daed S. Touretzky

Propagation Filters in PDS Networks for Sequencingand Ambiguity Resolution ......... ............................ 233

Ronald A. Sumada and Michael G. Dyer

A Segment-based Automatic Language Identification System .... ........... 241ieshwant K Muthusamy and Ronald A. Cole

Part V TEMPORAL SEQUENCES

The Efficient Learning of Multiple Task Sequences ..... ............... 251Sarinder P1 Singh

Practical Issues ir Temporal Difference Learning ...... .................. ... 259Gerald Tesaurv

HARMONET: A Neural Net for Harmonizing Choralesin the Style of J.S. Bach ........ .............................. 267

Hermann Hill Johannes Feulner and Wolfram Menzel

Induction of Multiscale Temporal Structure .......................... 275Michael C Mozer

Network Model of State-Dependent Sequencing ........................ 283Jeffrey P Sutton, Adam N. Mamelak, and]. Allan Hobson

Learning Unambiguous Reduced Sequence Descriptions .... ............. 291Jurgmn Schmidhuber

Part VI RECURRENT NETWORKS

Recurrent Networks and NARMA Modeling ...... ................... 301

Jerome Conner, Les E Atlas. and Douglas R. Martin

vii

Induction of Finite-State Automata Using Second-Order Recurrent Networks . . 309R/.mond L. Warrus, and Gary M. Kuhn

Extracting and Learning an Unknown Grammarwith Recurrent Neural Networks ....... .......................... 317

C.L. Gil, C . Miller, D. Chen, G.Z Sun, H. H.(Chen, and Y.C Lee

Operators and curried functions: Training and analysisof simple recurrent networks ........ ............................ 325

Janet Wile and Anthony Bloesch

Green's Function Method for Fast On-line Learning Algorithmof Recurrent Neural Networks ........ ........................... 333

Guo-Zheng Sun, Hsing-Hen Chen. and Yee-Chun Lee

Dynamically-Adaptive Winner-Take-All Networks ...................... 341Trent E. Lange

PartVII VISION

Information Processing to Create Eye Movements ...................... 351David A. Robinson

Decoding of Neuronal Signals in Visual Pattern Recognition ............... 356Emad N. Eskandar, Barry j Richmo4 John A. HertzILance M. Optican. and Tmrlh KjAr

Learning How to Teach or Selecting Minimal Surface Data ................ 364Davi G;'ger and Ricardo A. Marques Pereira

Learning to Make Coherent Predictions in Domains with Discontinuities . . . . 372Suzanna Becker and Geoffrey E Hinton

Recurrent Eye Tracking Network Using a Distributed Representationof Image Motion ......... .................................. 380

PA. Wola, S.G. Lisber, and TJ. Sejnowski

Against Edges: Function Approximation with Multiple Support Maps ......... 388Trevor Darrell and Alex Pendand

Markov Random Fields Can Bridge Levels of Abstraction ............... 396Paul R. Cooper and Peter N. Prokopowuiz

Illumination and View Position in 3D Visual Recognition ................. 404Amnon Shashua

Hierarchical Transformation of Space in the Visual System ................. 412Alexandre Pouget, Stephen A. Fisher, and Teience. Sjnoulki

VISIT A Neural Model of Covert Visual Attention ...................... 420Subutai Abmad

Visual Grammars and their Neural Nets ....... ...................... 428Eric Mjoknesi

stiii.

Learning to Segment Images Using Dynamic Feature Binding ........... 436Michael C Moto, Richard S Zeml, and Madrie B&hrmann

Combined Neural Network and Rule-Based Frameworkfor Probabilistic Pattern Recognition and Discovery . .............. 444

Hayit K Griwupan. Rodney Goodman. and Rama Chellappa

Linear Cperater for Object Recognition ............... .... . 452Ranen &Bri and Shsmon UIman

3D Object Recognition Using Unsupervised Feature Extraction ............. 460Nathan Infrator. Josh I Gold, Heinrich H. Bazhoff, and Shimon Edlman

Part VIII OPTICAL CHARACTER RECOGNITION

Structural Risk Minimization for Character Recognition ............... 471. Guyon. V Vapnk. BA Boser, L. Boouou. and SA. S4l0a

Image Segmentation with Networks of Variable Scales ................... 480Hans P Graft Craig R Noh* and Jan Ben

Multi-Digit Recognition Using a Space Displacement Neural Network ........ 488Ofer Matan. Christopher J. C Burges, Yann Le Cun, and John S. Denker

A Self-Organi.-ng Integrated Segmentation and Recognition Neural Net ..... .. 496

Jim Keeler and David E RumelUart

Recognizing Ovedapping Hand-Printed Charactersby Centered-Object Integrated Segmentation and Recognition .............. 504

Gale L. Martin and Mosfq Rashid

Adaptive Elastic Mode!: for Hand-Printed Character Recognition ............ 512Geoffivy E Hinton, Christophe K ! Williams, and Mich/l D. Retvow

Part IX CONTROL AND PLANNING

Obstacle Avoidance through Reinforcement Learning .................. 523TonyJ Prescon and John E. W Mayhew

Active Exploration in Dynamic Environments ...... ................... 531Sabasnan A. Thru n and Knut Moller

Oscillatory Neural Fields for Globally Optimal Path Planning ............. 539Michael Lemmon

Recognition of Manipulated Objects by Motor Learning .... ............. 547Hiroaki Gomi and Minuo Kawato

Refining PID Controllers using Neural Networks .................... .. 555Gary M. Soon, Jfde W Shavlik. and W Harmon Ray

Fast Learning with Predictive Forward Models ....................... 563Carlos Brody

iI

Fast, Robust Adaptive Control by Learning only Forward Models .. 7.1.... 571Andnpr W Moore

Reverse TDNN: An Architecture for Trajecory Generation ............ 579Pamece Simard and Yann Le GCn

Learning Global Direct Inverse Kinematics .$............. . 589David DeMers and Kenneth Kruz-Delgado

A Neural Net Model for Adaptive Control of Saccadic Accuracyby Primate Cerebellum and Brainstem ....... 595

Paul Dean, John Eo W' ayhew. and Pat Langdan

Lear:iing in the Vestibular System: Simulations of Vestibular CompensationUsing Recurrent Back-Propagation ....... ......................... 603

Thomai! Anastasio

A Cortico-Cerebellar Model that Learns to Generate Distributed MotorCommands to Control a Kinematic Arm . . ... .............. 611

N.E, Berrhierr S.P Singh. A.G. Barto. andj.C Houk

A Computational Mechanism to Account for Averaged ModifiedHand Trajectories ....... .................................. 619

Ealan A. Henis and Tamar Flash

Simulation of Optimal Movements Using theMinim um-Muscle-Tension-Change Model .... ..................... 627

Menashe Dornay Yoji Uno, Misuo Kawato. and Ryoji Suzuki

Part X APPLICATIONS

ANN Based Classification for Heart Defibrillators .................... 637M. Jabi, S. Pakardu P. Leong Z. Chi, B. Flower, and Y Xie

Neural Network Diagnosis of Avascular Necrosisfrom Magnetic Resonance Images ....... ......................... 645

Armando Manduca, Paul Christ and Richard Ehman

Neural Network Analysis of Even: Related Potentialsand Electroencephalogram Predicts Vigilance .......... ...... . 651

Rita Venturini. William W Lytton, and Terrencel. Sejnowsks

Neural Control for Rolling Mills: Incorporating Domain Theoriesto Overcome Data Deficiency ........ ........................... 659

Martin Roscheisen. Reimar Hofmann. and Volker Tresp

Fault Diagnosis of Antenna Pointing Systems Using Hybrid Neural Networkand Signal Processing Models ....... ........................... 667

Padhraic Smyth and Jeff Melhtrom

Multimodular Architecture for Remote Sensing Options ............. 675Sylvie Thiria, Carlo, Meuia, Fouad Badran, Michel Crnpon

x

Principled Architecture Selection for Neural Networks: Applicationto Corporate Bond Rating Prediction .......... ........ 683

John Moody and joachim Urans

Adaptive Development of Connectionist Decodersfor Complex Error-Correcting Codes . ...................... 691

Sheri L. Gish and Maro Blaum

Application of Neural Network Methodology to the Modellingof the Yield Strength in a Steel Rolling Plate Mill ................. 698

Ah Chung Tsoi

Computer Recognition of Wave Location in Graphical Databy a Neural Network ........ ................................ 706

Donald T Freeman

A Neural Network for Motion Detection of Drift-Balanced Stimuli ........ .714Hilary Tunley

Neural Network Routing for Random Multistage Interconnection Networks . . . 722Mark W Goudrrau and C. Lee Gilet

Networks for the Separation of Sources that are Superimposed and Delayed . . . 730John C. Plan and Federico Faggin

Part XI IMPLEMENTATION

CCD Neural Network Processors for Pattern Recognition ............... 741Alice M. Chiang. Michael L. Chuang, and Jefy R. LaFranchie

A Parallel Analog CCD/CMOS Signal Processor .... ................. 748Charles F Neugebauer and Amnon Yariv

Direction Selective Silicon Retina that uses Null Inhibition ............... 756Ronald G. Bnuon and Tobi Delbruck

A Contrast Sensitive Silicon Retina with Reciprocal Synapses ............. 764Kwabena A. Boahen and Andreas G. Andreou

A Neurocomputer Board Based on the ANNA Neural Network Chip ....... .773Eduard Sakicnger, Bernhard E. Boser; and Lawrence D. Jackel

Software for ANN training on a Ring Array Processor .................. 781Phil Kohn, Jef Bilmes. Nelon Morgan. and james Beck

Constrained Optimization Applied to the Parameter Setting Problemfor Analog Circuits ....................................... 789

David Kirk, Kurt Flucher, Lloyd Warn, and Alan Barr

Segmentation Circuits Using Constrained Optimization ................. 79'John G. Harri

xi

Analog LSI Implementation of an Auto-Adaptive Networkfor Real-Time Separation of Independent Signals ................ 805

Marc H Cohen, Pbllipe 0. Pouliqutn, and Andreas G. Andrru

Temporal Adaptation in a Silicon Auditory Nerve ..... ................. 813John L=a ro

Optical Implementation of a Self-Organizing Feature Extractor ............. 821Dana Z Anderson. Clau Benkert, Verna HebLrm Ju-Seog Jang,Don Montgomer); and Mark Saffman

Part XII LEARNING AND GENERALIZATIONPrinciples of Risk Minimization for Learning Theory ....... . ... . 831

V 'apnik

Bayesian Model Comparison and Backprop Nets . . . .. ......... 839David J.C MacKay

The Eflective Number of Parameters: An Analysis of Generalizationand Regularization in Nonlinear Learning Systems ..................... 847

John E. Moody

Estimating Average-Case Learning Curves Using Bayesian.Statistical Physics and VC Dimension Methods ...... .................. 855

David Hausler, Michael Kearmu. Manjred Opper, and Robert Schapire

Constant-Time Loading of Shallow I -Dimensional Networks ............... 863STephen Judd

Experimental Evaluation of Learning in a Neural Microsystem .............. 871Joshua ALkpecmr Anthony Jayakumar, and Stephan Luna

Threshold Network Learning in the Presence of Equivalences ............... 879John Shawue- Taylor

Gradient Descent: Second Order Momentum and Saturating Error ........... 887Barak Peardmuner

Tangent Prop-A formalism for specifying selected invariancesin an adaptive network ........ ............................... 895

Pamte Siman4 Bernard Victorri, Yann Le Cun. and John Denker

Polynomial Uniform Convergence of Relative Frequencies to Probabilities . . . . 904Alberto Bertoni, Paola Campadelli, Anna Morpurgo, and Sandra Panw

Unsupervised learning of distributions on binary vectorsusing 2- layer networks ......... ............................... 912

Yodv Freund and David Hauiler

Incrementally Learning Time-varying Half-planes ..... ................. 920Andhony Kis&, Thomas Peucke, and Ron L. Rivet

'Ui

The VC-Dimension versus the Statistical Capacity of Multilayer Networks .... 928Chuany Ji and Dememr !~a~ru

Some Approximation Properties of Projection Pursuit Learning Networks .... 936Ying Zhao and Chrstopher G. Atkezon

Neural Computing with Small Weights ...................... 944Kai- Yeung Siu and ]ekoshua Bruck

A Simple Weight Decay Can Improve Generalization .................. 950Andm Krogh and John A. Hertz

Bast-First Model Merging for Dynamic Learning and Recognition .......... 958Stephen M. Omobundro

Part XlII ARCHITECTURES AND ALGORITHMS

Rule Induction through Integrated Symbolic and Subsymbolic Processing . . . 969Clayton McMi~lan, Michael C Moaw, and Paul Smlensky

Interpretation of Artificial Neural Networks:Mapping Knowledge-Based Neural Networks into Rules ................ 977

Geoffrey Towell and Jude W Sbavhk

Hierarchies of adaptive experts ................................. 985MiShael I Jordan and Robert A. Jacobs

Adaptive Soft Weight Tying using Gaussian Mixtures .................. 993Steven I Nowlan and Geoffrey E Hinton

Repeat Until Bored: A Pattern Selection Strategy .... ................. 1001Paul W Munro

Towards Faster Stochastic Gradient Search ......................... 1009Christian Darken and John Moody

Competitive Anti-Hebbian Learning of Invariants .................... 1017Nicol N. Schraudolph and Terrence J. Sjnowsk

Merging Constrained Optimisation with Deterministic Annealingto *Solve" Combinatorially Hard Problems ........................ 1025

Paul StolorzKernel Regression and Backpropagation Training with Noise ............. 1033

Pain Kiaisnnen and Lawe Holmsrom

Splines, Rational Functions and Neural Networks .................... 1040Robert C. Wiliamsn and Peter L. Bartlert

Networks with Learned Unit Response Functions .................... 1048John Moody and Norman Yarvin

Learning in Feedforward Networks with Nonsmooth Functions ........... 1056NhiobasJ Reddingrand T Doums

Diii

Iterative Construction of Sparm Polynomial Approximations ........... 1064Ternre D. Sanger, Richard S. Sutton. and ChnftopherJ Mathew

Node Splitting- A Contructive Algorithm for Feed-Forward Neural Networks 1072Mike Wynne-Jones

Information Measure Based Skeletonisation .................. 1080Sowmya Ramachandran and Lonen Y Pran

Data Analysis Using G/Splina ........................... 1088David Roger

Unsupervised Classifiers, Mutual Information and 'Phantom Targets' . ... . 1096John S. Bridle. Anthony JIR. Heading, and Davidj.C. MacKay

A Network of Localized Linear Discriminants ....................... 1!02Martin S. Glauman

A Weighted Probabilistic Neural Network ......................... 1110David Montana

Network generalization for production: Learning and producingstyled Ietterforms ...... .................................. 1118

Igor Grebrt, David G. Stork, Ron Kersing. and Steve Mims

Shooting Craps in Search of an Optimal Strategyfor Training Connectionist Pattern Classifiers .... ................... 1125

J.B. Hampshire II and B. VK Vijaya Kumar

Improving the Performance of Radial Basis Function Networksby Learning Center Locations ...... ........................... 1133

Dietrich Wemchmreck and Thomas Dietterich

A Topograhic Product for the Optimization of Self-Organizing Feature Maps . . . 1141Hans-Ulrich Buer, AlUa Pawdeik, and Theo Gri•el

Part XIV PERFORMANCE COMPARISONS

Human and Machine 'Quick Modelin ........................... 1151Jakob Bernasofi and Kad Gutafion

A Comparison of Projection Pursuit and Neural NetworkRegression Modeling ...................................... 1159

Jenq-Neng Hwang, Hang Li. Martin Matchler,R Douglu Martin. and Jim Schimert

Benchmarking Feed-Forward Neural Networks: Models and Measures ....... 1167Leonard G. C. Haemey

Keyword Index ......................................... 1175Author Index .......................................... 1184

DYv

PREFACE

This volume contains 144 papers summarizing the talks and posters presented at the fifthNIPS conference (short for "Neural Information Processing Systems--Natural and Syn-thetc"), held in Denver, Colorado, from 2-5 December 1991. Since its inception in 1987,the NIPS conference has attracted researchers from many disciplines who are applying theirexpertise to problems in the field of neural networks. The conference and the followingtwo-day workshop have become a forum for presenting the latest research results and forleading researchers to gather and exchange ideas.

The 1991 conference maintained the high level of excitement of its predecessors.Important new theoretical results were presented concerning the capability and generaliza-tion performance of networks. Of particular interest are papers included in this volume byVapnik, MacKay, Haussler, and others, which describe how to relate the complexirv ofnetworks to generalization performance on unseen test data. Many new network architec-tures were described. Some integrate expert system rules with networks, build hierarchies ofnetworks, use radial basis function hidden nodes, and impose pre-specified invariance on thefinal solution. Neurobiological papers analyzed and modeled neurons in the hippocampus, incat striate cortex, and in the blowfly. They also modeled biological networks that control eyemovement, form topological maps, and compensate for head movement. Successful applica-tions of neural networks were described in the areas of speech, vision, language. control.medical monitoring, and system diagnostics. Of particular interest was a paper by Tesauro,which demonstrated how a network could be trained to play backgammon at an expert level,papers by Jain, Watrous, and Giles, which described approaches to learning grammars:hybrid hidden-Markov-model/neural-network speech recognizers described by Haffner.Levin, Singer, Renals, and Bengio; papers on optical character recognition; a paper by Jabri,which describes a network to control a wearable heart defibrillator; a paper by Smyth fordiagnosis of large-dish antenna pointing systems; and a paper by Rtscheisen concerningcontrol of force on rollers in steel rolling mills. Papers also described new analog and digitalVLSI chips, systems for neural network implementation, and compared neural network andstatistical approaches to pattern classification.

IV

An historical milestone was reached this year, NIPS-91 was the fifth NIPS conferencesince the first conference was held in 1987. To mark this anniversary, we decided to reviewthe history of events that led to the foundation of the NIPS conference and to discuss theevolution of the conference since its foundation. The following history is based in part onthe recollections of Jim Bower, Larry Jackei, and Ed Posner. Some of this history was pre-sented by Larry Jackel at the opening banquet.

While the first NIPS conference met in 1987, its origins can be traced back to the"Hopfest" meetings named in honor of John Hopsfield, held at Calech. The first few,1984-1986, were organized by Ed Posner of Caltech. These meetings met m the fall andincluded researches mainly from the Caltech campus and JPL In 1985. L.arry Jackel of BellLabs and Demetri Psaltis of Caltech organized the first of what were to become the 'Snow-bird" meetings. The meetings were intended to be small informal workshops and convenedin Santa Barbara. Twenty people were invited, but news of the meeting spread by word ofmouth, so that attendance ended up growing to 60. In 1986, the meeting reconvened atSnowbird, which offered better snow conditions. Jackel, Psaltis, and the other organizersintended to keep the attendance down to 100 people, but the interest was so great that manypeople were turned away even after the attendance was capped at 160. The first Snowbirdproceedings was edited by John Denker of Bell Labs and published by the American Instituteof Physics (AIP) press.

In 1986, the Snowbird meeting was the only neural network conference, and it clearlycould not accommodate the exploding numbers of researchers becoming interested in thefidd and still maintain the character of a small workshop. To respond to demand, theorganizers decided to make Snowbird a more dosed meeting, but to set in motion organiza-tion of a large meeting that would be open to all interested. The goal was to have a non,commercial meeting, dedicated to scholarhip, which would capture some of the flavor of theworkshop. The Snowbird organizers nominated a committee with Ed Posner as GeneralChairman and Yaser Abu Mostafa as Program Chairman (both of Caltech), to organiz andrun the 1987 NIPS conference, which was officially sponsored by the IEEE InformationTheory Society. Denver was chosen as the site due to its central geographical location, ease ofaccess by air, and dose proximity to the mountains and the University of Colorado atBoulder.

The 1987 organizers designed the NIPS conference to have many of the advantages of aworkshop, while still accommodating a large audience. To maximize scientific interchange.they decided to limit the oral presentations to a single stream, have posters be the majority ofpresentations, and indude poster preview as well as formal poster sessions. Furthermore, a setof post-onference workshops was organized at the Copper Mountain ski resort aher themain conference to enable small groups to discuss specific topics. The 1987 conferenceproved to be a great success, with about 450 attendees and 91 papers making it into theproceedings. Dana 7- Anderson of CU Boulder edited the proceedings, which were pub-lished by the AIP press and are now informally known as NIPS Volume 0.

Since 1987, some changes and refinements have been made. but the basic aescrur ofthe conference has remained the same. The NIPS 1988 proceedings (NIPS Volume i. editedby David Touresmzy of Carnegie Mellon) were the first published by Morgan Kauftann.Also in 1988, the post-conference workshops were moved to Keystone, CO. The refinementprocess (three reviewers instead of two), a more cross-disciplinary grouping of presenta.

Zvi

aons, finer presentation catagories, and the addition of five-minute oral poster spo"lightprescntations. A major and very successful addition to the 1991 conference was the introduc-tion of a day of tutorials preeding the main conference. The 1991 workshops were held atVail, which proved to ba a popular move.

Finally, 1991 marked the draffing of artides of incorporation for the Neural Informa.tion Processing Systems Foundation, which will be responsible for the continuity of theNIPS conference in future years. The initial board of directors of the foundation consists ofthe 1987 to 1992 NIPS General Chairs (Ed Posner of Cal Tech, Terry Sejnowski of the SalkInstitute and UCSD. Scott Kirkpatrick of IBM, Richard Lippmann of MIT Lincoln Labs.John Moody of Yale, and Stephen Hanson of Siemens), a member of the IEEE InformationTheory Society (Terry Fine of Cornell), and our legal counsel (Philip Sotel).

The NIPS conference continues to be an exciting, successful meeting due to the effortsof a large group of people. We would first like to thank all the other members of the 1991program and organizing committees who helped make this conference possible, In particular,we would like to thank Renate Crowley of Siemens for her extensive work throughout theyear as the conference secretary and both Renate and Kate Fuqua of CU Boulder for runningthe conference desk so smoothly. Student contributions are an important part of the NIPSprogram, and we gratefully thank Tom McKenna of ONR and Steve Suddarth of AFOSRfor the student travel funding provided by their agencies. Finally, we thank everyone whoattended and submitted papers and the 105 referees who carefully read and reviewed 20papers each.

John Mmody

S&phn HkoanRjckanrd Lippmann

rKi"

NIPS-91 ORGANIZING COMMITTEE

General Chair John Moody, Yale UniversityProgram Chair Stephen Hanson, SiemensWorkshop CoChair Gerry Tesauro, IBM

Scott Kirkpatrick, IBMPublicity Chair John Pearson, David Sarnoff Research LabPublications Chair Richard Lippmann, MIT Lincoln LaboratoryTreasurer Bob Allen, BellcoreGovernmenEiCorporate Liaison Lee Giles, NEC Research InstituteLocal Arrangements Chair Mike Mozer, University of ColoradoIEEE Liaisons Rodney Goodman and Ed Posner, CaltechTutorials Chair John Moody, Yale UniversityAPS Liaisons Eric Baum, NEC Research Institute

Larry Jackel. AT&T Bell LabsNeurobiology Liaisons James Bower, Caltech

Tom Brown, Yale UniversityIEEE Liaisons Rodney Goodman and Ed Posner, CaltechOverseas Liaison (Japan) Mitsuo Kawato, ATR Research LaboratoriesOverseas Liaison (Australia) Maran Jabri, University of SydneyOverseas Liaison (United Kingdom) John Bridle. RSREOverseas Liaison (South America) Andreas Meier, Simon Bolivar University

NIPS-91 PROGRAM COMMITTEE

Program Chair Stephen Hanson, SiemensProgram CoChairs David Ackley, Bellcore

Pierre Baldi, JPL and CaltehWilliam Bialek, NEC Research InstituteLee Giles, NEC Research InstituteSteve Omohundro, ICSIMike Jordan, MITJohn Platt, SynapticsTerry Sejnowski, Salk InstituteDavid Stork. Ricoh and StanfordAlex Waibel, CMU

NIPS-91 Reviewers

Martin Brady, Penn. State Univ. Andrew Bato, Univ. of MassachusettsGary Cotrell, UCSD Judy Franklin, GTEJohn Denker, AT&T Lab Vijaykumar Gullipalli, Univ. of Massachu-John Hertz, NORDITA sertsJames Keeler, Advanced Computing Tech- Thomas Martinez, Beckman Institute

noiogy Kumpati Narendra. Yale Univ.Alan Lapedes, Los Alamos Richard Sutton, GTEDavid Rogers. RIACS Manod Tenorio, Purdue Univ.David G. Stork, Ricoh Lyle Ungar, Univ. of PennsylvaniaPeter Todd. Stanford Univ. Josh Alspector, BellcoreAndreas Weigend, Stanford Univ. Jim Burr. Stanford Univ.Sue Becker, Univ. of Toronto Federico Faggin, SynapticsDavid A. Cohn, Univ. of Washington Hans Peter GraL AT&TMichael Littman, Bellcore Kristina Johnson, Univ. of ColoradoDavid Plaut, CMU John Lazzaro, Univ. of ColoradoLori Pratt, Rutgers Univ. Dick Lyon, Apple ComputerJIirgen Schmidhuber, Technische Univ. Dan Schwartz, GTE

Muenchen Larry Abbott, Brandeis Univ.Rich Zemel, Univ. of Toronto Thomas Anastasio, Univ. of SouthernPaul Munro, Siemens CaliforniaBen Yuhas, Bellcore Joseph Artick, PrincetonSteve Gallant, Northeastern Univ. William Bialek, Univ. of California atMarwen Jabri, Sydney, Univ. BerkeleyGary M. Kuhn, CCRP-IDA A.B. Bonds, Vanderbilt Univ.Y.C. Lee, Univ. of Maryland James Bower, California Inst. of Tech.Ken Marko, Ford Motor Company Thomas Brown, Yale Univ.Michiel Noordewier, Rutgers Univ. Jack Cowan, Univ. of ChicagoKevin Lang, NEC Shawn Lockery. Salk InstituteSteven C. Suddarth, AFOSR Bartlett Mel, California Inst. of Tech.Robert Allen, Bellcore Kenneth Miller, California Inst. of Tech.Jonathan Bachrach Eric Schwartz. New York Univ. MedicalGary Dell, Beckman Institute CenterMichael G. Dyer, UCLA Terry Sejnowski, Salk InstituteJeff Elman, UCSD Gordon Shepherd, New HavenMichael Gasser, UCSD David van Essen, California Inst. of Tech.Lee Giles, NEC Frank Fallside. Univ. of CambridgeRobert Jacobs, MIT Herve Bourlard. PhillipsStephen Hanson. Siemens Patrick Haffner, Centre NationalChristopher Atkeson, MIT d'Erudes des Telecommunications

xim

Richard U.ppmann, MIT Lincoln Lab Pierre Baldi, California Inst. of Tech.Hong Leung, MIT Amir Atiya. Texas A&M Univ.John S. Bridle, RSRE Santosh S. Venkaresh, Univ. of Pennsyl-David Burr, Beilcore vaniaRichard Duubin, ICS1 Halbert W h ite. UCSDDavid Haussler, Univ. of California at Stephen Judd, Siemens

Santa Cruz Tom Petsche, SiemensTerry Sanger, MIT Subatai Ahirad, ICSIGerry Tesauro, IBM Watson Lab Joachim Buhmann. Univ. of SouthernEric Mjolsness, Yale University CalfforniaRonny Meir, Belcore Gene Gindi, Yale Univ.Sara Solla, Santa Fe Institute Geoff -inton, Univ. of TorontoFernando Pineda, Applied Physics Lab Nathan Intrator, Brown Univ.Steve Omohundro, ICS1

si