Adaptive Analog VLSI and Neural Nets Thesis

  • View
    215

  • Download
    0

Embed Size (px)

Text of Adaptive Analog VLSI and Neural Nets Thesis

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    1/127

    Adaptive Analog VLSI Signal Processing and Neural

    Networks

    A ThesisPresented to

    The Academic Faculty

    by

    Jeff Dugger

    In Partial Fulfillmentof the Requirements for the Degree

    Doctor of Philosophy

    School of Electrical and Computer EngineeringGeorgia Institute of Technology

    November 2003

    Copyright c 2003 by Jeff Dugger

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    2/127

    Adaptive Analog VLSI Signal Processing and Neural

    Networks

    Approved by:

    Professor Paul Hasler, Advisor

    Professor David Anderson

    Professor Mark Clements

    Professor Steve DeWeerth

    Professor Dieter Jaeger

    (Emory University)

    Date Approved 24 November 2003

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    3/127

    To my father, Don Dugger, and to my mother, Shirley Dugger.

    iii

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    4/127

    PREFACE

    While the digital world frantically pursues ever-faster clock speeds to satisfy demanding sig-

    nal processing applications, a quiet revolution in analog computing has been brewing, which

    promises to do more with less more sophisticated signal processing delivered at less power

    in a smaller space. Novel application of a digital memory technology, the floating-gate MOS-

    FET (used in EEPROMs), as an analog memory and computation device provides the basic

    building block of this technology. Utilization of inherent device physics provides the adap-

    tivity and programmability needed to realize compact reconfigurable analog VLSI systems.

    Floating-gate charge storage provides non-volatile memory for a matrix of coefficients, while

    the nonlinear current-voltage relation of the MOSFET provides signal-coefficient multipli-

    cation. Summation of products is achieved simply using Kirckhoffs Current Law. Matrix

    coefficients adapt according to a correlation learning rule which utilizes physical device phe-

    nomena (electron tunneling and hot-electron injection) to program floating-gate charge. All

    of this functionality costs only four transistors per coefficient, each operating at nanowatts

    of power consumption. The resultant adaptive analog matrix-vector operations form the

    core of a novel analog VLSI signal-processing model, which is called computing in memory.

    Peripheral circuitry determines learning behavior, controls programmability, and expands

    core matrix functionality.

    iv

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    5/127

    ACKNOWLEDGEMENTS

    I wish to thank my colleagues in the Integrated Computational Electronics lab for their

    encouragement and support, particularly Venkatesh Srinivasan for assistance with the design

    and construction of the adaptive test board, as well as producing some of the simulation

    results in Chapter 6.

    v

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    6/127

    TABLE OF CONTENTS

    DEDICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

    PREFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv

    ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

    LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

    LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

    SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix

    I INTRODUCTION TO ADAPTIVE ELECTRONIC SYSTEMS . . . . 1

    1.1 Basic Neural Network Theory . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.1 Feedforward Computation . . . . . . . . . . . . . . . . . . . . . . . 3

    1.1.2 Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

    1.2 VLSI Implementations of Neural Networks . . . . . . . . . . . . . . . . . . 9

    1.2.1 Neuron Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

    1.2.2 Synapse Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

    1.2.3 Analog vs. Digital Circuits . . . . . . . . . . . . . . . . . . . . . . . 13

    1.2.4 Memories for VLSI Neural Networks . . . . . . . . . . . . . . . . . 14

    1.2.5 Floating-Gate Technology for VLSI Neural Memories . . . . . . . . 15

    II FLOATING-GATE SYNAPSE FUNDAMENTALS . . . . . . . . . . . . 17

    2.1 Floating-Gate Transistor Basics . . . . . . . . . . . . . . . . . . . . . . . . 18

    2.2 Charge Adaptation through Electron Tunneling and Hot-Electron Injection 20

    2.2.1 Electron Tunneling . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

    2.2.2 Hot-Electron Injection . . . . . . . . . . . . . . . . . . . . . . . . . 22

    2.3 Investigating Signal-Adaptive Behavior in Floating-Gate Circuits . . . . . 232.3.1 Separation of Timescales . . . . . . . . . . . . . . . . . . . . . . . . 24

    2.3.2 Fast-Timescale Behavior . . . . . . . . . . . . . . . . . . . . . . . . 25

    2.3.3 Slow Timescale Behavior . . . . . . . . . . . . . . . . . . . . . . . . 26

    2.3.4 The Autozeroing Floating-Gate Amplifier . . . . . . . . . . . . . . 28

    2.3.5 Source-Degenerated pFET Devices: Modeling and Behavior . . . . 30

    2.4 Concluding Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

    vi

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    7/127

    III THE FLOATING-GATE PFET CORRELATION LEARNING RULE 36

    3.1 Weight Update for the Continuously Adapting pFET Synapse . . . . . . . 36

    3.2 Effects of Drain Voltage on the Equilibrium Weight . . . . . . . . . . . . . 37

    3.3 Effects of Gate Voltage on the Equilibrium Weight . . . . . . . . . . . . . 393.4 Equilibrium Weight is Determined by Correlations Between Gate and Drain

    Voltages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

    3.5 Hebbian Learning Rule from Approximated Weight Dynamics . . . . . . . 43

    3.6 Concluding Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

    IV FROM FLOATING-GATE SYNAPSES TO FLOATING-GATE NODES 46

    4.1 A Simple Two-Input Floating-Gate Node . . . . . . . . . . . . . . . . . . . 47

    4.1.1 Simple Floating-Gate Learning Node Experiments . . . . . . . . . . 50

    4.1.2 Learning a Square Wave from Sinusoidal Inputs . . . . . . . . . . . 51

    4.2 Removal of Non-Ideal Effects in the Correlation Learning Rule . . . . . . . 53

    4.2.1 Harmonic Distortion, Gate Variance, and Gate Pre-Distortion . . . 54

    4.2.2 Drain Variance Effects, Drain Pre-Distortion, and Signal Correlations 57

    4.2.3 Cancelling Constant Offset and Gate Variance in the Weight . . . . 59

    4.3 Concluding Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

    V GENERAL LEAST-MEAN-SQUARES LEARNING IN FLOATING-GATE

    NODES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

    5.1 Issue of Weight Decay for Physical LMS computation structures . . . . . . 62

    5.2 The LMS Floating-Gate Synapse Circuit . . . . . . . . . . . . . . . . . . . 66

    5.2.1 Feedforward Synapse Computation . . . . . . . . . . . . . . . . . . 67

    5.2.2 Weight Adaptation and the Floating-Gate Correlation Learning Rule 70

    5.2.3 Amplitude Correlation Experiments . . . . . . . . . . . . . . . . . . 73

    5.3 The Least-Mean-Square (LMS) Learning Rule . . . . . . . . . . . . . . . . 76

    5.4 The Two-Input Adaptive Node . . . . . . . . . . . . . . . . . . . . . . . . 77

    5.5 Concluding Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

    VI FROM SIMPLE NODES TO ADAPTIVE NETWORKS . . . . . . . . 82

    6.1 The n-Input Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

    6.2 A Multiple-Input/Multiple-Node Adaptive Floating-Gate Integrated Circuit 84

    6.2.1 Hardware Test and Development System . . . . . . . . . . . . . . . 85

    vii

  • 8/9/2019 Adaptive Analog VLSI and Neural Nets Thesis

    8/127

    6.3 All-Transistor Synapse Circuit Model and Simulation . . . . . . . . . . . . 87

    6.4 Fourier Series Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

    6.5 Example Application: Adaptive Channel Equalization . . . . . . . . . . . . 93

    6.6 Concluding Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

    VII CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

    APPENDIX A OJAS RULE DERIVATION . . .