· Taylor 4 of 249 09/28/10 ABSTRACT This research develops and investigates an algorithm for the self-organized development of a classification network. This idea for this classific

Systems Research Institute Polish Academy of Sciences

Victor B. Taylor, MSc

Ph.D. Thesis

HeBIS: A Biologically Inspired Data Classification System

Supervisor: Prof. Dr. hab. Inż. Janusz Kacprzyk Afiliacja

Systems Research Institute

Polish Academy of Sciences

Warsaw, September 2010

Taylor 2 of 249

09/28/10

Taylor 3 of 249

09/28/10

Taylor 4 of 249

09/28/10

ABSTRACT

This research develops and investigates an algorithm for the self-organized development of a

classification network. This idea for this classification network, known as HeBIS (Heterogeneous

Biologically Inspired System), is based on a heterogeneous mixture of intelligent yet simple

processing units (cells) that can potentially consist of several types of machine learning constructs. A

list of these constructs can include self-organizing feature maps (SOFM), artificial neural networks

(ANN), and support vector machines (SVM) that communicate with each other via the diffusion of

artificial proteins. The context for the self-organization of the network and the communication

between the processing cells is that of an artificial genetic regulatory network (GRN). An evolved

GRN based on an artificial chemistry of simulated proteins is used as the controller. Artificial genes

within each processing cell are essentially excitatory and inhibitory switches that control the

concentration and diffusion of artificial proteins throughout a simulated environmental lattice. This

GRN controls both the growth of the classification network and the specific behaviors of the

individual processing cells. These controls also use artificial chemistry analogs of problem

descriptors such as second-order statistics.

Self-organization and evolution of the network occur on several levels: the high-level topology of the

network as well as parameters and behaviors that affect the internal organization of each processing

cell. The artificial proteins used for communications and the transfer of regulatory information

between and within the processing elements are also evolved as are the environmental proteins used

to represent the input feature vector for the set of training and test exemplars. An evolutionary

process incorporating particle swarm optimization is used to define an artificial genome that defines

these elements as well as the classification information that the network presents to the user.

Behaviors such as input/output signal conditioning, machine learning processing, and

environmental/regulatory communications are evolved and are used as the genome’s input to the

high-level evolutionary process.

The HeBIS algorithm architecture is discussed in detail and extensions for future research are

proposed. Performance of a classification network based on this novel technique with a single type of

cellular machine learning element, a SOFM, is examined. This performance is compared with that of

a baseline standalone SOFM. For this case study, the problem considered is how well HeBIS learns

the empirical algorithm for cloud/no-cloud pixel detection that is used by the National Aeronautics

Taylor 5 of 249

09/28/10

and Space Administration’s (NASA) for its multispectral optical datasets acquired from the Moderate

Resolution Imaging Spectroradiometer (MODIS) sensor on the earth-orbiting Aqua satellite.

Taylor 6 of 249

09/28/10

ACKNOWLEDGMENTS

Taylor 7 of 249

09/28/10

DECLARATION

Taylor 8 of 249

09/28/10

Table of Contents

ABSTRACT .......................................................................................................................................................................... 4

ACKNOWLEDGMENTS .................................................................................................................................................... 6

DECLARATION .................................................................................................................................................................. 7

TABLE OF CONTENTS ..................................................................................................................................................... 8

FIGURES ............................................................................................................................................................................ 11

TABLES .............................................................................................................................................................................. 16

TERMS AND ACRONYMS .............................................................................................................................................. 18

1. INTRODUCTION ...................................................................................................................................................... 19

1.1. PROBLEM STATEMENT ........................................................................................................................................... 20 1.2. DELIMITATIONS OF THE RESEARCH........................................................................................................................ 21 1.3. KEY CONTRIBUTIONS ............................................................................................................................................ 21 1.4. ORGANIZATION OF THIS THESIS ............................................................................................................................. 21

2. LITERATURE REVIEW .......................................................................................................................................... 24

2.1. MACHINE LEARNING AND SELF-ORGANIZATION .................................................................................................... 25 2.1.1. Classification overview ................................................................................................................................. 25 2.1.2. Artificial neural networks ............................................................................................................................. 28 2.1.3. Self-organizing feature maps ........................................................................................................................ 34

2.2. EVOLUTIONARY COMPUTATION............................................................................................................................. 40 2.2.1. Particle swarm optimization ......................................................................................................................... 40

2.3. BIOLOGICAL AND ARTIFICIAL EVOLUTIONARY DEVELOPMENT .............................................................................. 43 2.4. SUMMARY ............................................................................................................................................................. 51

3. HETEROGENEOUS BIOLOGICALLY INSPIRED SYSTEM (HEBIS) ............................................................ 54

3.1. OVERVIEW ............................................................................................................................................................ 54 3.2. FUNDAMENTALS .................................................................................................................................................... 55

3.2.1. Processing cell .............................................................................................................................................. 55 3.2.2. Environment.................................................................................................................................................. 57

3.2.2.1. 1-D environment ........................................................................................................................................................ 58 3.1.1.1. 2-D environment ........................................................................................................................................................ 58 3.1.1.2. 3-D environment ........................................................................................................................................................ 59

3.1.2. Genetic regulatory network .......................................................................................................................... 59 3.1.2.1. Gene coding ............................................................................................................................................................... 60 3.1.2.2. Protein communications ............................................................................................................................................. 64

3.1.3. Basic cell processing .................................................................................................................................... 66 3.1.3.1. Cell genome ............................................................................................................................................................... 66 3.1.3.2. Intrinsic behaviors ...................................................................................................................................................... 67

3.1.3.2.1. NumberProteinsInCell ......................................................................................................................................... 67 3.1.3.2.2. NumberProteinsInLocalEnviro ............................................................................................................................ 67 3.1.3.2.3. ConcentrationStandardDeviationLocalEnviro ..................................................................................................... 67 3.1.3.2.4. ConcentrationMeanLocalEnviro .......................................................................................................................... 68 3.1.3.2.5. ConcentrationMaxLocalEnviro ............................................................................................................................ 68 3.1.3.2.6. ConcentrationMinLocalEnviro ............................................................................................................................ 68 3.1.3.2.7. KillSelf ................................................................................................................................................................ 68 3.1.3.2.8. NumberFeatures ................................................................................................................................................... 68

3.1.3.3. Learned behaviors ...................................................................................................................................................... 69 3.1.3.3.1. AddCell................................................................................................................................................................ 69 3.1.3.3.2. PruneSelf ............................................................................................................................................................. 69 3.1.3.3.3. ChangeToSOFMAndTrain .................................................................................................................................. 69 3.1.3.3.4. Classify ................................................................................................................................................................ 70

3.1.3.4. Cell types ................................................................................................................................................................... 70 3.1.3.4.1. SOFM .................................................................................................................................................................. 70 3.1.3.4.2. Pass-Thru ............................................................................................................................................................. 70

3.2. INPUT FEATURE VECTOR REPRESENTATIONS.......................................................................................................... 71

Taylor 9 of 249

09/28/10

3.2.1. Direct feature-to-protein mapping ................................................................................................................ 71 3.3. PATTERN TRAINING FOR CLASSIFICATION .............................................................................................................. 72

3.3.1. Self-organization principles .......................................................................................................................... 73 3.3.1.1. Protein analogs of statistical features ......................................................................................................................... 74 3.3.1.2. Cellular fission and death ........................................................................................................................................... 74

3.3.2. Particle swarm optimization ......................................................................................................................... 75 3.3.3. Training algorithm ....................................................................................................................................... 75

3.3.3.1. Training Algorithm: Presentation of training vectors and classes to the system ........................................................ 75 3.4. OUTPUT CODING .................................................................................................................................................... 77 3.5. POST PROCESSING OF CLASSIFICATION RESULTS .................................................................................................... 77

4. SIMULATIONS AND ANALYSES .......................................................................................................................... 78

4.1. SIMULATION LIMITS .............................................................................................................................................. 78 4.2. GENERAL METHODOLOGY ..................................................................................................................................... 80

4.2.1. Remote sensing cloud/no-cloud problem ...................................................................................................... 80 4.2.1.1. Description ................................................................................................................................................................. 80 4.2.1.2. Sensor and datasets .................................................................................................................................................... 80

4.3. THE CONSTRUCTION OF SIMPLE GENETIC REGULATORY NETWORKS ...................................................................... 95 4.3.1. Introduction/Methodology ............................................................................................................................ 95 4.3.2. Experiments 1 and 2 – Proteins .................................................................................................................... 96

4.3.2.1. Setup .......................................................................................................................................................................... 96 4.3.2.2. Experiments 1 and 2 results and discussion ............................................................................................................... 97 4.3.2.3. Experiments 1 and 2 conclusions ............................................................................................................................. 100

4.3.3. Experiments 3 and 4 - Protein Chemistry ................................................................................................... 101 4.3.3.1. Setup ........................................................................................................................................................................ 101 4.3.3.2. Experiments 3 and 4 results and discussion ............................................................................................................. 102 4.3.3.3. Experiments 3 and 4 conclusions ............................................................................................................................. 113

4.3.4. Experiment 5 - Gene activation .................................................................................................................. 114 4.3.4.1. Setup ........................................................................................................................................................................ 114 4.3.4.2. Experiment 5 results and discussion ......................................................................................................................... 115 4.3.4.3. Experiment 5 conclusions ........................................................................................................................................ 127

4.4. SELF-ORGANIZATION IN THE HEBIS ENVIRONMENT ............................................................................................ 128 4.4.1. Introduction/Methodology .......................................................................................................................... 128 4.4.2. Fitness function description ........................................................................................................................ 129 4.4.3. Experiment 6 - Swarm fitness characterization .......................................................................................... 132

4.4.3.1. Setup ........................................................................................................................................................................ 132 4.4.3.2. Experiment 6 results and discussion ......................................................................................................................... 132 4.4.3.3. Experiment 6 conclusions ........................................................................................................................................ 135

4.4.4. Experiment 7 - Initial location of processing cells ..................................................................................... 136 4.4.4.1. Setup ........................................................................................................................................................................ 136 4.4.4.2. Experiment 7 results and discussion ......................................................................................................................... 136 4.4.4.3. Experiment 7 conclusions ........................................................................................................................................ 137

4.4.5. Experiment 8 - Cellular actions .................................................................................................................. 138 4.4.5.1. Setup ........................................................................................................................................................................ 138 4.4.5.2. Experiment 8 results and discussion ......................................................................................................................... 139 4.4.5.3. Experiment 8 conclusions ........................................................................................................................................ 141

4.4.6. Experiment 9 - Protein statistical analogs.................................................................................................. 141 4.4.6.1. Setup ........................................................................................................................................................................ 141 4.4.6.2. Experiment 9 results and discussion ......................................................................................................................... 142 4.4.6.3. Experiment 9 conclusions ........................................................................................................................................ 143

4.4.7. Experiment 10 - Output protein comparison .............................................................................................. 143 4.4.7.1. Setup ........................................................................................................................................................................ 144 4.4.7.2. Experiment 10 results and discussion ....................................................................................................................... 144 4.4.7.3. Experiment 10 conclusions ...................................................................................................................................... 145

4.5. CLASSIFICATION ACCURACY ............................................................................................................................... 146 4.5.1. Introduction ................................................................................................................................................ 146 4.5.2. Training algorithm parameters description ................................................................................................ 146 4.5.3. Fully-engaged HeBIS.................................................................................................................................. 147

4.5.3.1. Experiment 11 - Size of geographic processing environment .................................................................................. 147 4.5.3.1.1. Setup .................................................................................................................................................................. 148 4.5.3.1.2. Experiment 11 results and discussion ................................................................................................................ 149 4.5.3.1.3. Experiment 11 conclusions ................................................................................................................................ 153

4.5.3.2. Experiment 12 - Size of intracellular SOFM kernel ................................................................................................. 153 4.5.3.2.1. Setup .................................................................................................................................................................. 153

Taylor 10 of 249

09/28/10

4.5.3.2.2. Experiment 12 results and discussion ................................................................................................................ 154 4.5.3.2.3. Experiment 12 conclusions ................................................................................................................................ 160

4.5.3.3. Experiment 13 - Protein chemistry reaction probability ........................................................................................... 161 4.5.3.3.1. Setup .................................................................................................................................................................. 161 4.5.3.3.2. Experiment 13 results and discussion ................................................................................................................ 162 4.5.3.3.3. Experiment 13 conclusions ................................................................................................................................ 167

4.5.3.4. Experiment 14 - Shotgun .......................................................................................................................................... 168 4.5.3.4.1. Setup .................................................................................................................................................................. 168 4.5.3.4.2. Experiment 14 results and discussion ................................................................................................................ 171 4.5.3.4.3. Experiment 14 conclusions ................................................................................................................................ 194

4.6. CLASSIFICATION ROBUSTNESS ............................................................................................................................. 196 4.6.1. Introduction/Methodology .......................................................................................................................... 196 4.6.2. Experiment 15 - Noise ................................................................................................................................ 196

4.6.2.1. Setup ........................................................................................................................................................................ 196 4.6.2.2. Experiment 15 results and discussion ....................................................................................................................... 198 4.6.2.3. Experiment 15 conclusions ...................................................................................................................................... 199

4.6.3. Experiment 16 - Missing features ............................................................................................................... 200 4.6.3.1. Setup ........................................................................................................................................................................ 200 4.6.3.2. Experiment 16 results and discussion ....................................................................................................................... 200 4.6.3.3. Experiment 16 conclusions ...................................................................................................................................... 203

5. SUMMARY AND CONCLUSIONS ....................................................................................................................... 204

6. FUTURE RESEARCH ............................................................................................................................................ 209

6.1. FITNESS FUNCTIONS ............................................................................................................................................ 209 6.2. ADDITIONAL MACHINE LEARNING KERNELS ........................................................................................................ 209 6.3. MODULARITY AND LEARNED FUNCTIONALITY .................................................................................................... 209 6.4. ADDITIONAL CELLULAR ACTIONS ........................................................................................................................ 210

6.4.1. Scale-free networks ..................................................................................................................................... 211 6.4.2. Mutual information ..................................................................................................................................... 211 6.4.3. Cellular instantiation .................................................................................................................................. 212

6.5. GRAPHICS PROCESSING UNIT .............................................................................................................................. 213 6.6. PROTEIN-BASED COMMUNICATIONS FOR ARTIFICIAL DEVICES IN A BIOLOGICAL SYSTEM .................................... 214

7. APPENDICES .......................................................................................................................................................... 215

7.1. DATA ................................................................................................................................................................... 216 7.2. GENOME MAPPINGS FOR SHOTGUN DATA ............................................................................................................ 234 7.3. PROTEIN DIFFUSION EXAMPLE ............................................................................................................................. 235 7.4. HEBIS FITNESS FUNCTION DETAILS ..................................................................................................................... 238 7.5. HEBIS SHOTGUN EXPERIMENT CORRELATION MAPS AND PDF .......................................................................... 240 7.6. HEBIS TRAINING CYCLE DETAILS ........................................................................................................................ 241

BIBLIOGRAPHY ............................................................................................................................................................ 242

Taylor 11 of 249

09/28/10

Figures

FIGURE 1. PERFORMANCE OF EMPIRICAL LEARNING SYSTEMS. ............................................................................................. 26 FIGURE 2. AN ARTIFICIAL NEURON. ...................................................................................................................................... 29 FIGURE 3. ARTIFICIAL FEEDFORWARD NEURAL NETWORK. ................................................................................................. 29 FIGURE 4. SELF-ORGANIZING FEATURE MAP. ...................................................................................................................... 36 FIGURE 5. SELF-ORGANIZING FEATURE MAP TOPOLOGY PRESERVATION EXAMPLE [61]. ................................................... 37 FIGURE 6. PARTICLE SWARM OPTIMIZATION ALGORITHM [90]. ............................................................................................ 42 FIGURE 7. ENVIRONMENTAL LATTICE AND PROCESSING CELL OVERVIEW. ........................................................................... 55 FIGURE 8. MAJOR BLOCKS OF CELL FUNCTIONALITY. ........................................................................................................... 56 FIGURE 9. LINEAR GRID NUMBERING SCHEME. ..................................................................................................................... 58 FIGURE 10. PLANAR GRID NUMBERING SCHEME. .................................................................................................................. 59 FIGURE 11. THREE-DIMENSIONAL LATTICE NUMBERING SCHEME. ........................................................................................ 59 FIGURE 12. GENOME/GENE PROTEIN HIERARCHY. ................................................................................................................ 61 FIGURE 13. STANDARD REGULATORY/ENVIRONMENTAL AND SWITCH PROTEIN DESCRIPTIONS. ........................................... 61 FIGURE 14. DIRECT FEATURE-TO-PROTEIN MAPPING. ........................................................................................................... 72 FIGURE 15. SCHEMATIC OF THE TRAINING ALGORITHM. ....................................................................................................... 76 FIGURE 16. TRAINING/TESTING PIXEL AND ITS RELATIONSHIP TO ITS SURROUNDING GEOGRAPHIC PIXELS. ......................... 78 FIGURE 17. TRAINING/TESTING PIXEL AND THE SURROUNDING MULTISPECTRAL INFORMATION. ......................................... 79 FIGURE 18. PSEUDOCOLOR IMAGE FOR A2002193183000 DATASET. GREY AND WHITE COLORS CORRESPOND TO CLOUD

PIXELS, BLACK CORRESPONDS TO WATER, AND GREEN AND BROWN REFER TO LAND PIXELS. ........................................ 84 FIGURE 19. GROUND TRUTH (CLOUD/NO-CLOUD) FOR A2002193183000 DATASET. RED CORRESPONDS TO LAND PIXELS

WHEREAS WHITE PIXELS REFERENCE CLOUDS AND BLACK CORRESPONDS TO WATER. ................................................... 85 FIGURE 20. LAND MASK (CLOUD/NO-CLOUD) FOR A2002193183000 DATASET. RED CORRESPONDS TO LAND PIXELS AND

BLACK CORRESPONDS TO WATER PIXELS. ...................................................................................................................... 86 FIGURE 21. CLOUD/NO-CLOUD CLASS BREAKDOWN ACCORDING TO SPECIFIC WAVELENGTH-BAND FEATURE. ..................... 87 FIGURE 22. 412 NM CLOUD/NO-CLOUD SCATTER PLOT WITH CLOUD (C0) PIXELS REPRESENTED AS 1 AND NO-CLOUD (C0)

PIXELS REPRESENTED AS -1 ON THE ABSCISSA. MAGNITUDES ARE LOG-NORMALIZED AND BIASED AND SCALED TO FALL

WITHIN THE RANGE [0, 1]. ............................................................................................................................................. 88 FIGURE 23. 443 NM CLOUD/NO-CLOUD SCATTER PLOT WITH CLOUD (C0) PIXELS REPRESENTED AS 1 AND NO-CLOUD (C0)


















WITHIN THE RANGE [0, 1]. ............................................................................................................................................. 91

Taylor 12 of 249

09/28/10

FIGURE 32. 748 NM CLOUD/NO-CLOUD SCATTER PLOT WITH CLOUD (C0) PIXELS REPRESENTED AS 1 AND NO-CLOUD (C0)










WITHIN THE RANGE [0, 1]. ............................................................................................................................................. 94 FIGURE 37. BASELINE NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR ZERO-LENGTH GENOME IN EXPERIMENT 1.

VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. .................................................. 98 FIGURE 38. BASELINE NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR THE 3-GENE GENOME IN EXPERIMENT 2.

VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. .................................................. 98 FIGURE 39. NUMBER OF PROTEINS IN ENVIRONMENT COMPARED BETWEEN THE BASELINE GENOME FROM EXPERIMENT 1

AND THE MULTI-GENE GENOME FROM EXPERIMENT 2. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF

THE SAMPLE MEAN. ..................................................................................................................................................... 100 FIGURE 40. NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR A 0-GENE GENOME VS. CELLULAR ITERATION FOR

REACTION PROBABILITIES OF 0 %, 0.1 %, 1 %, AND 10 % WITH ERROR BARS REMOVED FOR CLARITY. ....................... 103 FIGURE 41. NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR A 3-GENE GENOME VS. CELLULAR ITERATION FOR

REACTION PROBABILITIES OF 0 %, 0.1 %, 1 %, AND 10 % WITH ERROR BARS REMOVED FOR CLARITY. ....................... 104 FIGURE 42. NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR A 0-GENE GENOME VS. CELLULAR ITERATION FOR A

REACTION PROBABILITY OF 0.1 %. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE

MEAN........................................................................................................................................................................... 105 FIGURE 43. NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR A 0-GENE GENOME VS. CELLULAR ITERATION FOR A



REACTION PROBABILITY OF 10 %. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. .................................................................................................................................................................................... 106

FIGURE 45. NUMBER OF PROTEINS IN ENVIRONMENTAL LATTICE FOR A 3-GENE GENOME VS. CELLULAR ITERATION FOR A





REACTION PROBABILITY OF 10 %. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. .................................................................................................................................................................................... 107

FIGURE 48. FITTING AND STATISTICAL INFORMATION FOR A 0-GENE GENOME IN AN ENVIRONMENT WITH A REACTION

PROBABILITY OF 0.1 %. VERTICAL BARS ON THE UPPER CHART CORRESPOND TO THE STANDARD DEVIATION OF THE

SAMPLE MEAN. ............................................................................................................................................................ 109 FIGURE 49. FITTING AND STATISTICAL INFORMATION FOR A 0-GENE GENOME IN AN ENVIRONMENT WITH A REACTION



PROBABILITY OF 10%. VERTICAL BARS ON THE UPPER CHART CORRESPOND TO THE STANDARD DEVIATION OF THE





SAMPLE MEAN. ............................................................................................................................................................ 112

Taylor 13 of 249

09/28/10

FIGURE 53. FITTING AND STATISTICAL INFORMATION FOR A 3-GENE GENOME IN AN ENVIRONMENT WITH A REACTION

PROBABILITY OF 10 %. VERTICAL BARS ON THE UPPER CHART CORRESPOND TO THE STANDARD DEVIATION OF THE

SAMPLE MEAN. ............................................................................................................................................................ 112 FIGURE 54. COMPARISON BETWEEN 0-GENE AND 3-GENE GENOMES FOR VARYING ENVIRONMENTAL REACTION

PROBABILITIES. ERROR BARS REMOVED FOR CLARITY. ............................................................................................... 113 FIGURE 55. EXAMPLE GENE ACTIVATION MAP. ................................................................................................................... 116 FIGURE 56. 3-GENE GENOME ACTIVATION VS. ITERATION FOR C0 (178) USING GENOMES 93 AND 146 IN (A) AND (B),

RESPECTIVELY. ............................................................................................................................................................ 117 FIGURE 57. 3-GENE GENOME ACTIVATION VS. ITERATION FOR C0 (142) USING GENOMES 9 AND 147, RESPECTIVELY, IN (A)

AND (B). ...................................................................................................................................................................... 117 FIGURE 58. 3-GENE GENOME ACTIVATION VS. CELLULAR ITERATION FOR C0 (48) USING GENOME 178. ............................. 118 FIGURE 59. 10-GENE GENOME ACTIVATION VS. CELLULAR ITERATION FOR ORIGINAL AND TWO CLONED CELLS WITHIN THE

ENVIRONMENTAL MATRIX- C0 (42) TEST PIXEL FOR GENOME 117. THE ORIGINAL GENOME IS PRESENTED IN (A) WHILE

THE TWO CLONED GENOMES ARE PRESENTED IN (B) AND (C). ...................................................................................... 119 FIGURE 60. 10-GENE GENOME ACTIVATION VS. CELLULAR ITERATION FOR ORIGINAL AND TWO CLONED CELLS WITHIN THE

ENVIRONMENTAL MATRIX- C0 (50) TEST PIXEL USING GENOME 117. THE ACTIVATION FOR THE ORIGINAL CELL IS

SHOWN IN (A) AND THE ACTIVATIONS FOR THE CLONED CELLS ARE SHOWN IN (B) AND (C). ........................................ 120 FIGURE 61. 10-GENE GENOME ACTIVATION VS. CELLULAR ITERATION FOR ORIGINAL AND TWO CLONED CELLS WITHIN THE

ENVIRONMENTAL MATRIX- C1 (51) TEST PIXEL USING GENOME 117. THE ORIGINAL CELL ACTIVATION IS PRESENTED IN

(A) AND THE ACTIVATIONS FOR THE CLONED CELLS ARE LISTED IN (B) AND (C). ......................................................... 120 FIGURE 62. 10-GENE GENOME ACTIVATION VS. CELLULAR ITERATION FOR THE SAME TEST PIXEL C0 (42). GENOMES 91,

123, AND 135 ARE DISPLAYED IN (A), (B), AND (C), RESPECTIVELY. GENOMES 123 AND 135 SHOW ORIGINAL AND TWO

CLONED CELL ACTIVATIONS FOR EACH OF THESE GENOMES. ....................................................................................... 122 FIGURE 63. 40-GENE GENOME ACTIVATION FOR GENOME 30 FOR TEST PIXELS C0 (26) IN THE TOP IMAGE (A) AND C1 (25) IN

THE BOTTOM IMAGE (B). THE GENOME SHOWS MULTI-GENE ACTIVATION FOR C0 AND SINGLE GENE ACTIVATION FOR

C1 WITH DIFFERING RESPONSES. ................................................................................................................................. 123 FIGURE 64. 40-GENE GENOME ACTIVATION FOR GENOME 66. THE TOP (A) AND MIDDLE(B) IMAGES ARE THE MULTI-GENE

ACTIVATION PROFILES FOR C0 (30) WITH AN ORIGINAL AND CLONED CELL. THE BOTTOM (C) IMAGE SHOW THE

ACTIVATION FOR THE SAME GENOME, BUT FOR A C1 (31). .......................................................................................... 124 FIGURE 65. 40-GENE GENOME ACTIVATION OF GENOME 90 FOR TEST PIXELS C0 (30), C1 (31), C0 (32), C0 (34), C0 (54),

AND C1 (55), RESPECTIVELY FROM TOP TO BOTTOM, IN FIGURES (A) – (F). .................................................................. 126 FIGURE 66. DECISION REGION MAPPING AND BOUNDARY BASED ON THE VALUE OF Corr

C0 max. ....................................... 130

FIGURE 67. AVERAGE BEST GENOME FITNESS VS. BREED # FOR 1-PARTICLE PSO SWARM. VERTICAL BARS CORRESPOND TO

THE STANDARD DEVIATION OF THE SAMPLE MEAN. ..................................................................................................... 133 FIGURE 68. AVERAGE BEST GENOME FITNESS VS. BREED # FOR 100-PARTICLE PSO SWARM. VERTICAL BARS CORRESPOND

TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. ................................................................................................ 133 FIGURE 69. AVERAGE BEST GENOME FITNESS VS. BREED # FOR 250-PARTICLE PSO SWARM. VERTICAL BARS CORRESPOND

TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. ................................................................................................ 134 FIGURE 70. AVERAGE BEST GENOME FITNESS VS. BREED # FOR 500-PARTICLE PSO SWARM. VERTICAL BARS CORRESPOND

TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. ................................................................................................ 134 FIGURE 71. BEST AVERAGE PEAK GENOME FITNESS VS. THE NUMBER OF PARTICLES IN THE PSO SWARM. VERTICAL BARS

CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN OF THE PEAK FITNESS FOR EACH SWARM TESTED. .................................................................................................................................................................................... 135

FIGURE 72. CV AVERAGE FITNESS VS. INITIAL CELL LOCATION FROM BEST BRED GENOME. VERTICAL BARS CORRESPOND

TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. ................................................................................................ 137 FIGURE 73. CV AVERAGE FITNESS VS. ACTIVATED CELLULAR ACTION. VERTICAL BARS CORRESPOND TO THE STANDARD

DEVIATION OF THE SAMPLE MEAN. .............................................................................................................................. 139 FIGURE 74. CV AVERAGE FITNESS VS. ACTIVITY LEVEL OF CELLULAR PROTEIN STATISTICS. VERTICAL BARS CORRESPOND

TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. ................................................................................................ 142 FIGURE 75. CV AVERAGE FITNESS VS. STATIC OR PSO-EVOLVED SETTING OF THE OUTPUT C0/C1 PROTEIN. VERTICAL BARS

CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. .......................................................................... 145 FIGURE 76. MULTI-SPECTRAL DATA CUBE FOR A 5X5 GEOGRAPHIC REGION WITH 15 BANDS OF MULTISPECTRAL DATA. ... 149 FIGURE 77. FULL-IMAGE AVERAGE CLASSIFICATION ACCURACY VS. SIZE OF GEOGRAPHIC REGIONS SURROUNDING TEST

PIXEL. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. ..................................... 150 FIGURE 78. FULL-IMAGE CLASSIFICATION ACCURACY VS. FITNESS FOR A 3X3 GEOGRAPHIC REGION SURROUNDING TEXT

PIXEL. .......................................................................................................................................................................... 151 FIGURE 79. FULL-IMAGE CLASSIFICATION ACCURACY VS. FITNESS FOR A 5X5 GEOGRAPHIC REGION SURROUNDING TEXT

PIXEL. .......................................................................................................................................................................... 152 FIGURE 80. CLASSIFICATION ACCURACY VS. FITNESS FOR CASE WITH 0 NEURONS IN INTRACELLULAR SOFM. .................. 155 FIGURE 81. CLASSIFICATION ACCURACY VS. FITNESS FOR CASE WITH 1 NEURON IN INTRACELLULAR SOFM. ................... 156

Taylor 14 of 249

09/28/10

FIGURE 82. CLASSIFICATION ACCURACY VS. FITNESS FOR CASE WITH 4 NEURONS IN INTRACELLULAR SOFM. ................. 157 FIGURE 83. CLASSIFICATION ACCURACY VS. FITNESS FOR CASE WITH 9 NEURONS IN INTRACELLULAR SOFM. ................. 159 FIGURE 84. CLASSIFICATION ACCURACY VS. FITNESS FOR CASE WITH 81 NEURONS IN INTRACELLULAR SOFM. ............... 160 FIGURE 85. FULL-IMAGE AVERAGE CLASSIFICATION ACCURACY VS. THE NUMBER OF NEURONS IN THE INTRACELLULAR

SOFM. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN OF THE CLASSIFICATION

ACCURACY. ................................................................................................................................................................. 160 FIGURE 86. CLASSIFICATION ACCURACY VS. FITNESS WITH 0.0 REACTION PROBABILITY. .................................................. 163 FIGURE 87. CLASSIFICATION ACCURACY VS. FITNESS WITH 0.001 REACTION PROBABILITY. .............................................. 164 FIGURE 88. CLASSIFICATION ACCURACY VS. FITNESS WITH 0.01 REACTION PROBABILITY. ................................................ 165 FIGURE 89. CLASSIFICATION ACCURACY VS. FITNESS WITH 0.1 REACTION PROBABILITY. .................................................. 166 FIGURE 90. FULL-IMAGE CLASSIFICATION ACCURACY VS. THE PROBABILITY OF PROTEIN REACTION IN THE

ENVIRONMENTAL LATTICE. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN. .... 167 FIGURE 91. SAMPLE ROC CURVE WITH FALSE-POSITIVE RATE ALONG THE ABSCISSA AND TRUE-POSITIVE RATE AS THE

ORDINATE. ................................................................................................................................................................... 170 FIGURE 92. HEBIS CLASSIFICATION IMAGERY AND ROC FOR 2010_05_28_07_58_58_61 TEST. THIS IS HEBIS SELECTED

TRIAL # 0. .................................................................................................................................................................... 175 FIGURE 93. HEBIS CLASSIFICATION IMAGERY AND ROC FOR 2010_06_11_01_18_20_77 TEST. THIS IS HEBIS SELECTED










TRIAL # 10. .................................................................................................................................................................. 185 FIGURE 103. COMPARISON PLOT OF HEBIS CLASSIFICATION ACCURACY VS. FITNESS OF THE GENOME FOR 200 TRIALS.

PROTEIN CHEMISTRY IS DEACTIVATED. ....................................................................................................................... 189 FIGURE 104. COMPARISON PLOT OF HEBIS CLASSIFICATION ACCURACY VS. FITNESS OF THE GENOME FOR 79 TRIALS WITH

PROTEIN CHEMISTRY ACTIVATED. ............................................................................................................................... 189 FIGURE 105. HEBIS CLASSIFICATION ACCURACY VS. REACTION PROBABILITY FOR 79 TRIALS. PROTEIN CHEMISTRY IS

ACTIVATED. ................................................................................................................................................................. 190 FIGURE 106. HEBIS CLASSIFICATION ACCURACY VS. REACTION PROBABILITY AND FITNESS FOR 79 TRIALS. PROTEIN

CHEMISTRY IS ACTIVATED. .......................................................................................................................................... 191 FIGURE 107 HEBIS CLASSIFICATION ACCURACY VS. REACTION PROBABILITY AND MINIMUM PROTEIN CORRELATION FOR 79

TRIALS. PROTEIN CHEMISTRY IS ACTIVATED. .............................................................................................................. 192 FIGURE 108. CLASSIFICATION ACCURACY VS. ENVIRONMENTAL DIFFUSION RATE FOR 200 SHOTGUN TRIALS. PROTEIN

CHEMISTRY IS DEACTIVATED. ...................................................................................................................................... 193 FIGURE 109. CLASSIFICATION ACCURACY VS. GENOME FITNESS AND ENVIRONMENTAL DIFFUSION RATE FOR 200 SHOTGUN

TRIALS. PROTEIN CHEMISTRY IS DEACTIVATED. ......................................................................................................... 193 FIGURE 110. NOISE COMPARISON PLOT FOR CLASSIFICATION ACCURACY VS. NOISE STANDARD DEVIATION FOR BOTH HEBIS

AND SOFM TRIALS. VERTICAL BARS CORRESPOND TO THE STANDARD DEVIATION OF THE SAMPLE MEAN OF

CLASSIFICATION ACCURACY. ....................................................................................................................................... 199 FIGURE 111. COMPARISON OF BEFORE AND AFTER CLASSIFICATION ACCURACY FOR THE MODIS BAND 16 KNOCKOUT. .. 201 FIGURE 112. COMPARISON OF BEFORE AND AFTER CLASSIFICATION ACCURACY FOR THE MODIS BAND 7 KNOCKOUT. .... 202 FIGURE 113. ONE METHOD OF PRESENTING TRAINING/TEST DATA TO HEBIS. EACH BEHAVIOR IS TRAINED SEPARATELY,

CANDIDATE GENOMES ARE CREATED, AND THE CANDIDATES THEN UNDERGO EVOLUTIONARY OPTIMIZATION IN A FINAL

CV/GA LOOP. ............................................................................................................................................................. 210 FIGURE 114. PROCESSING MODEL FOR RESEARCH INFRASTRUCTURE. ................................................................................ 214

Taylor 15 of 249

09/28/10

FIGURE 115. PARAMETER ACTIVATION MAPS FOR GENOMES DISCOVERED DURING SHOTGUN EXPERIMENTS. THE COLOR BAR

FROM TABLE 81 IS APPLICABLE TO THE EVOLVED ELEMENTS OF THE GENOMES IN THIS FIGURE. ................................. 234 FIGURE 116. SINGLE PROTEIN DIFFUSION FROM FOUR SITES WITHIN AN 11X11X11 CUBIC ENVIRONMENT. THIS FRAME

SHOWS FOUR INITIAL SITES OF PROTEIN ACTIVATION AT THE BEGINNING OF THE SIMULATION .................................... 235 FIGURE 117. FRAME #2 IN THE SIMULATION THIS FRAME IS A SNAPSHOT OF ACTIVITY IN THE ENVIRONMENTAL LATTICE

AFTER ITERATION 2 OF THE DIFFUSION SIMULATION. THE TWO RED PROTEIN SITES ARE STILL ACTIVELY PRODUCING

PROTEINS WHEREAS THE TWO GREEN SITES ARE DECAYING. THE LIGHT BLUE COLOR REPRESENTS SITES WITHIN THE

LATTICE THAT HAVE THE LOWEST NON-ZERO PROTEIN CONCENTRATIONS AT THIS POINT IN THE SIMULATION. .......... 235 FIGURE 118. FRAME #6. THE SITES COLORED RED ARE STILL ACTIVELY PRODUCING WHEREAS THE LIGHT-BLUE-COLORED

AND DARK-BLUE-COLORED SITES POSSESS LOWER CONCENTRATIONS OF THE SIMULATED PROTEIN. THE DARKER BLUE

SITES CONTAIN LOWER CONCENTRATIONS OF PROTEIN THAN THE LIGHT-BLUE SITES. THIS IS THE SNAPSHOT FROM

ITERATION 6 OF THE SIMULATION. ............................................................................................................................... 236 FIGURE 119. FRAME #26. AFTER 26 ITERATIONS, THE ARTIFICIAL PROTEIN HAS DIFFUSED THROUGHOUT A LARGE PORTION

OF THE 11X11X11 ENVIRONMENTAL MATRIX. HOTTER COLORS (E.G. RED, YELLOW, GREEN) CORRESPOND TO HIGHER

CONCENTRATIONS OF THE PROTEINS WHEREAS COOLER COLORS (E.G. LIGHT BLUE, BLUE) CORRESPOND TO AREAS OF

RELATIVELY LOW CONCENTRATIONS. .......................................................................................................................... 236 FIGURE 120. FRAME #40. AT ITERATION 40, THE PROTEIN HAS DIFFUSED THROUGHOUT THE ENVIRONMENTAL LATTICE.

THE RED SITES ARE THE LOCATIONS OF THE ORIGINAL AND CONTINUING PROTEIN SOURCES. HOTTER COLORS

CORRESPOND TO HIGHER PROTEIN CONCENTRATIONS WHEREAS COOLER COLORS CORRESPOND TO LOWER

CONCENTRATIONS. ...................................................................................................................................................... 237 FIGURE 121. REGIONS OF EQUIVALENT Corr

C0 max FOR THE FITNESS FUNCTION. .............................................................. 238

FIGURE 122. Θcorr

2 PORTION OF FITNESS FUNCTION. ........................................................................................................ 238

FIGURE 123. Magcorr

2 PORTION OF FITNESS FUNCTION. ...................................................................................................... 239

FIGURE 124. CORRELATION COEFFICIENT GRID FOR PROCESSING PARAMETERS AND CLASSIFICATION RESULTS. PARAMETERS AND RESULTS ARE NUMBERED FROM 1 TO 26. ....................................................................................... 240

FIGURE 125. SIGNIFICANCE P-VALUE GRID FOR PROCESSING PARAMETERS AND CLASSIFICATION RESULTS OBTAINED WITH A

STUDENT’S T-TEST. PARAMETERS AND RESULTS ARE NUMBERED FROM 1 TO 26. ....................................................... 240 FIGURE 126. HEBIS CLASSIFICATION TRAINING CYCLE. ..................................................................................................... 241

Taylor 16 of 249

09/28/10

Tables

TABLE 1. BITWISE XOR FUNCTIONAL MAPPING ................................................................................................................... 65 TABLE 2. ORBITAL INFORMATION FOR NASA'S AQUA SATELLITE. ...................................................................................... 80 TABLE 3. NOMINAL RESOLUTIONS FOR THE MODIS SENSOR. ............................................................................................. 81 TABLE 4. 36 BANDS OF MULTISPECTRAL DATA FROM MODIS. ........................................................................................... 81 TABLE 5. NASA/CEOS DATASET LEVEL DEFINITION ......................................................................................................... 82 TABLE 6. 17 BANDS FROM MODIS FOR A2002193183000 LAC_X_NIR. ........................................................................... 82 TABLE 7. MULTISPECTRAL BANDS USED FROM MODIS [111]............................................................................................... 83 TABLE 8. SIMULATION PARAMETERS FOR EXPERIMENT 1. ................................................................................................... 96 TABLE 9. SIMULATION PARAMETERS FOR EXPERIMENT 2. .................................................................................................... 97 TABLE 10. STATISTICAL SUMMARY FOR EXPERIMENTS 1 AND 2. ......................................................................................... 99 TABLE 11. SIMULATION PARAMETERS FOR EXPERIMENT 3. ............................................................................................... 101 TABLE 12. SIMULATION PARAMETERS FOR EXPERIMENT 4. ............................................................................................... 102 TABLE 13. STATISTICAL SUMMARY FOR EXPERIMENT 3. ................................................................................................... 108 TABLE 14. STATISTICAL SUMMARY FOR EXPERIMENT 4. ................................................................................................... 108 TABLE 15. SIMULATION PARAMETERS FOR EXPERIMENT 5. ............................................................................................... 115 TABLE 16. SIMULATION PARAMETERS FOR SELF-ORGANIZATION EXPERIMENTS............................................................... 129 TABLE 17. EXPERIMENT 6 TRIAL DISTRIBUTION ................................................................................................................ 132 TABLE 18. TRIAL DISTRIBUTION FOR EXPERIMENT 7. ......................................................................................................... 136 TABLE 19. TRIAL DISTRIBUTION FOR EXPERIMENT 8 ......................................................................................................... 139 TABLE 20. EXPERIMENT 9 PARAMETERS ............................................................................................................................ 142 TABLE 21. EXPERIMENT 10 PARAMETERS .......................................................................................................................... 144 TABLE 22. RANGE OF PERTINENT HEBIS TRAINING PARAMETERS FOR CLASSIFICATION .................................................. 147 TABLE 23. PERTINENT SOFM TRAINING PARAMETERS FOR CLASSIFICATION.................................................................... 147 TABLE 24. TRIAL DISTRIBUTION ACROSS GEOGRAPHIC REGION SIZE FOR EXPERIMENT 11. .............................................. 148 TABLE 25. SIMULATION PARAMETERS FOR EXPERIMENT 11. ............................................................................................. 148 TABLE 26. CONFUSION MATRIX FOR 3X3 GEOGRAPHIC REGION ........................................................................................ 150 TABLE 27. CONFUSION MATRIX FOR 5X5 GEOGRAPHIC REGION ........................................................................................ 152 TABLE 28. HEBIS KERNEL SIZES FOR THE INTRACELLULAR SOFM IN EXPERIMENT 12. ................................................... 153 TABLE 29. SIMULATION PARAMETERS FOR EXPERIMENT 12. ............................................................................................. 154 TABLE 30. CONFUSION MATRIX FOR INTRACELLULAR SOFM WITH 0 NEURONS ............................................................... 154 TABLE 31. CONFUSION MATRIX FOR INTRACELLULAR SOFM WITH 1 NEURON ................................................................. 156 TABLE 32. CONFUSION MATRIX FOR INTRACELLULAR SOFM WITH 4 NEURONS ............................................................... 157 TABLE 33. CONFUSION MATRIX FOR INTRACELLULAR SOFM WITH 9 NEURONS ............................................................... 158 TABLE 34. CONFUSION MATRIX FOR INTRACELLULAR SOFM WITH 81 NEURONS ............................................................. 159 TABLE 35. PROTEIN REACTION PROBABILITY DISTRIBUTION FOR EXPERIMENT 13. ........................................................... 161 TABLE 36. SIMULATION PARAMETERS FOR EXPERIMENT 13. ............................................................................................. 162 TABLE 37. CONFUSION MATRIX FOR 0.0 PROTEIN REACTION PROBABILITY ...................................................................... 162 TABLE 38. CONFUSION MATRIX FOR 0.001 PROTEIN REACTION PROBABILITY .................................................................. 163 TABLE 39. CONFUSION MATRIX FOR 0.01 PROTEIN REACTION PROBABILITY .................................................................... 165 TABLE 40. CONFUSION MATRIX FOR 0.1 PROTEIN REACTION PROBABILITY ...................................................................... 166 TABLE 41. LIST OF HEBIS PARAMETERS TO RANDOMIZE FOR EXPERIMENT 14. ............................................................... 169 TABLE 42. LIST OF SOFM PARAMETERS TO RANDOMIZE FOR EXPERIMENT 14. ................................................................ 169 TABLE 43. DISTRIBUTION OF CLASS AND INFRASTRUCTURE PIXELS IN A2002193183000................................................. 171 TABLE 44. OPERATIONAL PARAMETERS FOR SELECTED HEBIS SHOTGUN EXPERIMENTS ................................................. 172 TABLE 45. OPERATIONAL PARAMETERS FOR SOFM EXPERIMENTS ................................................................................... 172 TABLE 46. CLASSIFICATION RESULTS FOR SELECTED HEBIS SHOTGUN EXPERIMENTS ..................................................... 173 TABLE 47. CLASSIFICATION RESULTS FOR SELECTED SOFM EXPERIMENTS ...................................................................... 173 TABLE 48. FEATURE AND RESULT INDICES FOR CORRELATION COEFFICIENT AND P-VALUE MATRICES ........................... 187 TABLE 49. BEST CLASSIFICATION ACCURACIES FOR THE SELECTED HEBIS AND SOFM EXAMPLES ................................. 195 TABLE 50. DATASET DEFINITIONS FOR EXPERIMENT 15. .................................................................................................... 197 TABLE 51. SIMULATION PARAMETERS FOR THE HEBIS "BEST" GENOME FOR EXPERIMENT 15. ....................................... 197 TABLE 52. SIMULATION PARAMETERS FOR THE 2X1 SOFM "BEST" CODEBOOK FOR EXPERIMENT 15.............................. 197 TABLE 53. SIMULATION PARAMETERS FOR THE 3X1 SOFM "BEST" CODEBOOK FOR EXPERIMENT 15.............................. 197 TABLE 54. MISSING FEATURE COMPARISON FOR CLASSIFICATION ACCURACY USING HEBIS AND SOFM ALGORITHMS

WITH MODIS BAND 16 KNOCKOUT ............................................................................................................................ 201

Taylor 17 of 249

09/28/10

TABLE 55. MISSING FEATURE COMPARISON FOR CLASSIFICATION ACCURACY USING HEBIS AND SOFM ALGORITHMS

WITH MODIS BAND 7 KNOCKOUT .............................................................................................................................. 202 TABLE 56. EXPERIMENT 6 AGGREGATE BREEDING DATA. ................................................................................................. 216 TABLE 57. EXPERIMENT 7 DATA. ....................................................................................................................................... 216 TABLE 58. EXPERIMENT 8 DATA. ....................................................................................................................................... 217 TABLE 59. EXPERIMENT 9 DATA ........................................................................................................................................ 217 TABLE 60. EXPERIMENT 10 DATA. ..................................................................................................................................... 217 TABLE 61. EXPERIMENT 11 DATA FOR 3X3 AND 5X5 GEOGRAPHIC REGION COMPARISON ................................................ 217 TABLE 62. EXPERIMENT 11 SCATTER DATA FOR 3X3 GEOGRAPHIC REGION ...................................................................... 218 TABLE 63. EXPERIMENT 11 SCATTER DATA FOR 5X5 GEOGRAPHIC REGION ...................................................................... 219 TABLE 64. EXPERIMENT 12 - INTRACELLULAR SOFM DATA ............................................................................................. 219 TABLE 65. EXPERIMENT 12 SCATTER DATA FOR 0X0 INTRACELLULAR SOFM .................................................................. 220 TABLE 66. EXPERIMENT 12 SCATTER DATA FOR 1X1 INTRACELLULAR SOFM .................................................................. 220 TABLE 67. EXPERIMENT 12 SCATTER DATA FOR 2X2 INTRACELLULAR SOFM .................................................................. 221 TABLE 68. EXPERIMENT 12 SCATTER DATA FOR 3X3 INTRACELLULAR SOFM .................................................................. 221 TABLE 69. EXPERIMENT 12 SCATTER DATA FOR 9X9 INTRACELLULAR SOFM .................................................................. 222 TABLE 70. EXPERIMENT 13 AGGREGATE CLASSIFICATION DATA ...................................................................................... 222 TABLE 71. EXPERIMENT 13 DATA FOR 0.0 REACTION PROBABILITY .................................................................................. 223 TABLE 72. EXPERIMENT 13 DATA FOR 0.001 REACTION PROBABILITY .............................................................................. 223 TABLE 73. EXPERIMENT 13 DATA FOR 0.01 REACTION PROBABILITY ................................................................................ 224 TABLE 74. EXPERIMENT 13 DATA FOR 0.1 REACTION PROBABILITY .................................................................................. 224 TABLE 75. EXPERIMENT 14 - STATISTICAL SUMMARY DATA FOR HEBIS CLASSIFICATION ACCURACY AND FITNESS

SCATTER DATA ........................................................................................................................................................... 225 TABLE 76. EXPERIMENT 14 HEBIS SCATTER DATA FOR FITNESS AND CLASSIFICATION ACCURACY ................................. 225 TABLE 77. EXPERIMENT 14 - AGGREGATE CLASSIFICATION ACCURACY RESULTS FOR SOFM. ......................................... 230 TABLE 78. SELECTED RESULTS FROM EXPERIMENT 14 ...................................................................................................... 231 TABLE 79. DATA FOR EXPERIMENT 15- CLASSIFICATION ACCURACY FOR 0.1 PROBABILITY NOISE INJECTION WITH

VARYING NOISE STANDARD DEVIATIONS ................................................................................................................... 232 TABLE 80. DATA FOR EXPERIMENT 16 – CLASSIFICATION ACCURACY WITH MODIS BAND 15 KNOCKED OUT ................ 232 TABLE 81. COMPARISON OF SELECTED GENOMES FOR SHOTGUN EXPERIMENTS ............................................................... 233

Taylor 18 of 249

09/28/10

Terms and Acronyms ANN: Artificial Neural Network

CDF: Cumulative Distribution Function CEOS: Committee on Earth Observation Satellites

CV: Cross Validation

EC: Evolutionary Computation ED: Evolutionary Development ESA: European Space Agency Evodevo: Evolutionary Development GA: Genetic Algorithm

Genetic Algorithm: GRN: Genetic Regulatory Network

ML: Machine Learning MODIS: Moderate Resolution Imaging Spectroradiometer NASA: National Aeronautics and Space Administration

PDF: Probability Density Function PSO: Particle Swarm Optimization

ROC: Receiver Operating Characteristic curve

SOFM: Self Organizing Feature Map SVM: Support Vector Machine SI: Swarm Intelligence TOA: Top of Atmosphere TBD: To Be Determined

Taylor 19 of 249

09/28/10

1. Introduction

The need currently exists on a worldwide basis for the timely extraction of knowledge for the

management of natural resources. Extraction of this knowledge for local, regional, and global

applications is being driven by the desire to more precisely assess the issues associated with both

anthropogenic and natural drivers in the environment. Data are being collected from a multitude of

earth-orbiting sensor platforms from the European Space Agency (ESA), the National Aeronautics

and Space Administration (NASA) and a host of other national agencies as well as private concerns.

This torrent of raw data, for example, NASA’s Moderate Resolution Imaging Spectroradiometer

(MODIS) sensors routinely collect a terabyte of earth imagery on a daily basis, brings with it the

issue of extraction of usable knowledge from these information-rich datasets for efficient

management of the earth’s natural resources.

State-of-the-art on-orbit optical sensors routinely collect earth resource data that are multispectral or

hyperspectral in nature. This increase in the number of available bands of information promises to

provide better discrimination of desired classes only if an appropriate level of class precision is

available [1].

Remote sensing classification problems are difficult to solve because of many issues. A sampling of

these pitfalls includes spectral and spatial noise in the geographic areas of interest, loss of usable data

due to cloud cover, noisy multi-temporal datasets, and noisy human and machine-generated class

labels. A multitude of complex regional characteristics around the globe, e.g., the particulates in

different coastal regions, terrain effects, and open ocean effects and other disturbances also affect the

analytical chain associated with knowledge acquisition and analysis techniques. These problems

contribute to the difficulty of solving computational classification problems in the area of satellite-

based optical remote sensing.

This research examines a method through which classification knowledge for potential use in remote

sensing applications may be acquired from multi-spectral datasets that are routinely used by

researchers and government policy leaders. This research is based on contemporary machine learning

techniques that have been combined to develop a novel system that is based on recent thought in

biological evolutionary development. In particular, it is based on the observation that biological

evolution has provided life as we know it with successful means of navigating the data-rich pitfalls

and rewards associated with day-to-day survival in a harsh environment. It is hoped that the

application of the ideas associated with biologically complex structures beyond the typical realm of

Taylor 20 of 249

09/28/10

neural networks can provide novel means for processing human-produced, information-rich

constructions in the future.

The idea behind this dissertation is the research and development of an algorithm that creates a self-

organizing classification network. This classification network can be based on a mixture of

heterogeneous machine learning constructs such as self-organizing feature maps (SOFM), artificial

neural networks (ANN), and support vector machines (SVM) [2]. The biological context for self-

organization and communications between the simple processing constructs, or cells, in the network

is that of genetic regulatory networks and evolutionary development [3,4,5]. Each processing cell

consists of an artificial genome that has excitatory and inhibitory switches that are controlled through

the communication of artificial proteins in a simulated environmental lattice in which the cells reside.

Within each cell, protein switches control the expression of particular proteins from each processing

cell. These proteins diffuse through the lattice and in turn are used for communications between the

processing elements. This communication is based on an artificial protein chemistry with

concentration levels.

Self-organization and evolution of the network occur on several levels. The high-level topology of

the network- both the number and types of the simple processing cells (SOFM, ANN, or SVM)- can

change as can the internal organization of each processing element. At this lower processing-element

level, this could include parameters such as the choice of kernel used for a particular SVM element.

Artificial proteins used for communications between processing elements as well as between the

“outside” environment (the input data patterns) and the classification topology also adapt to the

application domain. These communications and environmental proteins are released into the

classification lattice if their corresponding genes are switched on and expressed.

1.1. Problem statement

Claim: Biological inspiration is a powerful paradigm in classification and hybridization introduces

interesting and useful qualities. It can provide powerful tools for solving relevant satellite image

classification problems.

We will investigate in detail as to whether the combination of an artificial genetic regulatory network

(GRN) and a basic machine learning element in a rudimentary self-organizing network is effective

when applied to binary classification in a multi-dimensional space; i.e. using multispectral feature

Taylor 21 of 249

09/28/10

vectors acquired from an optical satellite image. This detailed case study will examine how our novel

hybridization idea works and also how to set it up for a class of practical classification problems.

1.2. Delimitations of the research

• Simple GRNs will be constructed o Simplified artificial protein representation o Protein interaction through a simple protein “chemistry” o Size and complexity are limited by the available computational resources

• Self-organization o Facilitated by evolutionary computation; i.e. particle swarm optimization; and a small

set of self-organizing rules inspired by biological evolutionary development and statistical analysis

o Rudimentary classification training algorithm is based on repeated presentation of samples to the classification network in the PSO framework.

• Accuracy for a real-world binary cloud/no-cloud classification problem will be addressed

• Robustness for benchmark problems will be addressed in the areas of datasets with noisy features and datasets that have missing features

• A single binary classification using remotely sensed multispectral optical data will be examined

• Comparisons are limited to simple implementations of SOFM machine learning kernels with research into SVM and ANN kernels left for future research.

• It is not the purpose of this research to delve deeply into the merits of one SOFM training algorithm versus another one.

1.3. Key contributions

• Novel application of a hybridized biological construct with machine learning to a practical computational classification problem

• Determination of the effectiveness of a simplified GRN applied to multi-dimensional classification.

• Artificial proteins communicate classification information and results to and from the cellular machine learning kernels.

• Training of a GRN via particle swarm optimization (PSO).

• Application of a GRN-based classification system to a real-world multispectral remote sensing problem domain, i.e. cloud detection in optical satellite imagery.

• Performance comparison of HeBIS and a SOFM-only classification algorithm on a remotely-sensed multispectral dataset with unadulterated features, noisy features, and deleted features.

1.4. Organization of this thesis

The thesis is organized as follows:

Taylor 22 of 249

09/28/10

Chapter 1 outlines the objectives and limitations of this research and lists the key contributions of this

dissertation.

Chapter 2 reviews the current state of the art for classification systems based on machine learning

techniques that are embedded in self-organizing structures- as is the case with HeBIS. As such, this

chapter provides direct linkage to the origins of the core ideas used in the HeBIS architecture.

Specifically, the areas examined are Artificial Neural Networks (ANN) and Self-Organizing Feature

Maps (SOFM). ANNs and associated research into their self-organization are presented as important

background in addition to being an introduction to the SOFM theory. Promising new developments

are discussed as well as the problems associated with each of these machine learning paradigms.

Attempts to alleviate these problems through the use of a “bare bones” Genetic Regulatory Network

(GRN) based on an artificial protein chemistry form the basis for the remainder of the dissertation

using the HeBIS self-organizing system. Information on this GRN is presented from a computational

development viewpoint and an overview of the current state-of-the art in this research domain is also

included. This information is presented in the context of the cell-to-cell and intra-cell

communications that are enabled by a computational environment based on artificial proteins. A

literature overview of GRNs applied to classification problems is also presented.

In Chapter 3, a detailed architecture of the Heterogeneous Biologically Inspired System (HeBIS) is

presented. This treatment includes information on the protein environment and lattice structure of the

system; the artificial proteins, their different types and encodings, and the associated simulated

protein chemistry; and the basic processing cells. Also examined are the inherent and learned

behaviors that each cell can acquire through the protein reactions within the simulated environment.

Finally, the classification training algorithm is examined. The underlying particle swarm optimizer

that is used for system optimization is also discussed and sample training and classification

processing data flows are given to solidify the presentation of this material.

In Chapter 4, the simulation results are presented. Through these simulations, HeBIS classifications

are examined and compared to classifications based on SOFMs. Simulation results are also presented

which characterize salient properties of various instantiations of the HeBIS architecture.

Chapter 5 discusses and summarizes the comparison results in detail.

Taylor 23 of 249

09/28/10

Finally, in Chapter 6, the dissertation concludes with an outline of potential avenues of future work in

this research area.

Taylor 24 of 249

09/28/10

2. Literature review Hybridization Background

Algorithm Discussion GRN Analyses

GRN Training with PSO GRN Action Analyses

Remote Sensing Background Remote Sensing Application – Analyses and Comparisons

Robustness Analyses

The list of research areas that this dissertation touches upon is quite extensive. This stems from the

fact that the idea that biology has something to teach engineers and computer scientists has become

more accepted by researchers over the last several years. During the last decade, basic biological

research has become cheaper to perform and its volume has increased. This is now being coupled

with an exponential increase in computing power and information processing. The intersection of

these disciplines has seen a fertile exchange in ideas between the biological researchers and the

“information processors”. This dissertation is itself a result of this exchange and it focuses on the

attempt to apply biological principles to knowledge extraction; specifically the area of automated

multi-class classification. To do this, an overview of work that is directly applicable to this research

is required.

This review is divided into four sections that concentrate on machine learning and evolutionary

development within the context of self-organization and classification. Section 2.1 introduces

machine learning as it is applied in the specific domains of general classification, artificial neural

networks and self-organizing feature maps. Section 2.2 covers evolutionary computation with

emphasis on particle swarm optimization. Section 2.3 presents biological and artificial evolutionary

development and provides an overview of the research and applications associated with pattern

classification and creation. Finally, Section 2.4 summarizes the advantages and limitations of the

outlined techniques and proposes that there may be improvements in pattern classification if ideas

from these different research areas are blended together through HeBIS, the Heterogeneous

Biologically Inspired System.

HeBIS research is based on simple SOFM pattern recognition kernels (cells) with a GRN-based

communications infrastructure wrapped around them. The emphasis in this research is to determine

whether a GRN can be successfully used to create a classification network which can be used as the

basis for further research, not to examine the relative advantages or disadvantages of different

subclasses of this or other simple processing kernels.

Taylor 25 of 249

09/28/10

This review only provides a concise overview of these topics. Appropriate references are included

for further detailed examination.

2.1. Machine learning and self-organization

2.1.1. Classification overview

Machine learning entails methods by which computers may be programmed to learn. The accuracy

and precision of a computerized classification system are the primary attributes of such a system.

Learning tasks can be classified into analytical and empirical techniques. With analytical learning,

no external experiences – data and environmental descriptions- are required whereas empirical

learning explicitly requires the use of external data and experience [6]. This research is primarily

concerned with empirical learning in both supervised and unsupervised learning environments for

classification applications.

Classification is the process through which an object is mapped to a specific class within a set of

classes that has been defined for the problem. In this research, an object and its associated definition

or class is referred to as a labeled example or exemplar. The set of labeled exemplars constitute the

training set of data in which the object is a feature vector of many descriptive numerical features that

is mapped to a specific class label. These training data are applied to a given learning algorithm and

the result is a specific instantiation of a classifier. In turn, this classifier is evaluated for its precision

and accuracy by applying it to a test data set that is composed of a separate set of labeled examples

that have been taken from the same underlying statistical distribution as the training set.

Classifiers should generalize well to datasets that they have not been directly trained on. In other

words, a good classifier is one that, once it has been trained on a small training set, may be used to

effectively classify larger sets of data. Classification rate is the statistic that is the percentage of test

examples which are correctly classified. The misclassification rate is the converse, those test

exemplars that have been misclassified by the classifier [7]. These statistics are further refined in the

cases where successive classification decisions are not independent and when the classification

decisions are not equally important [6]. The latter leads to the development of the Receiver

Operating Characteristic curve, ROC, that is used to gauge classification performance for ranking

classifiers [8]. Through an ROC, the performance of a classifier is examined by changing the

threshold that is used to decide between two classes. As this threshold changes, one can construct an

Taylor 26 of 249

09/28/10

ROC from the false-positive classification rates and the corresponding true-positive classification

rates.

Empirical learning systems, whether supervised or unsupervised, trade performance between three

factors: the complexity of the classifier, the amount of training data, and the generalization ability of

the system when it is applied to new, unseen exemplars. As the classifier’s complexity increases, its

generalization ability increases, peaks, and then decreases. As more training data become available to

the classifier, more detailed information becomes available about the problem’s statistical manifold.

However, as the complexity of the classifier increases, the system’s generalization accuracy will

increase and then decrease after a certain point is reached. These points are noted in Figure 1.

Figure 1. Performance of empirical learning systems.

Taylor 27 of 249

09/28/10

Low variance and high variance classifiers are defined respectively as systems that exhibit a small

degree or a high degree of change in classification performance as different (and noisy) exemplars are

presented and tested [7].

A high bias system is defined as one which exhibits high classification precision on the problem, but

with low recall and a low bias system is one which exhibits low precision with high recall, with

recall =t p

t p + fn

ffffffffffffffffffffff,

(1)

precision =t p

t p + fp

ffffffffffffffffffffff,

(2)

where t p is the true-positive rate, fp is the false-positive classification rate, and f

n is the false-

negative classification rate.

A low bias system essentially can represent almost any classifier whereas a high bias system is not

complex enough to represent the optimal classifier.

Many mechanisms exist through which classifier complexity is matched to the complexity of the

training data [9,10,11].

Unsupervised learning requires no exemplar-class training pairs because the interrelationships

between the examples’ features are automatically categorized and clustered by these types of

algorithms according to a set of rules that is defined before the feature vectors are presented to the

learning system [12, 13].

Typically, machine learning algorithms make weak or no assumptions about the training data.

Therefore, machine learning techniques generally require a large number of training data so that the

problem’s statistical manifold can be adequately sampled. However, if domain knowledge is applied,

the size of the required data set (for a given level of classifier precision) is typically much less than it

Taylor 28 of 249

09/28/10

is in the case in which no knowledge is used. This introduction of bias into the system is risky,

however, since the a priori knowledge must be correct or the added bias may preclude the discovery

of an accurate classifier.

2.1.2. Artificial neural networks

Historically, research into artificial neural networks (ANNs) has been motivated by the differences

between mammalian brains and human-engineered digital computers. Researchers have typically

focused their efforts in two ways. The first is an attempt to better understand how the brain works by

simulating its topology through models of varying complexity. Secondly, researchers have attempted

to mimic the brain’s operation in a quest to improve engineered information processing systems.

It is through this second camp of researchers that the modern variants of artificial neural networks

have been utilized for complex and nonlinear applications such as pattern recognition. The

processing and self-organizing abilities of even the smallest mammalian brain outstrip current

supercomputers given almost any applicable performance metric.

A small sampling of the early work in the field includes [14,15,16,17,18,19].

A generic artificial neural network is composed of a collection of simple processing elements that are

interconnected. This type of architecture is one that is extended in this current work with the HeBIS

network’s processing cells that are interconnected through a GRN. Each of the simple processing

elements in an artificial neural network is called an artificial neuron and is based on a simple

mathematical model of a biological neuron.

Figure 2 shows an artificial neuron that receives a set of numerical inputs, applies a multiplicative

weighting function to each input and then sums these results over all of the weighted inputs to the

neuron.

Taylor 29 of 249

09/28/10

Figure 2. An artificial neuron.

This individual weight is called a synaptic weight and it mimics the excitatory and inhibitory

responses at the input of a biological neuron. In the artificial case, a negative-valued weight acts to

inhibit that input whereas a positive-valued weight excites that input in the artificial neuron. The

summed result of these weights is then nonlinearly mapped through a normalizing activation function

to the neuron’s output. This activation function is typically chosen to be a scaled sigmoid function.

At this point, the output signal (a number) is either passed on as an input to another neuron or is the

output of a layered feedforward network as in Figure 3.

Figure 3. Artificial Feedforward Neural Network.

A feedforward neural network is typically composed of three layers: an input layer, a hidden layer

and an output layer. In its most general form, the neurons in these layers are interconnected within

the layers and also between the layers via synapses that are excitatory or inhibitory. These responses

Taylor 30 of 249

09/28/10

are controlled by the previously mentioned synaptic weights. A properly-sized feedforward network

composed of three layers can theoretically approximate any arbitrary function [20]. Similarly, the

idea is that HeBIS forms a layered network of interconnected processing cells, albeit one that may not

be as apparent as that of an ANN.

ANN architectures are mature and can be differentiated according to the following characteristics:

• Neuron Models,

• Synaptic interconnection (network) models), and

• Training paradigms.

Common artificial neuron models use simplified versions of the actual operating properties inherent

in the biological neuron. For example, biological neurons appear to actually process their inputs and

outputs according to a pulsing signal model. This neuron model has been mostly ignored by the

computational research community but interest has increased recently [21]. Besides the common

sigmoidal activation functions, other nonlinear functions have also been examined [20].

Synaptic interconnection models define how the neurons in the different layers of a network are

connected to each other. Types of architectures include feedforward networks in which the inputs to

the network are processed by the input layer and then “fed forward” to the hidden layer and then the

output layer. Recurrent neural networks are oscillatory and function by feeding information

backwards through the network or directly back to the originating neuron [20,22]. Another model is

the Self-Organizing Feature Map which is described in detail in a separate section [23].

Many paradigms for the training of artificial neural networks’ synaptic weights have been researched.

Backpropagation is a workhorse training technique for the neural network community [24]. Other

neural network training techniques include [25, 26, 27, 28, 29, 30].

Researchers have also applied classical mathematical tools to the training issue. Sequential Monte

Carlo methods are used to train the ANN as each new training example is presented to the network

[31]. Iterative training such as this is useful in instances when the training datasets are large, consist

of thousands (or more) of features, and when the dataset’s statistics are time-varying and/or non-

Gaussian. [31] uses Monte Carlo sampling to characterize the training set’s probability distribution

for such an iterative and time-varying process. This technique, HySIR, was shown in 2000 to have

Taylor 31 of 249

09/28/10

only a linear dependence on the number of sampling trajectories used and thereby has realizable

computational requirements.

Non-traditional training methods are also used to train artificial neural networks- as are also used with

HeBIS. Vieira in 2005 [32] used a genetic algorithm (GA) coupled with elitism to train the Iterative

Artificial Neural Network (IANN). The authors researched the problem of fitting the size of a neural

network to the complexity of the problem at hand- an issue that HeBIS addresses through self-

organization. They proceed on the basis that ANN training based on high-dimensional data is

difficult and that an overly complex network is probably over-fitted to the training data. Network

pruning or dimensionality reduction of the training features are common techniques used to match the

complexities of the system to the problem [20]. The IANN uses an iterative training process on a

layered ANN in which each layer contains the same number of neurons. The number of layers, the

number of neurons in each layer, and the connection weights are determined with a genetic algorithm

that minimizes the quadratic error of the network’s output. Vieira believes that this iterative

algorithm’s advantages are its robustness, ease of implementation, and the ability to use high-

dimensional feature vectors, even when large numbers of features are irrelevant [32].

Evolutionary development (development or evo-devo) is described in detail in Section 2.3. However,

it is briefly mentioned in this section on neural networks because it has been found to be useful when

applied to the construction of a simulated robot’s morphology and the creation of its artificial “brain”

[33,34]. Development facilitates the creation of a compact set of rules that can be used to create

complex structures. The individual rules (roughly equivalent to biological genes) in the derived rule

set are not mapped on a one-to-one basis to the actual realized constructions (morphologies) that are

created by the interpretation of the rules. Four improvements in abilities and topology generation for

artificial neural networks point to the importance of incorporating evolutionary development into the

HeBIS algorithm.

First, large and complex networks can be created through the process of development by using a

small number of genes. Second, evolutionary development appears to be the basis of the gradual

development of biological morphological patterns that are complex and fractal such as the

mammalian brain’s synaptic connections. The term, fractal, in this research describes a pattern or

system in which structures or patterns are replicated on different scales. Applying such biological

design principles into human-engineered systems has been shown to be a valid effort [35, 36].

Artificial grammar systems such as L- systems have been found to be capable of introducing fractal-

Taylor 32 of 249

09/28/10

like morphologies in artificial systems [37, 38, 39]. With grammar-based systems, strings of

characters represent the state of the neural network (i.e. neuron weight, bias, and network topology)

and are changed according to a set of string parsing rules. These parsing rules modify the strings and

in turn, the instantiation of the neural network is changed. Third, because the linkages between

genotype (the set of processing rules) and phenotype (the realized structure: neural network, physical

body, etc.) in a development-based system are not direct but are complex, nonlinear, and indirect,

this allows radical alterations in neurological network architectures to be realized through slight

changes in the artificial genes. Finally, development allows the formation of modular functionality

within the network [40].

These works point to the utility of basing HeBIS on evolutionary development principles, in

particular with respect to ANNs.

Jung researched a topographical method for development-based neural network creation and reported

on it in [41] in 2005. His method has been shown to create artificial neural architectures that mimic

the neuron architecture patterns that are found in the mammalian visual cortex. Importantly, these

ANNs are based on a compact genetic representation that recreates modular fractal neural modules

(patterns) that are present in mammalian brains.

The Self-Optimizing Neural Network (SONN) is an ANN training paradigm through which a neural

network topology is optimized by creating subnetworks of the various training classes in a problem.

These subnetworks are created by statistically determining the important input features and then

combining and mapping these features through separate networks to the output. It is another method

by which a neural network’s complexity (i.e. topology and weights) may be tuned to the complexity

of the data, in this case without explicitly using biological developmental principles [42,43]. HeBIS

also has behaviors that allow for statistical sampling of the classification manifold, but from an

artificial protein perspective. This is discussed in detail in Chapter 3. The SONN is reported to have

no requirements for a prior configuration of training parameters and is directly applicable to different

datasets. The authors indicate that the technique derives minimally complex networks while still

retaining expressive generalization. The study’s first author, Hizyk, does mention that SONN

construction “can be compared to the processes that take place during brain development” [42]. It is

still unclear as to what SONN’s performance is when compared against classical ANN architectures,

but it is interesting nonetheless.

Taylor 33 of 249

09/28/10

Kitano’s Neurogenetic Learning algorithm (NGL) [44] is an extension of his grammar-encoding work

for neural networks that are trained by GA [45]. Through these methods, the creation of a neural

network topology and weight tuning are performed simultaneously. In [44], he showed that an

initialization of the neuronal weights decreases training convergence time. This network initialization

is controlled by a GA and a grammar and is used in the gradient-descent backpropagation algorithm

that trains the neural network.

Neural network classifiers have routinely been applied to problems in remote sensing of the

environment to perform, for example, land cover, ocean phytoplankton, and atmospheric composition

classifications [46, 47, 48, 49, 50, 51]. Recently, Chin implemented the Statistical Self-Organizing

Learning System (SSOLS) [52]. It combines a functional-link neural network [53] with a self-

organized hidden layer that adapts to the training dataset’s statistics. The number of neurons in the

hidden layer is increased according to the results of a series of statistical hypothesis tests performed

on the data as they are presented to the network. A combination of cross-validation and a Student’s

T-test is used to minimize the effects of overfitting and to keep the size of the network to a minimum.

SSOLS is applied to a hyperspectral dataset that consists of 191 optical bands in the 0.4 – 2.4 mµ

portion of the visible and infrared spectrum. The data are classified on the pixel level into seven land

cover classes that include background features, corn, soybean, wheat, alfalfa, pasture, and a sensor-

distortion class. In a comparison of training and testing accuracies between the SSOLS, a radial basis

function (RBF) support vector machine (SVM), a matched filter, and other classification methods, the

SSOLS achieved the highest accuracies in conjunction with the SVM.

Another type of artificial neural network is the functional neural network. This is a flat feedforward

network that has no hidden layer [53]. Each feature of the input vector that is presented to the

network undergoes a linear transformation with scaling and added bias. The transformed feature is

then mapped through a function that is typically a neuronal tanh sigmoid. The mapped features are

then simply scaled and summed to create the output of the network

The HeBIS processing network can also be somewhat described as a functional link network since

each cell encapsulates a specific mapping function.

In summary, artificial neural networks posses the following advantages and disadvantages:

Taylor 34 of 249

09/28/10

Advantages

• Mature training methods exist

• Straightforward iterative training

• Domain specific knowledge not required Disadvantages

• Long training times for datasets with large number of features

• Difficult to tune with respect to the choice of the number of elements in the middle layer

• Difficult to analyze; nonlinear analysis of neural weights

2.1.3. Self-organizing feature maps

As part of self-organizing systems, Self-Organizing Feature Maps (SOFM), also known as Self-

Organizing Maps (SOM), are introduced here separately from ANNs because of their importance to

this work. The HeBIS network can also be thought of as a SOFM, but with individual processing

elements that are more complicated than the simple SOFM neurons. Several of the general ideas in

this section are incorporated into the HeBIS architecture.

SOFMs are biologically inspired processes that construct structural representations of high-

dimensional data. The SOFM described by Kohonen [23] is composed of simple neurons embedded

in a lattice that is typically of much lower dimension than the input data. Metric relationships defined

between these neurons are referred to as weight vectors. These weights are modified as the feature

vectors are presented to the SOFM on a one-by-one basis. Presentation of a multi-dimensional

feature vector to the SOFM consists of mapping the vector to the neuron with the weights that

matches the vector most “closely” according to a defined distance metric; with the Euclidean metric

being commonly used. This matching process is somewhat analogous to vector quantization, but

with the addition of a weight-updating process that is defined over a subset of the surrounding

neurons. This is known as the neighborhood updating function. This neighborhood function can take

many forms [23].

After iterative presentation of data vectors is complete and the SOFM has converged [54], it now

contains a structural representation of the input dataset that maintains a degree of topological parity

with the original data. The weight vectors of the converged SOFM are collectively referred to as the

feature map. Typically, a SOFM is defined in two or three dimensions and is used in a variety of

applications. These include, visualization of the “important” similarity relationships between the

Taylor 35 of 249

09/28/10

input, high-dimensional data vectors, pattern classification, and function approximation [55, 56].

Figure 4 shows a schematic of the input/output relationships and a sample of a map whose neurons

are being used to map a multi-class classification problem for remote sensing purposes. The input

vector is represented as X with the associated weights, wij, describing the links from the input vector

to the output cells in the SOFM N.

More formally, a basic SOFM may be defined as a set of neurons that is arranged in a lattice of

specific shape and dimensionality. The neuron’s position in the network can be denoted by the

vector, s , and the weight vector of this neuron by ws. The set of all ws

for all the neurons in the

network comprise the feature map. A pattern vector taken from the input data space is denoted as p .

The SOFM is trained by taking a feature vector, p , from the input data space and determining which

of the neurons in the lattice is “closest” to the input pattern as

W c@PN

N

N

N

N

N= minS

W s@PN

N

N

N

N

N

(3)

where the weight vector, wc, is associated with the neuron at sc

that is found to be the closest match

as in ( ). The weight vector of the neurons located in a defined neighborhood of this closest neuron

are then updated with the neighborhood updating function. In the original incremental SOM [12],

this neighborhood function is updated at discrete time intervals, t, and is defined as

W n t + 1` a

=W n t0

b c

+ αh C,S n

b c

P t` a

@W n t` a

B C

(4)

where NW marks the weights in the defined neighborhood of neurons, NS , and h() is the relaxation

function that depends on the neurons’ position in the neighborhood of the closest matched neuron.

Taylor

The relaxation function, h(), is a smoothing kernel that is defined over the

neighborhood. Its spatial extent and behavior over time (for convergence h()

decreases as the distance metric increases) define the elasticity of the SOFM and how the mapped

lattice responds to input data vectors [23

as

Initialization of the SOFM is either random or may be chosen in some fashion if one has some

knowledge of the input data manifold [54

Figure 4. Self-Organizing Feature Map.

The relaxation function, h(), is a smoothing kernel that is defined over the

neighborhood. Its spatial extent and behavior over time (for convergence h()


sponds to input data vectors [23]. The learning rate, alpha, is bundled in this h() parameter

h ` a= αh ` a .

M is either random or may be chosen in some fashion if one has some

e of the input data manifold [54].

36 of 249

The relaxation function, h(), is a smoothing kernel that is defined over the neuron’s local

neighborhood. Its spatial extent and behavior over time (for convergence h() Q 0 as t Q 1 and h()


]. The learning rate, alpha, is bundled in this h() parameter

(5)

M is either random or may be chosen in some fashion if one has some

Taylor 37 of 249

09/28/10

Since its publication by Kohonen in 1982, many researchers have contributed to the theoretical

underpinnings of the SOFM. Many changes and enhancements to the basic algorithm have been

suggested [57, 58, 59].

One important feature of the SOFM is its ability to preserve topological relationships when high-

dimensional data are mapped to lower-dimensional or equi-dimensional structures. This preservation

of neighborhood has been studied extensively for regular lattices, but irregular lattices were also

examined [60,61]. In [61], the study’s authors, Neme and Miramontes, created many types or

irregular lattices, including several in which locality was defined as a small-world (1/f) network

where f is the distance between connected neurons in the network [62,63]. These SOFM lattices were

trained and topological preservation metrics were calculated for each. Mutual information between

the networks’ statistics and their degree of topological preservation was analyzed. Interestingly, the

report states that topological preservation was found to be better in non-regular lattices than in the

regular ones [61]. An example of topology preservation as training of the SOFM progresses is

presented in Figure 5.

Figure 5. Self-Organizing Feature Map Topology Preservation Example [61].

It is hoped that HeBIS will incorporate topological features of the data manifold into its self-

organized network of machine learning cells.

Vector quantization (VQ), SOFMs, and Learning Vector Quantization (LVQ) are closely related [23].

In [64], the Dynamic Time Warping algorithm (DTW) is proposed by which sequences of features

Taylor 38 of 249

09/28/10

may be applied to an SOFM for clustering and an LVQ for pattern recognition. DTW attempts to

compensate for and tolerate input sequence differences due to sequence lengths and spatial variation

by normalizing according to these sequence lengths.

Two limitations exist with the SOFM. The first of these limitations is the generic SOFM’s static map

architecture: the number of neurons and the orientation of the lattice are fixed before training begins.

With data that has unknown characteristics, it can be difficult to determine the SOFM’s architecture

that provides good training results. The second limitation is that the generic SOFM has no ability to

arrange the data according to their natural hierarchical relationships. Since the data from many

problem domains are hierarchical, this limitation precludes using SOFMs for satisfactory analyses in

such cases.

The Growing Hierarchical SOM (GHSOM) is one attempt to addresses these limitations [65]. It is an

outgrowth of prior SOFM enhancements [57,66] that attempted to also address these limitations.

GHSOM modulates both the size of the network at each layer and also decides which neuron will be

selected to seed a more detailed hierarchical map. These selections are dependent on data

representation quality measures for SOFMS such as the Mean Quantization Error (MQE) of a neuron

and the total Quantization Error (QE). With one of these measures and a threshold parameter as

another control, the depth of the hierarchy can be roughly set. The choice of the threshold parameter

can still be somewhat complicated.

The GHSOM and the Growing Neural Gas (GNG) [67] algorithms attempt to improve unsupervised

SOFM learning by dynamically adding neurons to the network. In the case of [65], [67] and others,

the network is only resized after a defined number of training iterations have occurred between

updates. Because of this, these networks may not respond well to data distributions that are non-

stationary.

Marsland, Shapiro, and Nehmzow have developed an unsupervised technique by which the size of the

network can change as the data statistics change [68]. Their Grown When Required (GWR)

algorithm responds to non-stationary aspects of the data by adding new neurons, but it stops its

growth when the data’s statistics have become stationary. The GWR was compared with the GNG

and the Restricted Coulomb Energy Classifier (RCE) [69] in Marsland’s study.

Taylor 39 of 249

09/28/10

The Generative Topographic Map (GTM) is an alternative to the SOM [70]. This algorithm, unlike

the SOFM, works by optimizing a specified objective function. Whereas training convergence is not

guaranteed for the SOFM, the authors state that it is guaranteed for the GTM. Also, the GTM

explicitly defines a probability density function (pdf) on the mapped data space, something which the

SOFM does not explicitly accomplish. Similarly, though, the GTM clusters input data, but

accomplishes this task through the modeling of a small number of latent variables.

Yet another extension to the SOFM algorithm is the SOM of SOMS, also known as SOM2 [71].

Whereas a SOFM operates by highlighting the differences and similarities between individual feature

vectors in a dataset, the SOM2 has a different focus. It attempts to examine the similarities and

differences directly between the distributions of classes (i.e. the different manifolds in a multi-

dimensional space) in the input data space.

SOFMs have also been used to cluster and aid in the analysis of temporal sequences of data [72,73].

One of the more recent, the Time Enhanced SOM (TESOM) is described in [74]. It requires the

tuning of many parameters for adequate training.

SOFMs have also been used in remote sensing applications. [75] discusses the application of an

SOFM to the classification of vegetative land cover with a combination of multispectral data from the

Airborne Thematic Mapper (ATM) and multifrequency/multipolarization data from the NASA

Airborne Synthetic Aperture Radar (AIRSAR). [76] compares classification results based on SOFM

and LVQ algorithms that are applied to a LANDSAT Thematic Mapper (TM) dataset and cloud cover

classification is discussed in [77] and [78].

In summary, self-organizing feature maps possess the following advantages and disadvantages:

Advantages

• Inter-feature relationships can be visualized

• Easy to implement

• Not computationally expensive

• Straightforward iterative training

• Simple, not very many training parameters to tune

• Domain specific knowledge is not required

• Relatively easy to analyze

Taylor 40 of 249

09/28/10

Disadvantages

• The size of the network is set before training and is not adaptable as the input statistics change

• Multi-dimensional -> lower-dimensional mapping(e.g. 3 dimension mapped to 2 dimensions) may gloss over critical feature relationships

• Convergence is not guaranteed

• Hierarchical data relationships are not mapped in the generic algorithm

2.2. Evolutionary computation

In the current work, Genetic algorithms (GA) and Particle Swarm Optimization (PSO) are introduced

primarily as tools to facilitate a broad search of potential HeBIS genomes. The evolved genome

denotes the proteins that are used in the HeBIS communications infrastructure, its Genetic Regulatory

Network (GRN). The genome also sets the evolved capabilities of each processing cell. This search

is necessary to provide a fast ability to examine a large set of innovative solutions. Both the GA and

PSO belong to the class of techniques in which populations of individuals are harnessed to find and

optimize problem solutions [79].

With a genetic algorithm, which is one of many optimization strategies in evolutionary computation

(EC), artificial evolution is used to search through the problem space [80]. A population of

chromosomes is mated and mutated using biological evolution as a guidebook. In Swarm

Intelligence (SI), PSO guides a population of potential problem solutions by communicating

information about the search space between members of the population [81]. Both methods have

been the subject of much research in the application and pure research communities in engineering as

well as the biological sciences. HeBIS uses a PSO as the basis of the training algorithm and hence

this will be discussed in further detail. For more details on genetic algorithms, please see

[82,83,84,85,80,86,87,88,89,90,91,92].

2.2.1. Particle swarm optimization

Particle Swarm Optimization is a variant of EC that is useful for multi-dimensional nonlinear

optimization [90]. The algorithm is traced back to Kennedy and Eberhart’s initial research in 1995

and is closely related to genetic algorithms and evolutionary programming [81]. It is also a

population-based optimization procedure, but the population’s members communicate information

Taylor 41 of 249

09/28/10

about the solution space to the entire population. The term swarm refers to the population’s members

(the particles) and is based on defining principles for swarms in the artificial life community [81].

PSO is different from the GA and evolutionary programming strategies in that no selection is

performed on the potential solutions in the population. Each particle is defined as a point in the

problem’s N-dimensional solution space and each particle flies (searches) through this solution space.

The velocity at which the particle flies through the space is dynamic and is adjusted according to the

particle’s best fitness value as well as the best-found fitness values of other particles within a defined

neighborhood. With the ith particle, Xi, a vector defined as

X i = x i1 ,x i2 , …,x iN

( )

(6)

in an N-dimensional space, the particle’s own best fitness value (at this time step in the iterative

process) is defined at the location

P i = pi1 , pi2 , … piN

( )

(7)

and the index over all P that represents the global best fitness value is denoted as g. Finally, with the

local velocity of Pi described as

V i = v i1 ,v i2 , …,v iN

( )

(8)

the update equations for the core PSO algorithm are

vin

= vin

+ C1 rand ` a pin@ x

in

` a

+ C 2 Rand ` a pgn@ xin

` a

(9)

and

xin

= xin

+ vin

(10)

Taylor 42 of 249

09/28/10

where C1 and C2 are constants and rand() and Rand() are random functions defined on [0,1].

Mutation in an EC context occurs in the PSO through the changes in velocity for each particle as

successive generations of particles are updated through (9) and (10).

PSO is competitive with GA on classical GA benchmarks [93]. Applications of particle swarm

optimization are diverse, ranging from recent research in multi-objective optimization [94] to remote

sensing applications such as ocean color reflectance inversion [95]. Research into hybrid

combination of PSO and GA is also ongoing [96].

Figure 6 shows a system of several swarm particles over 25 time iterations as they explore an

optimization problem that has a complicated three-dimensional manifold.

Figure 6. Particle Swarm Optimization Algorithm [90].

In summary, particle swarm optimization algorithms posses the following advantages and

disadvantages:

Advantages

• Easy to implement

• Simple algorithm can be applied to many different types of problem sets

• Domain specific knowledge is not required

• Fairly mature (real world and multi-objective problems)

Taylor 43 of 249

09/28/10

• Typically a faster optimization process than genetic algorithms

Disadvantages

• Accuracy tradeoffs required for global and local searching

• Not mature for multi-objective problems

2.3. Biological and artificial evolutionary development

Biological evolutionary development, commonly referred to as evo-devo, is the term for the processes

which shape the phenotypic development of an entity based on its genotype [97]. Although it is a

biological phenomenon, in this research it is directly applied to the creation of the HeBIS information

classification system. It provides the basis for self-organized control and communications in the

HeBIS classifier. Biological evolutionary development outlines the process through which a

fertilized human egg can grow from a single-cell entity to an entity that is composed of trillions of

independently functioning cells [98]. The fact that the instructions required to create a fully-

functioning and sentient human are encoded into such a compact structure is one of the bases of this

research. These compact structures, the cell’s DNA-based chromosomes, consist of a limited

alphabet of four distinct organic chemicals (complementary chemical base pairs consisting of

adenine, cytosine, thymine, and guanine) form a book of life by using only 4 billion characters.

These 4 billion characters encode for 20 000 to 30 000 genes within a collection of genes defined as a

genome. Each of these genes, when expressed, can form an organic chemical known as a protein.

From this relatively small number of genes that code for individual proteins, a complete and complex

organism is coded. These proteins constitute the control and communications infrastructure for

biological processes [98]. Evolutionary development attempts to explain the apparently huge

information density contained in DNA by placing this information in the context of a physical,

temporally changing, chemical environment. It is through this environment and the complex

signaling network which comprise it that the entity described by DNA may grow into a mature entity

composed of trillions of cells [98].

It is this physical environment which provides a rich level of chemical interaction, evolved complex

spatial and temporal protein-to-protein signaling (communications) networks, and a level of self-

organization that human engineers would like to exploit for their information processing systems.

The power of biological evolutionary development appears to stem from the constant reuse of

proteins within evolved signaling networks [99]. These evolved signaling networks are called genetic

regulatory networks (GRN). The term regulatory is used because it is through the GRN that a

physical body’s biological processes are regulated and controlled. It is the idea of a GRN, in an

Taylor 44 of 249

09/28/10

artificial form, that provides another of the bases for this research. The GRN and the expression of

proteins provide the control and signaling through which an actual organism is built This concept

may be used as the basis to build a physical organism such as evolved, fully functioning robots.

However, it can also be applied to design a control structure for information classification which is

relevant to this research

HeBIS’ self-organizing abilities are based on five key aspects of biological evolutionary development

as defined by Bentley and Kumar in [37]:

• Cleavage divisions- The process by which a zygote goes through a series of divisions which create more cells within the volume of the original cell.

• Pattern formation- The spatio-temporal process through which the layout and cellular activities of an organism are organized. The concept of positional information enables individual cells to be positioned within an organism according to prescribed chemical diffusion gradients.

• Morphogenesis- The process through which an organism grows and enables its 3-dimensional body plan.

• Cellular differentiation- This idea enables cells in the organism to acquire differing properties such as skin cells, kidney cells, or neurons. These properties are ultimately caused by the morphed changes in gene expression and signaling in the individual cell. They are caused by, among other factors, intercellular communication and asymmetric division during cell fission.

• Growth – The increase in mass of the organism through the pertinent examples of cell fission and proliferation.

These five fundamental aspects of evolutionary development utilize a cellular signaling mechanism

for both intra-cellular and inter-cellular communications. These signaling mechanisms consist of

both short-range and long-range responses and are collectively referred to as the cellular genetic

regulatory network (GRN) [98].

HeBIS incorporates an artificial GRN that is derived from biological GRN attributes. Because of

this, it is important that pertinent details about biological GRNs be discussed. What follows is an

overview of the salient biological concepts.

Cell signaling as enabled by the GRN, is predicated on the presence of the protein. A protein is a

linear organic polymer that is composed of 20 building blocks called amino acids. A protein is

composed of a sequence of these amino acids and, once created, acquires a distinct, complex three-

dimensional shape that defines how it will chemically react with other proteins. This research uses

Taylor 45 of 249

09/28/10

the construct of an artificial protein for communications of control information and data within the

HeBIS classification network.

A protein is ultimately produced within the cell by its ribosomes, but it is defined by a specific gene

located on the cell’s chromosomal DNA. Each DNA-encoded protein is defined by a sequence of

codons upon which each of the codons is defined as a 3-base pair sequence of DNA that codes for a

single amino acid. Artificial genes are used in an artificial genome in this research.

For a protein to be produced, the protein description as defined by its codons must be transcribed to

the ribosomes via an intermediary, messenger RNA or mRNA. However, before this occurs, the gene

that controls the protein must be activated or expressed [98]. This expression is controlled by a

region of DNA base pairs that is adjacent to the codons that code for the specific protein. This region

is referred to as a promoter region or the cis-regulatory region. This region responds to the presence

of particular proteins and acts as a switch that activates (turns on) or inhibits (turns off) protein

transcription by this gene. These “particular” genes are called regulatory proteins and are emitted by

other genes that have been expressed. This system provides a complex (and multi-tasked)

communications paradigm that can create control chains and negative and positive regulatory

feedback loops over long periods of time and across distant regions of the physical organism.

Through these GRN interactions that recursively control protein expression (and inhibition), an

organism undergoes development of its body plan, parts, and initial brain synaptic connections

(neurogenesis) [5]. Certain genes and their associated GRN have been found to have been conserved

across biological evolutionary time and across species. This set of genes, the hox genes, appears to

compose a compact and shared evolutionary toolkit that controls morphology [45]. Day-to-day

operation aspects of the organism, after it has developed fully, are also controlled by a GRN.

The idea of such a compactly coded, robust, and adaptive toolkit that is based on a GRN with

multiple, layered feedback and feedforward communications and control loops is very important to

this research. It is useful for artificial neurogenesis and also for the development of the HeBIS

artificial information processing structure.

The incorporation of a GRN into an evolutionary computation system allows a complex and

nonlinear mapping to occur between a genotype and the realized phenotype. This is important

because discovery and optimization systems based on evolutionary computation typically force the

genome to become more complex as the problem’s complexity increases. This is because the

Taylor 46 of 249

09/28/10

problem components are generally linearly mapped back to individual artificial genes. Therefore, as

the problem or design space becomes more complex, the number of required genes increases

prohibitively. This scaling issue is exacerbated by the issue that the human designer must also

explicitly inject her own knowledge into the situation with respect to the parameters that should be

searched or optimized to find a solution to the problem at hand. Quite often, lack of adequate

knowledge of the problem domain hinders the finding of a solution or biases solutions towards

human-realized alternatives. The EC workhorse, the genetic algorithm with one-to-one gene-to-

functionality mapping, is the epitome of this scaling problem.

Remote sensing classification is a complex problem. HeBIS research investigates the possibility of

melding the powerful nonlinear mapping ability of a GRN with evolutionary computation to create an

accurate and robust classifier.

Kumar in [37] has coined the term “computational development” to describe the application of

biological development principles to computer science problems. This term can be used to describe

that research and the resulting techniques which couple the EC-derived genotype to its physical or

neurological parameters. In [37], Kumar and Bentley define a set of advantages and disadvantages of

artificial systems that exploit biological development. In doing so, these systems exploit an important

commonality between biological development and evolutionary computation: the desire to

understand self-organized structure. A subset of these defined computational development

advantages and disadvantages includes:

Advantages

• Reduction of the size of the genotype

• Inherent emergence of complexity

• Complex phenotypes based on simple genotypes

• Hierarchical development with reuse of modular functionality

• Adaptability

• Noise robustness

• Regenerative capabilities of damaged functionality

• Regulatory capabilities

Disadvantages

• Computer-based evolution is difficult

• Analyses of the feedback and feedforward loops of the signaling networks within the evolved GRNs are difficult

• Computationally expensive (more than traditional EC methods)

Taylor 47 of 249

09/28/10

Much research remains in computational development, where biological development meets

evolutionary computation. Although the research areas in which biological principles are applied are

diverse and deep, the application of biological development is still limited to

• GRN studies and pattern creation,

• Evolved robotic morphology, and

• Artificial immunology.

Principles of biological development have been applied to the creation of artificial chemistries,

research into genetic regulatory networks and pattern creation, robotic morphology, control and

artificial immunology, and remote sensing.

Kitano in [100] was one of the first researchers to incorporate biological evolutionary development

into a computational model based on biological principles. At its core, this model was based on what

Kitano referred to as evolutionary large scale chaos. Through the Super Coupled Map (SCM),

individual cells reacted and communicated with each other on a two-dimensional lattice through

localized and global artificial enzyme reactions. Evolutionary large scale chaos was defined as the

application of a genetic algorithm to the evolution of a cellular metabolism. Kitano showed that

stable cell differentiation occurred as the simulations progressed. This differentiation was expressed

as the types of chemical activity present in the cells, specifically as the cell’s chromosomes

underwent expression and repression. HeBIS incorporates this idea of a lattice structure in which

communications between cells is based on interactions between artificial proteins.

Recently, in 2004, Madina, Ono, and Ikesani [101] studied the evolution of cellular processes that

were based on an artificial chemistry. They modeled their system using a 3-dimensional lattice of 643

(262 144) positions in which their artificial chemistry was based on the discrete diffusion of chemical

particles. While limited to 300 particles per position, the researchers were able to show the

spontaneous creation of membrane-like structures, or proto-cells, as they called them. Importantly,

they found that this formation of complex structures occurred from random initial conditions.

Although Kauffman pioneered the study of the genome as a complex network in 1993 [102] through

the use of random Boolean networks, the computing and biological communities took issue with his

modeling. The problem, it was believed, was that the random Boolean network was limited and did

not share many of the underlying processes of biological GRNs.

Taylor 48 of 249

09/28/10

However, in 1999, Torsen Reil at the University of Sussex expanded upon these initial GRN models

by incorporating a simplified but biologically plausible model [103]. In his model, he made use of

template matching and the concept of an artificial gene which was expressed or inhibited in an analog

of biological expression. Reil incorporated the idea that the artificial genes were switched on or off

through the manipulation of a unique switch region that was situated next to each artificial gene in

his artificial genomes. His simplified TATA box (the gene switch in the eukaryotic genome) which is

also known as a standard promoter used simple template matching and was activated if a matching

sequence in the genome was encountered.

Reil’s research is also notable because he discovered attractors of complex and dynamic gene

expression that ranged from chaotic behavior through complex behavior, and finally simple ordered

expression. The most interesting results were those complex instances in which many cycle-lengths

of expression were discovered to be inherent in the GRN. Furthermore, his research indicated that a

high degree of robustness is afforded by a GRN (both biological and artificial). It also indicated that

robust adaptation may occur when natural selection is allowed to act upon the raw properties inherent

in a GRN. The importance of scale-free or 1/f networks are also hinted at in Reil’s research results.

These are important results that are incorporated into HeBIS through the implementation of a simple

GRN.

An overview of formalisms in the study of genetic regulatory networks is contained in [104]. In this

2004 report, Geard outlined GRN modeling approaches and their various advantages and

disadvantages from the perspective of computational expense and biological accuracy.

Over the past few years, Bongard has specialized in the development of robotic morphologies and

neurogenesis for their control. His work is based on a genetic regulatory network in which simulated

genes produce transcription factors that directly affect phenotypic expression and the regulation of

other genes [4]. This GRN makes use of genes that possess promoter sites that are activated only if

the environmental concentration of the matching transcription factor is within an evolved range of

concentration values. Bongard’s research for his 2004 dissertation employed the concept of a two-

stage process of fitness determination. In the first stage, the simulated robot morphology (for a

simulated multi-articulated robot) is grown from an evolved GRN and in the second stage it is

evaluated in a virtual environment [33]. The GRNs were evolved with the use of a genetic algorithm

acting on a population of 300 genomes.

Taylor 49 of 249

09/28/10

Bongard discovered that the two-stage fitness determination process allowed evolution to experiment

with morphology and controller (neurogenesis) development separately and in a modular fashion.

The evolutionary success of the various morphologies was found to be due to the early evolution of

modular regulatory networks, the GRNs. Because of this, it was decided to use a two-stage fitness

process for testing HeBIS genomes.

Bentley has also used computational development as the basis for evolving robotic controllers, in this

case for a toy robot known as the wonderborg [3]. Locomotive actions for the robot are keyed

directly to action genes that are expressed through an evolved regulatory network. Information from

the physical world is represented in the environmental simulation as environmental proteins. This

research was the first to utilize a complex artificial chemistry based on the Mandelbrot fractal set. It

also highlighted the concept of a cell embedded in an environment that possesses internal and external

behaviors that are controlled by the evolved GRN. Through evolutionary computation, specifically a

genetic algorithm, GRN-based controllers were grown that successfully guided the robot through a

maze. The controllers are unique in the sense that they automatically created functional modules that

were used with variation for maze navigation. This concept of a module was not hard-wired into the

system, but emerged naturally as a consequence of using a GRN for signaling and regulation-

something which does not occur in GP (genetic programming) or parameterized L-systems [38].

Importantly, this research showed that evo-devo principles allow an information processing system to

efficiently build a system to find a solution [105].

Scaling issues in artificial neural networks have been examined in the context of rudimentary systems

that break the one-to-one genotype-phenotype mapping. In 1990, Hiroaki Kitano developed the

grammar encoding method to address the scalability issues of artificial neural networks that were

created with genetic algorithms [45]. Kitano's research addressed two issues with such a system

when scaled to large problems:

• ANN morphogenesis cannot be captured in a one-to-one mapping and,

• Complex problems lead to a corresponding increase in the complexity of the genome that the GA is optimizing.

The development of this grammar encoding method was based on Lindenmayer’s L-system [38] from

1968. The system was one of the first attempts to separate genotype development from phenotype

Taylor 50 of 249

09/28/10

evaluation. This also has been done recently in [33]. Kitano determined that this separation allows

his technique to generate more regular patterns than a direct-encoding method for the GA and that

subcircuits (modules) were formed and exploited by the GA. Kitano also discovered that this method

could be used to create a useful neural network using a shorter length of chromosome than the length

of chromosome that was required when using direct-encoded GA.

Evolutionary development combined with genetic programming has also been used to create simple

visual patterns. Pattern recognition and the ability to model patterns are essential to the operation of

an effective classifier. Therefore, this is why the creation of visual patterns is relevant to the HeBIS

classifier. It is possible that HeBIS will encapsulate or generate the pattern of the training data’s

manifold (or a statistical version of it) in the GRN. Designed to create French flag maps in a toy

environment, Miller and Banzhaf’s Cartesian Genetic Programming (CGP) was designed around

chemical diffusion that is much simpler than that used in HeBIS. Through the diffusion of chemicals

in a toy environment, a limited set of developmental rules was invoked in attempts to create the

desired pattern. The set of developmental rules was replicated in each artificial cell defined on a two-

dimensional Cartesian lattice in which each position in the lattice ultimately became part of an acyclic

graph that was self-organized to create the pattern. Although each cell ran the same program, the

local outcome of each was affected by the internal states of the neighboring cells. Cell differentiation

was displayed via the creation of a French flag pattern over the lattice. The research showed that

developmental principles can be successfully applied to genetic programming, albeit this was for a

rudimentary problem. Simple self-organization was demonstrated. Miller and Banzhaf suggested

that the idea of limiting the cells that can arbitrarily reproduce and differentiate (i.e. stem cells), could

be examined in future research. The authors believed that the cellular program was relatively small

for the level of complexity contained in the problem.

Artificial immunology has also been used as the basis for classification research. deCastro and Von

Zubin in 2001 [106] experimented with their artificial immune system model, aiNet in the context of

determining the distributions of test datasets and performing automated clustering of redundant data

samples. The network was designed more so to implement rudimentary learning capabilities instead

of being a completely biologically plausible model of the vertebrate immune system. The model’s

performance was analyzed for several benchmark problems that consisted of non-linearly separable

classes as well as the CHAINLINK problem which was formed from the point distribution of two

interwoven rings in 3-dimensional space. Results were comparable to a Kohonen SOFM trained on

the same problems, but the aiNet model is sensitive to a relatively larger number of tuning parameters

Taylor 51 of 249

09/28/10

than is the SOM. aiNet has high computational costs per training iteration that are O m2b c

where m

is the number of “memory” antibodies used by the system in a specific problem. Therefore, as the

complexity of the problem increases, the time required to train aiNet increases exponentially.

Unlike the artificial immunology classification research, HeBIS is more of a hybrid of biological and

ML principles. It does not attempt to map the ML constructs directly into biological analogs.

Not much research effort has been applied to the application of evolutionary development concepts to

classifying remote sensing imagery. One example uses the idea of an artificial immune system (AIS)

and applies it to this topic [107]. This presents an AIS that harnesses its various elements such as

antigens, antibodies, shape-space, and immune memory for the unsupervised classification of

multispectral optical satellite data. The algorithm, the Unsupervised Artificial Immune Classifier

(UAIC), classifies data on a pixel-by-pixel basis. The training data are mapped to the system’s

antigens and the classes/clusters are represented by UAIC’s antibodies. UAIC’s performance is

reported in comparisons with clustering algorithms such as K-Means, ISODATA, Fuzzy K-Means,

and a SOFM. The authors report that it outperforms these algorithms on the test dataset. However,

computation times for the algorithm are not adequately discussed. This has direct bearing on HeBIS

and indicates that the HeBIS algorithm may require significant computing resources for simulation.

2.4. Summary

It seems that biological development applied to classification problems may help by breaking the

one-to-one genotype-to-phenotype matching, but there are still problems [45]. One aspect of these

problems is the amount of computing power that is required to simulate large-scale GRN reactions in

an artificial chemistry [101]. Also, the evolutionary computation required to create a usable (and

complex) classification structure is clearly daunting with the computation power that is available to

the typical researcher.

The creation of a purely artificial-protein based classification system is difficult to realize, especially

one that can be as easily and as quickly trained as those produced by artificial neural networks, self-

organizing feature maps, and support vector machines. However, it is the combination of the SOFM

element with artificial biological development that is investigated in this work. It may mitigate the

previously outlined problems and produce a self-organizing classification network which possesses

Taylor 52 of 249

09/28/10

enough complexity and robustness to quickly and automatically adapt to different problems with high

precision. HeBIS is meant to examine a few of the basic issues associated with such a goal.

The modular building blocks for this architecture consist of evolved self-organizing feature maps. As

will be discussed in detail in later sections, communications between the processing cells are

conducted via artificial proteins that are diffused throughout the simulated environmental lattice.

Each cell in the artificial evolutionary development environment can have a SOFM machine learning

processing kernel. The communication among these cells will be defined by a fixed-length genome

that is evolved through evolutionary computation using the ideas of computational development. The

genes in each cell are associated with artificial proteins that will be activated or inhibited by evolved

genetic switches that are also contained within each processing cell’s genome [5,82].

A simple self-organizing algorithms will be examined for its potential contributions to this evolved

heterogeneous architecture that is based on computational development and artificial neurogenesis

[2,23,100]. General principles from the fields of computational development and genetic regulatory

networks will be used to guide the research towards developing complex learning structures that are

modular and robust [33]. HeBIS’ potential advantages and disadvantages are:

HeBIS Advantages

• Introduction of complexity via artificial protein chemistries and GRN signaling

• Potential for compact genomes because of the nonlinear mapping between genotype and the parameter (phenotype) space

• Graceful degradation of classification performance through the evolved GRN o Robustness o Noise

• Automatic modularization of functionality o Graceful degradation / robustness

� Missing data � Noisy data

o Adaptability � Damage-resistance- another module takes over processing

HeBIS Disadvantages

• Difficult to evolve (GRN)

• Difficult to analyze

• Computationally expensive to train and operate

Taylor 53 of 249

09/28/10

Taylor 54 of 249

09/28/10

3. Heterogeneous Biologically Inspired System (HeBIS) Hybridization Background

Algorithm Discussion Remote Sensing Background

GRN Analyses GRN Training with PSO

GRN Action Analyses Remote Sensing Application – Analyses and Comparisons

Robustness Analyses

3.1. Overview

HeBIS is a self-organizing classification system that is inspired by the ability of a live biological

organism to grow itself from the stage of a fertilized egg to the complete species’ phenotype through

the processing of the daughter genome that is obtained from the mating of its parents. The complex

development procedure is encoded through the total number of genes in the genome and the interplay

of these proteins in time and within the chemical environment in which they reside via a GRN. The

use of this GRN allows for a dense encoding of development information that is both self-organizing

and robust. It is also this network of protein-to-protein metabolic reactions that forms the basis and

feedback loops for the stable metabolic control of a human being. At its core, the HeBIS

environment for the classification of multispectral remote sensing data is inspired by the evolutionary

developmental power described and controlled by real-life biological GRNs.

HeBIS mimics portions of the mammalian protein signaling network from a GRN and embeds

machine learning elements in a simulated protein environment. An artificial GRN is evolved to

provide both the underlying communications infrastructure via protein-to-protein interactions as well

as to introduce a significant level of complexity into the system. This complexity provides the engine

for the self-organizing components of the classification network. Once trained, the evolved genome

is used to classify previously unseen examples from the problem at hand.

The HeBIS processing infrastructure is composed of the following functional areas, each of which

will be discussed in detail in this chapter:

• Simulated protein interaction environment

• Genetic regulatory network (GRN)

• Basic cell types, processing and behaviors

• Input feature vector and class representations

• Output class representation

• Classification training

• Evolutionary optimization

Taylor 55 of 249

09/28/10

Figure 7 provides an overview of the HeBIS environmental lattice in relation to its cells, genomes,

and proteins.

Outp

ut

ProcessingKernel

(ANN,SOM,or SVM)

Pro

tein

Hol

ogra

m

Environmental Protein

Regulatory Protein

Cell Receptor Protein

Pha

sed

Inpu

t Det

ecto

r

Environmental Protein

Cell Membrane

Control

Genome

Figure 7. Environmental lattice and processing cell overview.

In this chapter, the processing cell will be introduced as well as the simulated environment in which

artificial protein signaling occurs as well as the metabolic interactions between these proteins.

Proceeding from there, the artificial proteins will be described with an emphasis on the core

processing cell types and their intrinsic and learned behaviors. Input and output feature vector and

class representations will be described as well as the training method examined in this research.

Finally, implementation issues will be discussed.

3.2. Fundamentals

3.2.1. Processing cell

At this time, it is necessary to introduce information about the workhorse of the HeBIS environment,

the cell. Detailed functioning of the cell is included in later sections, but an overview is required to

provide the proper context for discussion of the HeBIS genetic regulatory network and its place in the

simulated environment. The major blocks of cell functionality to be discussed are portrayed in Figure

8.

Taylor 56 of 249

09/28/10

Figure 8. Major blocks of cell functionality.

The processing cell is an artificial entity which resides at a fixed spatial location within the

processing protein environment. The cell is delimited by a cell membrane that separates the cell’s

internal process from processes that are defined to exist only in the surrounding protein environment.

The interior volume delineated by the cell membrane is referred to as the cell’s cytoplasm. Major

structures contained within the cytoplasm are the cell receptor proteins, the control genome, the

output block, and the cell’s processing kernel. All these structures are purely artificial and mimic key

functions of their biological, real-world analogs.

The cell receptor proteins serve to map specific environmental proteins into the cell’s internal cell

control systems and the output system maps output results back out into the environment via

environmental proteins. Both the receptor and output blocks map numerical values to artificial

protein concentrations with appropriate scaling.

Each cell has a processing kernel which provides the machine learning functionality. This

functionality is currently limited to be a SOFM.

This architecture creates a genome within an individual cell. This genome has the ability to grow a

network of cells with processing machine learning kernels that will solve a specified classification

problem. The overlying structure provides the learning and analysis mechanism so that a genome

may be trained via the application of evolutionary computation, i.e. particle swarm optimization.

Taylor 57 of 249

09/28/10

The cell’s genome consists of genes which code for artificial proteins that are evolved by the HeBIS

training algorithm. It controls the execution of the cell’s innate and learned behaviors and it activates,

deactivates, and modifies these actions according to the concentrations of regulatory and recognized

environmental proteins. Through these interactions, a single cell self-organizes into a more complex

classification network.

The genome and its GRN that are created are optimized for a specific problem through the application

of a particle swarm optimizer to a specific training dataset taken from the problem domain.

These major functionalities are treated in more detail in the following sections of this dissertation.

3.2.2. Environment

The HeBIS protein environment is simulated and does not involve the use of actual proteins. The

environment is a discrete-time and discrete-space simulation that is based on a user-specified multi-

dimensional lattice structure. The environment comes in three flavors, a one-dimensional linear

array, a two-dimensional planar array, and a three-dimensional cubic lattice. However, only the

three-dimensional cubic lattice is examined in the current research. The discretized space coordinate

system that defines each structure is based on unit integers that are members of the set of counting

numbers, that is N such that n [0, ).

Simulation of the protein metabolic reactions and processing intervals occur in discrete unit time

steps such that t [0, ).

The environment uses an indexing arrangement in which the most fundamental processing element,

the cell, may be accessed and polled for its internal operating state. It also provides a simulated

medium through which artificial proteins diffuse, encounter each other, and react through the

simulated genetic regulatory network. 1-D, 2-D and 3-D lattices are now discussed in more detail.

Taylor 58 of 249

09/28/10

3.2.2.1. 1-D environment

The one-dimensional linear environment is pictured in Figure 9. At each integral unit along the

defined x-coordinate system, a processing cell may or may not exist. This is decided either through

the environment’s initialization or as the artificial GRN is processed. Therefore with the grid

numbering scheme defined in the figure, a maximum of 6 cells can exist in this example. Only six

cells can exist because boundary conditions are invoked in the environment so that a 1-unit buffer

zone always exists between the maximum extent of the cells and the edge of the environment. In this

case, the buffer positions are defined at the positions, x = 0 and x = 7. For purposes of protein

diffusion, the initial values at the beginning of the protein diffusion simulation are zero for the

boundary positions, x = 0 and x = 7. Protein concentrations for these boundary elements remain fixed

at zero through the simulation. Nearest-neighbor spatial relationships (to be defined later) between

cells on the linear grid are defined using a Euclidean L2 norm. For example, the nearest neighbors of

a cell located at position 3 are cells located at positions 2 and 4. Grid positions are numbered with

unit integers such that x 2 [0, Max -1] with Max defined as the maximum size of the linear grid.

Diffusion is defined only at the integral cell positions and not on the inter-cell regions, for example x

2 (0,1).

Figure 9. Linear grid numbering scheme.


The 2-dimensional planar environment is shown in Figure 10. It is gridded and enumerated in a

fashion similar to the 1-dimensional linear environment. In the 2-dimensional case, cells may exist

only at discrete unit intervals along the imposed x-y coordinate system. Therefore, cell number 7 is

located at the x-y position (1,1). Boundary conditions are set in a fashion similar to that of the 1-

dimensional lattice: a one-cell wide buffer zone surrounds those positions which cells may reside and

in which protein diffusion may occur. For the example in Figure 3, with Max = 24, the buffer zones

are defined as the elements with x = 0, y = [0,3]; x = 5, y = [0,3]; x = [0,5], y = 0; and x = [0,5], y = 3.

This leaves a potentially active processing region composed of (for this example) 8 positions, P, such

that P 2 { 7, 8, 9, 10, 13,14, 15, 16}. Boundary conditions for the buffer positions are set to 0 for

protein diffusion.

Taylor 59 of 249

09/28/10

Figure 10. Planar grid numbering scheme.


The third and final type of lattice is the 3-dimensional cubic lattice. It is an extension of the schemes

that have been outlined for the 1-dimensional and 2-dimensional environments. Presented in Figure

11, the same theory that regards boundary conditions and cell numbering in the lower dimensional

environments, applies to the 3-dimensional cubic lattice.

Figure 11. Three-dimensional lattice numbering scheme.

3.1.2. Genetic regulatory network

Biological development uses genes to define proteins that are in turn used for multiple overlaid

purposes [3]. These purposes include the activation of other genes, the suppression of other genes,

Taylor 60 of 249

09/28/10

and inter-cell and intra-cell signaling and communications. This GRN is a modular system that

recurrently uses the same gene for different purposes and for the creation of many different types of

proteins that have different and complex effects. These protein and metabolic interactions are

harnessed for complex and recurrent communications between the simple processing building blocks

in the learning system [108].

Artificial protein creation is activated or suppressed by other artificial proteins that are in turn

controlled by genetic switches present in the evolved processing cell’s fixed-length genome. These

proteins diffuse through the simulated environment which contains the network of modular

heterogeneous processing cells. The diffusion occurs at discrete time intervals and each protein’s

concentration decreases over a specified time interval [109]. Concentration is defined as a

mass/volume ratio and for HeBIS’ artificial proteins its unit is [generic-mass-unit/generic-unit-

volume]. Communications between processing elements is inherent in this system as the proteins and

their corresponding activation and inhibition genetic switches evolve during training [3].

3.1.2.1. Gene coding

Within each HeBIS cell is a fixed-length genome that consists of at minimum, one gene. The

ultimate number of genes that may be contained in the genome is unrestrained although it is fixed for

each research simulation. The hierarchy between the cell’s genome, its genes, and potential

expressed proteins is shown in Figure 12. This is a much simplified version of what occurs in a true

biological regulatory environment.

Taylor 61 of 249

09/28/10

Figure 12. Genome/Gene protein hierarchy.

Each gene portrayed in the figure contains several functional regions. These regions include the

switch protein, the gene type, the action that may be associated with the gene, and one or more

protein descriptions. The switch protein region is the description of the perfectly-matched protein

that activates the specific action or actions that have been associated with the gene. Biologically, the

switch protein (see Figure 13) defines the protein match for which proteins that are matched against

this template, must have a high affinity or else the action will not occur. Affinity is defined as the

degree to which a protein matches the switch protein template.

Figure 13. Standard regulatory/environmental and switch protein descriptions.

Next, the gene type region defines this gene as a combination of one or more of the following

attributes in an OR’d fashion: Action Gene || Regulatory Protein || Environmental Protein || Cell

Taylor 62 of 249

09/28/10

Receptor Protein. Once the gene is excited, the gene type region determines whether the cell starts or

stops production of regulatory and/or environmental proteins and also determines whether any of the

learned cell behaviors is executed.

The action region defines a diverse set of potential cellularactions. This list includes the following

cell behaviors that will be discussed in detail in a later section.

• AddCell

• PruneSelf

• ChangeToSOFMAndTrain

• Classify The protein description regions (regulatory, environmental, and cell receptor) within the gene

describe the proteins that are produced and released into either the cytoplasm or environment. The

regulatory and cell receptor proteins are only for intracellular release whereas the coded

environmental proteins are released directly into the environmental lattice in which the cells reside.

Two broad classes of protein codings exist: switch proteins and standard proteins. Switch codings

only describe switch proteins and standard codings describe environmental, regulatory, and cell

receptor proteins. Figure 13 portrays these codings and their important regions.

Both codings describe a protein as a vector of four integers, < p0, p1, p2, p3> . This vector defines a

point in a [1, 254] four-dimensional cube.

The difference in the two codings lies with the concentration region. In the case of a standard protein,

the single concentration value is a real number on the interval [0,1] that describes a current real-

valued concentration. For these regulatory, environmental, and cell receptor proteins, this number is

the value of the protein’s concentration as it is first produced by the gene at each excited timestep.

For example, when a specific environmental protein is produced, it is initially produced in the first

time step with the initial concentration indicated in the concentration mask. In subsequent timesteps,

if the gene remains excited then more of the protein will be created with this concentration level at

this point in the environmental lattice. As this protein diffuses through the cytoplasm or the

environment, concentration levels of this protein at all points in the lattice are updated.

Taylor 63 of 249

09/28/10

However, the switch protein requires minimum and maximum activation thresholds. If a test protein

matches the protein description template, <p0, p1, p2, p3> it still must have a local concentration at the

cell that is within the prescribed thresholds of the switch. If it does not, then the associated gene will

not express itself.

This similarity measure is currently implemented as a two-part function. More formally, the

determination of the switch’s activation is identical to the question of whether two proteins, P1 and

P2, are matches for each other. In this case, an exact match between the proteins is defined if and

only if

P1 ,P2

* +

P1

N

N

N

N

N

N P2

N

N

N

N

N

N

fffffffffffffffffffffffffffffff= cos θ

` a

= 1

(11)

and

P1

N

N

N

N

N

N

P2

N

N

N

N

N

N

fffffffffffffff= 1

,

(12)

where < , > is an inner product defined on the two vectors, || || is the L2 norm, and is the inner

angle between the two proteins, P1 and P2, when they are referenced as vectors.

The L2 norm is defined only over the domain of descriptive protein parameters; p0, p1, p2, and p3.

This norm is

P1

N

N

N

N

N

N= p02 + p1

2 + p22 + p3

2qwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww

.

(13) A partial match between the proteins and in turn between a switch and a test protein is enabled

through a constraint relaxation such that a gene switch is considered to be activated if

Taylor 64 of 249

09/28/10

1@∆1 ≤P0 ,Switch* +

P0

N

N

N

N

N

N SwitchN

N

N

N

fffffffffffffffffffffffffffffffffffffffffffff≤ 1

,

(14)

and

1@∆2 ≤P0

N

N

N

N

N

N

SwitchN

N

N

N

ffffffffffffffffffffffffffff≤ 1

.

(15)

The empirical effects of the choices of ∆1 and ∆2 are examined in the Simulations and Analyses

chapter.

Each of the two protein codings also has a protein ID region. This region is used purely for

bookkeeping purposes by the simulation’s infrastructure. It has no implied biological meaning or

utility.

3.1.2.2. Protein communications

Protein communications between the processing cells is founded on three points: protein diffusion

through the environment and within the cytoplasm, a defined artificial chemistry that allows the

proteins to react with each other and to create new proteins, and protein template matching.

• Diffusion of an artificial protein occurs in the 3-D environmental lattice. The protein

diffusion equation is simply the discretized scalar Laplacian. In a 3-dimensional

Cartesian coordinate system, this is defined as

(16)

where h = 2 and ψ x,y,z` a

is the value of the protein concentration at (x,y,z) in the

discrete spatial environment.

Taylor 65 of 249

09/28/10

• The HeBIS artificial protein chemistry is a simple one that bears little resemblance to its

biological model. The chemistry is defined and occurs with normal Gaussian probability

at a location in the environment where two or more proteins are present such that

p t` a

reaction=

1

2πpwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwffffffffffffffff

e@

t 2

2

ffffffff .

(17)

Note that the protein chemistry is not currently defined for intracellular reactions. If two

or more proteins exist at a specific location, a chemical reaction will occur between them

with a discrete Gaussian probability. The resultant new protein vector, <pnew0, pnew1,

pnew2, pnew3, ϕ new> is calculated by a simple exclusive-OR (XOR) operation on the bits of

each of the integer-valued parameters, p0… p3. The XOR operation is tabulated in Table

1.

Table 1. Bitwise XOR Functional Mapping

Bit0

Bit1

XOR

0 0 0

0 1 1

1 0 1

1 1 0

This protein chemistry is defined so that the chemical reaction depletes the protein

reactants that are available at the environmental location. This depletion assumes that the

proteins that are undergoing the reaction are reduced in concentration and that

“concentration” is conserved. As an example, assume that at location (2,3,2) in a cubic

environmental lattice that three proteins are being metabolized. With these proteins

defined as P0, P1, and P2 these proteins have concentrations of 1, 2, and 0.5 units

respectively. If the chemical reaction occurs, then it proceeds according to

1P 0 + 2P1 + 0.5P 2u 3.5P 3 .

(18)

Taylor 66 of 249

09/28/10

The concentration of P3, the newly formed protein, has an initial concentration of 3.5

units- the summation of all the reactants that are available for the metabolic function at

this location. In turn, the concentrations of P0, P1, and P2 at location (2,3,2) are reduced

to 0 before the next time step of the simulation occurs. Hence, concentration is conserved

through the metabolic reaction and a negative feedback loop for concentration levels is

imposed on the global environment.

• Protein matching, the final pillar of HeBIS communications, has been described in detail

in the previous section. Its purpose is to mimic the protein-based excitatory/inhibitory

functions that occur on a genetic level as well as the shape-based recognition that

biological proteins use.

3.1.3. Basic cell processing

The core of HeBIS is the basic processing cell. Embedded in the artificial environment, it contains

the system’s distributed intelligence for its interaction with the network’s GRN. The system’s self-

organizing features that allow it to learn are implemented at this level. The cell has many behaviors

that are controlled by an evolved fixed-length genome. These behaviors can be automatically

modified as processing continues. Some of these behaviors are immutable and are designed into the

system from the start and others are learned through interaction with the GRN and the proteins.

This section continues the discussion of HeBIS with details about the cell’s control genome, the

different cell types, intrinsic and learned behaviors, and available machine learning kernels. Initially,

the cell genome is discussed.

3.1.3.1. Cell genome

The role of the cell genome is to control the individual cell such that its local behaviors add structure

and ability to the entire classification network of cells. The classification network is built and

operates through the self-organized construction of complex behaviors. These behaviors are built

from a “dictionary” of simpler actions that are executed and controlled within each cell. Importantly,

the simple cellular actions can become more complex as cells interact with other cells via the artificial

GRN. With its individualized behaviors, the cell genome functions in two modes of operation. These

two modes, learning and processing (or production), are separate but not distinct from each other.

In the learning mode an initial cell genome with its variable number of genes is evolved to classify a

set of training examples via evolutionary computation. This evolved genome allows the network to

Taylor 67 of 249

09/28/10

grow (if necessary) from its original single cell beginnings, to a network that is sized and matched in

various aspects to the statistics of the problem. Secondly, in the processing/production operational

mode, the cell genome uses its evolved “control” program and GRN to classify unseen patterns from

the same problem domain. Thus, the evolved behaviors for a single cell interact in a complex fashion

with other cells that are autonomously added (or deleted) from the network.

The various behaviors and cell types are now examined in more detail.

3.1.3.2. Intrinsic behaviors

The cell’s intrinsic behaviors are actions which are performed at the beginning of each discrete time

step for each cell- after the proteins have chemically reacted with each other and they have diffused

further into the environment. These behaviors provide statistical information to the genome about the

proteins that are present both in the local environment as well as inside the cell.

The nomenclature for these intrinsic and learned behaviors is designed such that the names directly

referenced in the following sections are at the same time, function names programmed in the code.

The individual English words are allowed to run into each other to form a new, composite algorithm

name and topic heading.

3.1.3.2.1. NumberProteinsInCell

This behavior counts the proteins that are present in the cytoplasm. These proteins include all

internal regulatory proteins, cell receptor proteins, and environmental proteins that have been passed

into the cytoplasm. The number of these proteins is made available to the cell’s controller and to its

genome as a distinct cytoplasm protein.

3.1.3.2.2. NumberProteinsInLocalEnviro

This behavior counts the proteins that are present only in the local environment outside of the cell.

By definition, this includes only environmental proteins. The sampled proteins are only those that are

present in sufficient concentration at the cell’s location within the environment. The number of these

proteins is made available to the cell’s controller and its genome as a distinct cytoplasm protein.

3.1.3.2.3. ConcentrationStandardDeviationLocalEnviro

This behavior determines the concentrations of all the proteins that are present in the indexed

locations in the environment surrounding the location of this cell. A standard deviation is then

calculated from these concentrations via

Taylor 68 of 249

09/28/10

σconcentration local environment

=1

N@1ffffffffffffffffffX

n = 0

N@ 1

Conc n` a

@µlocal environment

b c2

h

j

i

k

0.5

,

(19)

and provided to the cell’s controller and its genome as a distinct cytoplasm protein.

3.1.3.2.4. ConcentrationMeanLocalEnviro

This behavior determines the mean of the concentrations of all the proteins that are present in the

environment at the cell’s location. This information is calculated with

µlocal environment

=1

N

fffffffXn = 0

N@ 1

Conc n` a

,

(20)

and provided to the cell’s controller and its genome as a distinct cytoplasm protein.

3.1.3.2.5. ConcentrationMaxLocalEnviro

This behavior determines the maximum concentration of all the proteins that are present in the

environment local to the individual cell. This information is provided to the cell’s controller and its


3.1.3.2.6. ConcentrationMinLocalEnviro

This behavior determines the minimum concentration of all the proteins that are present in the

environment local to the individual cell. This information is provided to the cell’s controller and its


3.1.3.2.7. KillSelf

This action allows the cell to commit suicide if a set number of discrete simulation time intervals

have occurred. The default behavior is that this feature is disabled.

3.1.3.2.8. NumberFeatures

This behavior determines the number of features that are being used within this time step by the cell’s

processing kernel. This information is provided to the cell’s controller and its genome as a distinct

cytoplasm protein.

Taylor 69 of 249

09/28/10

3.1.3.3. Learned behaviors

The cell’s learned behaviors are those actions which occur as a result of the cell’s interaction as part

of the genetic regulatory network. Each of these actions is a behavior that is expressed because a

specific gene (or gene sequence) is activated because the requisite proteins are present in the cell’s

cytoplasm in appropriate (evolved) concentrations. These learned behaviors may use information that

has been encoded as proteins by the cell’s intrinsic behaviors. These simple learned behaviors are

combined into more complex behaviors through the nonlinear GRN mapping. These behaviors

implement the communications and control for the classification network and they make use of

information provided by the intrinsic behaviors and the environmental GRN. Inhibition of a behavior

occurs when the excitatory proteins are no longer present in sufficient concentrations. These

behaviors take place after protein chemistry, protein diffusion, and the innate behaviors have

executed.

3.1.3.3.1. AddCell

The cell that executes this behavior adds a copy of itself to the environmental lattice immediately next

to itself. The exact location is one unit away from the initiating cell’s position and in the direction of

the average concentration gradient.

3.1.3.3.2. PruneSelf

The cell that executes this behavior deletes itself from the environmental lattice and ceases all

processing and GRN activities. Proteins that are being produced by this cell both within the

cytoplasm and in the environment cease production beginning at the next simulation time step.

3.1.3.3.3. ChangeToSOFMAndTrain

This behavior causes the cell to convert its processing kernel to a self-organizing feature map.

Training of the SOFM then proceeds and uses the protein holograms that are in the phased input

detector. If the cell is already an SOFM, the SOFM’s parameters are reset and training commences

during the current time interval.

Default SOFM parameters:

� Rectangular lattice � Lattice contains 9 elements in a 3x3 orientation � Simple neighborhood update function

Taylor 70 of 249

09/28/10

In subsequent test/production runs of this behavior, a feature that is not present in the protein-based

feature vector (exemplar) is replaced with an equivalent value of 0.0.

3.1.3.3.4. Classify

This behavior takes the cell’s input present in its phased input detector and uses that as input data as it

executes the processing kernel (SOFM). The resulting output is released into the environment as an

artificial protein.

3.1.3.4. Cell types

Two types of cells are used in HeBIS and they are defined by the type of machine learning processing

kernel that is active: the SOFM and Pass-Thru. By default, each position in the environmental lattice

is initialized as a Pass-Thru node. The Pass-Thru node exists to simply allow protein diffusion and

communications to occur in an uninhibited fashion through a position in the environmental lattice.

As the network’s structure organizes, pass-thru nodes can be converted to processing cells according

to the interactions of the genetic regulatory network.

3.1.3.4.1. SOFM

The SOFM cell type has a SOFM as its processing kernel. The kernel defaults to the following:

� Rectangular lattice � Lattice contains 9 elements in a 3x3 orientation � Simple neighborhood update function � Conservative learning rate

3.1.3.4.2. Pass-Thru

This is a cell which has no processing kernel. No GRN actions are performed by this cell. In

essence, it is invisible to the other cell types in the environmental network.

Taylor 71 of 249

09/28/10

3.2. Input feature vector representations

A single method is currently used to map test and training data from benchmark and real-world

datasets into artificial proteins that are used in the HeBIS environment. This is direct feature-to-

protein mapping. With this method, it is assumed that the data to be converted consists of N

exemplars. Each exemplar is defined by a feature vector which consists of M real-values and finite

entries (features). These features describe specific characteristics of the exemplar. With this

representation, the exemplar is referenced as a point in an M-dimensional feature space. The feature

vector may or may not be sparsely defined.

Given a set of exemplars, a modified set is created in a preprocessing stage that uses the collective

statistics of the feature vectors from the problem domain. This preprocessing stage normalizes the

range of each of the M features independently of the other M-1 features. This global normalization

shifts and scales each feature so that its normalized range now resides within [0,1] and has an

appropriately modified probability density function for each feature. The parameters used to

normalize the features are saved during preprocessing for later denormalization. Preprocessing

occurs for training, test, and operational data.

In this preprocessing stage, each feature is assigned to a unique protein description, <p0,p1,p2,p3>.

The class label is also assigned to a unique protein description.

3.2.1. Direct feature-to-protein mapping

This data mapping operates on a single training or test exemplar. Each feature and class label (if

available) of the normalized exemplar is mapped to the unique protein that has been defined to

represent that feature during preprocessing. The normalized value of the feature is directly mapped to

the concentration of the protein and the phase is set to 0.

The exemplar’s training class label is also normalized so that it is a discrete point in the interval [0,1].

For example, in a two-class problem with class labels C0 and C1, the normalized values of C0 and C1

are chosen as C0 = 0.25 and C1 = 0.75. These values are the centroids of two non-overlapping

subintervals of [0,1]. These two subintervals are [0,0.5) and [0.5,10.]. Multi-class labels are handled

by mapping the labels to the centroids of multiple non-overlapping intervals on [0,1]. Currently, the

subintervals are equally sized but it is foreseeable that the lengths of these subintervals could be set

according to some statistical measure of the training data. This issue will not be examined in this

research.

Taylor 72 of 249

09/28/10

Since the centroids of the subintervals are used, in some sense this class mapping strategy can also be

examined in the future as a fuzzy interpretation of the class label.

A similar process is applicable for multi-class problems where the number of classes is greater than

two.

This procedure for direct feature-to-protein mapping is outlined in Figure 14. The specific

feature/class protein mapping is arbitrarily established in advance during preprocessing. The only

constraint is that the feature/class protein mapping must be unique for each feature. Therefore, a

particular protein is always associated uniquely with a specific feature/class during the course of the

simulation.

Figure 14. Direct feature-to-protein mapping.

3.3. Pattern training for classification

In past sections of this chapter, the fundamental elements of the HeBIS architecture have been

introduced. This section aims to unify these elements under the umbrella of the network training

mechanism. These training mechanisms are based on the self-organization principles that control

each cell, the development of the GRN, and the overlying PSO that tunes the cell’s genome. Once

tuned, a single cell can potentially spawn a network of cells that in unison can perform data

classifications to varying degrees of success.

Taylor 73 of 249

09/28/10

This research examines the applicability of the HeBIS algorithms to the classification of sets of

features. HeBIS is oriented towards using biological principles to determine whether self-

organization ideas from biology may be used to create successful supervised classification networks

based on machine learning constructs. In this research, a supervised learning environment controlled

by a PSO is used. This optimizer tunes the gene and protein descriptions of the cell’s fixed-length

genome. It is this genome that is used to control the individual behavior and network organizational

abilities of the cell. These behaviors and abilities are the building blocks for complex behaviors that

may be used for classification as the cells in a network interact with each other.

By classification, it is meant that given a set of L exemplars, each of which consists of M real-valued

features, each of the exemplars can be mapped to a finite number of elements within a chosen set of

classes, C. Each of the L elements of this set, C, has a label associated with it, hence the term class

label.

This research determines if it is possible to self-organize a network of machine learning elements that

can map multidimensional feature vectors, exemplars, from the feature space to their correct class

labels. The mapping is determined in the network training phase using supervised learning, but the

network’s cellular topology, and the details of the ML intercommunications via the GRN are chosen

in an unsupervised manner through the optimization of the cell’s genome.

3.3.1. Self-organization principles

There are two important parts of the low-level HeBIS algorithm. The first is the development of a

genetic regulatory network. This network allows communications between the processing cells to be

encoded by an artificial protein chemistry (metabolism) with concentrations that are diffused

throughout an environmental lattice. These protein concentrations are both time and space

dependent. This communication is primarily defined by the cell’s genome.

The second part, which is discussed here, is the small set of self-organizing principles that are

investigated in this research. This small set of principles is not designed to be all inclusive, but is

meant to form the basis of this research which will determine if useful classification can occur based

on

Taylor 74 of 249

09/28/10

• a small set of rules

• biologically-inspired communications

• simple machine learning elements.

This set of self-organizing principles provides feedback to the cell for the local optimization of the

cell’s classification activities and defines the interactions between cells in the network. This set

includes

• Protein analogs of mathematical statistical features

• Cellular fission

• Cellular death

These self-organizing principles are now discussed.

3.3.1.1. Protein analogs of statistical features

These statistical analogs are the building blocks of the principles that follow. These analogs have

already been described but are enumerated again for convenience:

• NumberProteinsInCell – The number of distinct proteins that are present in the cell’s cytoplasm.

• NumberProteinsInLocalEnviro – The number of distinct proteins that are present in the cell’s external environment.

• ConcentrationStandardDeviationLocalEnviro – The standard deviation of the concentrations of the proteins that are present in the local environment.

• ConcentrationMeanLocalEnviro – The average concentration of all the proteins that are located in the cell’s external environment at a specific location.

• ConcentrationMaxLocalEnviro – The maximum concentration of all the proteins that are located in the cell’s external environmental lattice at the location of the cell.

• ConcentrationMinLocalEnviro – The minimum concentration of all the proteins that are located in the cell’s external environmental lattice at the location of the cell.

3.3.1.2. Cellular fission and death

Next are the principles of cellular fission and cellular death. These are expressed by the network’s

GRN and the evolved gene-protein process. There is also an inherent lifetime to each of the cells.

Once a cell is born through either cellular creation or cellular fission (a cloning procedure), the cell

has a finite number of simulation intervals during which it may live and remain active. Once it

reaches that limit it dies, unless it has already initiated its own death because of genes that have been

expressed through the GRN.

Taylor 75 of 249

09/28/10

Death allows inefficient or unused ML elements to be pruned from the classification network.

Fission allows a useful ML element (with its trained knowledge) to be utilized as a complete module

in another location in the classification topology.

3.3.2. Particle swarm optimization

A particle swarm algorithm is used to optimize the entire cellular genome so that it can grow a self-

organizing classification network. The HeBIS PSO optimizes the genes in the cellular genome at the

protein level. That is, the four parameters that describe each protein each comprise a dimension of

the support of the multi-dimensional space for the proteins in the genome as well as the proteins that

define the features of the training and testing exemplars. Protein output concentrations as well as

switch protein concentration ranges also add to the dimensionality of the PSO. More information on

the implementation is included in the Appendices.

3.3.3. Training algorithm

A single training algorithm is examined in this research for simplicity. In this case, the network’s

training is controlled by a PSO. This improves the network’s performance on unseen examples by

providing a better estimate of classification generalization performance.

3.3.3.1. Training Algorithm: Presentation of training vectors and classes to the system

The training algorithm is diagrammed in Figure 15 and its flow is as follows. For purposes of

illustration, this example is based on a pool of 100 training exemplars. Scaling to datasets that have a

smaller or larger number of points is straightforward.

Training Algorithm

1. Choose Q, the size of the particle swarm and N, the number of network training iterations. Set the particle average fitness values to 0.

2. Create a PSO population of Q particles. Initialize the parameters of each particle randomly with regards to initial values and velocities. 3. Initialize a HeBIS environment for each of the Q particles. Each environment will consist of a single processing cell with a single genome. The location of the processing cell and the genome’s parameters are taken from the particle parameters from the PSO. 4. Choose an exemplar from the training pool and present it to each of the environments that is controlled by one of the Q particles in the PSO.

Taylor 76 of 249

09/28/10

5. Begin the protein diffusion and self-organization process for each of the environments. Allow each simulation to continue for a fixed number of iterations or until all protein concentrations equal zero. 6. Determine the fitness of the genome associated with the ith PSO particle and record this. This fitness is determined by comparing the desired label of the exemplar to the label produced by the network. 7. Repeat steps 3, 4,5, and 6 until all the exemplars have been presented for training to the HeBIS network. 8. Average the fitness values of all the training data for each genome across all the training points presented to the network during this iteration of the training algorithm. 9. Update each of the Q PSO particles with its respective, averaged fitness. 10. Repeat Steps 2-9 for N iterations. 11. Save the best PSO-evolved genome.

Figure 15. Schematic of the training algorithm.

Taylor 77 of 249

09/28/10

Cross-validation (CV) can be used in conjunction with the training procedure to improve the

modeling HeBIS’ performance by providing a better estimate of the classifier’s generalization ability

[110].

3.4. Output coding

The classification result for a particular test case is a complex behavior that is evolved by the training

algorithm. The result is constrained to be a unique protein which maps class membership to a class

protein’s normalized concentration value 2 [0,1]. The concentration mapping is defined in the same

manner as for the input class mapping. To reiterate, class membership is defined through the class

u subinterval definition 2 [0,1]. The difference between the output and input class mapping is that

the input mapping is manually constrained to sub-intervals of [0,1], but the output mapping is a result

of the evolved network and its GRN. This mapping is evaluated and validated when the network’s

classification is tested during the evaluation phase of the training procedure.

3.5. Post processing of classification results

The detected output protein provides information about the classification of the test example.

Denormalization of the output protein concentration is accomplished simply by mapping the peak

concentration to the closest class centroid that has been previously defined for the input

normalization. For example, in a 2-class problem, the class centroids are defined as C0 = 0.25 and

C1 = 0.75. If the peak output protein concentration level is C = 0.20, then | C – C0 | < | C – C1 | and

the output class is found to be C0.

Since this is, at its core, a ranking classifier, it is possible to use data from the cross-validation

procedure to form a Receiver Operating Characteristic curve (ROC) that could be used to actively set

the output threshold so as to improve classification accuracy. This output threshold is the number in

the interval, [0,1], that can be used to set the classifier’s operating point- the estimate of the expected

number of correctly classified exemplars given the user-defined level of acceptable false positives.

Taylor 78 of 249

09/28/10

4. Simulations and analyses

4.1. Simulation limits

HeBIS with its diffusion of artificial proteins in a discrete time/space simulation, is currently a

processing-intensive application. Since the purpose of this research is to begin the exploration of this

classification paradigm, the number of testing and training exemplars was limited so as to provide

reasonable simulation times. Even so, with 20 or 40 multispectral center pixels and their surrounding

geographic samples used for training and cross-validation testing, simulation times were found to run

between 1.5 and 10 days on a modern multiprocessor computer. This depended on such parameters

as the number of particles in the PSO, the dimensionality of the tested HeBIS implementation (i.e.,

number of genes, number of features), and the size of the geographic region included in the

classification training process. Figure 16 and Figure 17 provide a strawman diagram for the

surrounding data cube that is associated with a specific pixel.

5.

Figure 16. Training/Testing pixel and its relationship to its surrounding geographic pixels.

Taylor 79 of 249

09/28/10

Figure 17. Training/Testing pixel and the surrounding multispectral information.

Initially limiting the exploration of the system in this manner allowed the creation of a base of

experience which will be used in future and more detailed HeBIS research.

Taylor 80 of 249

09/28/10

4.2. General methodology Hybridization Background




Robustness Analyses

4.2.1. Remote sensing cloud/no-cloud problem

4.2.1.1. Description

A remote sensing problem is used as a simplified benchmark for comparison and experiments in this

dissertation. We have constructed a problem that consists of learning whether a pixel in a remotely

sensed image is a no-cloud pixel or if it is a cloudy pixel. The baseline determination of whether the

pixel is considered to be cloud (Class C0) or no-cloud (Class C1) is empirically determined from the

data after it is processed from the raw radiance data. Thus, HeBIS is evaluated on its ability to learn

this empirical mapping.

Background on this dataset and its source are now discussed.

4.2.1.2. Sensor and datasets

The data used in this dissertation were acquired from NASA’s Moderate Resolution Imaging

Spectroradiometer (MODIS) on the Aqua satellite platform. Aqua’s orbital information is listed in

Table 2.

Table 2. Orbital Information for NASA's Aqua Satellite.

Parameter Value

Orbit 705 km circular

Equator Crossing 1:30 pm local

Orbit Type Ascending

Inclination Near-polar

Synchronicity Sun-synchronous

The MODIS sensor is capable of capturing data over a ground swath of 2 330 km crosstrack with a 10

km along-track swath measured at earth nadir. It is a multispectral optical sensor with 36 spectral

bands available with the best ground resolutions listed in Table 3 for the spectral bands.

Taylor 81 of 249

09/28/10

Table 3. Nominal Resolutions for the MODIS Sensor.

Band Resolution

1-2 250 m

3-7 500 m

8-36 1000 m

The full list of the available multispectral data available from MODIS is available in Table 4.

Table 4. 36 Bands of Multispectral Data from MODIS.

Band # Bandwidth [nm] Spectral Radiance [W]

1 620-670 21.8

2 841-876 24.7

3 459-479 35.3

4 545-565 29.0

5 1230-1250 5.4

6 1628-1652 7.3

7 2105-2155 1.0

8 405-420 44.9

9 438-448 41.9

10 483-493 32.1

11 526-536 27.9

12 546-556 21.0

13 662-672 9.5

14 673-683 8.7

15 743-753 10.2

16 862-877 6.2

17 890-920 10.0

18 931-941 3.6

19 915-965 15.0

20 3660-3840 0.45 (300K)

21 3929-3989 2.38 (335 K)

22 3929-3989 0.67 (300K)

23 4020-4080 0.78 (300 K)

24 4433-4498 0.17 (250 K)

25 4482-4549 0.59 (275 K)

26 1360-1390 6.0

27 6535-6895 1.16 (240 K)

28 7175-7475 2.18 (250 K)

29 8400-8700 9.58 (300 K)

30 9580-9880 3.69 (250 K)

31 10780-11280 9.55 (300 K)

32 11770-12270 8.94 (300 K)

33 13185-13485 4.52 (260 K)

34 13485-13785 3.76 (250 K)

35 13785-14085 3.11 (240 K)

36 14085-14385 2.08 (220 K)

Taylor 82 of 249

09/28/10

The HeBIS research dataset was created from top-of-the-atmosphere (TOA) radiances that were

derived from modified ocean Level 2 data products from MODIS [116]. Level 2 datasets consist of

calibrated radiances with geolocation references in addition to ancillary information about data

quality and error estimates. A definition of these NASA/CEOS data levels is contained in Table 5.

Table 5. NASA/CEOS Dataset Level Definition

Level Description

Level 1A Raw radiance counts from all bands Spacecraft and instrument telemetry for geolocation, calibration, and data processing

Level 1B Radiances that are calibrated and geolocated from the position of the sensor’s aperture Calibration data, quality flags, error estimates, Generated from Level 1A data

Level 2 Geophysical values for each pixel Values derived via Level 1A raw sensor information, atmospheric corrections, and bio-optical algorithms

Level 3 Generated from multiple Level 2 datasets All data from the following temporal periods:

1 day 8 days 1 month 1 year

Data are binned into non-overlapping bins across the earth’s ocean and land surfaces 9 km x 9 km bins 4 km x 4km bins

Since these data are nominally stored in hierarchical data format (HDF) files, a procedure was

generated to convert these files to a format accessible by HeBIS. This data conversion consisted of

acquiring the data from the NASA data archive, extracting the Level 2 data from the 36 bands, and

then converting the data for each pixel and band to floating point numbers. These 4-byte floating

points are then stored in flat files for easier customized access than the standard HDF format.

Although 36 multispectral bands are available, we only use those bands that are typically associated

with ocean remote sensing. These 17 bands are listed in Table 6.

Table 6. 17 Bands from MODIS for A2002193183000 LAC_x_NIR.

Band Name

Cloud Albedo

Lt 412

Lt 443

Lt 469

Lt 488

Lt 531

Lt 551

L5 555

L5 645

Taylor 83 of 249

09/28/10

Lt 667

Lt 678

Lt 748

Lt 859

Lt 869

Lt 1240

Lt 1640

Lt 2130

These intermediate data are used to create a cloud/no-cloud mask and then the final dataset of 15

channels is used for ingestion into HeBIS. The 15 final bands are included in Table 7.

Table 7. Multispectral bands used from MODIS [111]. HeBIS Band Index Primary Use MODIS Band Bandwidth [nm]

1 Land/Cloud/Aerosols Boundaries

1 620-670

2 2 841-876

3 Land/Cloud/Aerosols Properties 3 459-479

4 4 545-565

5 5 1230-1250

6 7 2105-2155

7 Ocean Color/Phytoplankton/Biogeochemistry 8 405-420

8 9 438-448

9 10 483-493

10 11 526-536

11 12 546-556

12 13 662-672

13 14 673-783

14 15 743-753

15 16 862-877

The HeBIS dataset is a 15-band dataset from MODIS that also includes a cloud/no-cloud pixel

classification created during Level-2 TOA processing. These bands consist of optical and short-wave

infrared radiances. The dataset is A2002193183000_cloud_albedo. It was acquired in 2002 on the

193rd day of the year over the Chesapeake Bay. The Bay is located in the Mid-Atlantic region of the

east coast of the United States. It consists of a processed swath that is 234 pixels wide and 430 pixels

long. Figure 18 shows the pseudocolor image for the visible bands, Figure 19 shows the cloud/no-

cloud ground truth, and Figure 20 shows the land mask for the dataset. A land mask was applied over

the region such that only observations over water (specifically the Bay) were included in the dataset.

Taylor

Figure 18. Pseudocolor image for A2002193183000 dataset.

black corresponds to water, and green and brown re

image for A2002193183000 dataset. Grey and white colors correspond to cloud pixels,

black corresponds to water, and green and brown refer to land pixels.

84 of 249

Grey and white colors correspond to cloud pixels,

Taylor

Figure 19. Ground truth (cloud/no

white pixels reference clouds and black corresponds to water.

. Ground truth (cloud/no-cloud) for A2002193183000 dataset. Red corresponds to land pixels whereas

white pixels reference clouds and black corresponds to water.

85 of 249

Red corresponds to land pixels whereas

Taylor

Figure 20. Land mask (cloud/no

corresponds to water pixels.

A cloud/no-cloud mask was used which

corrected for molecular atmospheric

is classified as a cloudy pixel if

Otherwise, if the approximated surface reflectance is less than or equal to 0.027, then the pixel is

classified as a no-cloud pixel.

to be 2105 nm; ρW

λ` a

is the approximated surface reflectance;

atmosphere reflectance; t g

surface to sensor; T is the direct transmittance of the atmosphere through Rayleigh and aerosols

scattering, from the surface to the sensor;

from the ocean’s surface; ρ

(cloud/no-cloud) for A2002193183000 dataset. Red corresponds to land pixels and black

was used which is based on MODIS band 16 reflectance

corrected for molecular atmospheric scattering and other scattering components

pixel if the approximated surface reflectance is greater than 0.027

ρW

λ` a

=ρ

tλ` a

t g λ` a

fffffffffffffffffff@T Aρ

gλ` a

@ρr

λ` a

h

j

i

k

1

td

λ` a

ffffffffffffffffff .

roximated surface reflectance is less than or equal to 0.027, then the pixel is

cloud pixel. In Equation (21), for MODIS data, λ is the wav

is the approximated surface reflectance; ρt

λ` a

λ` a

is the transmittance of the atmospheric gases, sun to surface and

is the direct transmittance of the atmosphere through Rayleigh and aerosols

scattering, from the surface to the sensor; ρg

λ` a

is the glint reflectance due to specular sun reflection

ρr

λ` a

is reflectance originated from molecular (Rayleigh) scattering; and

86 of 249

Red corresponds to land pixels and black

band 16 reflectance that has been

and other scattering components. A pixel over water

the approximated surface reflectance is greater than 0.027 [112] as in

(21)

roximated surface reflectance is less than or equal to 0.027, then the pixel is

is the wavelength and is defined a

is the total top-of-the-

is the transmittance of the atmospheric gases, sun to surface and

is the direct transmittance of the atmosphere through Rayleigh and aerosols

is the glint reflectance due to specular sun reflection

is reflectance originated from molecular (Rayleigh) scattering; and

Taylor 87 of 249

09/28/10

td

λ` a

is the total transmittance of the atmosphere through Rayleigh scattering from the surface to the

sensor.

Because the band data are exponentially distributed, log-normalization is used within HeBIS so that

the data may be more easily processed. This normalization process consists of applying the natural

log function to the data within each band and determining the minimum and maximum values of each

of these transformed bands. These band minima and maxima are then used to scale the range of the

normalized values for each band such that the resulting range 2 [0.0, 1.0].

Figure 21 shows a cloud/no-cloud class breakdown according to wavelength in the 15 bands that are

included in the dataset. Figure 22 through Figure 36 are detailed zooms of this plot. These are

scatter plots in which the cloud class (C0) is represented as a “1” on the abscissa and the no-cloud

class (C1) is plotted as a “-1”. The ordinate values are the log-normalized values of band magnitudes

for each of the pixels in the dataset.

Figure 21. Cloud/no-cloud class breakdown according to specific wavelength-band feature.

Taylor 88 of 249

09/28/10

Figure 22. 412 nm cloud/no-cloud scatter plot with cloud (C0) pixels represented as 1 and no-cloud (C0) pixels

represented as -1 on the abscissa. Magnitudes are log-normalized and biased and scaled to fall within the range

[0, 1].



[0, 1].

Taylor 89 of 249

09/28/10



[0, 1].



[0, 1].



[0, 1].

Taylor 90 of 249

09/28/10



[0, 1].



[0, 1].

Taylor 91 of 249

09/28/10



[0, 1].



[0, 1].



[0, 1].

Taylor 92 of 249

09/28/10



[0, 1].



[0, 1].

Taylor 93 of 249

09/28/10



[0, 1].



[0, 1].

Taylor 94 of 249

09/28/10



[0, 1].

Taylor 95 of 249

09/28/10

4.3. The construction of simple genetic regulatory networks Hybridization Background




Robustness Analyses

Now, we will begin the research experiments by examining the construction and efficacy of simple

genetic regulatory networks.

4.3.1. Introduction/Methodology

This section deals with the construction of simple genetic regulatory networks. Its purpose is the

determination as to whether an artificial GRN is being created within an experimental setup which

isolates a portion of the HeBIS operational environment. This portion is dedicated to management of

the protein environment.

We will examine HeBIS in both the context of the underpinnings of a GRN as well as examine

simple GRN processes in more detail. The underpinnings include diffusion of an artificial protein

through an artificial matrix in addition to a study of activation and inhibition of proteins. These

proteins are controlled through the protein switches which are contained within the genes which can

code for regulatory, environmental and output proteins. Following this, simple GRN processes will

be examined by characterizing gene activation and inhibition of selected artificial genomes.

Within this experimental setup, only genome complexity is examined. We want to determine if

complexity can be introduced through appropriate manipulation of our logic/control processor of

choice, the artificial genome. For our purposes, complexity is defined as follows:

• The number of proteins in the environmental matrix changes and becomes greater than the number contained in the matrix when a zero-length genome is used.

• Patterns of gene activation exist within the genome over the course of HeBIS simulations for non-zero-length genomes.

These gene-level patterns of activation can be used potentially for the stages of classification which

will be studied in later sections.

Taylor 96 of 249

09/28/10

Our baselines is the zero-length genome, i.e. a genome which has no explicit genes that can be sued

for interaction with environmental proteins.

4.3.2. Experiments 1 and 2 – Proteins

The protein simulations will only examine the proteins in the base GRN and they will not deal with

any intracellular processes. Thus, for this analysis, the intracellular SOFM kernel remains

deactivated.

Two experiments are conducted:

• Experiment 1: Baseline protein simulations, and

• Experiment 2: Baseline multi-gene protein simulations.

4.3.2.1. Setup

Experiment 1

In this test, 15 proteins corresponding to normalized multispectral channel information from the

cloud/no-cloud dataset are injected into the protein environment. The number of genes in the genome

is set to 0 genes and we will determine the number of proteins that are present in the environmental

matrix. This variable will be sampled periodically as the protein diffusion simulation progresses.

The simulations parameters for this experiment are listed in Table 8.

Table 8. Simulation Parameters for Experiment 1.

Simulation Parameter Value

# Trials 100

# Genes 0

Protein diffusion rate 0.1

Minimum protein environmental concentration 0.01

Maximum number of network diffusion iterations 1000

PSO breedings 10

Topology and size of artificial protein environment 3x3x17

Protein correlation (affinity) 0.9

Reaction probability 0.0 %

# PSO particles 200

This baseline will aid in the determination as to whether a GRN and associated complexity can be

introduced into the system by an appropriate genome. The PSO Breeding parameter is set to a non-

zero value so as to maintain constancy throughout the follow-on simulations.

Taylor 97 of 249

09/28/10

Experiment 2

This experiment also injects 15 normalized proteins that correspond to cloud/no-cloud spectral

information. However, the activated genome is one which possesses three (3) genes instead of zero

(0) genes as in Experiment 1. The number of proteins in the environmental matrix is tracked

periodically during the diffusion simulation. Simulation parameters for this experiment are listed in

Table 9.



# Trials 100

# Genes 3




PSO breedings 10




# PSO particles 200

Protein correlation refers to the affinity that two proteins have for each other as described in detail in

the algorithm section of this document. Higher values of correlation [0,1] imply that the proteins

only match if they are very similar to each other.

It will be determined in Experiment 2 if the HeBIS PSO interacting with a small multi-gene genome

can induce complexity through protein diffusion patterns and protein creation that are different from

the results obtained through the Experiment 1 baseline.

4.3.2.2. Experiments 1 and 2 results and discussion

The proteins that are used as input in these initial two experiments consist only of the artificial

proteins that have been mapped from the 15 spectral input bands for a test case of 18 pixels- 9 pairs

of C0 and C1 case exemplars. These input spectral proteins are assigned random protein parameters-

essentially the 4 bytes that describe the unique coding for the artificial protein. Since only 254 values

out of the possible 256 potential values for each description byte are used, this provides a potential

space of 64516 (2543) proteins which is somewhat less than 65536 that would be available if all 256

values for each byte were used. Each trial is conducted with a randomly chosen set of parameters

from this set.

Taylor 98 of 249

09/28/10

Figure 37 and Figure 38 show the average number of proteins and corresponding standard deviations

that are present in the environment for both the 0-gene (Experiment 1) and 3-gene (Experiment 2)

cases at the end of each iteration of the simulation.

Figure 37. Baseline number of proteins in environmental lattice for zero-length genome in Experiment 1. Vertical

bars correspond to the standard deviation of the sample mean.

Figure 38. Baseline number of proteins in environmental lattice for the 3-gene genome in Experiment 2. Vertical


Taylor 99 of 249

09/28/10

This information is derived by averaging the number of environmental proteins at each iteration

across all 200 particles of the particle swarm as well as across all 18 pixels that are used for the

specific trial. There is a small standard deviation among the trials after the end of iteration 0- the step

at which the 15 initial proteins have been injected into the environmental lattice. Higher standard

deviations are seen during the middle portion of the sequence of iterations as the proteins with lower

normalized concentrations are removed from the matrix through diffusion. The abrupt drop in the

number of proteins in the matrix at the end of the final iteration occurs when the protein

concentrations fall below the 0.01 minimum normalized concentration level. At that point, all the

proteins have concentrations less than this threshold and are subsequently removed from the matrix.

Table 10 summarizes the statistics that describe the diffusion characteristics for these initial two

experiments.

Table 10. Statistical Summary for Experiments 1 and 2.

Experiment 1 1 2 2

Data iteration # proteins iteration # proteins

Reaction 0 0 0 0

Max. 4.800e+001 1.497e+001 4.700e+001 1.495e+001 Mean 2.400e+001 9.844e+000 2.350e+001 9.898e+000 Median 2.400e+001 1.028e+001 2.350e+001 1.049e+001 Min. 0.000e+000 3.506e+000 0.000e+000 4.087e+000 Mode 0.000e+000 4.563e+000 0.000e+000 4.588e+000 Range 4.800e+001 1.146e+001 4.700e+001 1.086e+001 Standard deviation 1.4289e+001 4.290e+000 1.400e+001 4.204e+000

In Experiment 1, an average of 9.84 proteins remained active during the maximum 48 iterations that

occurred over the range of trials. Similarly, an average of 9.90 proteins had concentrations above the

minimum concentration threshold during a maximum of 47 iterations for the second experiment.

This slight difference is due to the fact that 200 different randomized 18-point training datasets were

used for the two experiments.

Taylor 100 of 249

09/28/10

A cubic equation was fitted to the averaged protein data for each of the two experiments. Norms of

the residuals from the fitted curves were 2.7599 and 2.6116, respectively. These residuals indicate

the “goodness of fit” of the curve and the average deviations of the data from this curve.

Another highlight is that the average number of proteins in the lattice as a function of iteration

number is dependent on the diffusion parameter from (16). 0.1 was used for this parameter and was

chosen to determine whether the system was working and not necessarily as an optimum. A diffusion

parameter greater than 0.1- the value used in these experiments- would maintain a number of

proteins in the environment lower than for these baselines. Conversely, a diffusion parameter with a

value less than the baseline would cause more proteins to remain in the environmental matrix for a

longer period of time. The effects on HeBIS operation by changing the diffusion parameter will be

examined in more detail later in this dissertation in Experiment 14.

4.3.2.3. Experiments 1 and 2 conclusions

It can be concluded that HeBIS’ protein diffusion works as it has been shown that artificial proteins

diffuse through the artificial environment. Figure 39 shows that the diffusion characteristics of the

protein environments are essentially the same for the 0-gene and 3-gene experiments.

Figure 39. Number of proteins in environment compared between the baseline genome from Experiment 1 and the

multi-gene genome from Experiment 2. Vertical bars correspond to the standard deviation of the sample mean.

Taylor 101 of 249

09/28/10

This was an expected result because the more complex parts of the GRN were deactivated for these

two experiments. An example of protein diffusion over time is presented in the Appendix in Figure

116 through Figure 120 , beginning on page 235.

4.3.3. Experiments 3 and 4 - Protein Chemistry

The next experimental phase examines the addition of complexity into the protein matrix and whether

this changes the complexity of the evolved genetic regulatory network. In this case, a straightforward

protein chemistry is introduced into the environment. This chemistry has been discussed prior to this

in 3.1.2.2 (page 64), the section that treats HeBIS protein communications in detail. The import

aspect of this chemistry is the probability of reaction and how it affects the creation of new proteins

both in a zero-gene instance and a multi-gene case. This probability of reaction can potentially be

thought of as a level of mutation in the GRN. This would roughly correspond to mutation in the

context of optimization based on genetic algorithms.

Two experiments are conducted:

• Experiment 3: Protein simulation with a zero-length genome and non-zero reaction probabilities.

• Experiment 4: Protein simulation with a multi-gene genome and non-zero reaction probabilities.

4.3.3.1. Setup

Experiment 3

This study is similar to Experiment 1 except for there being a non-zero protein reaction probability.

The number of proteins in the environmental matrix is also sampled periodically during the

simulation. 75 trials are split equally among three reaction probabilities: 0.1%, 1%, and 10%.

Simulation parameters for this experiment are listed in Table 11.



# Trials 75

# Genes 0




PSO breedings 10

Taylor 102 of 249

09/28/10



Reaction probability 0.1 %, 1 %, 10 %

# PSO particles 200

Protein creation and diffusion complexity are examined in light of the chosen reaction probabilities.

Experiment 4

This is similar to Experiment 2, however now we examine a non-zero reaction probability with a

multi-gene genome. As in Experiment 3, the number of environmental proteins is sampled

periodically and compared. Simulation parameters are listed in Table 12.



# Trials 75

# Genes 3




PSO breedings 10



Reaction probability 0.1 %, 1 %, 10 %

# PSO particles 200

With these data, we will examine whether the number of created proteins is different than those in the

Experiments in Section 4.3.2. Protein diffusion and creation patters are also briefly examined.

4.3.3.2. Experiments 3 and 4 results and discussion

Overall, there is a significant increase in the mean number of proteins in the environmental matrix for

experiments 3 and 4 when compared to that in experiments 1 and 2. The initial two experiments are

considered to be the baseline with 0% probability of protein chemistry occurring. As a reminder,

protein chemistry describes the potential creation and destruction of environmental proteins based

purely on “chemical” reactions between the proteins. This is different, but is also a component of the

creation and destruction of proteins based on an evolved GRN. Differences in protein

creation/destruction for experiments 3 and 4 are linked to the probability of a chemical reaction

Taylor 103 of 249

09/28/10

occurring over the simulation run. Cubic parameterized curves fitted to the average protein data is

also used as an additional proxy for the complexity in protein interactions that occurs in light of an

underlying probabilistic chemistry. The residuals associated with these fitted cubic equations and the

data indicate an overall error that is used as a type of texture to further describe the complexity

associated with protein interaction. The similar responses for the 0-gene and 3-gene trials for

experiments 3 and 4 were expected since an evolved GRN was not created or activated for these

trials.

Figure 40 and Figure 41 point out the averaged number of proteins in the environmental lattice for

the 0-gene and 3-gene genomes, respectively.

Figure 40. Number of proteins in environmental lattice for a 0-gene genome vs. cellular iteration for reaction

probabilities of 0 %, 0.1 %, 1 %, and 10 % with error bars removed for clarity.

Taylor 104 of 249

09/28/10

Figure 41. Number of proteins in environmental lattice for a 3-gene genome vs. cellular iteration for reaction

probabilities of 0 %, 0.1 %, 1 %, and 10 % with error bars removed for clarity.

Both show a significant change in the case in which the reaction probability is at its highest- 10%.

The mean number of active proteins for the baseline 0% (experiments 1 and 2), 0.1%, and 1.0%

reaction probabilities show a dependence on protein production (beyond the initial 15 multispectral

proteins injected at iteration 0) that increases slightly as the reaction probability increases.

The mean number of proteins in the lattice ranges from 9.8437 (0% probability case) to 14.4816

(10% probability case) for the 0-gene genome and 9.8983 (0% probability case) to 15.3183% (10%

probability case) for the 3-gene genome. Figure 42, Figure 43, Figure 44, Figure 45, Figure 46 and

Figure 47 highlight these results in more detail in addition to the high standard deviations relative to

the mean for the number of active environmental proteins.

Taylor 105 of 249

09/28/10

Figure 42. Number of proteins in environmental lattice for a 0-gene genome vs. cellular iteration for a reaction

probability of 0.1 %. Vertical bars correspond to the standard deviation of the sample mean.



Taylor 106 of 249

09/28/10


probability of 10 %. Vertical bars correspond to the standard deviation of the sample mean.



Taylor 107 of 249

09/28/10




probability of 10 %. Vertical bars correspond to the standard deviation of the sample mean.

Taylor 108 of 249

09/28/10

The 0.1% and 1% reactions (Figure 42, Figure 43, Figure 45 and Figure 46) did not add a large number

of extra proteins into the lattice compared to the 0% baselines in Figure 37 and Figure 38. However,

the data for the 10% reaction probability show a large increase in the average number of active

proteins before the protein concentrations drop to the minimum supported during the simulation.

This is apparent relative to the 0% baseline as well as the 0.1 % and 1.0% cases for both genomes.

On average, the 0-gene instance at 10% probability experienced 70 iterations whereas the 3-gene

instance at the same probability of reaction went through 76 iterations. Table 13 and Table 14

summarize other differences in the mean number of experimental iterations in addition to the mean

numbers of proteins present in the environmental lattice.

Table 13. Statistical Summary for Experiment 3.

Experiment 3 3 3 3 3 3

Data Type iteration #proteins iteration #proteins iteration #proteins

Reaction 10 10 1 1 0.1 0.1

Max. 6.900e+001 2.456e+001 4.700e+001 1.534e+001 4.700e+001 1.496e+001

Mean 3.450e+001 1.448e+001 2.350e+001 1.035e+001 2.350e+001 1.041e+001

Median 3.450e+001 1.526e+001 2.3500000e+001 1.113e+001 2.350e+001 1.104e+001

Min. 0.000e+000 2.000e+000 0.000e+000 3.438e+000 0.000e+000 4.438e+000

Mode 0.000e+000 3.800e+000 0.000e+000 1.496e+001 0.000e+000 4.563e+000

Range 6.900e+001 2.256e+001 4.700e+001 1.190e+001 4.700e+001 1.053e+001

Standard deviation 2.035e+001 7.030e+000 1.400e+001 4.348e+000 1.400e+001 3.845e+000

Table 14. Statistical Summary for Experiment 4.

Experiment 4 4 4 4 4 4

Data Type iteration #proteins iteration #proteins iteration #proteins

Reaction 10 10 1 1 0.1 0.1

Max. 7.600e+001 2.582e+001 4.700e+001 1.506e+001 4.800e+001 1.498e+001

Mean 3.800e+001 1.532e+001 2.350e+001 1.024e+001 2.400e+001 1.005e+001

Median 3.800e+001 1.768e+001 2.350e+001 1.181e+001 2.400e+001 1.049e+001

Min. 0.000e+000 1.000e+000 0.000e+000 3.000e+000 0.000e+000 4.778e+000

Mode 0.000e+000 1.000e+000 0.000e+000 3.556e+000 0.000e+000 5.182e+000

Range 7.600e+001 2.482e+001 4.700e+001 1.206e+001 4.800e+001 1.020e+001

Standard deviation 2.237e+001 7.711e+000 1.400e+001 4.637e+000 1.429e+001 3.677e+000

Taylor 109 of 249

09/28/10

Residual norm (texture) for the fitted cubic equations are presented in Figure 48, Figure 49, Figure

50, Figure 51, Figure 52 and Figure 53.

Figure 48. Fitting and statistical information for a 0-gene genome in an environment with a reaction probability of

0.1 %. Vertical bars on the upper chart correspond to the standard deviation of the sample mean.

Taylor 110 of 249

09/28/10




10%. Vertical bars on the upper chart correspond to the standard deviation of the sample mean.

Taylor 111 of 249

09/28/10



Taylor 112 of 249

09/28/10




10 %. Vertical bars on the upper chart correspond to the standard deviation of the sample mean.

Taylor 113 of 249

09/28/10

For the 0-gene genome, residuals increased from 2.618 through 3.635 and 13.9773 as the reaction

probability increased from 0.1 % to 10 %. This compares with 2.7599 for the 0% baseline from

experiment 1. Residuals for the 3-gene genome showed an increase from 3.0707 through 3.5035 and

14.9639 as the corresponding reaction probability increased from 0.1 % to 10%. The 10 % baseline

for 3 genes had a residual of 2.6116.

4.3.3.3. Experiments 3 and 4 conclusions

Comparing the deactivated protein chemistries in experiments 1 and 2 with the corresponding

experiments with the chemistries activated, it is seen that more proteins are available over time within

the lattice with nonzero reaction probabilities. The average number of iterations before the available

proteins fall below the minimum concentration threshold remained about the same for the 0 %, 0.1 %,

and 1.0 % cases, regardless of the number of genes present in the genome. However, with 10 %

reaction probability, the average number of lattice iterations increased significantly as can be seen in

Figure 54.

Figure 54. Comparison between 0-gene and 3-gene genomes for varying environmental reaction probabilities.

Error bars removed for clarity.

The mean number of active proteins as well as the maximum number of proteins tended to increase

for both genome cases as the probability of protein reaction increased. For the 0-gene instance (Table

10 and Table 13), the mean increased from 9.8437 (0 %) to 14.4818 (10 %) and the maximum

Taylor 114 of 249

09/28/10

number of proteins in the lattice increased from 14.965 ( 0%) to 24.56 (10 %). For the 3-gene

instance (Table 10 and Table 14), the mean increased from 9.8983 (0 %) to 15.3183 (10 %) and the

maximum number of proteins in the lattice increased from 14.945 (0 %) to 25.82 (10 %). Both

results show a significant change in environmental behavior when protein chemistry is activated.

Complexity of protein production also shows up as ripples in the averaged protein data in the

corresponding plots (Figure 42, Figure 43, Figure 44, Figure 45, Figure 46 and Figure 47) for the 0.1

%, 1.0 %, and 10 % activated protein chemistry experiments. Additionally, the levels of residual

norm increases as the reaction probabilities increase- as shown in the fitted cubic equations. This

indicate a complex change in the average number of proteins and a change in texture.

The additional proteins present in the matrix (compared to the 0 % baselines) potentially mean that

there are more possibilities for exploring the search space of solutions associated with the evolved

GRN. Additionally, the “mutations” produced by the protein reactions can potentially be used to

improve exploration within the protein parameter space for classification. These issues will be

explored in more detail in Experiment 14.

4.3.4. Experiment 5 - Gene activation

Genes in HeBIS can be thought of as being modules of logic and control that are modified by the

presence of evolved proteins that match the switch region in a specified gene. Investigation into the

workings of these genes in an intracellular context will be examined in more detail in a later section

of this dissertation. For now, it needs to be determined whether and to what extent gene activation

and inhibition can occur. Can gene activation/inhibition patterns occur within the HeBIS GRN?

For this examination, a single experiment will be conducted:

• Experiment 5: Gene Activation/Inhibition Complexity

4.3.4.1. Setup

Experiment 5

Periodic sampling of the genes’ expression in time occurs for a set of genomes. It is determined if

these genes have been activated or subsequently inhibited. These activations/inhibitions of the genes

are caused by the proteins that are instantiated in the protein diffusion matrix. The genes are allowed

Taylor 115 of 249

09/28/10

to activate and inhibit the functionality of evolved GRN proteins only for environmental protein

types. Regulatory proteins are not considered because they are constrained to intracellular

functionality that is not examined at this time. Cellular cloning is allowed at this stage. Other

simulation parameters for this experiment are listed in Table 15.



# Trials 3

# Genes (environmental) 3, 10, 40




PSO breedings 10




# PSO particles 200

Several genomes are examined for each of the trials. They are selected to highlight certain

characteristics of these simple, evolved GRNs. The fitness function used in this case is the standard

HeBIS fitness function which is discussed in detail in 4.4.2 on page 129.

4.3.4.2. Experiment 5 results and discussion

Examples of complex GRN behaviors are highlighted in this section by using genome activation

maps that are taken directly from the PSO at specified cellular iterations. These genome activation

maps summarize the activity of the evolved GRN in a graphical format. The before-mentioned

fitness function is based on a single protein output. This fitness function is use to examine whether

activation/inhibition of genes within evolved GRNs works., but not necessarily how well these

actions work.

An example of a gene activation map is in Figure 55. With a gene activation map, the response of the

entire genome (collection of genes) within a cell is summarized according to environmental and

cellular stimuli. Examining this image in more detail, from left to right, the vertical columns that are

separated by a thin black line represent a particular gene number. From top to bottom, the rows that

are separated by the black lines represent the activity level of the genes at that particular time iteration

of time within the simulation. Genes are indexed beginning with 0, for example Gene0, on the left-

hand side of the image and they increment as the map is traversed towards the right-hand side to a

Taylor

maximum of GeneN-1. Time indices begin at the

downwards. A white color in the intersection of gene number and

that that specific gene did not activate during that time inte

did activate during the interval. In this example, gene 1 (the second column) activated in iteration 0

and continued activating through iteration 20.

The first 9 genes are infrastructure genes within the genome and can also be expressed during this

experiment. For example, G

Gene1 is the output gene associated with Class

numbered beginning with Gene

genome refers to a genome that contains M environmental genes with a total of M+3 genes.

Genomes are numbered according to their

In Figure 56, Gene1 is activated for both imaged

and right-hand sides of the image. Activa

iteration 20 whereas activation of Gene1 in Genome

ime indices begin at the top with 0 and increment as the map is traversed

color in the intersection of gene number and cellular

that that specific gene did not activate during that time interval and a red color indicates that the gene


and continued activating through iteration 20.

Figure 55. Example gene activation map.


For example, Gene0, is the output gene associated with Class 0, C0, th

is the output gene associated with Class 1, C1, the no-cloud class. Environmental genes are

numbered beginning with Gene9. To simplify the nomenclature, for this experiment an M


according to their particle index in the PSO used in this experiment.

is activated for both imaged 3-gene genomes, 93 and 146, respectively on the left

hand sides of the image. Activation of Genome93 begins in cellular iteration 0 and ends in

iteration 20 whereas activation of Gene1 in Genome146 begins in iteration 6 and ends in iteration 12.

116 of 249

top with 0 and increment as the map is traversed

cellular iteration index indicates

rval and a red color indicates that the gene



Class 0, C0, the cloud class and

cloud class. Environmental genes are

To simplify the nomenclature, for this experiment an M-gene


index in the PSO used in this experiment.

genomes, 93 and 146, respectively on the left

begins in cellular iteration 0 and ends in

begins in iteration 6 and ends in iteration 12.

Taylor

Figure 56. 3-gene genome activation

respectively.

Examining Genome93 it is seen that the protein concentrations which activate the switch for the gene

evolved such that they activate once the test datum is injected into the environmental ma

of artificial proteins. Since protein chemistry is inactive in this example, Genome

and switching on due to an intermediate range of concentrations for the valid protein in the matrix.

This highlights the fact that diffu

progresses through the simulation.

Figure 57 highlights differing genetic responses between

being applied to the genomes as an

(a)

Figure 57. 3-gene genome activation vs.

(a) (b)

activation vs. iteration for C0 (178) using genomes 93 and 146

it is seen that the protein concentrations which activate the switch for the gene

evolved such that they activate once the test datum is injected into the environmental ma

of artificial proteins. Since protein chemistry is inactive in this example, Genome


This highlights the fact that diffusion is constantly decreasing protein concentrations as time

progresses through the simulation.

highlights differing genetic responses between 3-gene genomes, 9 and 147

he genomes as an exemplar from C0.

(a) (b)

activation vs. iteration for C0 (142) using genomes 9 and 147, respectively

117 of 249

93 and 146 in (a) and (b),

it is seen that the protein concentrations which activate the switch for the gene

evolved such that they activate once the test datum is injected into the environmental matrix as a set

of artificial proteins. Since protein chemistry is inactive in this example, Genome146 is thus reacting


sion is constantly decreasing protein concentrations as time

9 and 147, with pixel 142

(142) using genomes 9 and 147, respectively, in (a) and (b).

Taylor

The maps show that Gene1 activated with Genome

interesting point is that different genes activated and the delta time of the response for each of the

genomes was slightly different: Gene

after iteration 24 whereas Gene

A sustained gene expression response is exemplified in

genome, Gene2 is expressed for 30 cel

Figure 58. 3-gene genome activation vs. cellular iteration for C0 (48) using genome 178

The first example of cellular cloning is contained in

GRN that has been evolved with

which is from the C0 class. The left

cell whereas the middle and right

cloned cells that were created during the training of this genome.

activated with Genome9 and Gene5 activated with Genome


genomes was slightly different: Gene1 responded beginning at iteration 2 and ended its activation

Gene5 activated in iteration 0 and ended its activation after iteration 26.

A sustained gene expression response is exemplified in Figure 58 with 3-gene Genome

is expressed for 30 cellular iterations.

gene genome activation vs. cellular iteration for C0 (48) using genome 178

The first example of cellular cloning is contained in Figure 59. This gene activation map shows the

N that has been evolved with 10-gene Genome117. It produced 2 clones while training on pixel 42

which is from the C0 class. The left-hand portion of the figure is the activation map for the original

right-hand portions of the figure show the activation maps for the two

cloned cells that were created during the training of this genome.

118 of 249

ivated with Genome147. An


responded beginning at iteration 2 and ended its activation

activated in iteration 0 and ended its activation after iteration 26.

gene Genome178. Within the

gene genome activation vs. cellular iteration for C0 (48) using genome 178.

activation map shows the

. It produced 2 clones while training on pixel 42

hand portion of the figure is the activation map for the original

hand portions of the figure show the activation maps for the two

Taylor

(a)

Figure 59. 10-gene genome activation vs. cellular iteration for origina

environmental matrix- C0 (42) test pixel for genome 117

cloned genomes are presented in (b) and (c).

In this case, the same genome reacted

environmental lattice. The clonings occurred at different baseline times and are

can be considered as time-encoded patterns of protein concentrations within the lattice.

Although each cell (original and 2

for each of these three cells. Gene

gene activated in Clone0 between 2

24.

Complex behavior is further exemplified in

show an evolved GRN for 10

and pixel 51 from C1. Cloning occurred in addition to a modulation of the expression that differed

between the originals and clones in addition to across the classes. Since clones are placed in the

environmental lattice next to the original cell, only a single cell can occupy a specific location with

the environment.

(b) (c)

gene genome activation vs. cellular iteration for original and two cloned cells within the

C0 (42) test pixel for genome 117. The original genome is presented in (a) while the two

cloned genomes are presented in (b) and (c).

ase, the same genome reacted to differing concentrations at different

environmental lattice. The clonings occurred at different baseline times and are

encoded patterns of protein concentrations within the lattice.

Although each cell (original and 2 clones) all shared the same genome, Gene

for each of these three cells. Gene15 activated in the original cell between iterations

between 2 and 26; and the gene in Clone1 activated between iter

Complex behavior is further exemplified in Figure 60 and Figure 61. Respectively, these figures

10-gene Genome117 as it responded to the proteins from pixel 50 from C0



ice next to the original cell, only a single cell can occupy a specific location with

119 of 249

l and two cloned cells within the

The original genome is presented in (a) while the two

ns at different points in the

environmental lattice. The clonings occurred at different baseline times and are responding to what

encoded patterns of protein concentrations within the lattice.

clones) all shared the same genome, Gene15 responded differently

activated in the original cell between iterations 0 and 33; the

activated between iteration 0 and

Respectively, these figures

to the proteins from pixel 50 from C0



ice next to the original cell, only a single cell can occupy a specific location with

Taylor

(a)

Figure 60. 10-gene genome activation vs. cellular iteration

environmental matrix- C0 (50) test pixel using genome 117

and the activations for the cloned cells are shown in (b) and (c).

(a)

Figure 61. 10-gene genome a

environmental matrix- C1 (51) test pixel using genome 117

activations for the cloned cells are listed in (b) and (c).

The varying expression responses in

(or clone) within the matrix and the con

These 6 GRNs activate Gene

original cell expressed Gene

(b) (c)

activation vs. cellular iteration for original and two cloned cells within the

(50) test pixel using genome 117. The activation for the original cell is shown in (a)

and the activations for the cloned cells are shown in (b) and (c).

(b) (c)

gene genome activation vs. cellular iteration for original and two cloned cells within the

C1 (51) test pixel using genome 117. The original cell activation is presented in (a) and the

activations for the cloned cells are listed in (b) and (c).

The varying expression responses in Figure 60 and Figure 61 are dictated by the location of the cell

(or clone) within the matrix and the concentration of the evolved switch protein at that discrete point.

These 6 GRNs activate Gene15, but at different cellular iterations according

original cell expressed Gene15 between iterations 0 and 38, Clone0 expressed it between iterations 1

120 of 249

original and two cloned cells within the

The activation for the original cell is shown in (a)

ctivation vs. cellular iteration for original and two cloned cells within the

The original cell activation is presented in (a) and the

are dictated by the location of the cell

protein at that discrete point.

to the class. For C0, the

xpressed it between iterations 1

Taylor 121 of 249

09/28/10

and 32, and Clone1 expressed it between 0 and 31. With C1, the original cell expressed Gene15

between iterations 0 and 28, Clone0 expressed between iterations 3 and 20, and Clone1 expressed

between 0 and 18. As with the prior cloning example, a complex GRN has been created with a

temporal response that is different according to the class and also location within the lattice.

Figure 62 shows that further complexity is possible within the HeBIS GRN. This example consists of

the different expressions which occurred with the same training pixel, but with different 10-gene

genomes taken from the PSO. 10-gene Genomes 91, 123 and 135 were applied to pixel 42 from the

C0 (cloud) class. In this case, cloning has also occurred, but the expressions from the different

genomes are composed of different genes. The first map is for Genome91; the second, third and

fourth maps are for the original and two clones created by Genome123; and the fifth, sixth and seventh

maps are for the original and two clones created by Genome135.

Taylor

(a)

(b)

(c)

Figure 62. 10-gene genome activation vs. cellular iteration for

135 are displayed in (a), (b), and (c), respectively

activations for each of these genomes.

Genome91 shows a single gene response from Gene

differing GRN responses from the originals and the clones due to environmental pr

concentration differences due to location and protein diffusion within the lattice.

The first multiple-gene expression example is summarized in the GRN activation map in

Genome30 consists of 40 environmental genes for which multiple genes were expressed as the GRN

reacted differently to data from different classes.

gene genome activation vs. cellular iteration for the same test pixel C0 (42)

in (a), (b), and (c), respectively. Genomes 123 and 135 show original and two cloned cell

activations for each of these genomes.

shows a single gene response from Gene5 and the remainder of the maps highlight

differing GRN responses from the originals and the clones due to environmental pr


gene expression example is summarized in the GRN activation map in

nvironmental genes for which multiple genes were expressed as the GRN

reacted differently to data from different classes.

122 of 249

same test pixel C0 (42). Genomes 91, 123, and

23 and 135 show original and two cloned cell

and the remainder of the maps highlight

differing GRN responses from the originals and the clones due to environmental protein


gene expression example is summarized in the GRN activation map in Figure 63.

nvironmental genes for which multiple genes were expressed as the GRN

Taylor

(a)

(b)

Figure 63. 40-gene genome activation for genome 30 for test pixels C0 (26) in the top image

bottom image (b). The genome shows multi

responses.

The top image in the figure shows the map for pixel 26 which is from C0 and the bottom image

shows the map for pixel 25 which is from C1.

iterations 0 and 31 while also expressing Gene

expressed for the exemplar from C1 and it activated between iterations 0 and 24.

Further GRN complexity is shown with the example of multiple gene expression

cloning. This is exemplified in

Genome66 which is a 40-gene ge

gene genome activation for genome 30 for test pixels C0 (26) in the top image

The genome shows multi-gene activation for C0 and single gene activation for C1 with differing


l 25 which is from C1. The exemplar from C0 expressed Gene

iterations 0 and 31 while also expressing Gene35 between iterations 2 and 23. Only a single gene was


ther GRN complexity is shown with the example of multiple gene expression

cloning. This is exemplified in Figure 64 with complex temporal differences in expression for

gene genome.

123 of 249

gene genome activation for genome 30 for test pixels C0 (26) in the top image (a) and C1 (25) in the

gene activation for C0 and single gene activation for C1 with differing


The exemplar from C0 expressed Gene8 between

between iterations 2 and 23. Only a single gene was


ther GRN complexity is shown with the example of multiple gene expression occurring with

with complex temporal differences in expression for

Taylor

(a)

(b)

(c)

Figure 64. 40-gene genome activation for genome 66

activation profiles for C0 (30) with an orig

same genome, but for a C1 (31).

The GRN causes a clone to be created with the injection of class C0 proteins, pixel 30, at the

beginning of the simulation. However, the GRN for this genome does not create a clone with

genome activation for genome 66. The top (a) and middle(b) images are the multi

activation profiles for C0 (30) with an original and cloned cell. The bottom (c) image show the activation


beginning of the simulation. However, the GRN for this genome does not create a clone with

124 of 249

images are the multi-gene

image show the activation for the


beginning of the simulation. However, the GRN for this genome does not create a clone with the

Taylor 125 of 249

09/28/10

injection of a class C1 exemplar, pixel 31. The GRN also expresses 2 genes in the original cell, but

only a single gene in the cloned cell.

In detail, for the C0 pixel, genes 3 and 27 are expressed in the original cell, but only Gene3 is

expressed in Clone0. For the C1 pixel, only Gene3 is expressed. Also, the temporal responses differ

for Gene3: in the original cell with the C0 class, Gene3 was expressed between iterations 0 and 31

with Gene27 expressed for only a single iteration beginning at iteration 0. The clone saw an

expression of Gene3 that occurred between iterations 2 and 22. Yet another variation is seen with the

C1 pixel as Gene3 only expressed between iterations 0 and 22.

The final example of complex responses within evolved HeBIS GRNs is displayed in Figure 65. This

highlights different responses given that different class exemplars are applied to the same 40-gene

genome, Genome90.

Taylor

(a)

(b)

(c)

(d)

(e)

(f)

Figure 65. 40-gene genome activation

C1 (55), respectively from top to bottom

gene genome activation of genome 90 for test pixels C0 (30), C1 (31), C0 (32), C0 (34), C0 (54), and

top to bottom, in figures (a) – (f).

126 of 249

0 (32), C0 (34), C0 (54), and

Taylor 127 of 249

09/28/10

Six pixel in total were used in this instance, four from class C0 and 2 from class C1. These pixels

were pixel 30, 32, 34 and 54 for C0 and 31 and 55 for C1.

Class(pixel) C0(30), C1(31), C0(32) and C1(33) all cause the evolved GRN to express Gene25.

C0(54) and C1(55) express the GRN differently than the remaining four pixels. The GRN that results

from application of the first four pixels results in expression for a single iteration with a longer-term

expression for C1(55) at Gene36. The last two present GRNs that evolved one more generation

beyond the GRNs from the first four pixels. In the last two, Gene24 and Gene36 show a multiple-

gene expression for C0 whereas only Gene36 is expressed for the C1 exemplar.

4.3.4.3. Experiment 5 conclusions

The fitness function used in Experiment 5 provides for the evolution of a set of rudimentary GRNs

that express complex responses. However, the function is not necessarily geared towards

simultaneous multiple-gene expression. Routine multiple-gene expression could potentially be

accomplished with more complex fitness functions, however the flip-side of this is that the current

function quite possibly does not introduce unnecessary complexity into the structures required for

classification. This will be further discussed in the large-scale classification studies discussed in later

experiments within this thesis.

The PSO is providing a diversity of building-block GRN behaviors such as differing temporal

activations and cloning and follow-on experiments will determine whether these can be harnessed for

classification by adding more complex behaviors to the individual cell.

The observed cloning allows for a time-mediated response within an evolved GRN represented by a

specific genome. Figure 59, Figure 62 and Figure 65 show that the cloning behavior can provide

distinctly different responses for different class pixels.

Taylor 128 of 249

09/28/10

4.4. Self-organization in the HeBIS environment Hybridization Background




Robustness Analyses

We have now completed the examination of the genetic regulatory networks that HeBIS can create.

Now, we proceed to studying the presence and variability of self-organization with HeBIS. These

experiments examine this next layer of HeBIS’ functionality and builds from the GRN results. These

self-organization results will be used later as the basis for the detailed look at the utility of HeBIS for

a remote sensing classification application.

Within HeBIS, we examine its abilities to improve the fitness of the classifier genome as the genetic

regulatory network changes dynamically according to the different proteins that are present in the

system. We include the following list as a summation of HeBIS self-organization principles.

i. The number of particles in the particle swarm optimizer.

ii. Breed location of the initial processing cell within the protein diffusion matrix (environment).

iii. Cellular actions that are activated and inhibited according to the protein interactions that comprise the GRN.

iv. Cloning of a processing cell to another location in the environment. v. Protein statistical analogs that can potentially affect the GRN.

vi. Determination of output protein choice for the 2-class classification problems that we have chosen to study.

Although the size of the swarm in the PSO is not technically controlled from within HeBIS, it is

nonetheless the important basis that allows the creation of a genome with self-organization faculties.

Thus, we include this sizing parameter in our list and also examine it and its relation to the creation of

a genome which performs “good” classifications as defined by the fitness of the genome.


Self-organization is studied through an examination of evolved genomic fitness in the context of a

small-scale classification test based on cross-validation (CV). This CV is performed on a subset of

the training/test dataset and the results are compared when HeBIS self-organization training

Taylor 129 of 249

09/28/10

parameters are changed. A subset of the training data is used to facilitate a speed-up in the somewhat

lengthy training process. We compare the fitness of the best evolved genome given changes in these

parameters.

CV for all the experiment sin the self-organization section consist of 20 multispectral pixels sampled

uniformly from the optical cloud/no-cloud training dataset. A 10-fold CV is used which results in 20

test pixels. The base simulation parameters for these tests are listed in Table 16.

Table 16. Simulation Parameters for Self-Organization Experiments.


# Trials 100, 25, or 20

# Environmental genes 10




PSO breedings 10




# PSO particles 200

# Folds in the cross-validation 10

Test dataset size 20

4.4.2. Fitness function description

The fitness function is used to optimize a genome to produce an accurate classification result during

the training and testing phases. HeBIS uses a single fitness function to judge the efficacy of a

particular genome. This efficacy is based on whether the desired output protein is present in the

environmental protein matrix at an appropriate concentration. The fitness function rewards the

existence of this protein relative to the maximum correlation with the proteins that are present in the

matrix during the training phase. The fitness function also considers the closeness of the match of the

various proteins in the matrix relative to the 4-tuple descriptor of the output protein.

The fitness of the genome consists of a scalar value that is derived from the genome’s response to a

series of pairs of training pixels. Each pair of pixels contains a cloud pixel from class C0 and a no-

cloud pixel from class C1. The fitness scalar, Fitnessvalue

, is defined by

Taylor 130 of 249

09/28/10

if (Labeldesired class

== C0)

Fitnessvalue

= 2(CorrC0 max

– 0.5)

else if ( Labeldesired class

== C1)

Fitnessvalue

= 2(0.5 – CorrC0 max

)

(22)

where CorrC0 max

and fitnessValue are contained within the interval, [0.0, 1.0].

This provides a decision boundary equal to 0.5. This decision is dependent on CorrC0 max

for the

desired single output protein that distinguishes between classes C0 and C1. Figure 66 shows this

mapping range for CorrC0 max

.

Figure 66. Decision region mapping and boundary based on the value of CorrC0 max

.

CorrC0 max

is defined as

CorrC0 max = 0.5 Mag

corr

2+ Θ

2b c

(23) with

Magcorr

=Proteintest

N

N

N

N

N

N

ProteinC0 Output

N

N

N

N

N

N

fffffffffffffffffffffffffffffffffffffffffffffffffff for

Proteintest

ProteinC0 Output

N

N

N

N

N

N

fffffffffffffffffffffffffffffffffffffffffffffffffff2 0.0, 1.0B C

(24)

and

Magcorr

=1

ProteinN

N

N

N

ProteinC0 Output

N

N

N

N

N

N

ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff for

Proteintest

ProteinC0 Output

N

N

N

N

N

N

fffffffffffffffffffffffffffffffffffffffffffffffffff2 1.0,1b c

(25)

Taylor 131 of 249

09/28/10

and

Θcorr =@1

πfffff

θinner angle

+ 1.

(26)

.

Plots corresponding to equations (23), (25) and (26), respectively are included in the Appendices in

Figure 121, Figure 122, and Figure 123, beginning on page 238.

Taylor 132 of 249

09/28/10

4.4.3. Experiment 6 - Swarm fitness characterization Hybridization Background




Robustness Analyses

The fitness of the swarm is characterized through a single experiment:

• Experiment 6 Swarm fitness characterization.

4.4.3.1. Setup

Experiment 6

This is a characterization of the particle swarm optimizer and gauges the utility of PSO for optimizing

the fitness function of the genome. All the other self-organization parameters are set to the default

static values- no PSO evolution modifies these parameters.

The best genome fitness is acquired over each CV-fold and this is averaged over the trial to produce a

new averaged CV fitness value. These trials are accumulated as described in Table 17.

Table 17. Experiment 6 Trial Distribution

# Particles in swarm # Trials

1 25

100 25

250 25

500 25

All the training parameters are as defined in section 4.4.


Figure 67, Figure 68, Figure 69 and Figure 70 summarize the evolution of the best fitness achieved

within the particle swarm optimizer given a specific number of particles in the swarm. Ranging from

a minimum of 1 particle to a maximum of 500 particles, the plots show that there is a tendency for the

HeBIS PSO to perform better- that is to achieve higher fitness values- when there are more particles

being used by the optimizer. All four trial distributions indicate that the HeBIS PSO shows a

monotonic increase in average fitness from the first breeding through the end of the last breeding,

breeding 9.

Taylor 133 of 249

09/28/10

Figure 67. Average best genome fitness vs. breed # for 1-particle PSO swarm. Vertical bars correspond to the

standard deviation of the sample mean.



Taylor 134 of 249

09/28/10





For the 1-particle PSO, the swarm that has the worst optimized fitness, the breeding begins with an

average best fitness of 0.0 and by the end of the final breeding, it has risen to 0.0018308. In the case

of the 1-particle PSO, the fitness is changing due to the fact that several pixels are examined in CV

mode. This 1-particle fitness compares with the highest-achieved average best fitness of 0.0723 for

the 500-particle PSO. The standard deviations for the fitnesses of the genomes within the swarms are

also relatively high, thus indicating that the health of the swarms remains useful over the period of

breeding for the 100, 200 and 500-particle swarms. Standard deviations of the best average fitness

ranges from 0.0018 at the end of the second 1-particle PSO breeding to 0.0797 at the end of the tenth

Taylor 135 of 249

09/28/10

breeding of the 500-particle PSO. Table 56 on page 216 in the Appendices summarizes the breeding

data for the swarm in aggregate.

Figure 71 indicates the health of the HeBIS PSO. It shows that the best average peak fitness during

the averaged CV trials increases as the size of the swarm increases. The best average peak fitness

increases from 0.008308 in the case of the 1-particle swarm to 0.02386 for the 100-particle swarm ,

0.0503 for the 250-particle swarm, and finally to 0.0723 for the 500-particle swarm.

Figure 71. Best average peak genome fitness vs. the number of particles in the PSO swarm. Vertical bars

correspond to the standard deviation of the sample mean of the peak fitness for each swarm tested.


The HeBIS PSO works and is optimizing the limited portions of the fitness function that have been

activated for this experiment. The observed fitness values are low because of this, but nonetheless,

the optimizer appears to be working. Higher fitness values are seen in later experiments when more

complex cellular and GRN behaviors are activated. Thus, the PSO’s ability to search through a

complex search space for useful GRNs improves as the number of particles increases. This will be

discussed in further detail in the classification experiments.

Taylor 136 of 249

09/28/10

4.4.4. Experiment 7 - Initial location of processing cells

HeBIS allows the initial location of the first processing cell in the lattice to be randomly set and

evolved by the PSO. Subsequent breedings allow for this location to be optimized.

4.4.4.1. Setup

Experiment 7

This experiment determines whether any change in the genome’s fitness occurs over time as the PSO

iterates and the initial cell location is statically set to a specific position in the environmental lattice.

Only the initial location of the first (and only) processing cell is allowed to change via the PSO. The

best genome fitness information is acquired over each CV-fold and is then averaged. This new

averaged CV fitness represents the statistic for the trial.

300 trials are used where the initial locations can be varied over the 15 possible locations in a 3x3x17

environmental lattice for a single pixel. Table 18 shows the distribution of trials for this experiment.

Table 18. Trial distribution for Experiment 7.

# Trials Initial Cell Location (z-axis)

20 1

20 2

20 3

20 4

20 5

20 6

20 7

20 8

20 9

20 10

20 11

20 12

20 13

20 14

20 15


The plot in Figure 72 summarizes the results for changing the location of the initial cell among the 15

possible cell locations in the default HeBIS 3x3x17 environmental lattice. Only 15 possible cell

Taylor 137 of 249

09/28/10

locations are used because of the 2-location buffer zone in the environmental matrix. In this plot, we

see that the mean fitness values are associated with relatively large standard deviations.

Figure 72. CV average fitness vs. initial cell location from best bred genome. Vertical bars correspond to the


A peak does occur at location 9 which corresponds to HeBIS feature 9- a MODIS spectral bandwidth

of 483-493 nm. The mean peak fitness over 20 trials for this feature is 0.06287 with a standard

deviation of 0.02691.


As with Experiment 6, the mean fitnesses are low because the complex cellular and GRN behaviors

are not active. These trials do not show any significant differences in mean fitness between the initial

cell locations although the peak at HeBIS feature 9 is interesting.

Taylor 138 of 249

09/28/10

4.4.5. Experiment 8 - Cellular actions Hybridization Background




Robustness Analyses

Individual cellular actions form the backbone of the HeBIS algorithm. These actions are mediated by

the GRN as it responds to the proteins that are present in the environmental lattice and their

invocation is in turn evolved by the HeBIS PSO.

4.4.5.1. Setup

Experiment 8

The cellular actions experiment poses the question as to whether individual basic cellular actions can

produce changes in the corresponding fitness of a genome and its GRN. The genome is evolved by

application of the PSO and only a single cellular action is potentially allowed to activate during the

set of simulation trials for that particular action. This allows the determination as to whether these

individual cellular actions, which are used by the HeBIS self-organization building blocks, can

change the fitness of a simplified, bare-bones genome. Tied together by a fully activated GRN, we

will examine during the classification experiments whether these actions allow effective classification

of our cloud/no-cloud dataset.

The tested cellular actions include:

i. NOACTION ii. ADDCELL

iii. PRUNESELF iv. ACTIVATEENVIROPROTEIN v. INHIBITENVIROPROTEIN

vi. ACTIVATEREGPROTEIN vii. INHIBITREGPROTEIN

viii. CHANGETOSOFMANDTRAIN ix. CLASSIFY

The default simulation parameters listed in Table 16 on page 129 are used. The initial cell location

along the environment’s z-axis is set to be in the middle of the matrix at location 8. The best genome

fitness is acquired over each CV-fold and this is averaged over the 10 folds of the simulation. This

Taylor 139 of 249

09/28/10

new averaged CV fitness is calculated for each of the 25 trials for each tested cellular action listed in

Table 19.

Table 19. Trial Distribution for Experiment 8

Cellular Action Index Cellular Action # Trials

1 NOACTION 25

2 ADDCELL 25

3 PRUNESELF 25

4 ACTIVATEENVIROPROTEIN 25

5 INHIBITENVIROPROTEIN 25

6 ACTIVATEREGPROTEIN 25

7 INHIBITREGPROTEIN 25

8 CHANGETOSOFMANDTRAIN 25

9 CLASSIFY 25


Figure 73 displays a composite plot for comparison of all the cellular actions that are tested in this

experiment. Mean fitness and standard deviations for each of the cellular actions listed in Table 19 is

presented in the composite plot. The cellular action index is mapped to the abscissa of the plot. The

fitness function is the default HeBIS single-protein function that is based on environmental proteins

that are measured to be close to the defined mappings for the C0 and C1 output classes.

Figure 73. CV average fitness vs. activated cellular action. Vertical bars correspond to the standard deviation of

the sample mean.

Taylor 140 of 249

09/28/10

The baseline performance for this experiment consists only of the corresponding averaged fitness for

the NOACTION cellular action. This mean fitness is 0.02097 with a corresponding standard

deviation of 0.0358.

Referencing the composite plot in Figure 73, only three additional actions will be discussed in detail

with reference to the baseline. These actions are ADDCELL, ACTIVATEENVIROPROTEIN, and

CLASSIFY.

The first of these, ADDCELL, showed a decrease in the mean fitness to 0.01288 with a standard

deviation of approximately 0.0312. Referencing the relatively low fitness magnitudes from the

experimental data, this is a 38.5 % decrease in mean fitness from the NOACTION baseline.

The second action, ACTIVATEENVIROPROTEIN, shows the largest relative increase in mean

fitness when compared to the mean fitnesses of the other actions. Its mean fitness of 0.06888

corresponds to a 228 % (3.28x) increase in fitness compared to the NOACTION baseline. The

corresponding standard deviation is comparable to that of the other actions in this portion of the study

at 0.03725. This increase is primarily because in most of the trial cases, the only proteins in the

lattice were those that were initially injected into the lattice as the spectral proteins from the specific

exemplar that HeBIS trained with. Thus, proteins beyond this initial set of spectral proteins are

allowed to be expressed since the ACTIVATEENVIROPROTEIN action is active. This allows a

greater chance for the evolved and tested GRN to more closely match the desired output proteins that

are being measured and rated by the HeBIS fitness function as the PSO evolves the crippled GRN.

The CLASSIFY action shows no significant change in mean fitness when compared to the baseline.

CLASSIFY’s mean fitness over the 25 trials is 0.02082 which compares with 0.02097 for

NOACTION. This occurs primarily because of one reason. The first is that the CLASSIFY action

relies on a SOFM that is internal to each cell within the environmental lattice. This SOFM relies on

an internal codebook that has been trained on environmental proteins that are present in the lattice at

the position of the particular cell. However, with these CLASSIFY trials, no SOFM was trained

because the action, CHANGETOSOFMANDTRAIN is inactive. Thus, the classification process that

is started by the invocation of the CLASSIFY action, does not complete.

Taylor 141 of 249

09/28/10


Examining the composite plot in Figure 73, it is seen that the mutually exclusive invocation of the

different cellular actions does change the measured mean fitness. This reinforces the belief that the

activation or inhibition of these cellular actions by the selected genome and its GRN can be

effectively applied to our classification problem. These constituent behaviors are adding complexity

on their own to the evolved GRN. The use of these cellular actions makes sense when they are used

in conjunction with each other. Hence, the classification experiments, 11-16, examine changes in

both classification and fitness with these cellular actions acting in concert with the evolved GRN

controlling the self-organization of the system. Table 58 presents the data for this experiment on

page 217 in the Appendices.

4.4.6. Experiment 9 - Protein statistical analogs

Statistics corresponding to protein concentration information that is local to each cell can be released

into the protein environmental matrix as the cell iterates through processing. The question as to

whether these statistical analogs are released into the environment is answered by the PSO. Inclusion

of protein statistics may or may not help in the creation of a GRN that is more fit for classification.

4.4.6.1. Setup

Experiment 9

The Protein Statistic Effect experiment varies the creation and use of the cell’s protein statistics that

can be released into the environmental protein matrix. All other cellular actions are deactivated for

this experiment except for the activation and inhibition functions of the regulatory and environmental

proteins. The location of the initial cell is also constrained to be at location 8 which is in the middle

of the default environmental lattice.

The default simulation parameters for this experiment are listed in Table 16. The best genome fitness

is measured over the CV-fold and this value is averaged over the 10 folds. This new averaged fitness

represents the aggregate fitness for this trial.

50 trials are run with protein statistics activated and with the protein descriptors bred by the PSO. A

further 50 trials are run with protein statistics still activated, but with the protein descriptors statically

set to default values. Results from these runs are compared to the baseline 25 NOACTION trials

from Experiment 8. Parameters that are specific to this experiment are listed in Table 20.

Taylor 142 of 249

09/28/10

Table 20. Experiment 9 Parameters

Protein Statistics Index Protein Statistics Action Number of Trials

10 STATS_ACTIVE_STATIC 50

11 STATS_ACTIVE_BRED 50


The statistical protein analogs that area released by the individual cell into the environmental lattice

are those described in 3.1.3.2 and 3.3.1.1. Two cases exist for this experiment. In the first,

STATS_ACTIVE_STATIC, the four parameters that describe each of the statistical proteins remain

unmodified as the PSO iterates through successive breedings. Thus, in this case the proteins are

considered to be static. For the second, STATS_ACTIVE_BRED, the four parameters that describe

each of the statistical proteins are allowed to be modified and evolve as the PSO iterates through the

breedings, hence these are considered to be bred proteins. The NOACTION data from Experiment 8

are used as an appropriate baseline comparison since no protein statistics were active in Experiment

8. The results are plotted in Figure 74 with the protein statistics indices used as the abscissa.

Figure 74. CV average fitness vs. activity level of cellular protein statistics. Vertical bars correspond to the


Taylor 143 of 249

09/28/10

The mean fitness for STATS_ACTIVE_STATIC is little changed from the NOACTION case from

Experiment 8. It is 0.02075 with a standard deviation of 0.02635 over the 50 trials whereas the

NOACTION trials resulted in a mean fitness of 0.02097. However, STATS_ACTIVE_BRED shows

a significant change in mean fitness compared to both the NOACTION baseline and the static

statistical protein trials. The mean fitness for STATS_ACTIVE_BRED is 0.30357 with a standard

deviation of 0.0483. This is a 1363 % increase (14.6x) in mean fitness over that for the static

statistical protein case.


The inclusion of bred statistical proteins into the environmental lattice significantly changes the mean

fitness of the best genome and its corresponding GRN for the training set of protein-converted pixels.

The addition of the statistical proteins into the environmental lattice adds more proteins that can be

used by the GRN as it is evolved by the PSO according to the fitness function. Allowing the set of

statistics proteins to breed along with the other proteins in the GRN increases the chances for protein

expression and inhibition to occur which is useful for significantly changing the fitness. It should be

noted that these statistics proteins are always released into the environmental lattice without regards

to whether their switch proteins have been activated by other proteins present in the environment. In

this sense, they are considered to be always “on” and no matching of the switch protein is required for

expression.

However, the presence of these statistics is not enough to improve the fitness even more given the

number of breedings in the test. Providing more breedings could potentially allow these proteins to

be subverted and to allow them to be actively have their functionality taken over for improvement in

classification.

4.4.7. Experiment 10 - Output protein comparison

In this study, we examine the effects of HeBIS output proteins that are evolved with the PSO. This

evolution occurs in the context of the GRN. Conceivably, the output proteins can change according

to the GRN proteins encountered during training. Hence, we have another level of self-organization.

Taylor 144 of 249

09/28/10

4.4.7.1. Setup

Experiment 10

All other self-organizing cellular actions are deactivated except for the activation and inhibition

functions for the environmental and regulatory proteins. Protein statistics are deactivated. The

output genes are set to be either evolved or locked down to static values. The best genome fitness is

measured over each CV-fold and this is averaged to calculate the aggregate fitness for the test.

Evolved output proteins from the output genes are limited to only the environmental CO output

protein. In this case, C0 is the only protein used within the fitness function to differentiate between

the two classes. Also, the location of the initial cell is constrained to be at location 8 which is the

middle environmental element in the default environmental lattice.

The switch gene in both the fixed and evolved cases is always evolved. 50 trials are run in static

mode and 50 trials are run in PSO-evolved mode. Parameters specific to Experiment 10 are listed in

Table 21.

Table 21. Experiment 10 Parameters

Protein Statistics Index Protein Statistics Action Number of Trials

12 OUTPUT_ACTIVE_STATIC 50

13 OUTPUT_ACTIVE_BRED 50


Figure 75 plots the mean best fitness for the first case in which the output target protein for C0/C1

remains static during the PSO breedings and for the second case in which this C0/C1 target protein is

allowed to evolve. These output proteins are important because environmental proteins that closely

match the associated parameters for the C0/C1 protein are used directly by the HeBIS fitness function

to gauge classification success.

Taylor 145 of 249

09/28/10

Figure 75. CV average fitness vs. static or PSO-evolved setting of the output C0/C1 protein. Vertical bars

correspond to the standard deviation of the sample mean.

In both output protein cases compared to the NOACTION baseline in Figure 75, the mean value of

the best fitness is significantly different from the baseline fitness of 0.02097. For the

OUTPUT_ACTIVE_STATIC trials, the mean of the best fitness was 0.40678 and for the

OUTPUT_ACTIVE_BRED trials, the mean was 0.18473. Standard deviations were respectively

0.06797 and 0.0548.


It is interesting that there was a decrease in fitness with the bred output proteins when compared to

the trials in which static output proteins were used. This is the opposite outcome that occurred in

Experiment 9 in which the bred statistical proteins produced GRN fitnesses that were higher than

those from the static trials. This highlights the point that fixed C0 parameters are able to be utilized

more easily than the bred parameters because the fixed “target” more efficiently allows the PSO to

find higher fitness solutions given the low number of breedings in this controlled experiment: 10. An

important observation is that the trial GRNs are able to incorporate these fixed output protein

parameters more readily given the experimental constraints. In any case, complexity has been

introduced into the GRN with the addition of this particular constituent action.

Taylor 146 of 249

09/28/10

4.5. Classification accuracy Hybridization Background




Robustness Analyses

Classification accuracy of the HeBIS algorithm is now examined. In particular, comments will be

made with regards to this accuracy using our cloud/no-cloud dataset with respect to a few high-level

training parameters. In addition, we examine and compare the results from self-organizing feature

maps on the same dataset where applicable. These results are presented in the context of the HeBIS

algorithm being fully-engaged and active over a range of controllable simulation parameters.

These experiments consist of examples in which an entire test image is classified. Full-image

classification is used to mimic usage of the algorithm in the real world as well as to produce a larger

statistical base now that the core parts of HeBIS have been examined and shown to work individually.

In summary, we will accomplish the following in this section:

• Examine the performance of HeBIS at various points in the operating space of the algorithm,

• Compare HeBIS with a self-organizing feature map.

4.5.1. Introduction

4.5.2. Training algorithm parameters description

The pertinent training parameters for HeBIS and the SOFM simulations follow. Table 22 presents

the range of these parameters for HeBIS and similar information for the SOFM simulations is

contained in Table 23.

Taylor 147 of 249

09/28/10

Table 22. Range of Pertinent HeBIS Training Parameters for Classification

Simulation Parameter Description

Geographic size 3x3,5x5,7x7

Number of PSO breedings [0,20]

Number of PSO particles 400

Genome size [0,40]

Protein diffusion rate (cell) [0.0, 1.0]

Protein diffusion rate (environment) [0.0, 1.0]

Maximum number of protein diffusion iterations [0, 4000]

Minimum protein concentration (cell) 0.001, 0.01

Minimum protein concentration (environment) 0.001, 0.01

Topology and size of artificial protein environment 3x3x17, 5x5x17, 7x7,17

Protein chemical reaction probability [0.0, 1.0] %

Minimum protein correlation (affinity) [0.0, 0.9]

Active protein statistics ON/OFF

Protein magnitude [0,1] or [0,15]

SOFM kernel size 0,1,2,4,6,9

SOFM kernel topology 1x1, 1x2, 2x1, 2x2, 2x3, 3x2, 3x3

Full-size classification dataset size 100620 (234x430) pixels

Table 23. Pertinent SOFM Training Parameters for Classification

Simulation Parameter Description

Kernel size 1, 2, 6, 9

Number of coarse training iterations 100

Number of fine training iterations 1000

Coarse neighborhood size Limited by max distance within network topology

Fine neighborhood size 1

Neighborhood type square

4.5.3. Fully-engaged HeBIS

These experiments occur in the context of HeBIS being fully operational. In other words, all the self-

organization parameters are active and the underpinnings of the genetic regulatory network are also

active.

4.5.3.1. Experiment 11 - Size of geographic processing environment

In this experiment we examine the effects of choosing different sizes for the area that surrounds the

pixel that is classified. We examine whether surrounding pixels help to convey information to aid in

the HeBIS classification process.

Taylor 148 of 249

09/28/10

4.5.3.1.1. Setup

Experiment 11

Only HeBIS is considered in this experiment. 50 full-image trials are executed with the geographic

regions displayed in Table 24. Default C0/C1 descriptors are used and protein statistics are

deactivated.

Table 24. Trial Distribution across Geographic Region Size for Experiment 11.

Number of trials Geographic region size

25 3x3

25 5x5

The parameters listed in Table 25 are used for the training parameters for these trials.



# Trials 50





PSO breedings 10

Topology and size of artificial protein environment 3x3x17, 5x5x17



# PSO particles 400

# Folds in the cross-validation 10

Test dataset size 100620 (234x430) pixels

Note that the intracellular SOFM kernel used in this experiment is deactivated by setting its number

of neurons to 0. At this point, only the utility of the geographic size parameter is being tested.

The resulting input data cubes for each pixel to be classified is exemplified in Figure 76 for a

geographic size of 5x5 (25 pixels) with 15 multispectral features.

Taylor 149 of 249

09/28/10

Figure 76. Multi-spectral data cube for a 5x5 geographic region with 15 bands of multispectral data.

4.5.3.1.2. Experiment 11 results and discussion

A 3x3 geographic region of multispectral data contains no surrounding pixels that are usable by

HeBIS- only the central column of the features associated with the centrally located test pixel is used

for environmental protein diffusion. The 5x5 geographic region contains the test pixel and those

pixels that are immediately adjacent to the central test column. Thus, for the 3x3 region, the lattice

consists of 15 (1x1x15) discrete locations in which diffusion occurs and for the 5x5 region, the lattice

contains 135 (3x3x15) discrete diffusion points. A 7x7 geographic region contains the adjacent

pixels and the pixels adjacent to those for 375 (5x5x15) discrete points. Regions larger than 5x5 were

not considered because of simulation times increase exponentially as the size of the geographic region

increases.

Figure 77 summarizes the overall classification accuracy of the test image given the size of the test

geographic region; either 3x3 or 5x5. Overall, there was a decrease in average classification accuracy

from 91.47 % to 83.96 % as the size of the region increased.

Taylor 150 of 249

09/28/10

Figure 77. Full-image average classification accuracy vs. size of geographic regions surrounding test pixel. Vertical


The confusion matrix for the 3x3 region is presented in Table 26. It indicates relatively good overall

classification rates for both the C0 and C1 classes with 93.92 % of C0 and 88.25 % of C1 being

classified correctly. The percentage of C0 pixels that are incorrectly classified as C1 pixels is 6.08 %

and the percentage of C1 pixels that are misclassified is higher at 11.75 %.

Table 26. Confusion Matrix for 3x3 Geographic Region

C0pred C1pred

C0actual 93.92 % 6.078 %

C1actual 11.75 % 88.25 %

In Figure 78, a scatter plot of the 3x3 classification accuracies is presented with reference to the

fitness of the associated genome/GRN that was used to perform the classification. There is a single

outlier at an approximate classification accuracy of 60 %. This is due to a failed genome/GRN for

which the cloud class (C0) is completely classified as being from the no-cloud, C1, class. This failure

mode occurs when no significant concentrations of the C0 class protein are present in the

environmental lattice. In this case, zero concentration of the output protein for a particular test pixel

causes the pixel to be interpreted being from the C1 class as described previously in the decision

region map in Figure 66 on page 133.

Taylor 151 of 249

09/28/10

Figure 78. Full-image classification accuracy vs. fitness for a 3x3 geographic region surrounding text pixel.

The correlation coefficient for a linear fit for the 3x3 data is 0.2631, but with a confidence estimate of

only 79.6 %- low and not in line with a desired confidence on the order of 95 % (Student’s t-test).

Table 27 lists the confusion matrix for the 5x5 region. Once again, it indicates a relatively good

overall classification rate for the C0 class with 91.86 % of the C0 pixels being classified correctly.

However, the misclassification rate for C1 pixels being incorrectly classified as C0 pixels has

significantly increased to 25.89 % from the 3x3 11.75 % misclassification rate. The overall

misclassification rate for C0 pixels being classified as members of the C1 class has only slightly

increased for the 5x5 case to 8.144 % from 6.08 % in the 3x3 case.

Taylor 152 of 249

09/28/10

Table 27. Confusion Matrix for 5x5 Geographic Region

C0pred C1pred

C0actual 91.86 % 8.144 %

C1actual 25.89 % 74.11 %

Figure 79 displays a scatter plot for the 5x5 classification accuracies that is also with reference to the

fitness of the GRN used in the classification. Four outliers at the failure accuracy of approximately

60 % exist in a fashion similar to the outlier that is present in the 3x3 data.

Figure 79. Full-image classification accuracy vs. fitness for a 5x5 geographic region surrounding text pixel.

The correlation coefficient for a linear fit for the 5x5 data is higher at 0.3761 when compared to the

3x3 data. This indicates that some correlation exists between the genome’s fitness and its

classification performance with a confidence of 93.6 % (Student’s t-test).

Taylor 153 of 249

09/28/10

4.5.3.1.3. Experiment 11 conclusions

The 3x3 geographic region has been shown to perform better in overall classification accuracy than

the 5x5 region. The cloud (C0) classifications are similar for both the 3x3 and 5x5 regions, however

the reduction in accuracy for the 5x5 region is due to misclassifications of no-cloud (C1) pixels as

cloud (C0) pixels. Thus, the surrounding pixels do not appear to aid the HeBIS classification process

while it is using the parameters set forth for this experiment. Potentially, the accuracy might be

improved by increasing the number of breedings beyond the 10 allowed in this experiment or by

using regions that are larger than the larger 5x5 region that was tested in this experiment. However,

these larger regions were not studied because of the exponential increase in processing time required

for these larger-region simulations.

Also, Experiment 11’s parameters do not constitute an optimal set of processing parameters. The

shotgun experiments in Experiment 14 will help to shed light on this tuning process.

4.5.3.2. Experiment 12 - Size of intracellular SOFM kernel

The effects of varying the size of the HeBIS intracellular SOFM kernel is examined. For our

purposes, this size is defined as the number of neurons in the feature map. The 2-D topology of these

neurons is not considered in this experiment and thus a square topology is used as the default.

4.5.3.2.1. Setup

Experiment 12

Comparisons are made for different sizes of the SOFM kernel as shown in Table 28.

Table 28. HeBIS Kernel Sizes for the Intracellular SOFM in Experiment 12.

Kernel Number of Neurons Number of Trials

0x0 0 20

1x1 1 20

2x2 4 20

3x3 9 18

9x9 81 20

Taylor 154 of 249

09/28/10

Table 28 also shows the distribution of the 100 trials among the 5 kernel mappings that are tested.

Other training parameters used for these trials are included in Table 29.



# Trials 98





PSO breedings 10




# PSO particles 400

Test dataset size 100620 (234x430) pixels

Full-image testing is used to acquire classification accuracy information from each trial.


The confusion matrix for the case in which 0 neurons are used in the intracellular SOFM is presented

in Table 30.

Table 30. Confusion Matrix for Intracellular SOFM with 0 Neurons

C0pred C1pred

C0actual 93.12 % 6.885 %

C1actual 15.57 % 83.43 %

Classification rates for the cloud and no-cloud classes are 93.12 % and 83.43 % respectively. The

ratio of misclassifications is defined as

(27)

Taylor 155 of 249

09/28/10

and it provides an indication of the degree of difference between the two types of misclassification.

The ratio for this case is 2.26 with 6.89 % of C0 pixels being incorrectly assigned to C1 and 15.57 %

of C1 pixels being misclassified as belonging to class C0. Overall mean classification accuracy on

the entire test image for the 0-neuron classifier is 88.94 %.

A scatter plot for classification accuracy vs. fitness with 0 neurons in the intracellular SOFM is

displayed in Figure 80. A modest correlation correlation of 0.3609 exists between the two parameters

with a confidence of 88.2 % (Student’s t-test).

Figure 80. Classification accuracy vs. fitness for case with 0 neurons in intracellular SOFM.

Adding a single neuron to the SOFM improves the detection of class C0 slightly to 95.79 % when

compared to the classification for the 0-neuron case. However, detection of C1 becomes worse and

falls to 70.39 % compared to 83.43 % for the 0-neuron case. C0->C1 misclassification falls to 4.21%

from 6.89 % for the 0-neuron case, but C1->C0 misclassification increases when compared to the 0-

neuron baseline. In this case, C1->C0 misclassification increases from 15.57 % to 29.61 %. The

spread in the misclassification ratio increases to 7.03 compared to 2.26 for the baseline. These results

Taylor 156 of 249

09/28/10

are summarized in Table 31. Mean classification accuracy for the 1-neuron classifier falls slightly to

84.84 % from the 88.94 % attained with the 0-neuron classifier.

Table 31. Confusion Matrix for Intracellular SOFM with 1 Neuron

C0pred C1pred

C0actual 95.79 % 4.214 %

C1actual 29.61 % 70.39 %

The classification-fitness scatter plot in Figure 81 highlights a high linear correlation coefficient

index of 0.7421 that has a corresponding significance of greater than 99 % (Student’s t-test).

Figure 81. Classification accuracy vs. fitness for case with 1 neuron in intracellular SOFM.

Although the confusion matrix in Table 32 for the 4-neuron SOFM shows better performance than for

the 1-neuron instance, it is not better than the 0-neuron case.

Taylor 157 of 249

09/28/10


C0pred C1pred

C0actual 94.29 % 5.715 %

C1actual 21.78 % 78.22 %

HeBIS with a 4-neuron SOFM classifies C0 pixels correctly 94.29 % of the time with a detection rate

of C1 pixels that, at 78.22 %, is significantly higher than what occurred for the 2-neuron case. The

misclassification ratio is still higher than for the baseline SOFM, but it has improved to 3.81 from the

2-neuron SOFM’s 7.03 ratio. The corresponding misclassification details are that class C0 was

misclassified 5.72 % of the time and class C1 was misclassified 21.78 % of the time. Improvements

in the overall classification accuracy stem from the decrease in the misclassification of no-cloud

pixels into cloud pixels. This overall mean classification accuracy is approximately 87.36 % which is

better than the 84.84 % mean classification accuracy determined for the 1-neuron trials.

A moderate amount of linear correlation exists between classification accuracy and fitness as is

evidenced by the 0.5105 correlation coefficient which is found from the 4-neuron results with a 98%

significance (Student’s t-test). A linear fit through the scatter data is shown in Figure 82.


Taylor 158 of 249

09/28/10

The 9-neuron confusion matrix as listed in Table 33 shows that this classifier has the highest

aggregate classification rate on the test image of all the intracellular SOFMs examined. At 91.07%,

the classifier also has the lowest misclassification statistics. These are 4.81 % misclassification of C0

pixels as C1 pixels and 14.36 % for C1 pixels that have been incorrectly labeled as being members of

class C0. The misclassification ratio has dropped to 2.99 for this classifier which is better than for

both the 0-neuron and the 4-neuron classifiers.


C0pred C1pred

C0actual 95.1894 % 4.8106 %

C1actual 14.3611 % 85.6389 %

A low amount of linear correlation exists for the classification-fitness scatter data for the 9-neuron

classifier: 0.2576 with a low significance of approximately 69.8% (Student’s t-test). The plot of the

spread in data is presented in Figure 83 over the 20 trials for this portion of the experiment.

Taylor 159 of 249

09/28/10


The 81-neuron classifier has the fourth best performance with regards to mean classification

accuracy. At 87.16 %, only the 1-neuron classifier performs worse in aggregate. In Table 34, the

confusion matrix shows the lowest misclassification ratio, 2.80. This is lower than that of the best

aggregate classifier, HeBIS with 9 intracellular SOFM neurons. Although the ratio of

misclassifications is lower than the 9-neuron case, the individual misclassification errors are higher.

C0->C1 misclassifications increase to 7.23 % and C1->C0 misclassifications increase to 20.24 %.


C0pred C1pred

C0actual 92.77 % 7.229 %

C1actual 20.24 % 79.76 %

The scatter data for the 81-neuron classifier support a correlation coefficient that is the highest of the

studied SOFM classifiers at 0.7259 with the highest significance of greater than 99.0 % (Student’s t-

test). The linear fit is presented in Figure 84.

Taylor 160 of 249

09/28/10



Figure 85 summarizes the mean classification accuracies given the size of the intracellular SOFM

kernel used by the HeBIS algorithm.

Figure 85. Full-image average classification accuracy vs. the number of neurons in the intracellular SOFM.

Vertical bars correspond to the standard deviation of the sample mean of the classification accuracy.

Taylor 161 of 249

09/28/10

The best mean accuracy, 91.07 %, over the trials is obtained with the 9-neuron classifier and the

worst overall mean accuracy of 84.84 % was found with the 1-neuron classifier. From best mean

classification accuracy performance to the worst, the classifiers are ranked as follows: 9-neuron, 0-

neuron, 4-neuron, 81-neuron and 1-neuron.

Misclassifications of both types were found to be at a minimum with the 9-neuron classifier. It would

seem that the improvements in overall aggregate classification result from the decrease in C1->C0

misclassifications. The cloud class classifications (C0->C0) remain roughly the same at 93.12 %,

95.79 %, 94.29 %, 95.19 % and 92.77 % respectively for the 0-neuron, 1-neuron, 4-neuron, 9-neuron,

and 81-neuron classifiers. However, the standard deviations of mean classification accuracies are

high, most likely because of the relatively small number of trials conducted per SOFM kernel. The

number of trials used was due to time constraints for processing the simulations.

4.5.3.3. Experiment 13 - Protein chemistry reaction probability

The effects of protein chemistry on HeBIS classification accuracy is now examined.

4.5.3.3.1. Setup

Experiment 13

This experiment consists of fixing the primary HeBIS parameters while varying the probability that a

protein chemical reaction- beyond the genome-dictated interactions- can occur in the environmental

lattice. This is different from Experiment 2 in the sense that now the self-organization and GRN

modules are active. The trial distribution for the protein chemistry probabilities are listed in Table 35

and are similar to the trial distribution used in Experiment 4.

Table 35. Protein Reaction Probability Distribution for Experiment 13.

Number of trials Protein reaction probability

20 0.0 %

20 0.1 %

20 1.0 %

20 10 %

Taylor 162 of 249

09/28/10

The remainder of the HeBIS training parameters for this experiment are listed in Table 36.



# Trials 80

# Genes 3

Intracellular SOFM size 2x1




PSO breedings 10



Reaction probability 0.0, 0.1, 1.0, 10.0 %

# PSO particles 400

The size of the intracellular SOFM kernel is set at 2 neurons with a 2x1 topology. For this

experiment, training occurs over 18 pixels (9 pairs of C0/C1 pairs).

Classification accuracy data are acquired from testing a full image and these results are averaged over

similar trials. These averaged results are collected over the prescribed number of trials according to

the distribution in Table 35.


The baseline for Experiment 13 is the case in which there is a 0.0 % probability of chemistry

occurring between proteins in the environmental matrix. The mean classification accuracy for this

baseline HeBIS classifier is 87.1 %. In the confusion matrix listed in Table 37 for this classifier, it is

seen that this mean classification accuracy translates into 91.954 % correct classification of class C0

pixels and 80.704% correct classification of C1 pixels. The misclassification ratio for this classifier is

2.398 and this is indicative of the high misclassification rate, 19.296 %, of C1 pixels into C0 pixels.

Table 37. Confusion Matrix for 0.0 Protein Reaction Probability

C0pred C1pred

C0actual 91.95 % 8.0460 %

C1actual 19.30 % 80.70 %

Taylor 163 of 249

09/28/10

The correlation coefficient for this classifier with regards to classification accuracy and the fitness of

the associated GRN is 0.63 (with a confidence of 99.3 %; Student’s t-test) and this is indicative of a

fit that possesses a moderate amount of linearity. A linear fit through the classification and fitness

data is presented in Figure 86.

Figure 86. Classification accuracy vs. fitness with 0.0 reaction probability.

The second classifier has a 0.1% chance of a protein chemical reaction occurring in the environmental

lattice. This classifier has a mean classification accuracy of 93.88 % which is 6.78% higher than for

the baseline 0% classifier. The confusion matrix for the 0.1% classifier is listed in Table 38. This

classifier has the highest classification accuracy, 92.926 %, for no-cloud pixels for the four protein

chemistry classifiers studied. The misclassification rate for this classifier decreased to 1.31 from

2.398 for the baseline classifier with a deactivated protein reaction chemistry.


C0pred C1pred

C0actual 94.60 % 5.398 %

C1actual 7.074 % 92.93 %

Taylor 164 of 249

09/28/10

The low misclassification ratio indicates that the misclassification rates are roughly comparable: a

C1->C0 misclassification occurred over 7.07% of the C1 pixels and a C0->C1 misclassification

occurred over 5.40% of the C0 pixels.

Very little linear correlation seems to exist between the classification accuracies and the fitness of the

evolved 0.1 % reaction probability GRNs. The correlation coefficient is 0.2308 with a corresponding

confidence of 67.3% (Student’s t-test) - a relatively low and uncertain result. The scatter plot of the

trials for the application of this classifier is in Figure 87.


The third classifier, allows for proteins to react with a probability of 1% during the classification

process. It performs with a mean classification accuracy of 85.47 % with the decidedly mixed

confusion matrix in Table 39. Classification accuracies for C0 and C1 are 84.24% and 87.09%

respectively. For C0, this classifier performs 7.71% worse than the baseline classifier, but it performs

6.39 % better than the baseline classifier on the C1 class. The misclassification rate for C1->C0 is

12.91% compared to the baseline 19.30% which is a 6.4 % decrease whereas the misclassification

rate for C0->C1 increases by 7.71% to 15.76. The corresponding misclassification ratio is 0.8189

which indicates that the C0->C1 misclassification rate is now slightly higher than the C1->C0

misclassification rate.

Taylor 165 of 249

09/28/10


C0pred C1pred

C0actual 84.24 % 15.76 %

C1actual 12.91 % 87.09 %

There exists a high confidence (98.2%; Student’s t-test) that the correlation coefficient of 0.5239 for

the classification accuracy and fitness data indicates a moderate amount of linear correlation between

these parameters. Figure 88 shows a scatter plot of the classification accuracy vs. fitness data in

addition to a fitted line through the data.


The final classifier which possesses a 10% protein reaction probability has the highest C0

classification accuracy of 96.57 %. Unfortunately, as Table 40 shows, it also has the lowest C1

Taylor 166 of 249

09/28/10

classification accuracy of 71.75 %. Its misclassifications are unbalanced as is indicated by a

misclassification ratio of 8.239. Overall, the mean classification accuracy across the entire test image

and the trials stands at 85.87 %.


C0pred C1pred

C0actual 96.57 % 3.430 %

C1actual 28.25 % 71.75 %

Low confidence (49.3%; Student’s t-test) in the 0.1577 correlation coefficient leads to the linear fit of

the classification and fitness data that is presented in Figure 89.


Taylor 167 of 249

09/28/10

The results for the four reaction probability classifiers are summarized in Figure 90 in a mean

classification accuracy vs. probability of reaction plot. Peak mean classification accuracy occurred

with the 0.1 % probability classifier and the minimum mean classification accuracy occurred with the

1 % probability classifier. There is a higher standard deviation for the 1 % probability classifier than

the 0.1% and 10 % classifiers.

Figure 90. Full-image classification accuracy vs. the probability of protein reaction in the environmental lattice.

Vertical bars correspond to the standard deviation of the sample mean.


Similar classification accuracies for the 1% and 10% probability classifiers are based on different

failure modes. In the 1% case, there is an increase in both misclassification rates, C0->C1 and C1-

>C0. However, in the 10% case, there is a large increase in C1->C0 misclassifications with a

relatively small C0->C1 misclassification rate.

As the protein chemistry probability increases, C0->C1 misclassification decreases from the baseline

8.046% (0.0 % classifier) to 5.398 % (0.1% classifier). C0->C1 misclassification then increases by a

factor of three (3x) to 15.75% with the 1% classifier before falling back to a low of 3.43% with the

10% classifier.

Taylor 168 of 249

09/28/10

Also, as the probability increases, C1->C0 misclassification decreases from the baseline 19.30 % with

the 1% classifier to 7.074 % with the 0.1% classifier. C1->C0 misclassification then increases to

12.91% with the 1% classifier and peaks at 28.25 % with the 10 % classifier.

The minima for both types of misclassification rates coincide with the highest overall classification

accuracy with the 0.1% classifier. It appears that too much “mutation” in the form of protein reaction

rate decreases the classification accuracy below that of the baseline classifier in which no protein

chemistry is active. “Too much” in this case occurs with the 1% probability of protein chemistry

reaction. Therefore it seems that a small amount of protein reaction probability improves the overall

classification accuracy of the HeBIS algorithm. A small amount, 0.1%, of reaction probability causes

the subsequent creation of new intermediate proteins during the HeBIS classification process. It

appears that class C1 (no-cloud) is more sensitive to the creation of these extra intermediate proteins.

4.5.3.4. Experiment 14 - Shotgun

We are now ready to examine HeBIS with its self-organization and genetic regulatory network bases

activated. Comparisons between HeBIS and a SOFM are made where applicable.

4.5.3.4.1. Setup

Experiment 14

All training parameters are considered and can be modified randomly (uniform distribution for each

selected parameter) to determine classification accuracies. For this experiment, these accuracies are

derived from full classifications of the test image. This test image comes from our standard cloud/no-

cloud dataset.

The list of HeBIS parameters that are varied in these randomized trials is presented in Table 41.

Taylor 169 of 249

09/28/10

Table 41. List of HeBIS Parameters to Randomize for Experiment 14.

Simulation Parameter Range

Number of particles in PSO swarm [1,1000]

Number of PSO breedings [1,100]

Number of environmental/regulatory genes [2, 1000]

Intracellular SOFM kernel size 1x2, 2x1, 3x3, 5x5

Minimum protein correlation (affinity) [0, 1.0]

Protein diffusion rate (cell) [0, 1.0]

Protein diffusion rate (environment) [0, 1.0]

Minimum protein concentration (cell) [0, 1.0]

Minimum protein concentration (environment) [0, 1.0]

Protein magnitude [0, 1.0], [0, 15]

Protein statistics active On/Off

Similarly, Table 42 shows the parameters that are varied for the SOFM trials.

Table 42. List of SOFM Parameters to Randomize for Experiment 14.

Simulation Parameter Range

Kernel size 3x3, 5x5, 7x7, 9x9

Number of trials for coarse initialization 100

Number of trials for fine initialization 1000

Size of coarse neighborhood Limited to max L2 distance within network topology

Size of fine neighborhood 1

In both the HeBIS and SOFM trials, the classification accuracies are examined. Additionally, we also

analyze Receiver Operating Characteristic (ROC) curves to analyze threshold operating points for

classification for HeBIS [117].

ROC curves are useful in estimating the performance of ranking classifiers. They are based on the

concepts of precision, recall, sensitivity, and specificity which are taken from a classifier’s confusion

matrix and are defined in the following equations:

, (28)

, (29)

, (30)

Taylor 170 of 249

09/28/10

, (31)

, (32)

and

. (33)

The ROC curve graphically estimate the effect that increasing the false positive rate for a specific

classifier will have on the true positive rate. An example of a simple ROC curve is show in Figure

91.

Figure 91. Sample ROC curve with false-positive rate along the abscissa and true-positive rate as the ordinate.

The area under the curve (AUC) is a single number which is the area under the tradeoff curve in the

ROC analysis. It ranges from a minimum of 0 to a maximum of 1.0. Higher AUC implies a better

classifier as the area approaches the maximum. In other words, it measures the discrimination power

of the classifier which is the ability of the classifier to correctly classify both positive and negative

class members. Generally, the AUC is not a perfect measure of classification performance, but it is

useful nonetheless.

For these experiments, the AUC is approximated via the construction of trapezoids under the curve.

Finally, two more useful error characterizations are

Taylor 171 of 249

09/28/10

(34)

and

. (35)

.

Further details on the operating theory behind ROC curves are available from [117] and [118].


This experiment consists of both HeBIS and standalone SOFM trials on the test image from the

MODIS A2002193183000 dataset. The distribution of class pixels is listed in Table 1.

Table 43. Distribution of Class and Infrastructure Pixels in A2002193183000

Parameter Value

# C0 (cloud) pixels 19053

# C1 (no-cloud) pixels 14441

# Land pixels 67126

# Water pixels (C0 + C1) 33494

# Invalid pixels 0

The HeBIS trials in this experiment use a minimum environmental and intracellular protein

concentration of 0.01. HeBIS is trained on a total of 18 pixels- 9 pairs of C0/C1 pixels and the

standalone SOFMs are trained on 20 pixels that consist of 10 pairs of C0/C1 pixels. The previously

described 15 bands of spectral data are used as features for each of the class pixels. In the SOFM

case, each pixel consists of a single column of multispectral data without any information from the

surrounding pixels. This SOFM data format is used for training and testing as this is equivalent to a

3x3 HeBIS geographic region. Table 44 and Table 45 list more detailed operational parameters for

the selected HeBIS and standalone SOFM trials, respectively.

Taylor 172 of 249

09/28/10

Table 44. Operational Parameters for Selected HeBIS Shotgun Experiments

HeBISHeBISHeBISHeBIS

Sel.Sel.Sel.Sel. TrTrTrTrial #ial #ial #ial #

HeBISHeBISHeBISHeBIS

ExperimentExperimentExperimentExperiment LabelLabelLabelLabel

DeltaDeltaDeltaDelta

Sec.Sec.Sec.Sec.

BreedsBreedsBreedsBreeds MaxMaxMaxMax

Iter.Iter.Iter.Iter.

Part.Part.Part.Part. RRRReactioneactioneactioneaction

Prob.Prob.Prob.Prob.

OOOOutpututpututpututput

ProteinProteinProteinProtein ValueValueValueValue SwitchSwitchSwitchSwitch

MMMMinininin....

CorrCorrCorrCorr ValueValueValueValue

####

GenesGenesGenesGenes

SOFMSOFMSOFMSOFM

SizeXSizeXSizeXSizeX

SOFMSOFMSOFMSOFM

SizeYSizeYSizeYSizeY

Diff.Diff.Diff.Diff.

RateRateRateRate

Diff.Diff.Diff.Diff.

RateRateRateRate CellCellCellCell

StatStatStatStat

FlagFlagFlagFlag

0 05_28_07_58_58_61

1497 19 2770 197 0.000 0 0.9728 6 3 2 0.0432 0.7501 0

1 06_11_01_18_20_77

1417 19 2999 442 0.01841 1 0.9054 13 0 0 0.3654 0.5130 0

2 06_10_08_03_09_38

377 10 3431 120 0.02188 1 0.9806 14 1 1 0.4264 0.5133 1

3 06_09_16_37_07_17

554 10 1223 229 0.02685 0 0.9998 14 2 2 0.1645 0.5151 0

4 05_28_12_49_29_76

906 17 1184 110 0.000 1 0.9556 19 1 2 0.1226 0.4319 1

5 06_03_10_39_23_85

206 4 2943 270 0.000 0 0.9884 10 0 3 0.9508 0.9441 1

6 05_28_00_32_27_50

3310 16 348 426 0.000 0 0.9516 19 3 1 0.9432 0.8463 0

7 06_09_17_4

3_13_23

73 5 1264 24 0.002965 0 0.9849 15 1 0 0.9069 0.4742 1

8 05_28_12_42_02_75

447 15 779 104 0.000 1 0.9531 9 1 2 0.5592 0.0766 1

9 06_01_23_35_56_3

36881 17 591 436 0.000 0 0.9145 11 2 1 0.0152 0.2865 0

10 06_10_07_35_57_34

460 11 3859 209 0.02688 0 0.9756 4 1 3 0.8637 0.8795 1

Table 45. Operational Parameters for SOFM Experiments

SOFMSOFMSOFMSOFM TrialTrialTrialTrial ####

SOFMSOFMSOFMSOFM Experiment Experiment Experiment Experiment LabelLabelLabelLabel

SOFM SOFM SOFM SOFM SizeXSizeXSizeXSizeX

SOFM SOFM SOFM SOFM SizeYSizeYSizeYSizeY

1st stage 1st stage 1st stage 1st stage #iter.#iter.#iter.#iter.

1st stage 1st stage 1st stage 1st stage alphaalphaalphaalpha

1st stage 1st stage 1st stage 1st stage radiusradiusradiusradius

2nd stage 2nd stage 2nd stage 2nd stage #iter.#iter.#iter.#iter.

2nd stage 2nd stage 2nd stage 2nd stage alphaalphaalphaalpha

2nd stage 2nd stage 2nd stage 2nd stage radiusradiusradiusradius

0 2009.11.10 2 1 1000 0.05 2 100000 0.02 0.5

1 2009.11.11b 2 1 1000 0.05 2 100000 0.02 0.5

2 2009.06.17 2 1 100 0.05 2 1000 0.02 0.5

3 2009.11.11f 5 4 200 0.05 2 2000 0.02 0.5

4 2009.11.11e 3 2 200 0.05 2 2000 0.02 0.5

5 2009.11.11d 2 2 200 0.05 2 2000 0.02 0.5

6 2009.11.11c 3 1 200 0.05 2 2000 0.02 0.5

7 2009.11.11g 5 5 200 0.05 2 2000 0.02 0.5

8 2009.11.11h 8 7 200 0.05 2 2000 0.02 0.5

Taylor 173 of 249

09/28/10

Summaries of the classification results that are discussed in detail in this section are presented in the

following tables. Table 46 lists the results for the selected HeBIS experiments taken from the 279

that are available and Table 47 shows the SOFM experiments.

Table 46. Classification Results for Selected HeBIS Shotgun Experiments

HeBISHeBISHeBISHeBIS Sel.Sel.Sel.Sel. TrialTrialTrialTrial ####

HeBISHeBISHeBISHeBIS ExperimentExperimentExperimentExperiment LabelLabelLabelLabel

FitnessFitnessFitnessFitness % Correct% Correct% Correct% Correct AllAllAllAll VVVValidalidalidalid

% Incorrect% Incorrect% Incorrect% Incorrect AllAllAllAll VVVValidalidalidalid

%%%% C0C0C0C0 ValidValidValidValid CCCCorrectorrectorrectorrect

% C0% C0% C0% C0 VVVValid_alid_alid_alid_ IIIIncorrectncorrectncorrectncorrect

% C1% C1% C1% C1 VVVValid_alid_alid_alid_ CCCCorrectorrectorrectorrect

% C1% C1% C1% C1 VVVValidalidalidalid IIIIncorrectncorrectncorrectncorrect

0 05_28_07_58_58_61 0.000 43.12 56.88 0.000 100 100.0 0.000

1 06_11_01_18_20_77 0.1781 51.41 48.59 15.43 84.569 98.89 1.115

2 06_10_08_03_09_38 0.06596 56.88 43.12 100.0 0.000 0.000 100.0

3 06_09_16_37_07_17 0.2609 62.11 37.89 53.89 46.11 72.95 27.05

4 05_28_12_49_29_76 0.3987 88.98 11.02 81.64 18.36 98.67 1.330

5 06_03_10_39_23_85 0.6919 92.79 7.213 87.99 12.01 99.11 0.8864

6 05_28_00_32_27_50 0.7934 93.66 6.335 88.86 11.14 100.0 0.000

7 06_09_17_43_13_23 0.7637 94.28 5.723 92.27 7.726 96.92 3.082

8 05_28_12_42_02_75 0.5012 96.31 3.687 100.0 0.000 91.45 8.552

9 06_01_23_35_56_3 0.5382 98.38 1.618 99.61 0.3884 96.76 3.241

10 06_10_07_35_57_34 0.7135 99.15 0.8509 98.51 1.485 99.99 0.01385

Table 47. Classification Results for Selected SOFM Experiments

SOFMSOFMSOFMSOFM TrialTrialTrialTrial

####

SOSOSOSOFMFMFMFM Experiment LabelExperiment LabelExperiment LabelExperiment Label

%Correct Total%Correct Total%Correct Total%Correct Total %Incorrect Total%Incorrect Total%Incorrect Total%Incorrect Total %C0 Correct%C0 Correct%C0 Correct%C0 Correct %C0 Incorrect%C0 Incorrect%C0 Incorrect%C0 Incorrect %C1 Correct%C1 Correct%C1 Correct%C1 Correct %C1 Incorrect%C1 Incorrect%C1 Incorrect%C1 Incorrect

0 2009.11.10 63.408 36.59 35.67 64.33 100.0 0.000

1 2009.11.11b 63.408 36.59 35.67 64.33 100.0 0.000

2 2009.06.17 63.77 36.23 36.315 63.69 100.0 0.000

3 2009.11.11f 94.85 5.153 97.71 2.288 91.07 8.933

4 2009.11.11e 95.34 4.661 95.90 4.089 94.59 5.415

5 2009.11.11d 97.12 2.884 99.00 0.9970 94.63 5.374

6 2009.11.11c 97.18 2.821 98.73 1.275 95.14 4.861

7 2009.11.11g 97.20 2.669 95.49 4.509 99.45 0.2424

8 2009.11.11h 97.23 2.771 96.13 3.868 98.68 1.323

All the HeBIS trials used a classification discrimination threshold of 0.5 to classify C0/C1 according

to a single output protein. Note that the ROCs in the detailed examinations present the effects of

varying the threshold so as to tradeoff between classification recall and precision.

Taylor 174 of 249

09/28/10

Briefly examining Table 46 and Table 47, it is interesting to note that comparing both the HeBIS and

standalone SOFM trials with 2x1 SOFM kernels, it is seen that HeBIS outperforms the standalone

SOFM by approximately 33% in overall classification accuracy. This is with an intracellular SOFM

kernel which has the same topology as that of the standalone SOFM classifier, 2x1.

The following examination consists of several types of imagery for each of the listed trials. For each

listed HeBIS example, a figure is presented which contains four images. The classification image

shows those pixels in white that were classified as cloud (C0) pixels. Black pixels in this image

correspond to pixels that were classified as no-cloud (C1) pixels. Red pixels delineate land pixels

that are not considered during the classification. The second image is the difference image that

presents the misclassification errors. A blue pixel in this image shows a cloud pixel that was

misclassified as no-cloud (C0->C1) and a green pixel shows a no-cloud pixel that was misclassified

as a cloud pixel (C1->C0). The third image is the ground truth for the C0 and C1 class distribution.

White indicates cloud (C0) and black shows the no-cloud (C1) class with red as land. The fourth

image is the land mask that presents the distribution of water and land pixels with land pixels

represented as red and water pixels represented as black. There is also an ROC plot associated with

each trial directly underneath the four descriptive images.

Figure 92 shows the imagery for HeBIS selected trial # 0. Only 43.12 % of the pixels were correctly

classified and this was broken down as being 0% of the cloud pixels being classified correctly

although 100 % of the no-cloud pixels were correct. The difference image in this figure highlights the

C0->C1 misclassifications as blue pixels- the entire cloud field is blue in the difference image. The

AUC for the ROC is 0.5 which indicates that the classifier has no discrimination ability between

cloudy and no-cloud (clear-weather) pixels.

Taylor 175 of 249

09/28/10

2010_05_28_07_58_58_61

Classification Difference Cloud/No-Cloud

Ground Truth Land/Water Mask

Figure 92. HeBIS classification imagery and ROC for 2010_05_28_07_58_58_61 test. This is HeBIS selected trial

# 0.

Figure 93 summarizes the results for HeBIS trial # 1. In the classification image we see the

beginnings of an ability to classify clouds. In this trial, the overall classification rate was 51.4 %. of

the overall pixels being correctly classified as being cloudy- an increase of 8.28% from the 43.12 %

in trial #0. Of these correctly classified pixels, 15.43 % of the cloudy pixels were classified correctly

and 98.89 % of the clear-weather pixels were determined correctly. The AUC is 0.604 and the

resulting ROC curve shows a rise in recall at low precision. At the 1 % precision (FP) point, this

classifier possesses approximately 15 % of recall (TP).

Taylor 176 of 249

09/28/10

2010_06_11_01_18_77




# 1.

In Figure 94 we see that the AUC for HeBIS trial # 2 has risen to 0.854. This 0.25 increase in area

under the curve when compared to the AUC for trial #1 results in an improvement in recall to 6% at a

value of 1% precision, but this only results in an increase to 56.88 % for overall classification

performance. More telling is the fact that the breakdown of the misclassification results show that

although 100 % of C0 pixels were classified correctly, no C1 pixels were correct. This translates to

Taylor 177 of 249

09/28/10

the classification image being purely white over water regions and also to the no-cloud areas in the

difference image being filled with green as an indication of the C1->C0 misclassifications. This is an

example of a useless classifier in the sense that all pixels are considered to be cloudy, hence no

optical satellite data would be usable from this image based on the results of this classifier.

2010_06_10_08_03_09_38




# 2.

Taylor 178 of 249

09/28/10

The third selected HeBIS trial summarized in Figure 95, has a C0 classification rate of 53.89 % with

a C1 rate of 72.95 % which results in an overall classification rate of 62.11 %. It is noticeable in the

classification and difference images that there is a mix of C0->C1 and C1->C0 misclassifications.

The error field is dominated by the C0->C1 misclassifications that are highlighted by the speckled

blue field in the difference image. The AUC for this classifier is 0.757 and the ROC has a multi-step

response to varying FP and TP operating points. At a precision of 0.5%, the corresponding recall is

30 %, for a 20% precision, the recall rises to 50%, for a 28% precision, recall rises further to 54%. At

a 31% precision, the recall steps to 78 %.

2010_06_09_16_37_07_17




# 3.

Taylor 179 of 249

09/28/10

A rise to 88.98% overall classification accuracy is seen with the trial #4 classifier which is

summarized in Figure 96. This result consists of 81.64% of class C0 pixels being classified correctly

with a correspondingly large percentage of C1 pixels, 98.67%, being classified correctly as clear-

weather. The ROC curve shows a large increase in low-precision recall compared to the prior trials.

For example, the recall at the 1% precision level has risen to 82%. The green in the difference image

shows the C1->C0 misclassifications. In this image, this type of misclassification is observed to be

mostly in a few areas that are close to the coasts and also in a few areas in open water in the lower

section of the Chesapeake Bay. A rise in AUC to 0.902 corresponds to the improvement in

classification accuracies across both classes.

2010_05_28_12_49_29_76




# 4.

Taylor 180 of 249

09/28/10

Figure 97 shows that the classifier from trial # 5 has an overall classification accuracy of 92.79%.

This corresponds to 87.99 % of C0 pixels being classified correctly as cloudy and 99.11% of all C1

pixels being classified correctly as clear-weather pixels. The AUC of 0.984 correlates with what is so

far, the highest recall for the given lowest precision. For example, the classifier performs with 88%

recall with approximately 0% precision. The difference image indicates that more clouds are being

classified correctly in the confined water areas in the central north-south-oriented inlet and at the end

of the confined inlet in the northeast section of the image.

2010_06_03_10_39_23_85




# 5.

Taylor 181 of 249

09/28/10

HeBIS trial # 6 is summarized in Figure 98. An incremental change in overall classification

accuracy, 93.66%, is seen in comparison to the 92.79% of trial # 5. The 93.66% classification

accuracy is due to 88.86 % of the C0 class and 100% of the C1 class being classified correctly. The

AUC is 0.986 and subsequently, this classifier shows good performance at the low-precision end of

the ROC curve. At an approximate 0% precision, recall has increased by 0.5% to 88.5 % compared

to trial #5. Examining the difference image, it is noticed that the same regions that were classified

correctly for trial #5 were also classified correctly for this trial. However, more of the C0 pixels that

are present at the end of the northeastern water inlet are classified correctly as being cloudy. The

same occurs at the end of the lower-most southwestern water inlet.

2010_05_28_00_32_27_50




# 6.

Taylor 182 of 249

09/28/10

Trial # 7 shows an aggregate classification rate of 94.28 % with a split of 92.27 % of C0 pixels and

96.92 % of C1 pixels being classified correctly. The summary imagery for this trial are shown in

Figure 99. Out of all the trials listed so far, this one has the highest C1->C0 misclassification rate of

3.08 %. We also see the highest C0 classification rate of 92.27 % which compares to 88.86 % for

trial # 6 and 81.64 % for trial # 4. The 7.73 % misclassification rate for C0->C1 is presented in the

difference image as blue pixels. The subsequent 91% recall at an approximate 0 % precision level is

associated with this trial’s AUC of 0.986 which is essentially the same as for selected trial # 6.

2010_06_09_17_43_13_23




# 7.

Taylor 183 of 249

09/28/10

Figure 100 presents selected trial # 8 with an aggregate classification accuracy of 96.31%. 100% of

cloudy (C0) pixels and 91.45% of clear-weather (C1) pixels are classified correctly. This particular

example has the largest C1->C0 misclassification rate of 8.55% across all the selected trials. These

misclassified pixels show up as green in the difference image and they are largely limited to open-

water regions with a few clusters present in the coastal regions. Examining the ROC curve, we see

that 0% recall occurs at the 0 % precision level, but importantly, recall rises quickly to 100 % at the

low-end precision of 8.5%. The 0.957 AUC has decreased compared to that in trials # 6 and # 7.

2010_05_28_12_42_02_75




# 8.

Taylor 184 of 249

09/28/10

In Figure 101, the trial #9 data and imagery support an overall classification rate of 98.38 % with

99.62 % and 96.67 % of all C0 and C1 pixels, respectively, being classified correctly. This

corresponds to an overall misclassification rate of 1.62 % for which the misclassifications for C1->

C0 have decreased from the previous example’s 8.55 % to 3.24 %. These C1->C0 misclassifications

are clustered in the southwestern and northeastern water inlets along the coastal regions and near the

eastern shore inlets and small peninsulas. With an AUC that has increased significantly to 0.982

from trial # 8’s 0.957, there is a similarly approximate 0% recall at 0% precision. However, with a

modest shift in precision to the 4% level, recall jumps dramatically to the 100% level- this is much

better than the ROC performance from trial # 8.

2010_06_01_23_35_56_3




# 9.

Taylor 185 of 249

09/28/10

Finally, Figure 102 presents the summary imagery and ROC for selected HeBIS trial # 10 which has

the best overall classification accuracy of 99.15 %. This corresponds to C0 pixels being correctly

classified 98.51 % of the time and C1 pixels being correctly classified 99.99 % of the time. This trial

also has the lowest average (unweighted according to class balance) misclassification rate across the

two classes of 1.49 % for C0->C1 and 0.014 % for C1-> C0. These misclassification rates

correspond to 284 cloudy pixels being mistakenly classified as clear-weather and 202 clear-weather

pixels being misclassified as cloudy pixels. The corresponding improvement in recall to 98 % at the

0 % precision level which is the best of these trials examined in detail. The AUC is 0.997.

2010_06_10_07_35_57_34




# 10.

Taylor 186 of 249

09/28/10

Table 81 in the Appendix on page 233 lists the evolved genomes for the selected HeBIS trials in

graphical format. In this table, the image associated with each dataset displays the evolvable

parameters from each genome. The hotter colors, for example red, correspond to higher values for

that particular parameter and cooler colors (e.g. blue) correspond to lower relative values for the

parameter. Two color bars are shown because two different parameter ranges exist for different

parameter classes within the genome. For example, the real-valued range [0, 1.0] represents, for

example, expressed protein concentration and the lower and upper switch concentrations. The range

[1, 254] represents integral values that are used to define the 4 descriptive parameters for each

evolved protein in the genome.

Taylor 187 of 249

09/28/10

Table 48 lists the operational parameters and classification results that are used to create the graphical

correlation coefficient matrix in Figure 124 on page 240 in the Appendices and its corresponding p-

value matrix which is also in the Appendices in Figure 125 on page 240. These matrices are for the

279 HeBIS shotgun trials- 200 with protein chemistry deactivated and 79 with protein chemistry

activated.

Table 48. Feature and Result Indices for Correlation Coefficient and P-Value Matrices

Correlation Feature IndexCorrelation Feature IndexCorrelation Feature IndexCorrelation Feature Index FeatureFeatureFeatureFeature and Resultsand Resultsand Resultsand Results

1 Delta Sec

2 # Breeds

3 Max. # Iterations

4 # Particles

5 Reaction Probability

6 Output Protein Switch

7 Min. Correlation Value

8 Number Genes

9 SOFM SizeX

10 SOFM SizeY

11 Diffusion Rate Enviro

12 Diffusion Rate Cell

13 Statistics Flag

14 Fitness

15 # Correct All Valid

16 % Correct All Valid

17 # Incorrect All Valid

18 % Incorrect All Valid

19 # C0 Valid Correct

20 % C0 Valid Correct

21 # C0 Valid Incorrect

22 % C0 Valid Incorrect

23 # C1 Valid Correct

24 % C1 Valid Correct

25 # C1 Valid Incorrect

26 % C1 Valid Incorrect

Taylor 188 of 249

09/28/10

Referencing these figures and the corresponding correlation data, we see that the fitness parameter

(14) is positively correlated with the overall classification accuracy (16). The correlation coefficient

is 0.8183 with a corresponding significance greater than 99.0 % (Student’s t-test). The linear

correlation coefficient for fitness (14) and C0 classification accuracy (20) shows negligible

correlation with a coefficient equal to 0.0526, although the test of this significance is fairly low at

62.82% (Student’s t-test).

A moderate positive correlation of 0.4315 exists between the C0 classification accuracy (22) and the

C1 classification accuracy (24). The significance of this test is strong at greater than 99.0%

(Student’s t-test). Also, a strong correlation coefficient of 0.7140 exists between fitness (14) and

class C1 classification accuracy (24) and it has a corresponding p-value that is close to 0.0 (Student’s

t-test).

The environmental protein diffusion rate (11) is positively correlated with the overall classification

accuracy (16). The correlation coefficient is 0.6130 with a corresponding significance greater than

99.0% (Student’s t-test). However, this diffusion rate (11) is negatively correlated with the overall

misclassification rate (17). The correlation coefficient for this pair of parameters is -0.6130 and its

significance is greater than 99.0% (Student’s t-test). Finally, reaction probability (5) and the class C0

classification rate (20) are paired and examined. The correlation coefficient for this pairing shows a

moderate amount of negative correlation at -0.2525 with a significance that is greater than 99.0%

(Student’s t-test).

Figure 103 continues the shotgun analysis with a plot that compares HeBIS’ overall classification

accuracy with genome fitness for the 200 trials in which protein chemistry is deactivated. The

corresponding fit through the scatter data indicates a good linear fit because of the 0.8213 correlation

coefficient between the two parameters that has greater than 99.0% significance (Student’s t-test).

Taylor 189 of 249

09/28/10

Figure 103. Comparison plot of HeBIS classification accuracy vs. fitness of the genome for 200 trials. Protein

chemistry is deactivated.

Figure 104 plots the same parameters as in Figure 103, but this one is for the 79 trials in which

protein chemistry has been activated. The overall classification accuracy and fitness pair also has a

high positive correlation coefficient, 0.8214 with a significance that is greater than 99.0% (Student’s

t-test).

Figure 104. Comparison plot of HeBIS classification accuracy vs. fitness of the genome for 79 trials with protein

chemistry activated.

Taylor 190 of 249

09/28/10

In Figure 105, classification accuracy is plotted versus reaction probability with an activated protein

chemistry. Reaction probability is the probability that protein chemistry occurs between two proteins

which in turn creates a new third protein in the environmental lattice. Examining the correlation

between these parameters, we see that the correlation coefficient is small, 0.0617 with a significance

greater than 99.0% (Student’s t-test). Thus, little linear correlation is present and the fit has little

predictive value.

Figure 105. HeBIS classification accuracy vs. reaction probability for 79 trials. Protein chemistry is activated.

Conversely, Figure 106 and its correlation data present little significance (41.1 %) (Student’s t-test) to

the slight linear correlation (0.158) between reaction probability and fitness for genomes with

activated protein chemistries.

Taylor 191 of 249

09/28/10

Figure 106. HeBIS classification accuracy vs. reaction probability and fitness for 79 trials. Protein chemistry is

activated.

Another pairing of parameters for which there is no strong correlation is classification accuracy and

minimum protein correlation with protein chemistry activated. A plot of these parameters in addition

to classification accuracy is presented in Figure 107.

Taylor 192 of 249

09/28/10

Figure 107 HeBIS classification accuracy vs. reaction probability and minimum protein correlation for 79 trials.

Protein chemistry is activated.

There is little correlation (0.1070) between the two parameters and the significance of the test is only

65.21% (Student’s t-test)for the data in Figure 107.

Classification accuracy plotted versus environmental protein diffusion rate is shown in Figure 108 for

the 200 trials in which the protein chemistry was inactive. At 0.6313, there appears to be a moderate

amount of linear correlation between the members of the pair as indicated by a significance greater

than 99.0% (Student’s t-test).

Taylor 193 of 249

09/28/10

Figure 108. Classification accuracy vs. environmental diffusion rate for 200 shotgun trials. Protein chemistry is

deactivated. Now, in Figure 109, classification accuracy versus fitness and environmental diffusion rate is plotted.

We’ve already noticed a strong correlation between fitness and classification accuracy with prior data

and now the data shows a high correlation (0.6313) with a significance greater than 99.0% (Student’s

t-test) for the pairing of classification accuracy and environmental diffusion rate.

Figure 109. Classification accuracy vs. genome fitness and environmental diffusion rate for 200 shotgun trials.

Protein chemistry is deactivated.

Taylor 194 of 249

09/28/10

Finally, Figure 115 on page 234 in the Appendices displays a comparison plot of all 279 genomes that

were evolved for the HeBIS shotgun experiments. The color bar from Table 81 on page 233 is

applicable to the evolved elements of the genomes in this figure.


HeBIS’ aggregate classification accuracy ranges from 43.12 % to 99.15 given all 279 shotgun trials.

The worst HeBIS trial is 2010_05_28_07_58_58_61. The standalone SOFM’s aggregate

classification accuracy ranges from 63.41% to 97.23%. The highest overall HeBIS classification rate

occurs with the protein chemistry activated occurs with trial 2010_06_10_07_35_37_34. It would

appear that both the standalone SOFM and HeBIS have more difficulty with classifying the more

generic class, C1- the clear-weather class- compared to classifying the cloud class, C0.

The data show that classification accuracy of class C1 dips down to the 0% level whereas C0

classification accuracy is predominantly high and usually greater than 87.99 %. The increase in

overall classification rate for HeBIS occurs mostly because C0 classification improves from a low of

0% to a high of 99.61 % of all C0 pixels being classified correctly. Most C1 misclassification rates

are fairly low and in between 0 and 8.55 % for the selected examples.

The fitness function is working and providing the GRN with a basis for producing accurate

classifications of cloud pixels. This is indicated by the generally high level of correlation between the

fitness and classification accuracy parameters for the trials in which the protein chemistry was either

activated or deactivated. Successful classifications also point to the success of the fitness function

and show that the function can drive the evolution of a good genome and GRN for this simple

classification problem.

Aggregate classification accuracies compared to protein diffusion rates show the presence of a slight

positively correlated relationship. As the protein diffusion rate increases, the classification rate is

found to increase also. Thus, with the lower diffusion rates, the proteins remain for a longer period of

time in the environmental matrix and conversely with higher diffusion rates, the proteins disappear

more quickly.

Taylor 195 of 249

09/28/10

A summary of the best classification accuracies for the selected trials is listed in Table 49. The top

three highest classification rates were achieved with the HeBIS algorithm and the standalone SOFM

had the lowest classification accuracies.

Table 49. Best Classification Accuracies for the Selected HeBIS and SOFM Examples

Algorithm Trial Ranking Classification Accuracy [%]

HeBIS 1 99.15

HeBIS 2 98.38

HeBIS 3 98.10

SOFM 4 97.23

SOFM 5 97.20

SOFM 6 97.18

Taylor 196 of 249

09/28/10

4.6. Classification robustness Hybridization Background




Robustness Analyses


For this dissertation, classification robustness examines the ability of HeBIS to correctly classify a

corrupted test image. Corruption is divided into two types for these experiments and subsequent

analyses:

• Addition of noise to the multispectral pixel features,

• Features which are missing from the full multispectral set for each pixel.

Both HeBIS and a baseline SOFM are compared where possible.

4.6.2. Experiment 15 - Noise

The first experiment is the noise addition test.

4.6.2.1. Setup

Experiment 15

This tests a single “best” genome that is applied to the classification of a full test image over a

number of trials. Equivalently, a single “best” SOFM codebook is used to determine the SOFM’s

response to noise-corrupted input data.

An equivalent amount of noise is added to both the SOFM and HeBIS test datasets after both have

been trained on a noise-free baseline cloud/no-cloud dataset.

The noise-addition methodology is such that Gaussian noise is added to the spectral feature data with

0.1 probability across the 15 bands at varying levels of standard deviation for the noise. 10 datasets

are created for each of the 4 levels of standard deviation for the injected noise. Three classifications

are performed on each of these 10 datasets- one classification using the “best” HeBIS genome, a

classification with the “best” 2x1 SOFM codebook, and a classification with the “best” 3x1 SOFM

codebook. The noise is added to the datasets before log-normalization occurs. The 40 datasets are

constructed with the 4 noise parameter sets listed in Table 50.

Taylor 197 of 249

09/28/10

Table 50. Dataset definitions for Experiment 15.

Dataset # Mean Standard Deviation

1 0 0.0

2 0 0.01

3 0 0.1

4 0 0.2

The parameters for the HeBIS genome are listed in Table 51 and the parameters for the SOFM

codebooks are in Table 52 and Table 53.

Table 51. Simulation Parameters for the HeBIS "Best" Genome for Experiment 15.


# Genes 3

Intracellular SOFM size 2x1




PSO breedings 10




# PSO particles 400

Table 52. Simulation Parameters for the 2x1 SOFM "Best" Codebook for Experiment 15.


Kernel size 2x1



Size of coarse neighborhood 2

Size of fine neighborhood 0.5

Table 53. Simulation Parameters for the 3x1 SOFM "Best" Codebook for Experiment 15.


Kernel size 3x1



Size of coarse neighborhood 2

Size of fine neighborhood 0.5

Taylor 198 of 249

09/28/10


Figure 110 summarizes the mean classification results for the three classifiers considered in this

experiment: the highest performing SOFM classifier with a 2x1 neuron topology, the highest

performing SOFM classifier with a 3x1 neuron topology and one of the highest performing HeBIS

classifiers from Experiment 14.

Baseline comparisons of the cases with a noise standard deviation of 0.0, essentially with noise

injection deactivated, rank the 3x1 SOFM classifier with the highest mean classification accuracy of

97.16 %. The next highest is the HeBIS classifier that classified 96.2 % of the C0 and C1 pixels

correctly. In last place came the 2x1 SOFM classifier with an aggregate classification rate of 63.78%.

Standard deviation of the averaged classification accuracies was equal to 0.0 for all three classifiers.

This occurred because the 10 datasets used for each of the classifiers are static and no protein

chemistry is active for the HeBIS classifier.

For a noise standard deviation of 0.01, the 3x1 SOFM classifier once again has the highest mean

classification accuracy of 97.18 %- slightly higher by 0.01 % when compared to its baseline with

deactivated noise injection. The HeBIS classifier attained 96.2 % overall classification accuracy

which was essentially unchanged from its baseline. The 2x1 SOFM classifier’s mean classification

accuracy also remained unchanged at 63.78 %. Standard deviations of the mean classification

accuracy were negligible with values of 8.1828x10-5 for the 3x1 SOFM, 7.4832x10-5 for HeBIS and

0.0 for the 2x1 SOFM.

With noise injected at a level with the standard deviation equal to 0.1, the HeBIS classifier attained

the best mean classification accuracy of 98.31 %. This was an improvement of approximately 2%

over its performance with the lower level of standard deviation for the noise. The 3x1 SOFM

achieved the next best mean classification accuracy with 95.07 % which was a reduction of 2.11 %

compared to the 3x1 SOFM classifier with a noise standard deviation of 0.01. The 2x1 SOFM

classifier maintained its performance with a 63.78 % mean classification accuracy. Standard

deviations for this level of noise injection were 0.0039 for the 3x1 SOFM, 0.0057 for HeBIS and 0.0

for the 2x1 SOFM classifier.

Taylor 199 of 249

09/28/10

At an injected noise level of 0.2, the HeBIS classifier maintained its lead position over the other two

classifiers by performing at an 85.87 % mean classification accuracy. This was a reduction of 9.2%

from its performance with the 0.1 inject noise level. The next best performer was the 3x1 SOFM in

which it attained 81.69 % classification accuracy- a reduction of 13.38 % from its performance with

the 0.1 noise level. The 2x1 SOFM classifier increased its mean classification accuracy to 67.29 %

from 63.78 % which it attained with the lower 0.1 noise level. Standard deviations of the

classification accuracies over the trials increased for all classifiers. The 3x1 SOFM data indicated a

standard deviation of accuracy of 0.031, 0.0731 for HeBIS and 0.0227 for the 2x1 SOFM classifier.

Figure 110. Noise comparison plot for classification accuracy vs. noise standard deviation for both HeBIS and

SOFM trials. Vertical bars correspond to the standard deviation of the sample mean of classification accuracy.


At the baseline noise level with a standard deviation of 0.0, the difference between the classification

averages is less than 1% between HeBIS and the 3x1 SOFM kernel. The 2x1 kernel is less resistant

to noise because its mapping is more coarse than the 3x1 kernel.

HeBIS maintains a significant 3.24 – 4.18 % improvement in classification accuracy over the next

best SOFM for noise levels that have a standard deviation that is greater than or equal to 0.1. Raw

data from these trials are included in the Appendices in Table 79 on page 232.

Taylor 200 of 249

09/28/10

4.6.3. Experiment 16 - Missing features

In the second robustness experiment, we examine performance with missing features in the

classification test dataset.

4.6.3.1. Setup

Experiment 16

This also tests a single “best” HeBIS genome and a “best” SOFM codebook on a full-sized test image

from the cloud/no-cloud dataset. In this case, the features associated with spectral wavelength 862-

877 nm (MODIS Band 16) and wavelength 2105-2155 nm (MODIS Band 7) are negated in separate

trials by zeroing the band value after log-normalization of the testing dataset has been applied. These

two bands are chosen because they are both used for cloud detection in NASA operational

processing, although we primarily use MODIS band 7 in our case study.

The same genome and SOFM codebooks (2x1 and 3x1) used in the noise test (Experiment 15) are

applied in this experiment. All the pixels in the test image have the chosen cloud-detection feature

“knocked out” and the resulting classification accuracies are collected for 10 trials of HeBIS and a

single SOFM trial for comparison. Also, HeBIS and SOFM classification results for this

genome/codebook applied to the non-knocked-out datasets with no injected noise from Experiment

15 are examined as baselines.


The bar chart in Figure 111 presents a before-and-after comparison of mean classification accuracies

with the tests conducted with MODIS band 16 knocked out. The three classifiers examined are a

high performing HeBIS classifier with protein chemistry deactivated, the highest performing 3x1

SOFM classifier and the highest performing 2x1 SOFM classifier. The 3x1 SOFM has the best

before-knockout mean classification accuracy with a 97.78% success rate which falls by 7.73% to

89.45 % accuracy after the knockout. The HeBIS classifier has a baseline mean classification

accuracy of 96.2% and after the knockout it remains unchanged. Before the band 16 knockout, the

2x1 SOFM classifier achieved a 63.78% success rate in aggregate classification over the entire test

image, however after the knockout this decreased slightly to 63.44%.

Taylor 201 of 249

09/28/10

Figure 111. Comparison of before and after classification accuracy for the MODIS band 16 knockout.

These classification results for the MODIS band 16 knockout are summarized in Table 54.

Table 54. Missing Feature Comparison for Classification Accuracy Using HeBIS and SOFM

Algorithms with MODIS Band 16 Knockout

Algorithm Mean Standard Deviation

HeBIS Baseline (exp. 15, 0.0) 0.9620 0.0000

SOFM 2x1 Baseline (exp. 15, 0.0) 0.6378 0.0000


HeBIS Knockout (10 trials) 0.9620 0.0000

SOFM 2x1 Knockout 0.6344 0.0000


In Figure 112, a comparison of before-and-after classification results for the same three classifiers is

presented but for the MODIS band 7 knockout.

Taylor 202 of 249

09/28/10

Figure 112. Comparison of before and after classification accuracy for the MODIS band 7 knockout.

Each of the three classifiers essentially maintains its performance regardless of whether or not band 7

is modified. In decreasing order of mean classification performance, the 3x1 SOFM classifier

achieved 97.18 % classification accuracy before the knockout and it slightly increased by 0.20 %

after the modification. HeBIS came in as a close second with a mean classification accuracy before

the knockout of 96.20 % which it maintained after the knockout. The 2x1 SOFM came in last place

with a slight decrease from 63.78 % mean classification accuracy before the knockout to 63.68 %

accuracy after the knockout. Table 55 summarizes these results for the MODIS band 7 trials.

Standard deviations of the trials for both MODIS band 16 and 7 are 0 because the same dataset was

used and the knockouts were not random, but static and no protein chemistry was active.

Table 55. Missing Feature Comparison for Classification Accuracy Using HeBIS and SOFM

Algorithms with MODIS Band 7 Knockout

Algorithm Mean Standard Deviation

HeBIS Baseline (exp. 15, 0.0) 0.9620 0.0000



HeBIS Knockout (10 trials) 0.9620 0.0000



Taylor 203 of 249

09/28/10


It is apparent that the coarse 2x1 SOFM map has essentially no significant dependencies on MODIS

bands 7 or 16 although there was a very slight (0.10%) decrease in classification accuracy after

MODIS band 7 was modified. The coarse mapping of the 2x1 kernel helps in the cases of both band

7 or band 16 being modified. For the 3x1 SOFM kernel, its mapping does not depend on band 7,

however it is dependent on band 16 as indicated in Figure 111. The HeBIS GRN has derived an

algorithm that is not explicitly dependent on the band (or potential band) used to create the cloud/no-

cloud ground truth data.

Taylor 204 of 249

09/28/10

5. Summary and Conclusions

The focus of this thesis has been the novel hybridization of machine learning and biological

principles towards solving a relevant satellite image classification problem. This work proposed an

image classification system (HeBIS) built upon artificial protein diffusion, several principles of

evolutionary development, a control structure evolved from an artificial genetic regulatory network,

and the inclusion of a SOFM as a basic machine learning element. To our knowledge, this was the

first study of the evolution of genetic regulatory networks for control of cellular classification kernels

in the optical remote sensing domain.

The effectiveness of a genetic regulatory network applied to binary multi-dimensional classification

was examined in detail and the following contributions were made:

• Novel application of a biological construct to a practical computational classification problem

• Determination of the effectiveness of a simplified GRN applied to multi-dimensional classification

• Artificial proteins communicate classification information and results to and from the cellular machine learning kernels.

• Training of a GRN via particle swarm optimization (PSO).

• Application of a GRN-based classification system to a real-world multispectral remote sensing problem domain, i.e. cloud detection in optical satellite imagery.

• Performance comparison of HeBIS and a SOFM-only classification algorithm on a remotely-sensed multispectral dataset with unadulterated features, noisy features and deleted features.

These contributions follow from the experiments within the HeBIS environment that were conducted.

The literature review in Chapter 2 provides the theoretical basis for HeBIS. This literature review is

vital to the underpinnings of HeBIS as the system is based on the fusion of several, somewhat

disparate concepts: machine learning kernels, evolutionary development (evo-devo), genetic

regulatory networks and biological proteins, self-organization and evolutionary computation in the

form of particle swarm optimization.

Chapter 3 introduced the details of HeBIS.

The sixteen experiments in Chapter 4 were designed to characterize the behavior of HeBIS in a

detailed case study. Beginning with Experiment 1 and proceeding through Experiment 16, different

and increasingly more complex aspects were examined. This began with the investigation of the

Taylor 205 of 249

09/28/10

creation and diffusion of artificial proteins in an artificial lattice, proceeded to the mapping of an

artificial protein chemistry onto the created proteins, and examined the creation of simple genetic

regulatory networks. Then, the effects of PSO on the evolution of genomes and their corresponding

GRNs were investigated. The effects of different self-organizing and cellular behaviors were

examined and finally, these pieces were combined and studied under various classification scenarios.

In these classification tests, the fitness of the evolved GRNs was examined as well as classification

accuracy among the binary classes.

Experiments 1-4 examined protein creation, diffusion and the effects of a simple protein chemistry

overlaid on the basic genetic-based protein interactions. These experiments specifically targeted the

case in which only low-level spectral proteins were present in the environmental lattice because

protein activations/inhibitions were deactivated. Experiments 1 and 2 showed that the HeBIS

environmental proteins were able to diffuse through the artificial lattice whereas Experiments 3 and 4

showed that the protein chemistry varied the static protein interaction and diffusion performance that

was discovered in the first two experiments. Through these four experiments, it was determined that

the environmental proteins could interact in the lattice and create new proteins. This was an

important conclusion as the more complicated HeBIS behaviors all build upon these low-level protein

interactions.

Experiment 5 examined HeBIS in more detail with respect to gene activation and the creation of

simple genetic regulatory networks. This experiment determined if gene activation could occur and

whether it could be used as the basis for GRNs. The experiment showed that it was possible for

genes (and their resultant proteins) that were evolved with evolutionary computation (PSO in the case

of HeBIS) to become activated. A diversity of GRN behaviors were also shown to exist within the

evolved GRNs. This behavioral diversity consisted of differing genetic temporal activations for the

same gene, cellular cloning, and differing pixel responses at the GRN level. It was also determined

that the HeBIS fitness function is not necessarily geared towards simultaneous, multiple-gene

expression. In any case, the presence of these diverse behaviors provided the basis for the follow-on

experiments.

Genome/GRN optimization via evolutionary computation (PSO) was the subject of Experiment 6.

The utility of the PSO as an optimizer in HeBIS’ artificial protein environment was examined. It was

determined that the PSO was useful as an agent of optimization for HeBIS. In particular, search

diversity within the particle swarm was such that the overall fitness of the swarm increased both as

Taylor 206 of 249

09/28/10

the PSO iterated through successive breedings and as the number of particles in the swarm was

increased.

Self-organization and its effect on the GRN were examined in Experiments 7-10. The various self-

organization and cellular actions were examined individually and mutually exclusive of the other

actions in the set. The fitness of the GRN was evaluated for the following cases: cell location

(Experiment 7), cellular behaviors (Experiment 8), the injection of protein statistics into the

environment (Experiment 9), and the activation of output proteins (Experiment 10).

Experiment 7 varied the location of the initial starter cell and it was found that this choice did not

have a significant effect on the fitness of the evolved GRN- when this action was evaluated in

isolation of the other actions.

Experiment 8 tested the individual effects of various cellular actions. It was determined that the

independent use of these actions changed the GRN fitness when compared against the baseline in

which no cellular actions were initiated. Therefore, the activation and inhibition of these cellular

actions were found to have significant utility with respect to changes in the evolution of more

complicated classification behaviors.

In Experiment 9, the inclusion of statistical information into the artificial lattice was investigated.

This information consisted of protein analogs of local second-order statistics that were injected into

the environment during the simulation. Comparisons were made between the cases in which the

parameters of these statistics were either evolved by the PSO or were forced to remain static during

the simulation. The results indicate that bred statistical protein parameters were more likely to

significantly affect the fitness of the GRN than the static parameters were. This was potentially due

to the bred statistical proteins being subverted by the PSO to improve the fitness of the corresponding

GRN.

Experiment 10 compared the use of static versus bred output protein parameters. The results of this

experiment indicated that static output protein parameters helped to produce higher GRN fitnesses

than when bred output protein parameters were used.

Up to this point of the research, only individual actions and behaviors were tested in isolation so as to

determine the “impulse response”, per say, of HeBIS. With Experiments 11-16, HeBIS’ behaviors

Taylor 207 of 249

09/28/10

and modules were allowed to act in concert during the study of cloud/no-cloud classification of a full-

sized test image. In these experiments, genomes/GRNs were evolved for the purpose of

classification.

The effect of including more or less information from pixels that were geographically close to the test

pixel was examined in Experiment 11. For the limited number of cases studied, it was found that

classification accuracy actually decreased as the size of the included geographic region increased.

Experiment 12 surveyed the effect on classification accuracy due to changes in the size of the

intracellular SOFM kernel in HeBIS. Significant dependencies were found in the full-image test

classification that were based on the change in kernel size. In particular, misclassifications from class

to class were found to be the important component associated with classification success.

Experiment 13 added the before-studied protein chemistry to the gene-dictated protein interactions

from the GRN and examined the subsequent effects on classification accuracy. The conclusion was

that the mutations (new proteins) introduced into the HeBIS environment from the protein chemistry

were useful for improving classification accuracy on the full test image. However, it was found that

accuracy decreased if the probability of protein reaction increased beyond a threshold.

Fully-active HeBIS and standalone SOFMs were compared in Experiment 14. Analyses consisted of

confusion matrices, presentation of ROC curves, and visual examination of the classified imagery. In

this experiment, HeBIS was found to classify the test image as well or better than the standalone

SOFM trials. In particular, the simple HeBIS fitness function worked well enough to successfully

grade the evolved GRNs used for classification.

The final two experiments, Experiment 15 and 16, judged the robustness of HeBIS under varying

levels of injected noise and also with the information from certain features deleted entirely from the

simulation. In both cases, HeBIS was found to classify the full test image as good or better than the

standalone SOFM classifiers.

Throughout the development and research of HeBIS we have obtained results that are encouraging.

We have determined that biological inspiration hybridized with machine learning is definitely a

powerful paradigm with many interesting and useful qualities. The application of HeBIS to a real-

world classification problem in this case study demonstrates that these techniques can provide the

Taylor 208 of 249

09/28/10

basis for successfully solving relevant satellite image classification problems. There are many

theoretical and practical avenues for future research now that the HeBIS demonstration system has

been developed.

Taylor 209 of 249

09/28/10

6. Future research

This thesis opens up several possibilities for future research. In addition to applying the classifier to

other problem domains, these possibilities include the exploration of alternative fitness functions, the

inclusion of varied machine learning kernels on the cellular level, research into learned functional

modularity, the addition of new cellular actions, the use of fast processors to improve simulation turn-

around time, and finally the exploration of the potential for using protein communications in

biological systems.

6.1. Fitness functions

The current HeBIS fitness function produces simple GRNs that classify the cloud/no-cloud problem

well. However, classification accuracy vs. fitness scatter plots indicate that the current function is

somewhat noisy and that other fitness functions might improve classification performance. Also, the

HeBIS fitness function does not force a multiple-gene response and it could be interesting to explore

functions that evoke more complex GRNs such as this.

The application of Novelty Search may be useful in producing more complex GRNs [119]. Novelty

search encourages exploration of complexity without the explicit definition of fitness. In this case,

novelty search could be applied in place of the HeBIS fitness function and it could allow the system

to explore diverse operating points in an evaluated “action space”. With HeBIS, the PSO could still

remain the driver of diverse exploration under this potential avenue of research.

6.2. Additional machine learning kernels

Multiple, different kernels linked with the HeBIS GRN and its protein communications could be used

as the basis for protein-based aggregate classifiers. Machine learning kernels such as support vector

machines and artificial neural networks are obvious candidates for inclusion as both have been

extensively used in remote sensing applications [120, 121, 122].

6.3. Modularity and learned functionality

Modularity analyses could be used to uncover linkages between cluster of protein interactions along a

protein interaction pathway. These modularity analyses could be borrowed from computational

biology research [123, 124].

Additionally, learned modular functionality could be an interesting avenue of exploration through the

application of NeuroEvolution of Augmenting Topologies (NEAT) [125, 126]. Through NEAT,

complex control and classification problems can be split into separate sub-problems for which the

Taylor 210 of 249

09/28/10

first sub-problem is learned. This learned structure is then used as the basis for learning the next sub-

problem with learning of the remaining sub-problems following in an iterative fashion.

HeBIS already incorporates ideas of modularity with its independent cells that can have different

behaviors depending on the local protein concentrations and gradients. Modular learning in the

system could be invoked with separate sub-problem training such as that portrayed in Figure 113.

Figure 113. One method of presenting training/test data to HeBIS. Each behavior is trained separately, candidate

genomes are created, and the candidates then undergo evolutionary optimization in a final CV/GA loop.

6.4. Additional cellular actions

New cellular actions can be incorporated easily in HeBIS and would provide another venue for

potential future research. New cellular actions could include the application of scale-free network

theory to protein communications, the addition of the concept of mutual information for protein

interactions, research into new cellular instantiation algorithms, and adaptive sizing of the

intracellular SOFM kernel.

Taylor 211 of 249

09/28/10

6.4.1. Scale-free networks

The cell could be modified to promote scale-free protein communication links between other cells in

the HeBIS GRN (network). Scale-free networks are seen throughout biology, particularly in neural

connection networks [127, 128]. Also known as “1/f scaling”, where f is the distance between cells.

In future research, this distance could be correlated with inter-cell protein concentrations as follows:

The number of protein communication links is defined as the number of proteins from one cell that

are released into the environment that affect another cell in the environment. The number of these

communication links for cells that are close to each other in the environmental lattice is large and this

number decreases as the distance between communicating cells increases. However, this value never

falls completely to zero as the distance increases, instead it falls off as

(36)

where s is a scaling factor that accounts for the small network sizes used in the simulations.

The distance is defined as the distance between two cells in normalized units of length- 1 unit of

integral distance in the environmental lattice.

A cell in the HeBIS network could attempt to produce a scale-free environment by monitoring the

environmental proteins that are in its local neighborhood. The concentrations of these proteins could

be used to probabilistically determine which of these environmental proteins may be used as switch

templates for intracellular behaviors. Environmental proteins with high concentrations will have a

lower probability of being used as a switch template than those proteins with low concentrations.

A scale-free network could potentially allow information to efficiently move between upper and

lower levels of a network hierarchy as well as across the network.

6.4.2. Mutual information

Mutual information also provides a venue for future research [129]. Mutual information is a measure

of the amount of information that can be determined about one random variable given information

about another random variable.

Taylor 212 of 249

09/28/10

It is defined as

I X Y` a

=Xx,y

p y` a

p x|yb c

logp x|yb c

p x` a

ffffffffffffffffffffffh

l

j

i

m

k, (37)

and equivalently,

Xx,y

p x,y` a

logp x,y` a

p x` a

p y` a

fffffffffffffffffffffffffffffffffh

j

i

k. (38)

For this research, mutual information could be defined between different proteins and their

concentrations. The mathematical analog is that the protein is similar to a random variable where its

concentration is similar to the sampled value of the random variable. Through this protein mutual

information, the cell could attempt to use those proteins that appear to have more information content

than others. This process could occur purely from the perspective of the local cell and the

environmental proteins that it has seen in prior time steps. This is another method by which

irrelevant features in the protein feature vector may be discarded, thus potentially improving the

classification by decreasing the effect of the curse of dimensionality in large-dimensional spaces.

Another method through which this feature reduction could occur is through the GRN- both from the

unmapping of feature proteins that are not considered important by the network as well as the fact

that the added complexity of the GRN itself could allow for better management of the curse.

6.4.3. Cellular instantiation

Currently, new cells are instantiated purely when the corresponding switch for the ADDCELL action

is activated by specific evolved proteins within the artificial environment. One idea to extend this

paradigm is to always force cellular instantiation when protein concentration reaches a particular

level in the environment. Acting in conjunction (or separately) with the current ADDCELL action,

this type of instantiation could provide a more direct link to spectral texture information from the

input features as it is encoded by the prevalence of the artificial spectral proteins in different

geographic regions of the image. A second avenue of exploration is to base cellular instantiation at

the beginning of classification on the number of input domain features. This jumpstart could

potentially simplify the evolution of a suitable GRN for complex classification problems.

Taylor 213 of 249

09/28/10

6.5. Graphics Processing Unit

The amount of evolutionary computation required to evolve the HeBIS network requires a robust

system with high processing throughput. Because of this, future research is predicated on building a

processing system based on a high-performance graphics processing units (GPU) that can provide

higher sustained floating point processing throughput compared to a state-of-the-art CPU [12,113].

The computational hardware required for fast iteration through the HeBIS training and testing cycles

requires a processing capability that is several times greater than what is currently available in a state-

of-the-art i7-based desktop computer. This need can be addressed either through the acquisition of

time on a supercomputer, the creation of a relatively small processing cluster, or through the

innovative use of commodity computer graphics chips. By using a commodity graphics processing

unit (GPU) that is readily available for high-end desktop gaming and visualization computers, the

promise is that an order of magnitude increase in computation throughput can be achieved through

the use of a relatively inexpensive high-end graphics card [113]. As an example, harnessing the

specialized processor hardware on the $450 ATI X800 XT GPU will provide 63 GFLOPs of

sustained processing throughput compared to a peak of 14.8 GFLOPs for a 3.7 GHz Pentium 4

computer [113]. The “secret” behind this computational cornucopia is that the hardware structure in

the GPU is optimized for high-speed parallel mathematical operations on pixels whereas the typical

CPU is optimized for sequential code. This highlights the problem that is associated with the use of a

GPU as an inexpensive, desktop supercomputer: How does one efficiently map a general purpose

computation to hardware that has been designed for efficient graphics rendering and pixel

manipulation? This computational problem is being addressed by researchers and has resulted in

several high-level programming languages that are similar to C/C++. Examples include ‘sh’ from the

University of Waterloo and Brook from Stanford [113]. Although these (and other) languages are

still no panacea for the GPU computational mapping problem, they have recently been used for

evolutionary computation applications. [114] and [115] are examples of genetic algorithms that have

been ported to a GPU. Because of this, it is believed that future development and analysis of HeBIS-

inspired learning algorithms should be implemented on a GPU so as to enable quick iteration and

testing for research purposes.

Taylor 214 of 249

09/28/10

Figure 114. Processing model for research infrastructure.

6.6. Protein-based communications for artificial devices in a biological system

There is a potential that the HeBIS protein-based communications and classification could be used for

controlling human-made devices in biological systems, e.g. a human body. These devices could

conceivably be designed to respond to the local concentrations and gradients of specific proteins and

to perform internal computations based on a codebook of simple, artificial proteins.

Taylor 215 of 249

09/28/10

7. Appendices

Taylor 216 of 249

09/28/10

7.1. Data

Table 56. Experiment 6 Aggregate Breeding Data.

Breed Average

Best

Fitness

(1)

Best

Fitness

SDV

(1)

Average Best

Fitness

(100)

Best

Fitness

SDV

(100)

Average Best

Fitness

(250)

Best

Fitness

SDV

(250)

Average Best

Fitness

(500)

Best

Fitness

SDV

(500)

0 0 0 0 0 0 0 0 0

1 0.00027655 0.00177126 0.00464188 0.0117422 0.01555976 0.03278979 0.01398984 0.02187814

2 0.00085993 0.00487531 0.00901192 0.02481268 0.02385866 0.04189065 0.02643543 0.03746593

3 0.00098304 0.00506875 0.01277754 0.02968328 0.03147017 0.04936366 0.03700024 0.05196669

4 0.00118006 0.00548549 0.01675406 0.03625152 0.03616746 0.05304285 0.04489608 0.06000474

5 0.00137479 0.0059233 0.01812938 0.03766535 0.03998255 0.05612186 0.04783097 0.06140292

6 0.0015833 0.00645447 0.01954484 0.03929089 0.04166388 0.05620707 0.05216529 0.06440029

7 0.00168214 0.00655383 0.02140479 0.04321551 0.0434137 0.05645667 0.05899074 0.06897797

8 0.00176476 0.00667782 0.02267569 0.04443178 0.04584552 0.05630136 0.06503971 0.07238485

9 0.0018308 0.00677083 0.02385972 0.04586475 0.05028934 0.05904922 0.0723335 0.07971921

Table 57. Experiment 7 Data.

Feature Index Mean Fitness Fitness Standard Deviation

1 0.039564735 0.031173286

2 0.0510463 0.048527315

3 0.04498486 0.026785561

4 0.036350745 0.022174415

5 0.04720598 0.031625397

6 0.04069623 0.024782471

7 0.041230425 0.032343847

8 0.05012401 0.037711992

9 0.062870475 0.026917273

10 0.05576429 0.033785508

11 0.05222655 0.02367747

12 0.04195883 0.026432196

13 0.045210895 0.028171183

14 0.04615925 0.031231122

15 0.038886235 0.023972954

Taylor 217 of 249

09/28/10


Genetic Action Index Genetic Action Mean Fitness Fitness Standard Deviation

1 NO_ACTION 0.020973752 0.035764639

2 ADD_CELL 0.012882272 0.031065069

3 PRUNE_SELF 0.015769768 0.028084739

4 ACTIVATEENVIROPROTEIN 0.068884336 0.037255274

5 INHIBITENVIROPROTEIN 0.029064268 0.049464757

6 ACTIVATEREGPROTEIN 0.030676984 0.040561945

7 INHIBITREGPROTEIN 0.01569026 0.033994529

8 CHANGETOSOFMANDTRAIN 0.036585236 0.037505919

9 CLASSIFY 0.020820384 0.035469301

Table 59. Experiment 9 Data

Protein Statistics Index Protein Statistics Action Mean Fitness Fitness Standard Deviation

10 STATS_ACTIVE_STATIC 0.020746292 0.026351562

11 STATS_ACTIVE_BRED 0.303572754 0.048304406


Protein Output Index Protein Output Action Mean Fitness Fitness Standard Deviation

12 OUTPUT_ACTIVE_STATIC 0.406778162 0.067970135

13 OUTPUT_ACTIVE_BRED 0.18472541 0.054844318

Table 61. Experiment 11 Data for 3x3 and 5x5 Geographic Region Comparison

Number of Neurons Classification Accuracy Mean Classification Accuracy Standard Deviation

3x3 0.9147 0.0812

5x5 0.8396 0.1361

Taylor 218 of 249

09/28/10

Table 62. Experiment 11 Scatter Data for 3x3 Geographic Region

Trial Index Fitness Classification Accuracy

0 0.6210 0.9699

1 0.5719 0.9296

2 0.2208 0.9307

3 0.5971 0.8596

4 0.6307 0.9755

5 0.3914 0.9182

6 0.5943 0.9699

7 0.4180 0.9182

8 0.6386 0.9307

9 0.5956 0.8849

10 0.5461 0.9699

11 0.4488 0.8596

12 0.5418 0.9256

13 0.6250 0.9307

14 0.4414 0.5688

15 0.5816 0.9112

16 0.5040 0.9699

17 0.5790 0.9182

18 0.6234 0.9755

19 0.4105 0.8860

20 0.5035 0.9755

21 0.6005 0.8602

22 0.6473 0.9307

23 0.6057 0.9299

24 0.6042 0.9699

Taylor 219 of 249

09/28/10

Table 63. Experiment 11 Scatter Data for 5x5 Geographic Region


0 0.5444 0.8594

1 0.5146 0.9537

2 0.1974 0.9072

3 0.5073 0.9041

4 0.4320 0.7085

5 0.4898 0.5595

6 0.6136 0.9106

7 0.6091 0.9041

8 0.4648 0.5603

9 0.3714 0.5551

10 0.4566 0.9356

11 0.4599 0.8193

12 0.4796 0.8841

13 0.6337 0.9035

14 0.4145 0.5603

15 0.5397 0.9488

16 0.5960 0.9439

17 0.4394 0.8689

18 0.5477 0.9642

19 0.5349 0.9426

20 0.4859 0.8741

21 0.4761 0.8717

22 0.6164 0.9488

23 0.4752 0.8732

24 0.3505 0.8282

Table 64. Experiment 12 - Intracellular SOFM Data

# Neurons in Intracellular SOFM Classification Accuracy Mean Classification Accuracy Standard Deviation

0 0.8894 0.1123

1 0.8484 0.1678

4 0.8736 0.1371

9 0.9107 0.0921

81 0.8716 0.1362

Taylor 220 of 249

09/28/10

Table 65. Experiment 12 Scatter Data for 0x0 Intracellular SOFM


0 0.1300 0.9307

1 0.5400 0.8527

2 0.1600 0.5809

3 0.5700 0.9307

4 0.5200 0.9301

5 0.5400 0.8849

6 0.6000 0.9699

7 0.6400 0.8596

8 0.4500 0.8860

9 0.5300 0.9301

10 0.1700 0.9182

11 0.5000 0.8849

12 0.6200 0.9699

13 0.6500 0.9755

14 0.5700 0.9699

15 0.6200 0.9182

16 0.4800 0.9307

17 0.4500 0.9182

18 0.5900 0.9699

19 0.5100 0.5766



0 0.0800 0.5722

1 0.5800 0.9307

2 0.0700 0.5704

3 0.2700 0.9755

4 0.5600 0.9755

5 0.6100 0.9699

6 0.4900 0.5713

7 0.5600 0.9182

8 0.6200 0.9699

9 0.0800 0.5704

10 0.5200 0.9480

11 0.6000 0.9755

12 0.4700 0.9182

13 0.5900 0.9128

14 0.5100 0.8860

15 0.5600 0.9307

16 0.6200 0.9755

17 0.6000 0.9755

18 0.6000 0.8527

19 0.4200 0.5690

Taylor 221 of 249

09/28/10



0 0.1300 0.9182

1 0.4300 0.9182

2 0.4600 0.9307

3 0.6200 0.9182

4 0.6400 0.9755

5 0.3600 0.8527

6 0.5000 0.9301

7 0.5700 0.9755

8 0.5100 0.8860

9 0.2700 0.9307

10 0.1900 0.9307

11 0 0.5688

12 0.5300 0.8596

13 0.5700 0.9699

14 0.3300 0.5694

15 0.5400 0.9699

16 0.4000 0.5694

17 0.5000 0.9699

18 0.5800 0.9755

19 0.5300 0.8527



0 0.6000 0.9307

1 0.4700 0.9182

2 0.4800 0.9699

3 0.5700 0.9301

4 0.1700 0.9182

5 0.4800 0.9015

6 0.6200 0.8596

7 0.5200 0.9755

8 0.5100 0.9755

9 0.6300 0.9301

10 0.4400 0.8596

11 0.4900 0.9307

12 0.2400 0.9699

13 0.6100 0.9307

14 0.3300 0.5688

15 0.4200 0.9301

16 0.6300 0.9755

17 0.4800 0.9182

Taylor 222 of 249

09/28/10



0 0.1400 0.5809

1 0.5800 0.9699

2 0.5700 0.8934

3 0.6600 0.8849

4 0.5700 0.9755

5 0.6000 0.9755

6 0.5800 0.9755

7 0.5600 0.8527

8 0.4700 0.5689

9 0.5000 0.9301

10 0.5300 0.9699

11 0.5900 0.8602

12 0.5300 0.8602

13 0.4400 0.8527

14 0.5100 0.9699

15 0.5300 0.9182

16 0.5200 0.9182

17 0.6400 0.9307

18 0.3700 0.5691

19 0.5600 0.9755

Table 70. Experiment 13 Aggregate Classification Data

Classification Accuracy Mean Classification Accuracy Mean Classification Accuracy Standard Deviation

0.0 0.8710 0.1163

0.0010 0.9388 0.0397

0.0100 0.8547 0.1298

0.1000 0.8587 0.0590

Taylor 223 of 249

09/28/10

Table 71. Experiment 13 Data for 0.0 Reaction Probability


0 0.44 0.8527

1 0.65 0.9699

2 0.46 0.8527

3 0.51 0.8602

4 0.54 0.9699

5 0.58 0.8849

6 0.52 0.9755

7 0.45 0.9301

8 0.6 0.8849

9 0.59 0.9301

10 0.56 0.9307

11 0.52 0.8849

12 0.59 0.9699

13 0.3 0.7603

14 0.6 0.8596

15 0.61 0.9301

16 0.4 0.5692

17 0.59 0.8527

18 0.44 0.5768

19 0.57 0.9755



0 0.5200 0.9179

1 0.5900 0.9699

2 0.5700 0.9301

3 0.5400 0.9675

4 0.5700 0.9301

5 0.6600 0.9697

6 0.5300 0.9755

7 0.6300 0.9307

8 0.6000 0.9979

9 0.4600 0.9698

10 0.5400 0.9591

11 0.6500 0.9685

12 0.6600 0.8618

13 0.5300 0.9755

14 0.6600 0.9263

15 0.4600 0.8587

16 0.4500 0.9185

17 0.4600 0.8988

18 0.5000 0.8848

19 0.6000 0.9649

Taylor 224 of 249

09/28/10



0 0.6100 0.8905

1 0.5400 0.9733

2 0.6400 0.9024

3 0.2600 0.6263

4 0.2500 0.9685

5 0.3600 0.8812

6 0.5300 0.7710

7 0.2900 0.6460

8 0.5200 0.9670

9 0.5800 0.9141

10 0.3900 0.8814

11 0.6100 0.9746

12 0.6500 0.8537

13 0.4900 0.5754

14 0.5100 0.8971

15 0.6500 0.9257

16 0.6500 0.9695

17 0.6400 0.8626

18 0.3000 0.6457

19 0.6400 0.9682



0 0.4200 0.7717

1 0.5600 0.8577

2 0.4500 0.8443

3 0.5600 0.9217

4 0.4400 0.9075

5 0.4000 0.8372

6 0.6000 0.8839

7 0.5100 0.7744

8 0.5000 0.7864

9 0.4700 0.9323

10 0.4600 0.9343

11 0.6000 0.8880

12 0.5600 0.8590

13 0.6100 0.8876

14 0.4600 0.7847

15 0.4000 0.9026

16 0.5800 0.8928

17 0.5800 0.8441

18 0.5200 0.9194

19 0.5200 0.7437

Taylor 225 of 249

09/28/10

Table 75. Experiment 14 - Statistical Summary Data for HeBIS Classification Accuracy and

Fitness Scatter Data

Statistic Fitness Classification Accuracy

min 0 0.4312

max 0.82 0.9631

mean 0.3034 0.7123

median 0.265 0.6361

mode 0 0.5688

std 0.2425 0.1515

range 0.82 0.5319

Table 76. Experiment 14 HeBIS Scatter Data for Fitness and Classification Accuracy


0 0.35 0.8684

1 0.36 0.8838

2 0 0.5688

3 0.18 0.5688

4 0.37 0.5688

5 0 0.57

6 0.39 0.674

7 0.67 0.7889

8 0.2 0.6198

9 0 0.5699

10 0 0.5688

11 0.12 0.621

12 0.18 0.9203

13 0.07 0.6121

14 0 0.5716

15 0.17 0.6427

16 0.36 0.569

17 0.56 0.919

18 0.12 0.569

19 0.24 0.5884

20 0.07 0.5711

21 0.47 0.9288

22 0.3 0.7117

23 0.37 0.7632

24 0.26 0.6105

25 0 0.5688

26 0 0.5688

27 0.73 0.8951

28 0 0.5688

29 0.15 0.6726

30 0.08 0.5831

31 0.32 0.6716

32 0.1 0.5953

33 0.59 0.6755

Taylor 226 of 249

09/28/10

34 0.22 0.5688

35 0.64 0.9343

36 0.82 0.9444

37 0.2 0.6895

38 0 0.569

39 0.65 0.8639

40 0.06 0.6134

41 0.27 0.7418

42 0.32 0.569

43 0.65 0.9479

44 0.38 0.6085

45 0.47 0.9029

46 0.02 0.5852

47 0.64 0.8562

48 0.07 0.6295

49 0.21 0.7469

50 0.79 0.9366

51 0 0.5688

52 0.31 0.8761

53 0.55 0.9232

54 0.42 0.8246

55 0.06 0.5688

56 0.76 0.9446

57 0.49 0.8262

58 0.51 0.8336

59 0.54 0.9073

60 0.38 0.6833

61 0 0.4312

62 0.63 0.8887

63 0.7 0.8272

64 0.15 0.5688

65 0.26 0.9243

66 0.42 0.8811

67 0 0.5793

68 0.05 0.5691

69 0.06 0.5902

70 0.08 0.5869

71 0.71 0.9339

72 0.53 0.9042

73 0 0.569

74 0.8 0.8719

75 0.5 0.9631

76 0.4 0.8898

77 0.17 0.5688

78 0.17 0.9009

79 0.39 0.5688

80 0.69 0.9294

81 0.53 0.9136

82 0.08 0.6078

83 0.69 0.9284

84 0 0.5688

85 0 0.5688

86 0.13 0.607

87 0.26 0.7946

88 0.24 0.5727

89 0.07 0.5734

90 0.69 0.9229

91 0.45 0.5688

Taylor 227 of 249

09/28/10

92 0.28 0.5688

93 0.55 0.8152

94 0.22 0.7591

95 0.16 0.5932

96 0.29 0.5688

97 0.43 0.903

98 0.3 0.569

99 0.05 0.569

100 0.3512 0.569

101 0.1675 0.5688

102 0.8094 0.7428

103 0.5382 0.9838

104 0.384 0.937

105 0.0635 0.5697

106 0.3124 0.6927

107 0.0075 0.5688

108 0.3182 0.5904

109 0 0.569

110 0.2841 0.7454

111 0.0955 0.639

112 0 0.5688

113 0.597 0.8123

114 0.0541 0.5945

115 0 0.5688

116 0.644 0.9519

117 0.6829 0.8592

118 0.2669 0.7003

119 0.5179 0.8729

120 0.4633 0.8294

121 0.2206 0.7428

122 0.2768 0.767

123 0.0236 0.5881

124 0 0.5688

125 0.5824 0.8975

126 0.753 0.7552

127 0.5752 0.9372

128 0.3422 0.5713

129 0.6215 0.9183

130 0 0.5688

131 0.8747 0.9308

132 0.5956 0.893

133 0.1371 0.721

134 0.4704 0.7995

135 0 0.5688

136 0.7757 0.8628

137 0.2016 0.5688

138 0 0.5691

139 0.1274 0.5688

140 0.5313 0.7463

141 0.0852 0.5688

142 0.072 0.5689

143 0.0406 0.5761

144 0.6003 0.9181

145 0 0.569

146 0.2368 0.7162

147 0.7508 0.9179

148 0.5126 0.7956

149 0.1095 0.5689

Taylor 228 of 249

09/28/10

150 0.1983 0.5814

151 0.0687 0.5814

152 0.2488 0.7254

153 0.0701 0.5892

154 0.5141 0.8566

155 0.076 0.6006

156 0.1471 0.576

157 0.5882 0.8395

158 0 0.5688

159 0.6008 0.6286

160 0.1597 0.6639

161 0 0.5688

162 0.5688 0.9198

163 0.6854 0.9321

164 0.2625 0.655

165 0.4991 0.981

166 0.0682 0.5688

167 0.2776 0.5748

168 0.6011 0.9228

169 0.3835 0.5767

170 0.6597 0.8832

171 0.1813 0.5688

172 0.4722 0.8957

173 0.1865 0.6232

174 0.4205 0.758

175 0.6355 0.7591

176 0 0.5688

177 0.0717 0.5811

178 0.611 0.9339

179 0.4897 0.768

180 0.2623 0.6364

181 0.1793 0.5688

182 0.4865 0.782

183 0.2089 0.6704

184 0.1413 0.6657

185 0.6919 0.9279

186 0 0.5714

187 0.852 0.9811

188 0.3809 0.9444

189 0.0795 0.5688

190 0 0.5696

191 0.6799 0.8897

192 0.4964 0.6963

193 0.2776 0.5764

194 0.3512 0.8427

195 0.7043 0.9405

196 0.1496 0.6303

197 0.3157 0.6895

198 0.5535 0.9443

199 0 0.5752

200 0.5244 0.9232

201 0.1725 0.5688

202 0.0944 0.5193

203 0.5127 0.87

204 0.4279 0.8524

205 0.16 0.5688

206 0.1586 0.5706

207 0.6497 0.9426

Taylor 229 of 249

09/28/10

208 0.5101 0.8879

209 0.0601 0.5688

210 0.5363 0.9291

211 0.1667 0.6643

212 0.5847 0.8678

213 0.1706 0.6508

214 0.4105 0.7503

215 0.393 0.8961

216 0.1094 0.5708

217 0.2609 0.6211

218 0.2866 0.7646

219 0.2873 0.7791

220 0.4912 0.6764

221 0.0912 0.5365

222 0.7354 0.9081

223 0.7637 0.9428

224 0.5285 0.9397

225 0.0925 0.6266

226 0.4085 0.6736

227 0.3312 0.7676

228 0.5134 0.9169

229 0.3269 0.6624

230 0.3437 0.5688

231 0.2975 0.5728

232 0.2059 0.6038

233 0.3235 0.5688

234 0.7135 0.9915

235 0.1953 0.5688

236 0.5159 0.9009

237 0.1986 0.5714

238 0.066 0.5688

239 0.301 0.923

240 0.2326 0.6269

241 0.0978 0.5688

242 0.6236 0.9259

243 0.1691 0.5688

244 0.3569 0.5792

245 0.6249 0.7031

246 0.7232 0.9076

247 0.5947 0.8508

248 0.3011 0.7383

249 0.5918 0.7605

250 0.8685 0.9284

251 0.1299 0.7746

252 0.3794 0.9368

253 0.1588 0.6353

254 0.2952 0.6571

255 0.4624 0.8175

256 0.4037 0.8389

257 0.0248 0.5703

258 0.2766 0.6795

259 0.3142 0.5751

260 0.1762 0.5279

261 0.1342 0.5688

262 0.219 0.6046

263 0.4781 0.9275

264 0.7801 0.8865

265 0.6403 0.8943

Taylor 230 of 249

09/28/10

266 0.3154 0.6276

267 0.1297 0.5738

268 0.552 0.91

269 0.2652 0.7138

270 0.5755 0.8408

271 0.0558 0.4681

272 0.2192 0.5688

273 0.6579 0.931

274 0.473 0.8005

275 0.4082 0.9295

276 0.8855 0.8565

277 0.1781 0.5141

278 0.1648 0.5763

Table 77. Experiment 14 - Aggregate Classification Accuracy Results for SOFM.

Trial

#

Classification

Accuracy

SOFM

Topology

# of

Neurons

1st stage

#iterations

1st

stage

alpha

1st

stage

radius

2nd stage

#iterations

2nd

stage

alpha

2nd

stage

radius

600 0.637726 2x1 2 100 0.05 2 1000 0.02 0.5

601 0.634084 2x1 2 1000 0.05 2 100000 0.02 0.5

602 0.634084 2x1 2 1000 0.05 2 100000 0.02 0.5

603 0.971786 3x1 3 200 0.05 2 2000 0.02 0.5

604 0.971159 2x2 4 200 0.05 2 2000 0.02 0.5

605 0.953395 3x2 6 200 0.05 2 2000 0.02 0.5

606 0.948468 5x4 20 200 0.05 2 2000 0.02 0.5

607 0.971995 5x5 25 200 0.05 2 2000 0.02 0.5

608 0.972294 8x7 56 200 0.05 2 2000 0.02 0.5

Taylor 231 of 249

09/28/10

Table 78. Selected Results from Experiment 14

Test Dataset c_50 c_61 c_75 c_76 d_3 d_85 e_23 e_34 e_38 e_77

Delta_sec 3310 1497 447 906 36881 206 73 460 377 1417

superBreedingCounterMax

16 19 15 17 17 4 5 11 10 19

counterMax 348 2770 779 1184 591 2943 1264 3859 3431 2999

numberParticles 426 197 104 110 436 270 24 209 120 442

reactionProbability 0 0 0 0 0 0 0 0.03 0.02 0.02

outputProteinValueSwitch

0 0 1 1 0 0 0 0 1 1

minCorrelationValue 0.95 0.97 0.95 0.96 0.91 0.99 0.98 0.98 0.98 0.91

numberGenes 19 6 9 19 11 10 15 4 14 13

sofmSizeX 3 3 1 1 2 0 1 1 1 0

sofmSizeY 50 61 75 76 3 85 23 34 38 77

diffusionRate 0.94 0.04 0.56 0.12 0.02 0.95 0.91 0.86 0.43 0.37

diffusionRateCell 0.85 0.75 0.08 0.43 0.29 0.94 0.47 0.88 0.51 0.51

statFlagCell 0 0 1 1 0 1 1 1 1 0

Fitness 0.79 0 0.5 0.4 0.54 0.69 0.76 0.71 0.07 0.18

#Correct_all_valid 31372 14441 32259 29803 32952 31078 31577 33209 19053 17220

%Correct_all_valid 93.66 43.12 96.31 88.98 98.38 92.79 94.28 99.15 56.88 51.41

#Incorrect_all_valid 2122 19053 1235 3691 542 2416 1917 285 14441 16274

%Incorrect_all_valid 6.34 56.88 3.69 11.02 1.62 7.21 5.72 0.85 43.12 48.59

#C0_valid_correct 16931 0 19053 15554 18979 16765 17581 18770 19053 2940

%C0_valid_correct 88.86 0 100 81.64 99.61 87.99 92.27 98.51 100 15.43

#C0_valid_incorrect 2122 19053 0 3499 74 2288 1472 283 0 16113

%C0_valid_incorrect 11.14 100 0 18.36 0.39 12.01 7.73 1.49 0 84.57

#C1_valid_correct 14441 14441 13206 14249 13973 14313 13996 14439 0 14280

%C1_valid_correct 100 100 91.45 98.67 96.76 99.11 96.92 99.99 0 98.89

#C1_valid_incorrect 0 0 1235 192 468 128 445 2 14441 161

%C1_valid_incorrect 0 0 8.55 1.33 3.24 0.89 3.08 0.01 100 1.11

Taylor 232 of 249

09/28/10

Table 79. Data for Experiment 15- Classification Accuracy for 0.1 Probability Noise Injection

with Varying Noise Standard Deviations

Trial 0.0_2x1 0.01_2x1 0.1_2x1 0.2_2x1 0.0_3x1 0.01_3x1 0.1_3x1 0.2_3x1 0.0_hebis 0.01_hebis 0.1_hebis 0.2_hebis

0 0 0.6378 0.6378 0.6378 0.7258 0.9718 0.9716 0.9502 0.7841 0.962 0.9607 0.9783 1 1 0.6378 0.6378 0.6378 0.6875 0.9718 0.9716 0.9476 0.8135 0.962 0.9606 0.9849 2 2 0.6378 0.6378 0.6378 0.6488 0.9718 0.9717 0.9505 0.8455 0.962 0.9605 0.9862 3 3 0.6378 0.6378 0.6378 0.662 0.9718 0.9717 0.9525 0.8193 0.962 0.9607 0.9813 4 4 0.6378 0.6378 0.6378 0.6703 0.9718 0.9716 0.9528 0.8329 0.962 0.9606 0.9818 5 5 0.6378 0.6378 0.6378 0.6554 0.9718 0.9716 0.9554 0.8377 0.962 0.9607 0.9836 6 6 0.6378 0.6378 0.6378 0.6609 0.9718 0.9718 0.957 0.8342 0.962 0.9607 0.9942 7 7 0.6378 0.6378 0.6378 0.6691 0.9718 0.9717 0.9459 0.8094 0.962 0.9607 0.9837 8 8 0.6378 0.6378 0.6378 0.6896 0.9718 0.9718 0.945 0.7469 0.962 0.9606 0.9854 9 9 0.6378 0.6378 0.6378 0.66 0.9718 0.9718 0.9506 0.8451 0.962 0.9606 0.9719

Table 80. Data for Experiment 16 – Classification Accuracy with MODIS Band 15 Knocked

Out

Trial 2x1 SOFM 3x1 SOFM HeBIS

0 0.634387 0.894503 0.962017

1 0.634387 0.894503 0.962017

2 0.634387 0.894503 0.962017

3 0.634387 0.894503 0.962017

4 0.634387 0.894503 0.962017

5 0.634387 0.894503 0.962017

6 0.634387 0.894503 0.962017

7 0.634387 0.894503 0.962017

8 0.634387 0.894503 0.962017

9 0.634387 0.894503 0.962017

Taylor

Table 81. Comparison of Selected Genomes for Shotgu

HeBIS

Sel.

Trial

#

Dataset

0 2010_05_28_07_58_58_61

1 2010_06_11_01_18_20_77

2 2010_06_10_08_03_09_38

2a 2010_06_09_16_37_07_17

3 2010_05_28_12_49_29_76

4 2010_06_03_10_39_23_85

5 2010_05_28_00_32_27_50

6 2010_06_09_17_43_13_23

7 2010_05_28_12_42_02_75

8 2010_06_01_233_35_56_3

9 2010_06_10_07_35_57_34

. Comparison of Selected Genomes for Shotgun Experiments

Genome

233 of 249

n Experiments

Colorbars

Taylor

7.2. Genome mappings for

Figure 115. Parameter activation maps for genomes discovere

Table 81 is applicable to the evolved elements of the genomes in this figure.

Genome mappings for shotgun data

. Parameter activation maps for genomes discovered during shotgun experiments. The color bar from

is applicable to the evolved elements of the genomes in this figure.

234 of 249

d during shotgun experiments. The color bar from

Taylor

7.3. Protein diffusion example

Figure 116. Single protein diffusion from four sites within an

initial sites of protein activation at the beginning of the simulation

Figure 117. Frame #2 in the simulation

iteration 2 of the diffusion simulation

two green sites are decaying. The light blue color represents sites within the lattice that hav

protein concentrations at this point in the simulation.

Protein diffusion example

ffusion from four sites within an 11x11x11 cubic environment. This frame

of protein activation at the beginning of the simulation

#2 in the simulation This frame is a snapshot of activity in the environmental lattice after

of the diffusion simulation. The two red protein sites are still actively producing proteins whereas the

The light blue color represents sites within the lattice that hav

protein concentrations at this point in the simulation.

235 of 249

11x11x11 cubic environment. This frame shows four

environmental lattice after

. The two red protein sites are still actively producing proteins whereas the

The light blue color represents sites within the lattice that have the lowest non-zero

Taylor 236 of 249

09/28/10

Figure 118. Frame #6. The sites colored red are still actively producing whereas the light-blue-colored and dark-

blue-colored sites possess lower concentrations of the simulated protein. The darker blue sites contain lower

concentrations of protein than the light-blue sites. This is the snapshot from iteration 6 of the simulation.

Figure 119. Frame #26. After 26 iterations, the artificial protein has diffused throughout a large portion of the

11x11x11 environmental matrix. Hotter colors (e.g. red, yellow, green) correspond to higher concentrations of the

proteins whereas cooler colors (e.g. light blue, blue) correspond to areas of relatively low concentrations.

Taylor 237 of 249

09/28/10

Figure 120. Frame #40. At iteration 40, the protein has diffused throughout the environmental lattice. The red

sites are the locations of the original and continuing protein sources. Hotter colors correspond to higher protein

concentrations whereas cooler colors correspond to lower concentrations.

Taylor 238 of 249

09/28/10

7.4. HeBIS fitness function details

Figure 121. Regions of equivalent CorrC0 max

for the fitness function.

Figure 122. Θcorr

2 portion of fitness function.

Taylor 239 of 249

09/28/10

Figure 123. Magcorr

2 portion of fitness function.

Taylor 240 of 249

09/28/10

7.5. HeBIS Shotgun Experiment Correlation Maps and PDF

Figure 124. Correlation coefficient grid for processing parameters and classification results. Parameters and

results are numbered from 1 to 26.

Figure 125. Significance p-value grid for processing parameters and classification results obtained with a

Student’s t-test. Parameters and results are numbered from 1 to 26.

Taylor 241 of 249

09/28/10

7.6. HeBIS training cycle details

Figure 126. HeBIS classification training cycle.

Taylor 242 of 249

09/28/10

Bibliography [1] D.A. Landgrebe, “Multispectral Land Sensing: Where From, Where to?”, IEEE Transactions on Geoscience and Remote Sensing, vol. 43, no. 3, pp. 414-421, Mar. 2005. [2] A. Ranganathan and Z. Kira, “Self-Organization in Artificial Intelligence and the Brain”, College of Computing, Georgia Institute of Technology, 2003. [3] P. Bentley, “Evolving Fractal Gene Regulatory Networks for Robot Control”, Department of Computer Science, University College London, 2003. [4] J. Bongard, “Evolving Modular Genetic Regulatory Networks”, Artificial Intelligence Laboratory, University of Zurich. [5] S. B. Carroll, “Endless Forms Most Beautiful”, W.M. Norton & Company, New York, 2005. [6] T.G. Dietterich, “Machine Learning”, Department of Computer Science, Oregon State University, 2001. [7] R.P. Lippmann, Pattern Recognition, class, MIT Lincoln Laboratory, 2001. [8] Alek Kolcz, System Architect, AOL, private communication, 2004. [9] Y. Yingjie, C. Hinde, and D. Gillingwater, “Improved neural network training using redundant structure,” Proc. of the Int. Joint Conference on Neural Networks 2003, vol. 3, pp. 2023-2027, 2003. [10] R. Reed, “Pruning algorithms- a survey”, IEEE Transactions on Neural Networks, vol. 4, no. 5, pp. 740-747, Sept. 1993. [11] S. Osowski and K. Siwek, “Regularisation of neural networks for improved load forecasting in the power system”, IEEE Proceedings on Generation, Transmission and Distribution, vol. 149, no. 3, pp. 340-344, May 2002. [12] C.-A. Bohn, Kohonen, "Feature Mapping through Graphics Hardware”, Computational Intelligence and Neuroscience (ICCIN ’98), 1998. [13] H. Weiming, D. Xie, T. Tieniu, and S. Maybank, “Learning activity patterns using fuzzy self-organizing neural network”, IEEE Trans. On Systems, Man and Cybernetics, Part B, vo. 34, no. 3, pp. 1618-1626, June 2004. [14] W.S. McCulloch and W.S. Pitts, “A Logical Calculus of the Ideas Immanent in Nervous Activity”, Bulletin of Mathematical Biophysics, vol. 5, pp. 115-133, 1943. [15] M.L. Minsky, “Steps Toward Artificial Intelligence”, Proceedings of the Institute of Radio Engineers, vol. 49, no. 1, pp. 8-30, January 1961. [16] J.J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities”, Proceedings of the National Academy of Sciences, USA, vol. 79, no. 8, pp. 2554-2558, 1982. [17] D.O. Hebb, “The Organization of Behavior: A Neuropsychological Theory”, Wiley, New York, 1949. [18] D.E. Rumelhart and J.L. McClelland, “Parallel Distributed Processing: Explorations in the Microstructure of Cognition Vol. 1”, MIT Press, Cambridge, MA, 1986. [19] G.A. Carpenter and S. Grossberg, “ART 3: Hierarchical Search Using Chemical Transmitters in Self-Organizing Pattern Recognition Architectures”, Neural Networks, vol. 3, pp.129-152, 1990. [20] S. Haykin, “Neural Networks: A Comprehensive Foundation”, Prentice Hall, Upper Saddle River, NJ, 1999.

Taylor 243 of 249

09/28/10

[21] “Pulsed Neural Networks”, ed. W. Maass and C.M. Bishop, MIT Press, Cambridge, MA, 1999. [22] Gomez, J., “Co-Evolving Recurrent Neurons Learn Deep Memory POMDPs”, GECCO ’05, June 25-29, 2005, Washington, DC, 2005. [23] T. Kohonen, “Self-Organizing Maps”, Springer-Verlag, Berlin, 2

nd. Edition, 1997.

[24] D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning Internal Representations by Error Propagation,” Parallel Distributed Processing: Explorations in The Microstructure of Cognition, D.E. Rumelhart and J.L. McClelland, eds., vol. 1, pp. 318-362, Cambridge, Mass.: MIT Press, 1986. [25] M. Kukar and I. Kononenko, “Cost-Sensitive Learning with Neural Network,” Proc. 13

th European Conf.

Artificial Intelligence, pp. 445-449, 1998. [26] S. Lawrence, I. Burns, A. Back, A.C. Tsoi, and C.L. Giles,” Neural Network Classification and Prior Class Probabilities,” Lecture Notes in Computer Science 1524, G.B. Orr and K.-R. Muller, eds., pp. 299-313, Berlin: Spring, 1998. [27] Z.-H. Zhou and X.-Y. Liu, “Training Cost-Sensitive Neural Networks with Methods Addressing the Class Imbalance Problem,” IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 1, pp. 63-77, January 2006. [28] J. Sum, C.-S. Leung, G.H. Young, and K. Wing-Kay, “On the Kalman filtering method in neural network training”, IEEE Transactions on Neural Networks, vol. 10, no. 1, pp. 161-166, Jan. 1999. [29] G. Lera and M. Pinzolas, “Neighborhood based Levenberg-Marquardt algorithm for neural network training”, IEEE Transactions on Neural Networks, vol. 13, no. 5, pp. 1200-1203, Sep 2002. [30] J. Go, G. Han, H. Kim, and C. Lee, “Multigradient: A New Neural Network Learning Algorithm for Pattern Classification”, IEEE Transactions on Geoscience and Remote Sensing, vol. 39, no. 5, pp. 986-993, May 2001. [31] J.F.G. de Freitas et al, “Sequential Monte Carlo Methods to Train Neural Network Models”, Neural Computation, vol. 12, 2000, pp. 955-993. [32] A. Vieira, “An Iterative Artificial Neural Network for High Dimensional Data Analysis”, Springer Verlag, Berlin, 2005. [33] J.C. Bongard, “Incremental Approaches to the Combined Evolution of a Robot’s Body and Brain”, University of Zurich, dissertation, 2003. [34] Peter J. Bentley, “Fractal Proteins”, Genetic Programming and Evolvable Machines Journal, Kluwer Academic Publishers, London, v5, pp. 71-101. [35] J.M. Benyus, “Biomimicry: Innovation Inspired by Nature”, Harper Perennial, London, 2002. [36] S. A. Brueckner et al., editors, “Engineering Self-Organising Systems – Methodologies and Applications”, Lecture Notes in Artificial Intelligence, LNAI 3464, Springer, Heidelberg, 2005. [37] S. Kumar and P.J. Bentley, “On Growth, Form and Computers”, Elsevier Academic Press, Amsterdam, 2003. [38] A. Lindenmayer, “Mathematical models for cellular interaction in development, parts I and II”, Journal of Theoretical Biology, vol. 18, pp. 280-315. [39] Hunt, J.E., & Cooke, D.E., “Learning Using an Artificial Immune System”, Journal of Network and Computer Applications, vol. 19, no. 2, pp. 189-212, 1996. [40] R. Calabretta, et al, “Duplication of Modules Facilitates the Evolution of Functional Specialization”, Artificial Life, vol. 6, pp. 69-84, 2000.

Taylor 244 of 249

09/28/10

[41] Sung Young Jung, “A Topographical Method for the Development of Neural Networks for Artificial Brain Evolution”, Artificial Life, vol. 11, pp. 293-316, 2005. [42] A. Horzyk and R. Tadeusiewicz, “Self-Optimizing Neural Networks”, Lecture Notes in Computer Science, vol. 3173, pp. 150-155, Springer Verlag, Berlin, 2004. [43] A. Horzyk and R. Tadeusiewicz, “A new Extension of Self-Optimizing Neural Networks for Topology Estimation”, ICANN 2005, pp. 415-420, Springer Verlag, Berlin, 2005. [44] H. Kitano, “Neurogenetic learning: an integrated method of designing and training neural networks using genetic algorithms”, Physica D, pp. 225-238, Aug. 1, 1994. [45] H. Kitano, “Designing Neural Networks Using Genetic Algorithms with Graph Generation System”, Complex Systems, vol. 4, pp. 461-476, 1990. [46] F. Del Frate et al, “Wheat cycle monitoring using radar data and a neural network trained by a model”, IEEE Transactions on Geoscience and Remote Sensing, vol. 42, no. 1, pp.35-44, Jan. 2004. [47] P. Cipollini, G. Corsini, M. Diani, and R. Grasso, “Retrieval of Sea Water Optically Active Parameters from Hyperspectral Data by Means of Generalized Radial Basis Function Neural Networks”, IEEE Transactions on Geoscience and Remote Sensing, vol. 39, no. 7, pp. 1508-1524, July 2001. [48] R. Lepage, R.G. Rouhana, B. St.-Onge, R. Noumeir, and R. Desjardins, “Cellular Neural Network for Automated Detection of Geological Lineaments on Radarsat Images”, IEEE Transactions on Geoscience and Remote Sensing, vol. 38, no. 3, pp. 1224-1233, May 2000. [49] V.M. Krasnopolsky and H. Schiller, “Some Neural Network Applications in Environmental Sciences. Part I: Forward and Inverse Problems in Geophysical Remote Measurements”, Neural Networks, vol. 16, no. 3-4, pp. 321-334, April-May 2003. [50] A. Niang, L. Gross, S. Thiria, F. Badran, and C. Moulin, “Automatic Neural Classification of Ocean Colour Reflectance Spectra at the Top of the Atmosphere”, Remote Sensing of Environment, vol. 86, no. 2, pp. 257-271, 30 July 2003. [51] Z. Liu, A. Liu, C. Wang, and Z. Niu, “Evolving Neural Network Using Real Coded Genetic Algorithm (GA) for Multispectral Image Classification”, Future Generation of Computer Systems, vol. 20, no. 7, pp. 1119-1129, 1 October 2004. [52] H.-M. Chi and O.K. Ersoy, “A Statistical Self-Organizing Learning System for Remote Sensing Classification”, IEEE Transactions on Geoscience and Remote Sensing, vol. 43, no. 8, pp. 1890-1900, Aug. 2005. [53] Y.-H. Pao and Y. Takefuji, “Functional-Link Net Computing: Theory, System Architecture, and Functionalities”, Computer, pp. 76-79, May 1992. [54] E. Erwin, K. Obermayer, and K. Schulten, “Self-organizing maps: ordering, convergence properties and energy functions”, Biological Cybernetics, vol. 67, pp. 47-55, 1992. [55] T. Kohenen, “Speech recognition based on topology preserving neural maps”, Neural Computing Architectures, ed. I. Aleksander, pp. 26-40, MIT Press, Cambridge, 1989. [56] F. Favata and R. Walker, “A Study of the Application of Kohonen-type Neural Networks to the Traveling Salesman Problem”, Biological. Cybernetics, vol. 64, pp. 463-468, 1991. [57] D. Alahakoon, S.K. Halgamuge, and B. Srinivasan, “Dynamic Self-Organizing Maps with Controlled Growth for Knowledge Discovery”, IEEE Trans. On Neural Networks, vol. 11, no. 3, pp. 601-614, May 2000. [58] K.H. Low, W.K. Leow, and M.H. Ang, “An Ensemble of Cooperative Extended Kohonen Maps for Complex Robot Motion Tasks”,Neural Computation, vol. 17, pp. 1411-1445, June 2005.

Taylor 245 of 249

09/28/10

[59] T.M. Martinetz and K.J. Schulten, “A ‘neural-gas’ network learns topologies”, Artificial Neural Networks, ed. Kohonen, T. Makisara, K., Simula, O., and Kangas, J., pp. 397-402, North-Holland, Amsterdam, 1991. [60] T. Villman, et al, “Topology Preservation in Self-Organizing Feature Maps: Exact Definition and Measurement”, IEEE Trans. On Neural Networks, vol. 8, no. 2, pp. 256-266, March 1997. [61] A. Neme and P. Miramontes, “Statistical Properties of Lattices Affect Topographic Error in Self-organizing Maps”, Springer-Verlag, Berlin, 2005. [62] S. Strogatz, “Exploring Complex Networks”, Nature, vol. 410, pp. 268-276, 2001. [63] D. Watts and S. Strogatz, “Collective dynamics of ‘small-world’ networks”, Nature, vol. 393, pp. 440-442, 1998. [64] P. Somervuo, Kohonen, T., “Self-Organizing Maps and Learning Vector Quantization for Feature Sequences”, Neural Processing Letters, vol. 10, pp. 151-159, 1999. [65] A. Rauber, D. Merkl, and M. Dittenbach, “ The Growing Hierarchical Self-Organizing Map: Exploratory Analysis of High-Dimensional Data”, IEEE Trans. On Neural Network, vol. 13, no. 6, pp. 1331-1341, November 2002. [66] B. Fritzke, “Growing Grid – A Self-Organizing Network with Constant Range and Adaptation Strengths”, Neural Processing Letters, vol. 2, no. 5, pp. 1-5, 1995. [67] B. Fritzke, “Growing Neural Gas Network Learns Topologies”, Advances in Neural Information Processing Systems 7, ed. G. Tesauro, MIT Press, Cambridge MA, 1995. [68] S. Marsland, J. Shapiro, U. Nehmzow, “A self-organising network that grows when required”, Neural Networks, Vol. 15, 2002, pp. 1041-1058. [69] A. Kurz, “Constructing Maps for Mobile Robot Navigation Based on Ultrasonic Range Data”, IEEE Trans. On Systems, Man and Cybernetics – Part B: Cybernetics, vol. 26, no. 2, pp. 233-242, 1996. [70] C.M. Bishop, M. Svensen, and C.K.I. Williams, “GTM: A Principled Alternative to the Self-Organizing Map”, Advances in Neural Information Processing Systems 9, MIT Press, 1997. [71] T. Furukawa, “SOM of SOMs: Self-organizing Map Which Maps a Group of Self-organizing Maps”, Springer-Verlag, Berlin, 2005. [72] T. Kohonen, “The Hypermap Architecture”, Artificial Neural Networks, ed. K.Mkisara, Ol. Simula, and J. Kangas, pp. 1357-1360, 1991. [73] G. Vauche, N. Mozayyani, V. Alanou, and J.F. Dreyfus, “A Spatio-Temporal Data-Coding Applied to Kohonen Maps”, Proc. of the International Conference on Artificial Neural Networks, 2:75-79, 1995. [74] P. Campoy and C.J. Vicente, “Residual Activity in the Neurons Allows SOMs to Learn Temporal Order”, Springer-Verlag, Berlin, 2005. [75] C.H. Chen and B. Shrestha, “Classification of Multi-Sensor Remote Sensing Images Using Self-Organizing Feature Maps and Radial Basis Function Networks”, IGARSS 2000, vol. 2, 24-28, pp. 711-713, July 2000. [76] B. Solaiman and M.C. Mouchot, “A Comparative Study of Conventional and Neural Network Classification of Multispectral Data”, IGARSS ’94, vo. 3, 8-12, pp. 1413-1415, August 1994. [77] C.I. Christodoulou, S.C. Michaelides, and C.S. Pattichis, “Multifeature Texture Analysis for the Classification of Clouds in Satellite Imagery”, IEEE Trans. on Geoscience and Remote Sensing, vol. 41, no. 11, pp. 2662-2668, November 2003.

Taylor 246 of 249

09/28/10

[78] E.J. Kwiatkowska, “Cloud Cover Classification using SOM”, dissertation, University of Bradford, UK, 1997. [79] A.P. Engelbrecht, “Computational Intelligence: An Introduction”, Wiley, New York, 2002. [80] D.B. Fogel, “An Introduction to Simulated Evolutionary Optimization”, IEEE Trans. On Neural Networks, vol. 5, no. 1, pp. 3-14, January 1994. [81] J. Kennedy and R. Eberhart, “Particle Swarm Optimization”, Proc. IEEE INt. Conference on Neural Networks, vol. IV, Perth, Australia, 1995, pp. 1942-1948. [82] P. Bentley, “Evolving Fractal Proteins”, Department of Computer Science, University College London, 2003. [83] Amor, H.B., Rettinger, A., “Intelligent Exploration for Genetic Algorithms – Using Self-Organizing Maps in Evolutionary Computation”, Proc. GECCO ’05, 2005. [84] D.B. Fogel (ed.), “Evolutionary Computation: The Fossil Record”, IEEE Press, 1998. [85] J.H. Holland, “Adaptation in Natural and Artificial Systems”, 2

nd Edition, MIT Press, Cambridge, MA, 1992.

[86] W. Atmar, “Notes on the Simulation of Evolution”, IEEE Trans. On Neural Networks, vol. 5, no. 1, pp. 130-147, January 1994. [87] C. A. Coello, “A Short Tutorial on Evolutionary Multiobjective Optimization”, eds. E. Zitzler et al, Lecture Notes in Computer Science 1993, pp. 21-40, Springer-Verlag, Berlin, 2001. [88] N.R. Harvey et al, “Comparison of GENIE and Conventional Supervised Classifiers for Multispectral Image Feature Extraction”, IEEE Trans. on Geoscience and Remote Sensing, vol. 40, no. 2, pp. 393-404, February 2002. [89] B.C.K. Tso and P.M. Mather, “Classification of Multisource Remote Sensing Imagery Using a Genetic Algorithm and Markov Random Fields”, IEEE Trans. on Geoscience and Remote Sensing, vol. 37, no. 3, pp. 1255-1266, May 1990. [90] R.C. Eberhart and Y. Shi, ed., Special Issue on Particle Swarm Optimization, IEEE Trans. on Evolutionary Computation, vol. 8, no.3, June 2004. [91] A.S. Fraser, “Simulation of Genetic Systems”, J. of Theor. Biology, vol. 2, pp. 329-346, 1962. [92] H.J. Bremermann, “Optimization Through Evolution and Recombination,” in Self-Organizing Systems, eds. M.C. Yovits, G.T. Jacobi, and G.D. Goldstone, Spartan Books, Washington, D.C., pp. 93-106, 1962. [93] Y. Shi and R.C. Eberhart, “Empirical Study of Particle Swarm Optimization”, Proc. of the 1999 Congress on Evolutionary Computation, Washington, DC, vol. 3, pp. 1945-1950, July 1999. [94] C.A. Coello, G.T. Pulido, and M.S. Lechuga, “Handling Multiple Objectives with Particle Swarm Optimization”, IEEE Trans. on Evolutionary Computation, vol. 8, no. 3, pp. 256-279, June 2004. [95] W.H. Slade, H.W. Ressom, M.T. Mujavi, and R.L. Miller, “Inversion of Ocean Color Observationis Using Particle Swarm Optimization”, IEEE Trans. on Geoscience and Remote Sensing, vol. 42, no. 9, pp. 1915-1923, September 2004. [96] C.-F. Juang, “A Hybrid of Genetic Algorithm and Particle Swarm Optimization for Recurrent Network Design”, IEEE. Trans. on Systems, Man, and Cybernetics – Part B: Cybernetics, vol. 34, no. 2, pp. 997-1006, April 2004. [97] K. P. White, “Functional Genomics and the Study of Development, Variation, and Evolution”, Nature Reviews Genetics, July 2001, vol. 2, pp. 528-537. [98] T. D. Pollard and W.C. Earnshaw, “Cell Biology”, Saunders, Philadelphia, 2002.

Taylor 247 of 249

09/28/10

[99] S.B. Carroll, “Endless forms: the evolution of gene regulation and morphological diversity”, Cell, vol. 101, pp. 577-580, 2000. [100] H. Kitano, “A Simple Model of Neurogenesis and Cell Differentiation based on Evolutionary Large-Scale Chaos”, Artificial Life, vol. 2, no. 1, 1995. [101] D. Madina et al, “Cellular Evolution in a 3D Lattice Artificial Chemistry, 2004. [102] S. Kauffman, The Origins of Order, Oxford University Press, 1993. [103] T. Reil, “Dynamics of Gene Expression in an Artificial Genome – Implications for Biological and Artificial Ontogeny”, Proceedings of the 5

th European Conference on Advances in Artificial Life, Lecture Notes in

Computer Science, vol. 1674, 1999, pp. 457-466, Springer-Verlag, London. [104] N. Geard, “Modeling Gene Regulatory Networks: Systems Biology to Complex Systems”, ACCS Draft Technical Report, May 6, 2004. [105] Peter J. Bentley, “Evolving beyond perfection: an investigation of the effects of long-term evolution on fractal gene regulatory networks”, BioSystems, vol. 76, 2004, pp. 291-301. [106] L. N. de Castro and F.J. Von Zuben, “aiNet: An Artificial Immune Network for Data Analysis”, chapter in Data Mining: A Heuristic Approach, H. A. Abbass ed., Idea Group Publishing, USA, March 2001. [107] Y. Zhong, L. Zhang, B. Huang, and P. Li, “An Unsupervised Artificial Immune Classifier for Multi/Hyperspectral Remote Sensing Imagery”, IEEE Trans. On Geoscience and Remote Sensing, vol. 44, no. 2, February 2006. [108] L.A.N. Amaral, et al, “Emergence of complex dynamics in a simple model of signaling networks”, Proceedings of the National Academy of Sciences, vol. 101, no. 44, 2004. [109] J.C. Astor and C. Adami, “Development and Evolution of Neural Networks in an Artificial Chemistry”, arXiv:adap-org/9807003, Jul. 17, 1998. [110] R. Kohavi, “A Study of Cross-Validation and Boostrap for Accuracy Estimation and Model Selection”, International Joint Conference on Artificial Intelligence (IJCAI), 1985, Los Angeles, California, USA. [111] NASA MODIS website, http://www.modis.gsfc.nasa.gov, acquired July 30, 2009. [112] H.R. Gordon and M. Wang, “Retrieval of water-leaving radiance and aerosol optical thickness over the oceans with SeaWiFS: a preliminary algorithm”, Applied Optics, vol. 33, no. 3, January 1994. [113] J. D. Owens, et al, “A Survey of General-Purpose Computation on Graphics Hardware”, EUROGRAPHICS 2005. [114] Qizhi Yu, Chongcheng Chen, Zhigeng Pan, Parallel Genetic Algorithms on Programmable Graphics Hardware, Lecture Notes in Computer Science, vol. 3612, pp. 1051 – 1059, Jul 2005. [115] K.-L. Fok, T.-T Wong, and M.-L. Wong, “Evolutionary Computing on Consumer-Level Graphics Hardware”, to appear in IEEE Intelligent Systems, 2005. [116] Conversation with Ewa Kwiatkowska, August 19, 2009. [117] A.P. Bradley, “The Use Of The Area Under The ROC Curve In The Evaluation of Machine Learning Algorithms”, Pattern Recognition, vol. 30, no. 7, pp. 1145-1159, 1997. [118] P.A. Flasch, “Tutorial on The Many Faces of ROC Analysis in Machine Learning”, ICML 2004, July 4-8, 2004, Banff, Alberta, Canada.

Taylor 248 of 249

09/28/10

[119] J. Lehman and K.O. Stanley, “Revising the Evolutionary Computation Abstraction: Minimal Criteria Novelty Search”, Proceedings of GECCO 2010, July 7-11, Portland, Oregon USA. [120] B. Scholkpf, C.J.C. Burges, and A.J. Smola, “Advances in Kernel Methods: Support Vector Learning,” MIT Press, Cambridge MA, 1998. [121] A.N. Srivastava, N.C. Oza, and J. Stroeve, “Virtual Sensors: Using Data Mining Techniques to Efficiently Estimate Remote Sensing Spectra,” IEEE Trans. On Geoscience and Remote Sensing, vol. 43, no. 3, pp. 590-600, March 2005. [122] E.J. Kwiatkowska, and J.S. Fargion, “Application of Machine-Learning Techniques Toward the Creation of a Consistent and Calibrated Global Chlorophyll Concentration Baseline Dataset”, IEEE Trans. On Geoscience and Remote Sensing, vol. 41, no. 12, pp. 2844-2860. [123] K.Y. Yip, et al, “Improved Reconstruction of In Silico Gene Regulatory Networks by Integrating Knockout and Perturbation Data”, PLoS ONE , vol. 5, no. 1, January 2010. [124] S. Pinkert, et al, “Protein Interaction Networks- More than Mere Modules”, PloS Computation Biology, vol. 6, no. 1, January 2010. [125] K.O. Stanley and R. Miikkulainen, “Competitive Coevolution Trhough Evolutionary Complexification”, Journal of Artificial Intelligence Research, vol. 21, pp. 63-100, 2004. [126] K.O. Stanley, et al “Real-time Neuroevolution in the NERO Video Game”, IEEE Transactions on Evolutionary Computation, vol. 9, no. 6, pp. 653-668, December 2005. [127] A.-L. Barabasi and E. Bonabeau, “Scale-Free Networks”, Scientific American, pp. 60-69, May 2003. [128] Watts, D. and Strogatz, S., “Collective dynamics of ‘small-world’ networks”, Nature, vol. 393, pp. 440-442, 1998. [129] J.D. Gibson, Principles of Digital and Analog Communications, Second Edition, Prentice Hall, 1993.

Taylor 249 of 249

09/28/10

Documents

· Taylor 4 of 249 09/28/10 ABSTRACT This research develops and investigates an algorithm for the self-organized development of a classification network. This idea for this classific