Evaluating Classifiers for Disease Gene Discovery

Lon Turnbull and Kino Coursey

lt0013@unt.edu, kino@daxtrom.com

University of North Texas

Biocomputing Fall 2005

CSCD 4930.004/CSCE 5933.007Biol 4930.773/Biol 5905.773

Instructors: Armin Mikler and Kaja Abbas

Outline

• An interesting hypothesis

• What is a disease gene?

• Can disease genes be classified using machine learning tools?

• If so, can we do better?

• Classifiers + Data

• Analysis + Conclusions

Hypothesis

• It has been suggested that the genes which have some relationship to hereditary disease might have common variations in their DNA sequence structure.

What is a disease gene?

• Any gene that has mutated in such a way that the proteins created from it are dysfunctional.

What is a disease gene?

• Any gene that has mutated in such a way that the proteins created from it are dysfunctional.

• However, mutation can happen to any gene, so can one actually search for physical characteristics of a “disease” gene?

Reviewed Paper• A research group has used the alternating

decision tree algorithm from Weka to test the hypothesis.

• On average, 70% of the genes marked as disease phenotype were correctly identified with their automatic classifier they called PROSPECTR.

• They found that about 40% of their chosen features had statistically significant differences.

PROSPECTR results

Feature Ratio

Gene encodes signal peptide 2.06

Gene Length 1.42

5' CpG islands 1.33

Protein length 1.29

Exon Number 1.25

cDNA length 1.15

Distance to neighboring gene 1.13

3' UTR length 1.09

Question

Can we do better with other methods of classification?

Classification Methods

1. ADTree: alternating decision tree, optimized for two-class problems.

2. J48: a variant of classification 7.3. Logistic: Linear logistic regression.4. SMO: Sequential Minimal Optimization algorithm

for training a support vector classifier.5. Naïve Bayes: Standard probabilistic Naïve Bayes.6. Ibk-K: K-nearest neighbor classifier (k=5).7. PART: Obtains rules from partial decision trees

build using C4.5 heuristics.

Test Data

• A training set that consisted of 1,084 genes known to be associated with a disease and 1,084 genes not known to be associated with genes diseases.

• A set with 675 disease genes listed in the Human Gene Mutation Database (HGMD) and 675 genes not known to be involved in disease.

• A set based on oliongenic disorders. It contained 54 genes known to be associated with an oliongenic disorder and 54 genes not known to be associated with gene diseases.

Classifier interpretationThere are four possible results from a classification analysis. They are that a selected gene either:

1. Matches a disease gene.

2. Matches a non disease gene.

3. Is selected to match a disease gene but does not do so.

4. Is selected to match a non-disease gene but does not do so.

1 2 3 4 5 6 7

Training SetDisease classified as diseaseDisease classified as not diseaseNot disease classified as diseaseNot disease classified as not disease

Number of genes

Classifiers

Validity

• The analysis of an independent data set ought to produce similar results to the training set. If not the analysis is suspect.

1 2 3 4 5 6 7

Number of genes

Classifiers

1 2 3 4 5 6 7

HGMD SetDisease classified as diseaseDisease classified as not diseaseNot disease classified as diseaseNot disease classified as not disease

Number of Genes

Classifiers

Validity

• If the analysis is valid, we would expect that classification using the only the successful subset of features found by the PROSPECTR application would result in improved results.

• The removal of non-relevant features ought to decrease the number of mismatches.

1 2 3 4 5 6 7

ReducedTraining Set

Disease classified as diseaseDisease classified as not diseaseNot disease classified as diseaseNot disease classified as not disease

Number of Genes

Classifiers

1 2 3 4 5 6 7

Number of genes

Classifiers

1 2 3 4 5 6 7

Number of genes

Classifiers

1 2 3 4 5 6 7

Oligogenic SetDisease classified as diseaseDisease classified as not diseaseNot disease classified as diseaseNot disease classified as not disease

Number of Genes

Classifiers

1 2 3 4 5 6 7

ReducedTraining Set

Number of Genes

Classifiers

All data analyzed

1 2 3 4 5 6 7

ReducedHGMD Set

Number of Genes

Classifiers

1 2 3 4 5 6 7

ReducedOligogenic Set

Number of Genes

Classifiers

1 2 3 4 5 6 7

HGMD SetDisease classified as diseaseDisease classified as not diseaseNot disease classified as diseaseNot disease classified as not disease

Number of Genes

Classifiers

The Best Classifier Results

Classifier Percent total corrects

Difference with best features

J48 88.7 -15.1

PART 80.7 -10.6

ADTree 75.5 -3.1

Ibk-K 75.4 -0.32

Naïve Bayes 73.0 -12.0

SMO 72.3 -6.0

Logistic 70.0 -4.9

Conclusions• We have shown that classifier 2, performs

better than classifier 1, the one chosen by PROSPECTR method.

• The features that showed the largest differences in the PROPSPECTR study were most likely a statistical anomaly.

• It seems that using these machine learning methods to classify disease genes is not very productive. At best it needs to be combined with some other independent method.

References

• Euan Adie et. al., Speeding disease gene discovery by sequence based candidate prioritization, BMC Bioinformatics 2005, 6:55.

• Hammond MP, Birney E, Genome information resources - developments at Ensembl. Trends in Genetics 2004, 20:268-272.

• http://www.biomedcentral.com/1471-2105/6/55.

• http://www.ncbi.nlm.nih.gov/books/bv.fcgi?rid=gnd

• http://www.genetics.med.ed.ac.uk/prospectr/

Questions

What causes disease?

• Causes of disease are a continuum of genetic activity interacting with nongenetic factors.

The Metabolic Molecular Basis of Inherited Disease. Vol 1, Chapter 1. 8ed.

RC 627.8.M47.2001

• A collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a data set or called from your own Java code.

• Contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization.

• Well-suited for developing new machine learning schemes.

• Is open source software issued under the GNU General Public License.

Evaluating Classifiers for Disease Gene Discovery

Documents

Ideology Classifiers for Political Speech Stefan Kaufmannccc.inaoep.mx/~villasen/bib/Ideology Classifiers for Political Speech.pdf · Ideology classifiers for political research pg

Does one size really fit all? Evaluating classifiers in Bag-of-Visual-Words classification Christian Hentschel, Harald Sack Hasso Plattner Institute

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis

A Simulation-Based Approach for Evaluating Gene Expression

Applicability of histomorphometry analysis for evaluating ... · Applicability of histomorphometry analysis for evaluating alveolar bone regeneration after gene transfer Mariko Kawai

Classifiers in Kam-Tai Languages · 2013-01-23 · 4.2.7. Classifiers and temporal words ..... 169 4.2.8. Classifiers and pronouns ..... 171 4.2.9. Classifiers and numerals ... few

Gaussian Classifiers

More Classifiers

Boosting of classifiers

Gene and exon signatures as radiation biomarkers · • Samples from the same donor must be in the same partition ... Gene and exon signatures as classifiers Equal performance of

Clinical Appropriateness Guidelines Molecular Testing of Solid … · 2021. 1. 4. · Prostate cancer - post-diagnosis gene expression classifiers Skin cancer - basal and squamous

Gaussian Bayes Classifiers

5 classifiers

Topics in the Development and Validation of Gene Expression Profiling Based Predictive Classifiers Richard Simon, D.Sc. Chief, Biometric Research Branch

Reflux Classifiers

Lecture8 classifiers ldc_rules

classifiers kelompok 1

Safety Considerations for Gene Editing and Other Gene ... Huang_0.pdfgenomic modifications No ‘gold standard’ for evaluating large genomic modifications or genomic instability

Evaluating the Analytical Distribution of bicoid Gene ...eprints.bournemouth.ac.uk/29704/1/Metagene2.pdf · Evaluating the Analytical Distribution of bicoid Gene Expression Pro le

Evaluating gene flow and habitat connectivity between S ... · [Escrever o nome da empresa] Evaluating gene flow and habitat connectivity between S. salamandra lineages across heterogeneous