1

Class Imbalance in Text Classification

Project ID: 08

Elham JebalbareziNedjma Ousidhoum

2

Outline

• Class Imbalance• Algorithms for Class Imbalance• Text Classification• Feature selection for text classification• Experiments• Results• Discussion

3

The Class Imbalance Problem(1)

• Common problem in Machine Learning• Almost all the instances belong to one major class

and the rest belong to the minor class.• Imbalance Level= |Majority Class|/|Minority Class|.

It can be huge (order of 106).• Applications

detecting oil spills, text classification, fraud detection and many medical applications such as automatic diagnosis

4

The Class Imbalance Problem(2)

• Many classification algorithms are sensitive to the imbalanced class distribution

• Class imbalance is taken into account in the design of new classifiers

• Solutionscost-sensitive learning, data resampling,

feature selection.

5

Cost-Sensitive Algorithms

• Penalties assigned to mistakes made by classification algorithms.

• Assign different asymmetric misclassification costs to classes. The penalty is higher when the mistake is made on the minority class, to emphasize the correct classification of minority instances.

• Cost- sensitive learning does not modify the class distribution

6

Data Resampling

• Learning instances in the majority class and minority class are manipulated in order to balance the class distribution.

• Effective but may introduce noise or remove useful information.

7

Data ResamplingOversampling

• Duplicates the minority class for more effect on the machine learning algorithm.

• Might be effective but may be prone to overfitting.

• Variants: SMOTE (Synthetic Minority Oversampling Technique), MSMOTE (Modified SMOTE), …

8

Data ResamplingUndersampling

• Using a subset of the majority class to train the classifier.

• Many majority class examples are ignored so that the training set becomes more balanced and the training process becomes faster.

• Effective but may discard useful information.• There are variants of undersampling. E.g. One-

sided undersampling

9

Bagging/Boosting

• Bootstrapping is random sampling with replacement

• Bagging is aggregating classifiers induced over independently drawn bootstrap samples.

• Boosting is to focus on difficult samples by giving a higher weight parameter

10

Feature Selection

• Feature selection is able to improve the performance of naive Bayes and regularized logistic regression on imbalanced data.

• The challenges of feature selection and imbalanced data classification meet when the dataset to be analyzed is of high-dimensionality and highly imbalanced class distribution

11

Text Classification

• Sorting natural language texts or documents into predefined categories based on their content.

• Applications automatic indexing, document organization, text filtering, hierarchical categorization of

web pages, spam filtering, …• Class Imbalance is common in text classification

(e.g)

12

Feature Selection in Text Classification

• Common in text classification because it can improve text classification.

• Select features using different metrics (TF, Chi-square, information gain) for a nearly optimal classification

• We can use positive/negative features

• Combining positive and negative features might be useful

13

Experiments

• We implemented random oversampling, Random undersampling, SMOTE, MSMOTE, One sided Undersampling.

• Our approachWe combined feature selection and resampling by:

1. Calculating Term Frequency2. Applying a resampling Algorithm

• Dataset Reuters-21578.• Chosen Evaluation Metrics

precision=tp/tp+fp , recall=tp/tp+fn, f-measure=2.recall.precision/recall+precision

14

ExperimentsData

15

ExperimentsRandom Oversampling

16

ExperimentsSMOTE

17

ExperimentsMSMOTE

18

ExperimentsRandom Undersampling

19

ExperimentsOne-sided Undersampling

20

Results(1)

Without Sampling

Random oversampling

Random undersampling

one sided undersampling

smote msmote

Precision0.03191 0.09259 0.14772 0.15957 0.0909 0.0434

Recall0.14285 0.23809 0.61904 0.71428 0.2380 0.09523

F-Measure

0.05217 0.13333 0.2385 0.26086 0.1315 0.0597

No feature selection

21

Results(2)

Without Sampling

Random oversampling

Random undersampling

one sided undersampling

smote msmote

Precision1 0.6111 0.0884 0.0851 0.5 0.5384

Recall0.0476 0.5238 0.6190 0.7619 0.5238 0.3333

F-measure

0.0909 0.5641 0.1547 0.1531 0.5116 0.4117

100 features selected after using TF

22

Results(3)

Without Sampling

Random oversampling

Random undersampling

one sided undersampling

smote msmote

Precision0.0476 0.1777 0.0937 0.1666 0.16666 0.4

Recall0.0476 0.38095 0.2857 0.5238 0.3809 0.5714

F-Measure

0.0476 0.2424 0.1411 0.2528 0.2318 0.4705

500 features selected after using TF

23

Discussion

• Feature selection improves oversampling.

• Feature selection also improves undersampling recall.

• Adding more features does not always improve the results.

24

Thank you!