Data Analysis using WEKA Waikato Environment for Knowledge Analysis Weka is a popular suite of machine learning software written in Java, developed at the University of Waikato, New Zealand. This paper is attempts to show the use of this software for Cluster Analysis and Decision Tree Analysis. Prabhjot Singh Bhatia – 10BM60060 Vinod Gupta School of Management, Indian Institute of Technology Kharagpur India.

1

Table of Contents Introduction ........................................................................................................................................................ 3

Cluster Analysis .............................................................................................................................................................. 3

K Means Clustering ................................................................................................................................................... 3

Decision Trees ................................................................................................................................................................ 4

Features ....................................................................................................................................................................... 4

Cluster Analysis using WEKA ......................................................................................................................... 5

About the dataset ........................................................................................................................................................... 5

Steps for K Means clustering ....................................................................................................................................... 5

Interpretation ................................................................................................................................................................ 14

Cluster 0 .................................................................................................................................................................... 14

Cluster 1 .................................................................................................................................................................... 14

Cluster 2 .................................................................................................................................................................... 15

Cluster 3 .................................................................................................................................................................... 15

Cluster 4 .................................................................................................................................................................... 15

Decision trees using WEKA .......................................................................................................................... 16

About the Dataset ........................................................................................................................................................ 16

Steps for decision tree generation .............................................................................................................................. 16

Interpretation of Output ............................................................................................................................................. 21

Works Cited ...................................................................................................................................................... 23

2

Table of Figures Screenshot 1: The data file loaded into Weka. ............................................................................................... 6

Screenshot 2: The Visualize Tab. ..................................................................................................................... 6

Screenshot 3: Selecting Simple KMeans ......................................................................................................... 7

Screenshot 4: The Cluster Tab ......................................................................................................................... 7

Screenshot 5: KMeans Clustering Options .................................................................................................... 8

Screenshot 6: SimpleKMeans, 2 Clusters ....................................................................................................... 8

Screenshot 7: Simple KMeans, Results for 2-14 clusters ............................................................................. 9

Screenshot 8: Knee point for number of clusters ......................................................................................... 9

Screenshot 9: Visualizing 5 cluster solution ................................................................................................. 12

Screenshot 10: Visualization of Clusters - I ................................................................................................. 13

Screenshot 11: Visualization of Clusters - II ............................................................................................... 13

Screenshot 12: The data file loaded into Weka ........................................................................................... 17

Screenshot 13: Visualizing the given dataset ................................................................................................ 17

Screenshot 14: Selecting J48 Tree Algorithm .............................................................................................. 18

Screenshot 15: The Classify Tab .................................................................................................................... 18

Screenshot 16: The Output of J48 Algorithm ............................................................................................. 19

Screenshot 17: The Decision Tree Generated. ............................................................................................ 19

Screenshot 18: Dissection of the Textual Decision Tree Output ............................................................. 22

3

Introduction

Identifying patterns in data and being able to make predictions based on the patterns plays

significant role in all aspects of an industry or an individual business. A plethora of methods and

tools are available. This paper is an attempt to introduce two such methods namely Decision tree

classification and K Means clustering using a tool called Weka.

Cluster Analysis

As a part of exploratory data mining, cluster analysis is used to assign a set of objects to various

groups such that they are more similar to other objects within the group than those belonging to

another. Cluster Analysis as a statistical data analytical tool forms the backbone of various disciplines

including marketing research, pattern finding, intelligence gathering and image analysis etc.

Cluster analysis comprises of a set of algorithms, all used in specific situations. These include

Hierarchical Clustering, K-Means clustering etc.

K Means Clustering

This method has the objective of classifying a set of n objects into k clusters, based on the closeness

to the cluster centers. The closeness to cluster centers is measured by the use of a standard distance

algorithm, eg. Euclidean distance.

Features

KMeans clustering is computationally very fast, as compared to other types of clustering algorithms.

The number of clusters is expected as an input for the algorithm to work. This number “k” can be

determined by a number of methods:One of them uses hierarchical clustering first to determine k

and then continues with K Means. However, this method has a limitation of not being

computationally efficient and takes a long time to converge.

Another method, and the one that will be described here, is to use the Knee- point method. (Sugar,

Gareth, & James, 2003) This comprises of running the K means algorithm for k=1 to an arbitrary

number say 10-15. The distortion or average sum of squared errors are then plotted against number

of clusters to find a “knee” point. This gives the number of clusters.

4

KMeans Clustering Algorithm

The algorithm takes the number of clusters („k’) as a mandatory input. The algorithm follows the

following steps: (Matteucci)

1. „k‟ points are placed into space represented by the objects under consideration. These are the

initial cluster centres, and are the group centroids.

2. Based on a distance algorithm, the objects are assigned to the cluster closest to them.

3. Step 2 continues until all objects have been assigned to a cluster.

4. The positions of the cluster centroids are calculated again.

5. Steps 2 and 3 are repeated until the cluster centres no longer move.

6. This gives a partitioning of objects into distinct groups.

Decision Trees

The aim of this method is to predict what criteria is used to determine the outcome and in what

order. We start with a sample data called the training set and generate a model based on machine

learning. We later use this model in the field to predict the outcome on the basis of the input criteria.

The actual splitting of data into various groups is done on the basis of some rules. A majority of

them depend on a technique called recursive partitioning, since it is repeated for each sub-group of

data. Further partitioning is done till the point where such partitioning does not add value. There are

several algorithms for decision trees including J48, BTree, Random Forest, etc.

Features

Simple to understand and interpret.

No or little data preprocessing is required.

Ability to handle both numerical and categorical data.

Performs well with large data in a short time.

Trees can get very complex and very large to visualize

Attributes with more levels tend to bias the tree.

5

Cluster Analysis using WEKA

About the dataset

The dataset used is a german credit dataset, and is available from (Hofman). The data has been

compiled by Professor Dr. Hans Hofmann, Institut f"ur Statistik und "Okonometrie, University of

Hamburg, Germany. The dataset describes various customers on the basis of a set of 20 parameters,

including age, housing, number of years since being a resident, car owned, etc.

The data set also contains a “class” attribute to classify credit-worthiness. We do not include this

attribute in our clustering algorithm.

Steps for K Means clustering

1. Open the file named credit-g.arff, available from the source above. The file is pre-processed,

we can use the data as is, without need for preprocessing. (Screenshot 1)

2. Click on “Visualize” tab to have a look at the data. This shows the plots of one field against

every other field, colour coded by the class. (Screenshot 2)

3. Click on “Cluster” tab (Screenshot 4). Click on “Choose” and select “SimpleKMeans”

(Screenshot 3). By default, Weka chooses a commonly used value for its

parameters.(Screenshot 5) The various options are self-explanatory. One major decision

variable is the number of clusters. As described above, we need to perform the cluster

analysis.

4. Keep the number of clusters as 2, since a 1 cluster solution would not be appropriate in this

case. Click OK.

5. Click on Ignore attributes. If the data set contains any column from other classification

experiments, select that attribute from the list provided. This is needed to make the

clustering process unbiased.

6. Click “Start” to start the clustering algorithm. The Weka icon at the bottom right corner

keeps flipping while processing is still in progress. Once the icon stops, the analysis result is

displayed in the adjacent window. (Screenshot 6)

7. Click on the text box next to “Choose” button to get the list of options. Increase the

number of clusters to 3, and click OK. Repeat steps 5 and 6.

8. Similarly, repeat step 7 for cluster numbers upto 14.

9. Tabulate the “Average Within Cluster Sum of Squared Errors” (as seen in Screenshot 7) and

plot the graph using a spreadsheet program (Screenshot 8).

10. The knee point shows the number of clusters in the dataset. Use the output of 5 clusters as

the cluster solution.

11. Right Click on the 5 cluster solution and click on “Visualize Cluster

Assignments”.(Screenshot 9)

6

Screenshot 1: The data file loaded into Weka.

Screenshot 2: The Visualize Tab.

7

Screenshot 4: The Cluster Tab

Screenshot 3: Selecting Simple KMeans

8

Screenshot 5: KMeans Clustering Options

Screenshot 6: SimpleKMeans, 2 Clusters

9

Clusters Avg. Within Cluster Sum of Squared Error

2 5365.998 3 5145.269 4 4927.793 5 4691.713 6 4613.818 7 4530.644 8 4437.524 9 4273.035

10 4202.059 11 4197.927 12 4157.734 13 4113.83 14 4037.48

Screenshot 7: Simple KMeans, Results for 2-14 clusters

5, 4691.713078

4000

4200

4400

4600

4800

5000

5200

5400

5600

2 3 4 5 6 7 8 9 10 11 12 13 14

Avg

. W

ith

in C

lust

er

Su

m o

f Sq

uar

ed

Err

ors

No. of Clusters

Avg. Within Cluster Sum of Squared Error

Screenshot 8: Knee point for number of clusters

Knee Point

10

=== Run information ===

Scheme:weka.clusterers.SimpleKMeans -N 5 -A "weka.core.EuclideanDistance -R first-last" -I 500 -S

10

Relation: german_credit

Instances: 1000

Attributes: 21

checking_status

duration

credit_history

purpose

credit_amount

savings_status

employment

installment_commitment

personal_status

other_parties

residence_since

property_magnitude

age

other_payment_plans

housing

existing_credits

job

num_dependents

own_telephone

foreign_worker

Ignored:

class

Test mode:evaluate on training data

=== Model and evaluation on training set ===

kMeans

======

Number of iterations: 9

Within cluster sum of squared errors: 4691.713078260774

Missing values globally replaced with mean/mode

Cluster centroids: Cluster# Attribute Full Data 0 1 2 3 4 (1000) (220) (130) (230) (267) (153) =========================================================================================================================================================================================================================== checking_status no checking no checking <0 0<=X<200 no checking <0 duration 20.903 21.7409 27.4462 17.1522 20.779 19.9935 credit_history existing paid critical/other existing credit existing paid existing paid existing paid existing paid purpose radio/tv new car used car radio/tv radio/tv new car credit_amount 3271.258 3523.8136 5507.3846 2465.4739 2763.4307 3105.6471

11

savings_status <100 <100 <100 <100 <100 <100 employment 1<=X<4 1<=X<4 >=7 1<=X<4 >=7 <1 installment_commitment 2.973 3.0591 3.0769 2.8957 3.161 2.549 personal_status male single female div/dep/mar male single male single male single female div/dep/mar other_parties none none none none none none residence_since 2.845 2.5727 3.5769 2.3261 3.0225 3.085 property_magnitude car car no known property real estate life insurance car age 35.546 34.4091 44.4923 33.6652 37.7004 28.6471 other_payment_plans none none none none none none housing own own for free own own rent existing_credits 1.407 1.6318 1.4 1.2913 1.4045 1.268 job skilled skilled high qualif/self emp/mgmt unskilled resident skilled skilled num_dependents 1.155 1.0727 1.2462 1.2174 1.1835 1.0523 own_telephone none yes yes none none none foreign_worker yes yes yes yes yes yes

The above output in a cleaner form:

Cluster Number

Attribute Full Data 0 1 2 3 4

(1000) (220) (130) (230) (267) (153)

checking_status no checking

no checking <0 0<=X<200

no checking

<0

duration 20.903 21.7409 27.4462 17.1522 20.779 19.9935 credit_history existing

paid critical/other existing credit

existing paid

existing paid

existing paid

existing paid

purpose radio/tv new car used car radio/tv radio/tv new car credit_amount 3271.258 3523.8136 5507.3846 2465.4739 2763.430

7 3105.6471

savings_status <100 <100 <100 <100 <100 <100 employment 1<=X&lt

;4 1<=X<4 >=7 1<=X<4 >=7 <1

installment_commitment

2.973 3.0591 3.0769 2.8957 3.161 2.549

personal_status male single

female div/dep/mar

male single

male single male single

female div/dep/mar

other_parties none none none none none none residence_since 2.845 2.5727 3.5769 2.3261 3.0225 3.085 property_magnitude car car no known

property real estate life

insurance

car

age 35.546 34.4091 44.4923 33.6652 37.7004 28.6471 other_payment_plans none none none none none none housing own own for free own own rent existing_credits 1.407 1.6318 1.4 1.2913 1.4045 1.268 job skilled skilled high

qualif/self emp/mgmt

unskilled resident

skilled skilled

num_dependents 1.155 1.0727 1.2462 1.2174 1.1835 1.0523 own_telephone none yes yes none none none foreign_worker yes yes yes yes yes yes

Time taken to build model (full training data) : 0.08 seconds

12

=== Model and evaluation on training set ===

Clustered Instances

0 220 ( 22%)

1 130 ( 13%)

2 230 ( 23%)

3 267 ( 27%)

4 153 ( 15%)

Screenshot 9: Visualizing 5 cluster solution

13

Screenshot 10: Notice that a majority of the people in cluster 1 are highly qualified and those in cluster 2 are unemployed

Screenshot 11: Notice that a majority of the people in cluster 0 are holders of previous unpaid/delayed credit

14

Interpretation

As evident from the visualization, we can see the output has been classified into 5 clusters. The

algorithm took 9 iterations to complete and reach the result. The distortion (average within cluster

sum of squared errors) amongst the clusters themselves is 4691 units.

The cluster centers have been shown in the form of a table. The following results can be seen from

the cluster center table:

A majority of people in all clusters are foreign workers, with no other payment plans from

bank/store, and have atleast one family person dependent wholly on them. They also do not

have any guarantors or third parties to vouch for their loan. Moreover, all of these have very

little savings to their name.

Clusters 0 and 4 are female and are divorcees/dependent/married, others are single males.

Only Clusters 0 and 1 own a telephone.

Cluster 0

This set of customers comprise of middle aged married/divorcee female applicants, who

have been employed in a skilled job for the past 1-4 years, and have been resident at their

self-owned house for around 2.5 years. They currently have critically delayed existing credit

to their name. They are seeking a loan for a second new car, and are willing to commit about

3 % of their income towards installments, to be paid over 21 months. However, they do not

have a checking account where their salary is drawn. In the absence of a guarantor, this

seems to be a high risk category.

Cluster 1

This set of customers comprise of single male applicants in their mid-40s, who have been

employed in a high skilled job or self-employed for more than 7 years, and have been

resident at their freely provided/company provided house for around 3.5 years. They

currently have no existing credit to their name, but also do not have any property registered

to their name. Also, they have an overdrawn checking account. They are seeking a loan for a

used car, and are willing to commit about 3 % of their income towards installments, to be

paid over 27 months. However, they are asking for too high an amount for a used car,

considering other groups are asking only 60% of that amount for a new car. This is

15

indicative of intentions of either using the loan money for some other purpose or possible

fraud. In the absence of a guarantor, this seems to be a high risk category.

Cluster 2

This set of customers comprise of middle aged single male applicants, who have been

employed in an unskilled resident job for the past 1-4 years, and have been resident at their

self-owned house for around 2 years. They also own real estate on their name. They are

seeking a loan for a radio/TV, and are willing to commit about 2.8 % of their income

towards installments, to be paid over 17 months. They maintain a checking balance of 0-200

DeutscheMark in their account. In the absence of a guarantor, this seems to be a low risk

category

Cluster 3

This set of customers comprise of middle aged single male applicants, who have been

employed in a skilled job for over 7 years, and have been resident at their self-owned house

for over 3 years. Apart from this, they own nothing more than a life insurance policy to their

name. They are seeking a loan for a radio/TV, and are willing to commit about 3.1 % of

their income towards installments, to be paid over 20 months. However, they do not have a

checking account where their salary is drawn. In the absence of a guarantor, this seems to be

a medium risk category.

Cluster 4

This set of customers comprise of very young divorced/married female applicants, who have

been employed in a skilled job for less than 1 year, and have been resident at their rented

house for around 3 years. They are seeking a loan for a second new car, and are willing to

commit about 2.5 % of their income towards installments, to be paid over 20 months. Their

checking account is overdrawn. In the absence of a guarantor, this seems to be a high risk

category.

16

Decision trees using WEKA

About the Dataset

The data set is vote.arff, US Congressional Voting Records, is a part of the UCI Repository

(University of California Irvine), a repository of data available publicly.

The dataset is basically a training set used to classify voters on the basis of their likelihood to vote

for either democrats or republicans, on the basis of various criteria chosen from their manifesto.

These criteria include the parties‟ stands on various issues like cost sharing of drinking water project,

freezing physician fee, spending on education, among other factors.

Steps for decision tree generation

1. Launch Weka and open the “Explorer Window”. Click on “Open File” and browse to the

where “vote.arff” is stored. The data set opens up in Weka. (Screenshot 12)

2. Click “Visualize” tab to see the data in graphical form, plotted against each

attribute.(Screenshot 13)

3. Click on “Classify” tab. We see the “ZeroR” as the default classification rule. (Screenshot 15)

4. Click on “Choose” button and Select “J48” under “Trees”.(Screenshot 14)

5. Weka makes workable assumptions about the various configuration options for this

algorithm, so we can continue using these. Click on “Start” to start executing the algorithm.

When the algorithm has finished executing, we get the output, including the textual decision

tree(Screenshot 16)

6. To visualize the decision tree in graphical format, Right Click on the result in the left pane

and click “Visualize Tree”. A new windows is opened up showing the decision tree in

graphical format.(Screenshot 17)

17

Screenshot 12: The data file loaded into Weka

Screenshot 13: Visualizing the given dataset

18

Screenshot 15: The Classify Tab

Screenshot 14: Selecting J48 Tree Algorithm

19

Screenshot 16: The Output of J48 Algorithm

Screenshot 17: The Decision Tree Generated.

20

=== Run information ===

Scheme:weka.classifiers.trees.J48 -C 0.25 -M 2

Relation: vote

Instances: 435

Attributes: 17

handicapped-infants

water-project-cost-sharing

adoption-of-the-budget-resolution

physician-fee-freeze

el-salvador-aid

religious-groups-in-schools

anti-satellite-test-ban

aid-to-nicaraguan-contras

mx-missile

immigration

synfuels-corporation-cutback

education-spending

superfund-right-to-sue

crime

duty-free-exports

export-administration-act-south-africa

Class

Test mode:10-fold cross-validation

=== Classifier model (full training set) ===

J48 pruned tree

------------------

physician-fee-freeze = n: democrat (253.41/3.75)

physician-fee-freeze = y

| synfuels-corporation-cutback = n: republican (145.71/4.0)

| synfuels-corporation-cutback = y

| | mx-missile = n

| | | adoption-of-the-budget-resolution = n: republican (22.61/3.32)

| | | adoption-of-the-budget-resolution = y

| | | | anti-satellite-test-ban = n: democrat (5.04/0.02)

| | | | anti-satellite-test-ban = y: republican (2.21)

| | mx-missile = y: democrat (6.03/1.03)

Number of Leaves : 6

Size of the tree : 11

Time taken to build model: 0.06 seconds

21

=== Stratified cross-validation ===

=== Summary ===

Correctly Classified Instances 419 96.3218 %

Incorrectly Classified Instances 16 3.6782 %

Kappa statistic 0.9224

Mean absolute error 0.0611

Root mean squared error 0.1748

Relative absolute error 12.887 %

Root relative squared error 35.9085 %

Total Number of Instances 435

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure ROC Area Class

0.97 0.048 0.97 0.97 0.97 0.971 democrat

0.952 0.03 0.952 0.952 0.952 0.971

republican

Weighted Avg. 0.963 0.041 0.963 0.963 0.963 0.971

=== Confusion Matrix ===

a b <-- classified as

259 8 | a = democrat

8 160 | b = republican

Interpretation of Output

The complete output of the J48 Algorithm is shown in the previous section. The beginning defines

the summary of the dataset, which shows that the output was reached using the 10 fold Cross

validation method. To avoid ambiguity, the attributes worked upon are also shown.

Under the heading “Classifier model J48 Pruning tree” we see the decision tree in textual format.

The first split is based on whether the “Physician fee freeze” is part of the manifesto or not. If not,

the voter straight away votes for the Democrats.

A detailed description of the various elements of this textual decision tree are as follows(Screenshot

18):

1. The attribute which decides the outcome for that branch

2. The value of the attribute for that branch.

22

3. The colon is the separator for the class label assigned to a particular leaf.

4. The class label assigned to a particular leaf

5. The number of instances assigned for the leaf, as a decimal number

6. The number of instances wrongly classified, and assigned to this leaf.

The graphically formatted decision tree (Screenshot 17) is more visually appealing and has the same

components as above. As mentioned, the number of leaves and size of the tree are 6 and 11

respectively.

The next section of the output gives an idea about the performance measure of the tree. 3.6782%

instances are incorrectly classified. Also, from the confusion matrix, it can be seen that a total of 16

instances are incorrectly classified (8 from democrats and 8 from republicans).

A high value of Kappa statistic (92.24%) shows that the results generated from this output are quite

believable and this model is suitable for deployment in the real world.

This decision tree gives us a fair idea about the importance and order of various factors to be taken

into consideration by a voter while making a decision on whom to vote.

Screenshot 18: Dissection of the Textual Decision Tree Output

1 2 3 4 5 6

Branch

Leaf

23

Works Cited

Hofman, D. H. (n.d.). Institut f"ur Statistik und "Okonometrie, University of Hamburg, Germany; German

Credit Dataset. Retrieved April 16, 2012, from Auckland University Website:

http://www.stat.auckland.ac.nz/~reilly/credit-g.arff

Matteucci, M. (n.d.). A Tutorial on Clustering Algorithms: . Retrieved April 2, 2012, from Matteucci's

personal page at Politecnico di Milano Website:

http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/kmeans.html

Sugar, C. A., Gareth, & James, M. (2003). Finding the number of clusters in a data set: An

information theoretic approach. Journal of the American Statistical Association, Vol. 98, 750-763.

University of California Irvine, M. R. (n.d.). 1984 US State Congressional voting Records Dataset.

Retrieved April 16, 2012, from University of California, Irvine, Machine Learning Repository

Archive: http://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records

Wikipedia. (2012, March 28). Decision_tree_learning. Retrieved April 6, 2012, from Wikipedia:

http://en.wikipedia.org/wiki/Decision_tree_learning

Wikipedia. (2012, April 7). K-means_clustering. Retrieved April 9, 2012, from Wikipedia:

http://en.wikipedia.org/wiki/K-means_clustering