Data mining Computerassignment 3

Data mining ‘Epochs & Accuracy’ ~ Using the MultiLayerPerceptron function on a deliverd database ~

COMPUTER ASSIGNMENT 3 BARRY KOLLEE 10349863

Computerassignment 3

2

Exercise 1a): In the GUI screen you need to adjust the epochs (500 is too much). Start by choosing 5. Select Start. Select Accept. The output screen gives you the accuracy of the learned model. I’ve loaded up the soybean.arff file into weka and did the following:

• Used the MultiLayerPerceptron classifier • Set training set under test options

Set GUI on true I performed the MultiLayerPerceptron with an Epoch of 5. The accuracy of an epoch of 5 (5 iterations) is given in red. Epoch of 5

Correctly Classified Instances 534 78.1845 % Incorrectly Classified Instances 149 21.8155 %

Info about this evaluation

Epoch = 5 Error per Epoch = 0.0164204 Learning Rate = 0.3 Momentum = 0.2

Exercise 1b): Repeat the training with different settings (epochs 5, 10, 50 and 100) and see what happens to the performance.

Epoch set to 10




Epoch set to 50

Correctly Classified Instances 674 98.6823 % Incorrectly Classified Instances 9 1.3177 % Info about this evaluation



3

Epoch set to 100




Now we place all our findings of the different epochs into one table. Then it becomes clear which epoch gives us the highest accuracy. The learning rate within the underneath table has been set to 0.3.

Number of Epochs/iterations

Accuracy (number of correct classified instances)

5 78.1845 % 10 93.4114 % 50 98.6823 %

100 99.4143 % It becomes clear that, when we increase the number of epochs (iterations), the accuracy climbs up also. The increasing of the accuracy is mostly occurring when the epoch (number of iterations) is between 1 and 10. When we increase the number epochs to a value which is higher then 10, then there’s a significant lower increase of the accuracy. The percentage of incorrect classified instances, when using 100 as the input value of Epoch, has been decreased to 0.5657 %. So that might be ‘negligbible’. In conclusion we can state that using higher values for epoch wouldn’t be necessary with this learning rate.

If we also check the tables with ‘info about this evaluation’ we can also see that the error per epoch has been decreased significantly when we increased the number of epoch(s).


4

Exercise 2: Pick one epoch value, and start playing with the "learning rate" parameter, e.g. try 0.1, 0.3, and 0.6. Again, look for each value what happens to the accuracy.

I’ve chosen for using an epoch value of 10 within the three outputs listed below. The accuracy (number of correct and incorrect instances) is given in red.

Learning rate set to 0.1





Epoch set to 10








Now we place our findings in a table where we put the learning rate

Learning rate Accuracy (number of correct classified instances) 0.1 72.328 % 0.3 93.4114 % 0.6 93.4114 %

We see that increasing the learning rate from 0.1 to 0.3 has affected the accuracy in a good way. However we see that if we would increase the learning rate it’s value even more that we won’t have a higher accuracy. It might be interesting to take a look at a learning rate of 0.2 and check if the accuracy, when using a learning rate of 0.2, has been affected if we compare it to a learning rate of 0.1.


5

The result of a learning set which has been set to 0.2 is listed below. The accuracy is given in red.



Epoch = 10 Error per Epoch = 0.0112326 Learning Rate = 0.2 Momentum = 0.2 We see that the accuracy has climbed up significantly if we compare the learning rates of 0.1, 0.2 and 0.3.

learning rate 0.1 = 72.328 % learning rate 0.2 = 89.6047 % 89.6047 / 72.3280 ≈ 1.24 increase factor learning rate 0.2 = 89.6047 % learning rate 0.3 = 93.4114 % 93.4114 / 89.6047 ≈ 1.04 increase factor learning rate 0.1 = 72.328 % learning rate 0.3 = 93.4114 % 93.4114 / 89.6047 ≈ 1.29 increase factor Another conclusion we can make is that the highest increasing factor is between a learning rate of 0.1 and 0.2. When increasing the learning rate any further is has a small effect on the accuracy of our model. Another thing we can conclude is that the highest accuracy within this test has occurred when using a learning rate of 0.3. We did observed a decrease of the error per epoch of 0.0011 between the learning rates 0.3 and 0.6, but that hasn’t affected the accuracy (number of correctly classified instances).

1 See ‘info about this evaluation’ of learing rate 0.3 & 0.6. Difference between these values. 2 Unfortunately Weka was not able to set the learning rate to a higher value then 1.0. There was a fixed maximum set


6

Exercise 3. As a final experiment, pick one value for both epoch and learning rate, and play with the hiddenLayers (in the same window as where you set GUI to true). The default value is 'a', try setting it differently (e.g. 5 or 10) and try multiple values at the same time (e.g. 10,10). Again, look at performance of the model for various settings.

For the final experiment I’ve chosen to an epoch of 50 and a learning rate of 0.2.

These are the results when the hiddenLayers feature is set to it’s default value ‘a’. The accuracy is given in red.



Epoch = 50 Error per Epoch = 0.0008995 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘a’ Now we set the value of hiddenLayers to 1. The output is listed below (and the accuracy in red).



Epoch = 50 Error per Epoch = 0.0481054 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘1’ Now we set the hiddenLayer option to 3. The accuracy is given in red again



Epoch = 50 Error per Epoch = 0.0269675 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘3’ Now we set the hiddenLayer option to 5. The accuracy is given in red again.



Epoch = 50 Error per Epoch = 0.0173471 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘5’


7

Now we set the hiddenLayer value to 10. The accuracy is given in red (again):



Epoch = 50 Error per Epoch = 0.0062486 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘10’

Now we set the hiddenLayers option to 20. The accuracy is given in red again.

Correctly Classified Instances 667 97.6574 % Incorrectly Classified Instances 16 2.3426 % Info about this evaluation

Epoch = 50 Error per Epoch = 0.001507 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘20’ Now we do a final test to see if there’s still a significant increase within the accuracy. We set the hiddenLayers option to 30. The results are listed below with the accuracy in red.



Epoch = 50 Error per Epoch = 0.0014833 Learning Rate = 0.2 Momentum = 0.2 hiddenLayers = ‘30’ We can now make a clear table where we note all accuracy’s weighted to the hiddenLayers it’s input value. The underneath table shows us that the most significant increase of the accuracy is within 1 to 10 layers. After 20-30 hiddenLayers we have an almost perfect accuracy.

hiddenLayers value


‘a’ (default) 98.5359 %

1 13.3236 % 3 59.7365 % 5 69.9854 %

10 90.0439 % 20 97.6574 % 30 98.3895 %

I’ve also visualized the outputs. It’s becoming clear that the more hiddenLayers we put in the more connections can me made between the input attributes and the output attributes. We see that the red dots represents the number of hiddenLayers which we have put in.


8

Epoch of 1

Epoch of 3

Epoch of 10

Epoch of 20

Epoch of 30

Now we test the same number of hiddenLayers but then with a learning rate of 0.3 and we leave the epoch as it was (50 iterations). The accuracy of these output are listed below. If we compare these numbers to the previous table we can state that they’re pretty similar and that it’s hard to say how they differ.

hiddenLayers value


‘a’ (default) 98.6823 %

1 24.0117 % 3 55.7833 % 5 68.6676 %

10 91.8009 % 20 98.0966 % 30 98.6823 %

Now we try different settings for the hiddenLayer feature. Instead of adding one integer we add two. This will result in 2 layers of hiddenLayers. We leave the learning rate as 0.3 and use an epoch of 50 (50 iterations) as we have used in the previous examples also. We see that we get a ‘multi layer perceptron’.

Epoch set to 5,1

Epoch set to 1,5

Epoch set to 5,5


9

hiddenLayer value


1,1 26.2079 % 5,1 32.0644 %

10,1 26.2079 % 1,5 30.7467 % 5,5 62.0791 %

10,5 64.1288 % 1,10 32.7965 % 5,10 65.593 %

10,10 71.5959 % Now we do the same thing but we only adjust the learning rate to 0.6

hiddenLayer value


1,1 26.2079 % 5,1 32.7965 %

10,1 39.6779 % 1,5 26.2079 % 5,5 74.0849 %

10,5 79.063 % 1,10 23.1332 % 5,10 77.5988 %

10,10 94.2899 % Now we do the same thing but we only adjust the learning rate to 1.0. Unfortunately weka was unable to set a higher learning rate then 1.0. So 1.0 is the heighest value which we could pick.

hiddenLayer value


1,1 26.2079 % 5,1 38.9458 %

10,1 49.3411 % 1,5 28.1113 % 5,5 69.5461 %

10,5 80.9663 % 1,10 23.1332 % 5,10 76.2811 %

10,10 89.019 %

As a final test we want to know what the accuracy is we have a maximum value but below 1.0. So for this final test we set the learning rate to a value of 0.9.

hiddenLayer value


1,1 26.2079 % 5,1 27.0864 %

10,1 48.3163 % 1,5 28.5505 % 5,5 67.4963 %

10,5 82.4305 % 1,10 28.1113 % 5,10 73.3529 %

10,10 94.8755 % If we observe the tables which are listed above it’s hard to say where our increase of accuracy is. What we do see is that our maximum hiddenLayer gives us the highest accuracy. This counts for using both the learning rates 0.6 and 0.9 (94,8755 %).2

2 Unfortunately Weka was not able to set the learning rate to a higher value then 1.0. There was a fixed maximum set of 1.0 withing the program

Documents

Data mining Computerassignment 3