AN APPROACH IN DETERMINING THE TOPOLOGY OF MULTI-LAYER PERCEPTRON USING PRINCIPAL COMPONENT ANALYSIS AND CLUSTERING - Ibnu Choldun R., Muhammad
Setting the appropriate hyper-parameters is a challenge in the successful implementation of neural networks. One of the important hyper-parameters is the topology of the neural networks which has a high influence on the performance of the neural network. Determining the topology of neural networks means determining the number of hidden layers needed along with the number of neurons for each hidden layer. The approach or method in determining this topology can be done manually or automatically. Determination manually, for example through trial and error or the rule of thumb. Determining the topology through a trial and error approach, until finding an optimal setting is a time-consuming process. In the rule of thumb approach, the topology determination uses a certain formula based on the number of inputs or the number of outputs, its use is very dependent on the experience of the researcher. The automated approach is divided into two, namely the model-based approach and the non-model-based approach. An approach that is not model-based, for example, is grid search and random search, while a model-based approach, for example, is to use the particle swarm optimization (PSO) algorithm. In this automated approach, the topology is determined through a training process. Some existing manual approaches, the determination of the topology is only based on the number of inputs and the number of outputs, which is not generally applicable to different datasets. Although it is still a difficult research area, research to determine the neural networks topology must continue. One of the research opportunities to determine the number of hidden layers and the number of neurons is to use principal component analysis (PCA) and clustering. The use of PCA to determine the number of hidden layers based on analogizing the complexity of the hidden layer in neural network is in line with the cumulative variance in PCA. After the number of components is obtained, for each component a K-Means clustering process is carried out which is used to determine the number of neurons from the number of clusters formed. The purpose of this research is to produce an approach or method to determine the topology of neural networks, especially those in the form of Multi-Layer Perceptron (MLP). The resulting method is expected to be a guide for determining the topology of the neural network before the training process is carried out, especially for outputs that have a regression or classification function. The proposed approach is expected to provide stable performance on MLP. The datasets used in the experiment are 14 tabular datasets: 4 datasets for the classification objective function, 5 datasets for the regression objective function, and 5 datasets for the data with large number attributes classification objective function. For each of these datasets, PCA and K-Means clustering are applied and then the performance of each resulting topology is calculated. Performance is also compared with the previous method that determines the topology before the training process: the first 4 datasets are compared with the trial and error method, the other 10 datasets are compared with the rule of thumb method (Sartori method, Huang method, Tamura and Tateishi method, and Qian and Yong method). Performance for the classification objective function is measured using the level of accuracy, while for the regression objective function using the Root Means Square Error (RMSE). The experimental results show that in general the proposed approach or method has better performance (12 out of 14 datasets) than the comparison method. An experiment was also carried out for the regression objective function using the Forrest Type Mapping dataset and the results were compared with the Tej and Holban method. The results obtained from the research of Tej and Holban using clustering and multiple regression, the topology with the best performance is to use one hidden layer with 36 neurons. This topology gives the best accuracy value is 87.83%. By using the approach proposed by the researcher, the best topology is to use two hidden layers with the number of neurons being 37 and 33 respectively. This topology obtains the best accuracy of 88.79%.. The proposed approach shows that it has achieved the goal of being able to determine the topology of the neural network, especially the topology in the form of MLP. The cumulative variance of the principal component analysis can be used as a reference to determine the number of hidden layers in the neural network. From the research results, the cumulative variance used as a reference for the purpose of multi-class classification is 60% or 70%, for binary classification is 40%, while for the regression objective function is 80% or higher. So in general it can be said that the cumulative variance required for classification is lower than the cumulative variance required for regression.