Entropy Learning in Neural Network
Vol. 20 No. 3-4 (2003), Research Article
Vol. 20 No. 3-4 (2003)

Entropy Learning in Neural Network

Research Article
https://doi.org/10.29037/ajstd.362
Published 2017-12-27
Geok See Ng
Intelligent System Laboratory, School of Computer Engineering, Nanyang Technological University, Blk N4 #2A-32, Nanyang Avenue, Singapore 639798
D. Shi
Intelligent System Laboratory, School of Computer Engineering, Nanyang Technological University, Blk N4 #2A-32, Nanyang Avenue, Singapore 639798
A. Wahab
Intelligent System Laboratory, School of Computer Engineering, Nanyang Technological University, Blk N4 #2A-32, Nanyang Avenue, Singapore 639798
H. Singh
Intelligent System Laboratory, School of Computer Engineering, Nanyang Technological University, Blk N4 #2A-32, Nanyang Avenue, Singapore 639798

Abstract

In this paper, entropy term is used in the learning phase of a neural network.  As learning progresses, more hidden nodes get into saturation.  The early creation of such hidden nodes may impair generalisation.  Hence entropy approach is proposed to dampen the early creation of such nodes.  The entropy learning also helps to increase the importance of relevant nodes while dampening the less important nodes.  At the end of learning, the less important nodes can then be eliminated to reduce the memory requirements of the neural network.
https://doi.org/10.29037/ajstd.362
PDF

References

Muller, B. and Reinhardt, J. (1990), Springer Verlag.

Battiti, R.(1994), IEEE Transactions on Neural Network, vol. 5, no. 4, pp. 537-550.

Prat, L.Y.(1993), Computer Science Department, Rutgers University, ML-TR-37.

Deco, G., W.F. and Zimmermann, H.G. (1995), Neural Computation, vol. 7, pp. 86-107.

Burrows, T.L. and Niranjan, M. (1993), Technical report CUED/F-INFENG/TR158, Cambridge University Engineering Department.

Kamimura, R. and Nakanishi, S. (1995), IEICE Trans. Inf. and Syst., vol. E78, pp. 484489.

Kamimura, R. and Nakanishi, S. (1996), International Conference on Neural Information Processing, Hong Kong, pp. 145-150.

Kirpatrick, S., Gelatt Jr., C.D. and Vecchi, M.P. (1983), Science, vol. 220, pp. 671-680.

Ng, G.S., Erdogan, S.S. and Chan, K.H. (1996), Fourth International Conference On Automation, Robotics and Computer Vision, pp. 2472-2477.

Mozer, M.C. and Smolensky, P. (1989), Connection Science, vol. 1, no. 1, pp. 3-16.

Karnin, E.D. (1990), IEEE Transactions on Neural Networks, vol. 1, pp. 239-242.

Ruck, D.W., Rogers, S.K., Kabrisky, M., Priddy, K. L., and Tarr, G. L. (1992), TechnicalReport No 92-92, Department of Electrical and Computer Engineering, Air Force Institute of Technology.

Chung, F. L. and Lee, T. (1992), International Journal of Neural Systems, vol. 3, pp.

-314.

Shamir, N., Saad, D., and Marom, E. (1993), International Journal of Neural Systems, vol. 4, pp. 143-158.

Le, C. Y., Denker, J.S. and Solla, S.A. (1990), Edited by Touretzky, D.S., Morgan Kaufmann, San Mateo, pp. 598-605.

Fisher, R.A. (1936), Annals of Eugenics,vol. 7, pp. 179-188.

Downloads

Download data is not yet available.