| 3/2021 - 1 | View TOC | « Previous Article | Next Article » |
Robust 2-bit Quantization of Weights in Neural Network Modeled by Laplacian DistributionPERIC, Z.   , DENIC, B. , DINCIC, M. , NIKOLIC, J. |
View the paper record and citations in  |
| Click to see author's profile in SCOPUS, IEEE Xplore, Web of Science |
Download PDF  (1,299 KB) | Citation | Downloads: 1,330 | Views: 1,993 |
Author keywords
image classification, neural networks, quantization, signal to noise ratio, source coding
References keywords
neural(18), networks(14), learning(7), information(7), quantization(6), processing(6), systems(5), machine(5), deep(5), convolutional(5)
Blue keywords are present in both the references section and the paper title.
About this article
Date of Publication: 2021-08-31
Volume 21, Issue 3, Year 2021, On page(s): 3 - 10
ISSN: 1582-7445, e-ISSN: 1844-7600
Digital Object Identifier: 10.4316/AECE.2021.03001
Web of Science Accession Number: 000691632000001
SCOPUS ID: 85114815185
Abstract
Significant efforts are constantly involved in finding manners to decrease the number of bits required for quantization of neural network parameters. Although in addition to compression, in neural networks, the application of quantizer models that are robust to changes in the variance of input data is of great importance, to the best of authors knowledge, this topic has not been sufficiently researched so far. For that reason, in this paper we give preference to logarithmic companding scalar quantizer, which has shown the best robustness in high quality quantization of speech signals, modelled similarly as weights in neural networks, by Laplacian distribution. We explore its performance by performing the exact and asymptotic analysis for low resolution scenario with 2-bit quantization, where we draw firm conclusions about the usability of the exact performance analysis and design of our quantizer. Moreover, we provide a manner to increase the robustness of the quantizer we propose by involving additional adaptation of the key parameter. Theoretical and experimental results obtained by applying our quantizer in processing of neural network weights are very good matched, and, for that reason, we can expect that our proposal will find a way to practical implementation. |
| References | | | Cited By «-- Click to see who has cited this paper |
| [1] A. Zhang, Z. C. Lipton, M. Li, A. J. Smola, Dive into Deep Learning. Amazon Science, 2020.
[2] Z. Nagy. Artificial Intelligence and Machine Learning Fundamentals: Develop Real-World Applications Powered by the Latest AI Advances. Packt Publishing, 2018. [3] A. Krizhevsky, I. Sutskever, G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proc. of the International Conference on Neural Information Processing Systems, Harrahs and Harveys, Lake Tahoe, NV, USA, 2012, pp. 1097-1105 [4] G. Mukhtar, S. Farhan, "Convolutional neural network based prediction of conversion from mild cognitive impairment to Alzheimer's disease: A technique using hippocampus extracted from MRI", Advances in Electrical and Computer Engineering, vol. 20, no. 2, pp. 113-122, 2020. [CrossRef] [Full Text] [Web of Science Times Cited 8] [SCOPUS Times Cited 13] [5] S. Ren, K. He, R. Girshick, J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks," in Proc. of the Conference on Advances in neural information processing systems (NeurIPS), Montreal, Canada, 2015, pp. 91-99. [6] A. Conneau, H. Schwenk, L. Barrault, Y. Lecun, "Very deep convolutional networks for text classification," arXiv preprint arXiv: 1606.01781, 2016. [7] V. Delic, Z. Peric, M. Secujski, N. Jakovljevic, J. Nikolic, et. al, "Speech technology progress based on new machine learning paradigm," Computational Intelligence and Neuroscience, vol. 2019, Article ID 4368036, 19 pages, 2019. [CrossRef] [Web of Science Times Cited 30] [SCOPUS Times Cited 50] [8] A. Albu, R. E. Precup, T. A. Teban, "Results and challenges of artificial neural networks used for decision-making and control in medical applications," Facta Universitatis Series: Mechanical Engineering, vol. 17, no. 3, pp. 285-308, 2019. [CrossRef] [Web of Science Times Cited 96] [SCOPUS Times Cited 108] [9] M. U. Ahmed, S. Brickman, A. Dengg,N. Fasth, M. Mihajlovic, et. al., "A machine learning approach to classify pedestrians' events based on IMU and GPS," International Journal of Artificial Intelligence, vol. 17, no 2, pp. 154-167, 2019. [10] U. L. Yuhana, N. Z. Fanani, E. M. Yuniarno, S. Rochimah, L. T. Koczy, et.al., "Combining fuzzy signature and rough sets approach for predicting the minimum passing level of competency achievement," International Journal of Artificial Intelligence, vol. 18, no. 1, pp. 237-249, 2020. [11] Y. Sheng, H. Ma, W. Xia, "A Pointer neural network, for the vehicle routing problem with task priority and limited resources," Information Technology and Control, vol. 49, no.2, pp. 237-248, 2020. [CrossRef] [Web of Science Times Cited 6] [SCOPUS Times Cited 11] [12] Y. Li, Y. Bao, W. Chen, "Fixed-sign binary neural network: An efficient design of neural network for Internet-of-Things devices," IEEE Access, vol. 8, pp. 164858-164863, 2018. [CrossRef] [Web of Science Times Cited 8] [SCOPUS Times Cited 10] [13] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, Y. Bengio, "Quantized neural networks: Training neural networks with low precision weights and activations," Journal of Machine Learning Research, vol. 18, no. 1, pp. 6869-6898, 2017. [14] D. Lin, S. Talathi, S. Annapureddy, "Fixed point quantization of deep convolutional networks," in Proc. of the 33rd International Conference on Machine Learning, New York, NY, USA, 2016. [15] R. Banner, Y. Nahshan, E. Hoffer, D. Soudry, "ACIQ: Analytical Clipping for Integer Quantization of Neural Networks," arXiv preprint arXiv: 1810.05723, 2018. [16] L. Enderich, F. Timm, W. Burgard, "SYMOG: Learning symmetric mixture of Gaussian modes for improved fixed-point quantization," Neurocomputing, vol. 416, pp. 310-315, 2020. [CrossRef] [Web of Science Times Cited 5] [SCOPUS Times Cited 6] [17] A. Nannarelli, "Variable precision 16-bit floating-point vector unit for embedded processors," in Proc. of IEEE 27th Symposium on Computer Arithmetic, (ARITH 2020), Portland, OR, USA, 2020. [18] R. Banner, I. Hubara, E. Hoffer, D. Soudry, "Scalable methods for 8-bit training of neural networks," in Proc. of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada, 2018. [19] R. Banner, Y. Nahshan, D. Soudry, "Post training 4-bit quantization of convolutional networks for rapid-deployment," in Proc. of the 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 2019. [20] J. Choi, S. Venkataramani, V. Srinivasan, K. Gopalakrishnan, Z. Wang, et. al, "Accurate and efficient 2-bit quantized neural networks," in Proc. of the 2nd SysML Conference, Stanford, CA, USA, 2019. [21] L. Deng, P. Jiao, J. Pei, Z. Wu, G. Li, "GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework," Neural Networks, vol. 100, pp. 49-58, 2018. [CrossRef] [Web of Science Times Cited 89] [SCOPUS Times Cited 104] [22] H. Qina, R. Gonga, X. Liu, X. Baie, J. Songc, et. al, "Binary neural networks: A survey," Pattern Recognition, vol. 105, Article ID: 107281, 2020. [CrossRef] [Web of Science Times Cited 267] [SCOPUS Times Cited 340] [23] Z. Peric, B. Denic, M. Savic, V. Despotovic, "Design and analysis of binary scalar quantizer of Laplacian source with applications," Information, vol. 11, 18 pages, 2020. [CrossRef] [Web of Science Times Cited 11] [SCOPUS Times Cited 13] [24] N. S. Jayant, P. Noll, Digital Coding of Waveforms: Principles and Applications to Speech and Video. New Jersey, Prentice Hall, Chapter 4, pp. 115-188, 1984. [25] S. Na, "Asymptotic formulas for mismatched fixed-rate minimum MSE Laplacian Quantizers," IEEE Signal Processing Letters, vol. 15, pp. 13-16, 2008. [CrossRef] [Web of Science Times Cited 32] [SCOPUS Times Cited 36] [26] Z. Peric, G. Petkovic, B. Denic, A. Stanimirovic, V. Despotovic, et al., "Gaussian source coding using a simple switched quantization algorithm and variable length codewords," Advances in Electrical and Computer Engineering, vol. 20, no. 4, pp. 11-18, 2020. [CrossRef] [Full Text] [Web of Science Times Cited 1] [SCOPUS Times Cited 1] [27] S. Tomic, Z. Peric, M. Tancic, J. Nikolic, "Backward adaptive and quasi-logarithmic quantizer for sub-band coding of audio," Information Technology and Control, vol. 47, no. 1, pp. 131-139, 2018. [CrossRef] [Web of Science Times Cited 8] [SCOPUS Times Cited 21] [28] M. Dincic, Z. Peric, D. Denic, Z. Stamenkovic, "Design of robust quantizers for low-bit analog-to-digital converters for Gaussian source," Journal of Circuits, Systems and Computers, vol. 28, no. supp01, 1940002, 2019. [CrossRef] [Web of Science Times Cited 5] [SCOPUS Times Cited 5] [29] Z. Peric, J. Nikolic, D. Aleksic, A. Peric, "Symmetric quantile quantizer parameterization for the Laplacian source: Qualification for contemporary quantization solutions," Mathematical Problems in Engineering, vol. 2021, Article ID 6647135, 12 pages, 2021. [CrossRef] [Web of Science Times Cited 5] [SCOPUS Times Cited 5] [30] S. Gazor, W. Zhang, "Speech probability distribution," IEEE Signal Processing Letters, vol. 10, pp. 204-207, 2003. [CrossRef] [Web of Science Times Cited 209] [SCOPUS Times Cited 258] [31] Y. LeCun, C. Cortez, C. Burges, "The MNIST Handwritten Digit Database," available online: yann.lecun.co. Web of Science® Citations for all references: 780 TCR SCOPUS® Citations for all references: 981 TCR Web of Science® Average Citations per reference: 24 ACR SCOPUS® Average Citations per reference: 31 ACR TCR = Total Citations for References / ACR = Average Citations per Reference We introduced in 2010 - for the first time in scientific publishing, the term "References Weight", as a quantitative indication of the quality ... Read more Citations for references updated on 2024-10-12 17:32 in 105 seconds. Note1: Web of Science® is a registered trademark of Clarivate Analytics. Note2: SCOPUS® is a registered trademark of Elsevier B.V. Disclaimer: All queries to the respective databases were made by using the DOI record of every reference (where available). Due to technical problems beyond our control, the information is not always accurate. Please use the CrossRef link to visit the respective publisher site. |
Copyright ©2001-2024
Faculty of Electrical Engineering and Computer Science
Stefan cel Mare University of Suceava, Romania
All rights reserved: Advances in Electrical and Computer Engineering is a registered trademark of the Stefan cel Mare University of Suceava. No part of this publication may be reproduced, stored in a retrieval system, photocopied, recorded or archived, without the written permission from the Editor. When authors submit their papers for publication, they agree that the copyright for their article be transferred to the Faculty of Electrical Engineering and Computer Science, Stefan cel Mare University of Suceava, Romania, if and only if the articles are accepted for publication. The copyright covers the exclusive rights to reproduce and distribute the article, including reprints and translations.
Permission for other use: The copyright owner's consent does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific written permission must be obtained from the Editor for such copying. Direct linking to files hosted on this website is strictly prohibited.
Disclaimer: Whilst every effort is made by the publishers and editorial board to see that no inaccurate or misleading data, opinions or statements appear in this journal, they wish to make it clear that all information and opinions formulated in the articles, as well as linguistic accuracy, are the sole responsibility of the author.
Faculty of Electrical Engineering and Computer Science
Stefan cel Mare University of Suceava, Romania
All rights reserved: Advances in Electrical and Computer Engineering is a registered trademark of the Stefan cel Mare University of Suceava. No part of this publication may be reproduced, stored in a retrieval system, photocopied, recorded or archived, without the written permission from the Editor. When authors submit their papers for publication, they agree that the copyright for their article be transferred to the Faculty of Electrical Engineering and Computer Science, Stefan cel Mare University of Suceava, Romania, if and only if the articles are accepted for publication. The copyright covers the exclusive rights to reproduce and distribute the article, including reprints and translations.
Permission for other use: The copyright owner's consent does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific written permission must be obtained from the Editor for such copying. Direct linking to files hosted on this website is strictly prohibited.
Disclaimer: Whilst every effort is made by the publishers and editorial board to see that no inaccurate or misleading data, opinions or statements appear in this journal, they wish to make it clear that all information and opinions formulated in the articles, as well as linguistic accuracy, are the sole responsibility of the author.
