Gil Shomron, Freddy Gabbay, Samer Kurzum et al. · arXiv (Cornell University) · 2021
Quantization is a technique used in deep neural networks (DNNs) to increase execution performance and hardware efficiency. Uniform post-training quantization (PTQ) methods are common, since they can be implemented efficiently in hardware and do not require extensive hardware resources or a training set. Mapping FP32 models to INT8 using uniform PTQ yields models with negligible accuracy degradation; however, reducing precision below 8 bits with PTQ is challenging, as accuracy degradation becomes noticeable, due to the increase in quantization noise. In this paper, we propose a sparsity-aware quantization (SPARQ) method, in which the unstructured and dynamic activation sparsity is leveraged in different representation granularities. 4-bit quantization, for example, is employed by dynamically examining the bits of 8-bit values and choosing a window of 4 bits, while first skipping zero-value bits. Moreover, instead of quantizing activation-by-activation to 4 bits, we focus on pairs of 8-bit activations and examine whether one of the two is equal to zero. If one is equal to zero, the second can opportunistically use the other's 4-bit budget; if both do not equal zero, then each is dynamically quantized to 4 bits, as described. SPARQ achieves minor accuracy degradation and a practical hardware implementation. The code is available at https://github.com/gilshm/sparq.
Cited by 3Open access
Jaeseong You, Minseop Park, K. J. Lee et al. · arXiv (Cornell University) · 2024
This paper investigates three different parameterizations of asymmetric uniform quantization for quantization-aware training: (1) scale and offset, (2) minimum and maximum, and (3) beta and gamma. We perform a comprehensive comparative analysis of these parameterizations' influence on quantization-aware training, using both controlled experiments and real-world large language models. Our particular focus is on their changing behavior in response to critical training hyperparameters, bit width and learning rate. Based on our investigation, we propose best practices to stabilize and accelerate quantization-aware training with learnable asymmetric quantization ranges.
Cited by 0Open access
Hasindu Kariyawasam, Ramith Hettiarachchi, Quansan Yang et al. · Optics Express · 2025
Optical processors, built with "optical neurons," can perform large-scale high-dimensional linear operations at the speed of light. With the current advances in micro-fabrication, such optical processors can now be 3D-fabricated, but at limited precision, eventually leading to a model mismatch due to quantized optical weights. To address this issue, we propose a quantization-aware training framework. Our approach accounts for physical constraints during the training process, leading to robust designs. We numerically demonstrate that our approach can design state-of-the-art optical processors using diffractive networks for multiple tasks despite quantized learnable parameters. We thus lay the foundation upon which improved optical processors may be 3D-fabricated in the future.
Cited by 0Open access
Hasindu Kariyawasam, Ramith Hettiarachchi, Quansan Yang et al. · Figshare · 2026
Optical processors, built with "optical neurons," can perform large-scale high-dimensional linear operations at the speed of light. With the current advances in micro-fabrication, such optical processors can now be 3D-fabricated, but at limited precision, eventually leading to a model mismatch due to quantized optical weights. To address this issue, we propose a quantization-aware training framework. Our approach accounts for physical constraints during the training process, leading to robust designs. We numerically demonstrate that our approach can design state-of-the-art optical processors using diffractive networks for multiple tasks despite quantized learnable parameters. We thus lay the foundation upon which improved optical processors may be 3D-fabricated in the future.
Cited by 0Open access
Morreale, Luca, Ramos, Alberto Gil C. P., Chadwick, Malcolm et al. · ArXiv.org · 2025
State-of-the-art (SOTA) generative models have demonstrated impressive capabilities in image synthesis or text generation, often with a large capacity model. However, these large models cannot be deployed on smartphones due to the limited availability of on-board memory and computations. Quantization methods lower the precision of the model parameters, allowing for efficient computations, \eg, in \INT{8}. Although aggressive quantization addresses efficiency and memory constraints, preserving the quality of the model remains a challenge. To retain quality in previous aggressive quantization, we propose a new fractional bits quantization (\short) approach. The novelty is a simple yet effective idea: we progressively reduce the model's precision from 32 to 4 bits per parameter, and exploit the fractional bits during optimization to maintain high generation quality. We show that the \short{} yields improved quality on a variety of diffusion models, including SD3.5-Medium, Sana, \pixart, and FLUX.1-schnell, while achieving $4-7\%$ lower FiD than standard QAT. Finally, we deploy and run Sana on a Samsung S25U, which runs on the Qualcomm SM8750-AB Snapdragon 8 Elite Hexagon Tensor Processor (HTP).
Cited by 0Open access
Dushan N. Wadduwage, Hasindu Kariyawasam, Ramith Hettiarachchi et al. · Research Square · 2024
Cited by 2Open access
Hasindu Kariyawasam, Ramith Hettiarachchi, Quansan Yang et al. · Figshare · 2026
Optical processors, built with "optical neurons," can perform large-scale high-dimensional linear operations at the speed of light. With the current advances in micro-fabrication, such optical processors can now be 3D-fabricated, but at limited precision, eventually leading to a model mismatch due to quantized optical weights. To address this issue, we propose a quantization-aware training framework. Our approach accounts for physical constraints during the training process, leading to robust designs. We numerically demonstrate that our approach can design state-of-the-art optical processors using diffractive networks for multiple tasks despite quantized learnable parameters. We thus lay the foundation upon which improved optical processors may be 3D-fabricated in the future.
Cited by 0Open access
Hasindu Kariyawasam, Ramith Hettiarachchi, Quansan Yang et al. · Figshare · 2026
Optical processors, built with "optical neurons," can perform large-scale high-dimensional linear operations at the speed of light. With the current advances in micro-fabrication, such optical processors can now be 3D-fabricated, but at limited precision, eventually leading to a model mismatch due to quantized optical weights. To address this issue, we propose a quantization-aware training framework. Our approach accounts for physical constraints during the training process, leading to robust designs. We numerically demonstrate that our approach can design state-of-the-art optical processors using diffractive networks for multiple tasks despite quantized learnable parameters. We thus lay the foundation upon which improved optical processors may be 3D-fabricated in the future.
Cited by 0Open access
Cédric Gernigon, Silviu-Ioan Filip, Olivier Sentieys et al. · arXiv (Cornell University) · 2024
Large-scale deep neural networks (DNNs) have achieved remarkable success in many application scenarios. However, high computational complexity and energy costs of modern DNNs make their deployment on edge devices challenging. Model quantization is a common approach to deal with deployment constraints, but searching for optimized bit-widths can be challenging. In this work, we present Adaptive Bit-Width Quantization Aware Training (AdaQAT), a learning-based method that automatically optimizes weight and activation signal bit-widths during training for more efficient DNN inference. We use relaxed real-valued bit-widths that are updated using a gradient descent rule, but are otherwise discretized for all quantization operations. The result is a simple and flexible QAT approach for mixed-precision uniform quantization problems. Compared to other methods that are generally designed to be run on a pretrained network, AdaQAT works well in both training from scratch and fine-tuning scenarios. Initial results on the CIFAR-10 and ImageNet datasets using ResNet20 and ResNet18 models, respectively, indicate that our method is competitive with other state-of-the-art mixed-precision quantization approaches.
Cited by 4Open access
Soroush Tabesh, Mher Safaryan, Andrei Panferov et al. · ArXiv.org · 2025
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization. CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information. On the theoretical side, we introduce the notion of Pareto-optimal solutions for quantized optimization, and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting. In terms of implementation, our approach is optimizer-agnostic, but we provide a highly-efficient implementation that leverages Adam statistics. CAGE significantly improves upon the prior state-of-the-art methods in terms of accuracy, for similar computational cost: for QAT fine-tuning, it halves the compression accuracy loss relative to the prior best method, while for QAT pre-training of Llama models, its accuracy for 3-bit weights-and-activations (W3A3) matches the accuracy achieved at 4-bits (W4A4) with the prior best method. The official implementation can be found over https://github.com/IST-DASLab/CAGE .
Cited by 0Open access
Rodion Novkin, Hussam Amrouch, Florian Klemme · 2023
<p>Graph Neural Networks (GNNs) are one of the best-performing models for processing graph data. They are known to have considerable computational complexity, despite the smaller number of parameters compared to traditional Deep Neural Networks (DNNs). Operations-to-parameters ratio for GNNs can be tens and hundreds of times higher than for DNNs, depending on the input graph size. This complexity indicates the importance of arithmetic operation optimization within GNNs through model quantization and approximation. In this work, for the first time, we combine both approaches and implement quantization- and approximation-aware training for GNNs to sustain their accuracy under the errors induced by inexact multiplications. We employ matrix multiplication CUDA kernel to speed up the simulation of approximate multiplication within GNNs. Further, we demonstrate the execution speed, accuracy, and energy efficiency of GNNs with approximate multipliers in comparison with quantized low-bit GNNs. We evaluate the performance of state-of-the-art GNN architectures (i.e., GIN, SAGE, GCN, and GAT) on various datasets and tasks (i.e., Reddit-Binary, Collab for graph classification, Cora and PubMed for node classification) with a wide range of approximate multipliers. </p>
Cited by 2Open access
Rodion Novkin, Hussam Amrouch, Florian Klemme · 2023
<p>Graph Neural Networks (GNNs) are one of the best-performing models for processing graph data. They are known to have considerable computational complexity, despite the smaller number of parameters compared to traditional Deep Neural Networks (DNNs). Operations-to-parameters ratio for GNNs can be tens and hundreds of times higher than for DNNs, depending on the input graph size. This complexity indicates the importance of arithmetic operation optimization within GNNs through model quantization and approximation. In this work, for the first time, we combine both approaches and implement quantization- and approximation-aware training for GNNs to sustain their accuracy under the errors induced by inexact multiplications. We employ matrix multiplication CUDA kernel to speed up the simulation of approximate multiplication within GNNs. Further, we demonstrate the execution speed, accuracy, and energy efficiency of GNNs with approximate multipliers in comparison with quantized low-bit GNNs. We evaluate the performance of state-of-the-art GNN architectures (i.e., GIN, SAGE, GCN, and GAT) on various datasets and tasks (i.e., Reddit-Binary, Collab for graph classification, Cora and PubMed for node classification) with a wide range of approximate multipliers. </p>
Cited by 0Open access
Jiang, Jiacheng, Meng, Yuan, Tang, Chen et al. · arXiv (Cornell University) · 2025
Current quantization-aware training (QAT) methods primarily focus on enhancing the performance of quantized models on in-distribution (I.D) data, while overlooking the potential performance degradation on out-of-distribution (OOD) data. In this paper, we first substantiate this problem through rigorous experiment, showing that QAT can lead to a significant OOD generalization performance degradation. Further, we find the contradiction between the perspective that flatness of loss landscape gives rise to superior OOD generalization and the phenomenon that QAT lead to a sharp loss landscape, can cause the above problem. Therefore, we propose a flatness-oriented QAT method, FQAT, to achieve generalizable QAT. Specifically, i) FQAT introduces a layer-wise freezing mechanism to mitigate the gradient conflict issue between dual optimization objectives (i.e., vanilla QAT and flatness). ii) FQAT proposes an disorder-guided adaptive freezing algorithm to dynamically determines which layers to freeze at each training step, effectively addressing the challenges caused by interference between layers. A gradient disorder metric is designed to help the algorithm identify unstable layers during training. Extensive experiments on influential OOD benchmark demonstrate the superiority of our method over state-of-the-art baselines under both I.D and OOD image classification tasks.
Cited by 0Open access
Jiaqi Zhang · 2025
With the rapid expansion of the AI era, data volumes and computational demands have grown significantly. Quantization-Aware Training (QAT), introduced in 2018, has emerged as a mainstream approach for enabling the quantization of large-scale models while preserving accuracy under constrained hardware resources. In recent years (2024–2025), QAT has been extensively applied in domains such as image recognition, large language models, and diffusion models. However, these applications have also revealed critical challenges, including unstable gradient propagation, increased computational complexity, and significant accuracy degradation under low-bit quantization. Although numerous improved QAT methods have been proposed, they remain fragmented and lack a systematic review. This paper provides a comprehensive survey of QAT by (i) tracing its historical development, (ii) clarifying its core definition, (iii) contrasting QAT with Post-Training Quantization (PTQ), and (iv) summarizing its applications across major fields. Furthermore, we categorize existing literature into six primary problem areas and four classes of solution strategies. As a survey, this work aims to offer researchers a structured perspective on QAT and to serve as a reference for future efforts in optimizing quantization techniques and facilitating practical deployment.
Cited by 0Open access
Hang Wang, Menglong Lu, Zhen Huang et al. · 2025
In recent years, the scale and computational demand for deep neural network models have been continuously increasing, leading to a growing need for efficient model deployment methods. Model quantization technology is an efficient way for model compression. However, low-bit quantization of models may result in diminished model accuracy. Quantization-aware training (QAT) is a representative training method for alleviating the decline of model accuracy. Nevertheless, most QAT methods require a large amount of labeled datasets and lack generalizability for tasks with limited data. We propose a new approach by using self-training, a semi-supervised method, to meet the dataset requirements of QAT. Based on the analysis of the characteristics of self-training and quantization-aware training, we use a meta-learning module for pseudo-label dataset selection and employ quantization operations to introduce noise into the model to enhance training effectiveness. We conducted experimental evaluations of traditional quantization training methods and our method on four network models (ResNet-50, ResNet-101, MobileNet-v2, MobileNet-v3) with the Cifar100 dataset. Our method enhances the training performance of both full-precision and quantized models compared to traditional quantization training methods. Utilizing 30% of the labeled dataset, compared to QAT, our method demonstrates an average accuracy enhancement of 2.19% after model training, while the average accuracy decline during the transition from a full-precision model to an 8-bit model is diminished by 0.55%.
Cited by 0
International Research Journal of Modernization in Engineering Technology and Science · 2025
A major obstacle to agricultural output is grape leaf stress conditions.Conventional diagnosis techniques are time-consuming and necessitate domain knowledge.This study investigates the use of Quantization-Aware Training (QAT) to create an effective deep learning model for the classification of grape leaf diseases.The model is refined using a dataset supplied by the Indian Council of Agricultural Research (ICAR) using the ResNet50 architecture.Real-time illness diagnosis is made possible by the quantized TFLite model, which guarantees interoperability with edge devices such as IoT modules.What makes it perfect for deployment in settings with limited resources is high accuracy and a smaller model size.
Cited by 0Open access
Sangho An, Jong Won Shin, Jangho Kim · IEEE Access · 2025
The evolution of deep neural networks (DNNs) naturally leads to an increase in model size. This necessitates various model compression techniques, such as pruning and quantization, to reduce memory usage and power consumption. In particular, combining these compression techniques can achieve significant cost savings. However, we found that methods using dynamic pruning and quantization suffer from instability in training and poor generalization performance due to the effects of the two Straight Through Estimators (STE). To address this problem, we propose a Quantization-aware training with Dynamic and Static pruning (QADS) method that takes advantage of both pruning and quantization by performing STE operations only during quantization from a certain point in time. In our experiments, the proposed method exhibits more stable training compared to existing techniques and achieves performance improvements on the CIFAR-10/100, ImageNet, and Google Speech Command datasets. The code is provided at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Ahnho/Quantization-aware-training-with-Dynamic-and-Static-Pruning</uri>.
Cited by 6Open access
Qiaolin Li · Applied and Computational Engineering · 2023
In recent years, reinforcement learning has been applied to many areas, especially deep reinforcement learning. Deep learning-based methods use a large computing method known as Quantization Aware Training (QAT). For this project, QAT will be used to investigate the performance of Deep Q-Networks. This method of QAT can reduce memory usage of the network, but may affect the training performance of the network with some games.
Cited by 0Open access
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani et al. · 2025
Steven K. Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, Dharmendra Modha. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. 2025.
Cited by 0Open access
Jonghoon Kwak, Kyungho Kim, Sang-Seol Lee et al. · 2022 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · 2022
In this paper, we present a novel quantization-aware training method proceeding in a specific order in deep learning network. Conventional quantization techniques generally perform all layers at the same time. However, quantization sensitivity is different for each layer. For example, a quantization error increases as the quantizing bit-widths decrease, therefore the quantized model is hard to converge in the training process. To handle the issue, we propose the layer selection module which controls the order of layers to be quantized. Specifically, each layer is sequentially quantized in the order of selection so that the quantized model can be converged effectively. In our experiments, we show the quantization results on the CIFAR-10 image dataset. We also analyze the results between the different selection methods and verify the effectiveness of our proposed method.
Cited by 6