Paul M. Baggenstoss · arXiv (Cornell University) · 2021
In this paper, it is shown that an auto-encoder using optimal reconstruction significantly outperforms a conventional auto-encoder. Optimal reconstruction uses the conditional mean of the input given the features, under a maximum entropy prior distribution. The optimal reconstruction network, which is called deterministic projected belied network (D-PBN), resembles a standard reconstruction network, but with special non-linearities that mist be iteratively solved. The method, which can be seen as a generalization of maximum entropy image reconstruction, extends to multiple layers. In experiments, mean square reconstruction error reduced by up to a factor of two. The performance improvement diminishes for deeper networks, or for input data with unconstrained values (Gaussian assumption).
Cited by 1Open access
Christoph Bunte, Amos Lapidoth · IEEE Transactions on Information Theory · 2014
A task is randomly drawn from a finite set of tasks and is described using a fixed number of bits. All the tasks that share its description must be performed. Upper and lower bounds on the minimum pth moment of the number of performed tasks are derived. The case where a sequence of tasks is produced by a source and n tasks are jointly described using nR bits is considered. If R is larger than the Rényi entropy rate of the source of order 1/(1 + ρ) (provided it exists), then the ρth moment of the ratio of performed tasks to n can be driven to one as n tends to infinity. If R is smaller than the Rényi entropy rate, this moment tends to infinity. The results are generalized to account for the presence of side-information. In this more general setting, the key quantity is a conditional version of Rényi entropy that was introduced by Arimoto. For IID sources, two additional extensions are solved, one of a rate-distortion flavor and the other where different tasks may have different nonnegative costs. Finally, a divergence that was identified by Sundaresan as a mismatch penalty in the Massey-Arikan guessing problem is shown to play a similar role here.
Cited by 58Open access
Schwarz, H., Wiegand, T., Marpe, D. · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · 2012
Decomposing a value range of the respective syntax elements into a sequence of n partitions with coding the components of z laying within the respective partitions separately with at least one by VCL coding and with at least one by PIPE or entropy coding is used to greatly increase the compression efficiency at a moderate coding overhead since the coding scheme used may be better adapted to the syntax element statistics. Accordingly, in accordance with embodiments, syntax elements are decomposed into a respective number n of source symbols si with i=1...n, the respective number n of source symbols depending on as to which of a sequence of n partitions (140<SB>1-3</SB>) into which a value range of the respective syntax elements is sub-divided, a value z of the respective syntax elements falls into, so that a sum of values of the respective number of source symbols si yields z, and, if n > 1, for all i=1...n-1, the value of si corresponds to a range of the ith partition.
Cited by 0Open access
Cognitive technologies · 2006
Cited by 0
Seyedmohammad Seyedzadehdelcheh · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2023
Entropy agnostic data encoding includes: receiving, by an encoder, input data including a bit string; generating a plurality of candidate codewords, including encoding the input data bit string with a plurality of binary vectors, wherein the plurality of binary vectors includes a set of deterministic biased binary vectors and a set of random binary vectors; selecting, in dependence upon a predefined criteria, one of the plurality of candidate codewords; and transmitting the selected candidate codeword to a decoder.
Cited by 0Open access
Subhash Kak · 2021
<p>This paper considers the entropy perspective on the problem of noise in the circuit where the quantum data is prepared before it is sent forward to the error correction encoder. Since the errors in the circuits before the data qubits are converted to logical qubits cannot be corrected, there will be residual qubit errors in the processing system. This constitutes a great challenge for developing useful, scalable quantum computers. </p>
Cited by 2Open access
Myung-Sin Song · Birkhäuser Boston eBooks · 2008
Entropy encoding is a method of lossless compression that is performed on an image after the quantization stage. It enables one to represent an image in a more efficient way with less memory needed for storage or transmission. In this chapter, we will explore various schemes of entropy encoding and how they work mathematically where applicable.
Cited by 17
Stefano Melacci, Marco Gori · 2011
Following the basic principles of Information-Theoretic Learning (ITL), in this paper we propose Minimum Entropy Encoders (MEEs), a novel approach to data clustering. We consider a set of functions that project each input point onto a minimum entropy configuration (code). The encoding functions are modeled by kernel machines and the resulting code collects the cluster membership probabilities. Two regularizers are included to balance the distribution of the output features and favor smooth solutions, respectively, thus leading to an unconstrained optimization problem that can be efficiently solved by conjugate gradient or concave-convex procedures. The relationships with Maximum Margin Clustering algorithms are investigated, which show that MEEs overcomes some of the critical issues, such as the lack of a multi-class extension and the need to face problems with a large number of constraints. A massive evaluation on several benchmarks of the proposed approach shows improvements over state-of-the-art techniques, both in terms of accuracy and computational complexity.
Cited by 1
Magda Gregorová, Marc Desaules, Alexandros Kalousis · arXiv (Cornell University) · 2021
We consider the problem of learned transform compression where we learn both, the transform as well as the probability distribution over the discrete codes. We utilize a soft relaxation of the quantization operation to allow for back-propagation of gradients and employ vector (rather than scalar) quantization of the latent codes. Furthermore, we apply similar relaxation in the code probability assignments enabling direct optimization of the code entropy. To the best of our knowledge, this approach is completely novel. We conduct a set of proof-of concept experiments confirming the potency of our approaches.
Cited by 0Open access
Einat Granot-Atedgi (389933), Gašper Tkačik (213054), Ronen Segev (178535) et al. · Figshare · 2015
<p>The explicit dependence of single-neuron terms (, vertical axis, ‘T’ or ‘S’), and the absence or presence of pairwise terms (, horizontal axis, ‘1’ or ‘2’), together define the type of the maximum entropy model (e.g. pairwise SDME is ‘S2’). For completeness, the first row of the table includes static maximum entropy models of population vocabulary, , which have no explicit stimulus dependence. Full conditionally independent model (T1) reproduces exactly the instantaneous firing rate of every neuron, and thus fully captures the stimulus sensitivity, history effects, and adaptation on a single neuron level; for experimentally recorded rasters with stimulus repeats, simulated T1 rasters are often generated by taking the original data and, at each time point and for every neuron, randomly permuting the responses recorded on different stimulus repeats. “Total correlation” is the pairwise correlation matrix of activities, , averaged over all repetitions and all times in the experiment.</p>
Cited by 0Open access
Andrea Berdondini · 2021
ABSTRACT:
Cited by 0Open access
Hu, Yifan, Liang, Frank, Zhao, Dachuan et al. · ArXiv.org · 2025
Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream tasks. However, applying BPE to unsegmented languages such as Chinese presents significant challenges, as its frequency-driven merge operation is agnostic to linguistic boundaries. To address this, we propose two entropy-informed pre-tokenization strategies that guide BPE segmentation using unsupervised information-theoretic cues. The first approach uses pointwise mutual information and left/right entropy to identify coherent character spans, while the second leverages predictive entropy derived from a pretrained GPT-2 model to detect boundary uncertainty. We evaluate both methods on a subset of the PKU dataset and demonstrate substantial improvements in segmentation precision, recall, and F1 score compared to standard BPE. Our results suggest that entropy-guided pre-tokenization not only enhances alignment with gold-standard linguistic units but also offers a promising direction for improving tokenization quality in low-resource and multilingual settings.
Cited by 0Open access
Ryeong-Hee Gweon, Woo-Jin Han, Yung-Lyul Lee · Journal of Broadcast Engineering · 2011
ITU-T VCEG과 ISO/IEC MPEG은 공동으로 JCT-VC(Joint Collaborative Team on Video Coding) 를 구성하여 차세대 비디오 코덱 HEVC(High Efficiency Video Coding)에 대한 표준화를 진행하고 있다. HEVC에서 사용되고 있는 양자화 및 역 양자화는 TU(Transform Unit)내 모든 위치의 변환계수에 동일한 크기의 양자화 및 역 양자화를 수행한다. TU내의 변환계수는 동일하게 분포되어 있지 않으므로 모든 위치의 변환계수에 동일한 크기의 양자화 및 역 양자화를 수행하는 것은 비효율적이다. 또한 주사 순서상 뒤쪽에 있는 양자화 된 계수는 엔트로피 부호화 방법상 비효율적 일 수도 있다. 이 논문에서는 엔트로피 부호화의 효율성을 고려하여 TU 내 변환계수의 위치에 따라 다른 크기의 양자화를 수행하여 성능을 향상시키는 알고리즘을 제시한다. 이 알고리즘의 원리는 양자화 된 계수 분포의 통계적 특징에 따른 TU의 주사 순서에 따라 다른 크기의 양자화 및 역 양자화를 수행하는 것이다. 이 알고리즘은 평균 Y BD-rate 0.34% 의 압축률 향상을 나타내었다. MPEG and VCEG have constituted a collaboration team called JCT-VC(Joint Collaborative Team on Video Coding) and have been developing HEVC(High Efficiency Video Coding) standard. All transform coefficients in a TU(Transform Unit) have been equally quantized according to the quantization and inverse quantization method which is used in HEVC standard. Such an equal quantization is not efficient because the transformed coefficients in the TU are not eqully distributed. Furthermore, the quantized coefficients which is positioned in later scanning order cannot be efficient due to the entropy scanning method. We suggest an algorithm that transform coefficients are quantized at different values according to the position in TU considering a scanning order of entropy encoding to improve the coding efficiency. The principle of this algorithm is that quantization and inverse quantization are carried out according to the scanning order which is in accordance with the statistical characteristic of distribution of quantized transform coefficients. The proposed algorithm shows on the average of 0.34% Y BD-rate compression rate improvement.
Cited by 1Open access
Bruna Clara De Simone, F. De Luca, B. Maraviglia · Magnetic Resonance in Medicine · 1987
Abstract Results are reported which demonstrate that the maximum entropy method can be applied to phase‐encoding NMR imaging. © 1987 Academic Press, Inc.
Cited by 8
Palle E. T. Jorgensen, Myung-Sin Song · Journal of Mathematical Physics · 2007
By introducing Hilbert space and operators, we show how probabilities, approximations, and entropy encoding from signal and image processing allow precise formulas and quantitative estimates. Our main results yield orthogonal bases which optimize distinct measures of data encoding.
Cited by 28Open access
Andreas Soegandi · ComTech Computer Mathematics and Engineering Applications · 2010
The purpose of this study was to perform lossless compression on the uncompress audio file audio to minimize file size without reducing the quality. The application is developed using the entropy encoding compression method with rice coding technique. For the result, the compression ratio is good enough and easy to be developed because the algorithm is quite simple.
Cited by 0Open access
Myung-Sin Song · 2009
Cited by 3
Feng Ding, Yingbo Wang, Shuo Yu et al. · Knowledge-Based Systems · 2025
Cited by 0
Contemporary mathematics and its applications · 2023
Cited by 0
Palle E. T, Jorgensen, Myung-sin Song · 2007
Abstract. By introducing Hilbert space and operators, we show how probabilities, approximations and entropy encoding from signal and image processing allow precise formulas and quantitative estimates. Our main results yield orthogonal bases which optimize distinct measures of data encoding.
Cited by 0