Kishore Bingi, B Rajanarayan Prusty, Neeraj Gupta · 2024
Data preprocessing is the first step before data-driven model development. Outlier detection is one of the preprocessing activities. Most of the time, detected outliers can’t be excluded from the dataset; therefore, outlier correction becomes essential and has gained immense interest among data-related researchers. This chapter aims to highlight lucrative outlier detection and correction algorithms suitable for power and energy applications. The requirement for outlier detection and correction is highlighted, and well-established methods are introduced. Further, each method’s merits and demerits are elaborated. The lack of suitable metrics to judge a particular preprocessing method as a better method has led the authors to propose a new set of reliable metrics, which adds one more dimension to inculcate interest in novice researchers in data preprocessing. A critical analysis of well-established methods, metrics, and their applications is expected to benefit researchers in selecting apt algorithms based on the applications in the power and energy sectors.
Cited by 0
Nosel, Ingrid · 2013
Cited by 0Open access
strempfel, teja · Figshare · 2025
The code and data from fNIRS and EEG. Preprocessing was conducted in MATLAB, using EEGLAB and FieldTrip.
Cited by 0Open access
Jiawei Han, Micheline Kamber, Jian Pei · 2013
Cited by 0
Sławomir Przyłucki, Waldemar Wójcik, Andrzej Kotyra et al. · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2003
In the article we present the idea of the data preprocessing in the optoelectronic measurement system for monitoring the industrial gas pollutants. A single optical measurement path can carry a big amount of data in an entirely parallel manner (e.g. spectral response of light in an absorbing media). To meet the requirements coming from optical system properties, the new algorithm called Local Discriminant Basis with Feature Selection - LDBFS is proposed. It's based on discrete wavelet transform and multidimensional data decomposition techniques. In order to assess the performance of the proposed method, the article presents the simulation results of data analysis of gas spectral absorption.
Cited by 0
Tarkeshwar Barua, Kamal Kant Hiran, Ritesh Kumar Jain et al. · 2024
Cited by 0
Ana Đokić, Hana Stefanović, Dragana Dudić · 2025
Smart home systems generate large volumes of data from various sensors, such as temperature, humidity, motion, and energy consumption. Effective preprocessing is essential to enhance data quality, reduce noise, and enable accurate analysis. This paper examines Python-based data preprocessing techniques for smart home environments. The preprocessing workflow includes data cleaning, handling missing values, normalization, feature selection, and data transformation. Data cleaning addresses duplicate records, outliers, and inconsistencies, while missing values are imputed using statistical and machine learning approaches. Normalization techniques standardize sensor readings to ensure consistency across data points. Feature engineering and dimensionality reduction refine the dataset for improved predictive modeling. By enhancing data quality, preprocessing contributes to smarter home automation, efficient anomaly detection, and optimized energy management. This study underscores the critical role of preprocessing in smart home analytics, facilitating reliable and meaningful insights for decision-making. The proposed techniques enhance the integration of smart home data into machine learning models, driving advancements in intelligent home automation systems.
Cited by 0
Anne Richelle (6589178), Chintan Joshi (7010672), Nathan E. Lewis (599027) · Figshare · 2019
Cited by 0Open access
Lei Guo (36221), Jun Ge (298545), Li Cheng (228291) et al. · Figshare · 2025
A. Baseline characteristics of the training set prior to standardization; B. Baseline characteristics of the training set following standardization; C. Principal component analysis (PCA) of the training set colored by disease status; D. PCA of the training set colored by dataset source; E. Baseline characteristics of the validation set prior to standardization; F. Baseline characteristics of the validation set following standardization; G. PCA of the validation set before batch effect correction; H. PCA of the validation set after batch effect correction. (TIF)
Cited by 0Open access
Đorđe Miladinović (6611840), Christine Muheim (6611843), Stefan Bauer (24356) et al. · Figshare · 2019
<p>The figure depicts the creation of the three-channel two-dimensional input for the CNNs. In step (a) raw time series of EEG/EMG signals are separately transformed into the corresponding time-frequency domain (power spectrum density is computed) via a sequence of short Fourier transforms applied to overlapping Hamming windows. In step (b) EEG signals are band-pass filtered (0.5–24<i>Hz</i>) and EMG power is integrated over frequency range (0.5-30Hz) resulting in one-dimensional representation of muscle activity change over time. Furthermore, one-dimensional representation of EMG is converted into the two-dimensional one by a multiplication of the signal. Finally, in step (c) the data is log transformed and standardized per frequency component.</p>
Cited by 0Open access
U. K. Gehlich, H. Kuckuck · International Association of Geodesy symposia · 1993
Cited by 3
B. Marangelli · Image and Vision Computing · 1991
Cited by 1
Mohammadreza Fani Sani · 2020
Cited by 5
Kallin Westin, Lena · DiVA at Umeå University (Umeå University) · 2004
In this paper, several ways to handle missing data, e.g. removing cases, mean imputation, and multiple imputation, are described and discussed. The Pima-Indians-Diabetes data set is used as a case study. This particular data set is interesting to use since it has not been obvious to all users that it actually contains a substantial amount of missing data. The data set is described in detail and the methods for coping with missing data mentioned in the text is applied on the data set. The preprocessing perceptron is used to train decision support systems on the data sets. A sketch of a way to impute missing data using the preprocessing perceptron is also proposed and discussed. The accuracy of the trained decision support systems, at the optimal efficiency point, lied in the interval 76-82% for the different methods. The highest values were obtained when all missing data cases were removed both from the test and the training set. This is, however, not a good way to handle missing data since the resulting decision support system is biased. Furthermore it will not be able to handle missing data when used on real data in the future. The results of the remaining methods were surprisingly similar, a reason for this might be that the data set used is rather large. Differences between methods would probably be larger in a smaller data set with larger amount of missing data.
Cited by 0Open access
El Moghrabi, Rawan · Scholarship@Western (Western University) · 2022
Data preprocessing is an essential step when building machine learning solutions. It significantly impacts the success of machine learning modules and the output of these algorithms. Typically, data preprocessing is made-up of data sanitization, feature engineering, normalization, and transformation. This paper outlines the data preprocessing methodology implemented for a data-driven predictive maintenance solution. The above-mentioned project entails acquiring historical electrical data from industrial assets and creating a health index indicating each asset's remaining useful life. This solution is built using machine learning algorithms and requires several data processing steps to increase the solution's accuracy and efficiency. In this project, the preprocessing measures implemented are data sanitization, daylight savings transformation, feature encoding, and data normalization. The purpose and results of each of the above steps are explained to highlight the importance of data preprocessing in machine learning projects.
Cited by 0
Fenglian Pan (21246895), You Zhou (129411), Carolina Vivas-Valencia (13817348) et al. · Figshare · 2025
<div> Substance use disorder, particularly opioid-related, is a serious public health challenge in the U.S. Accurately predicting opioid overdose events and stratifying the risk of having such an event are critical for healthcare providers to deliver effective interventions in patients with opioid overdose. Despite a large body of literature investigating various risk factors for the prediction, the existing research to date has not explicitly investigated and quantitatively modeled how an individual’s past opioid overdose events affect future occurrences. In this paper, we proposed a covariate-adjusted triggering point process to simultaneously model the effect of various risk factors on opioid overdose events and the triggering mechanism among opioid overdose events. The prediction performance was assessed by the U.S. state-wise Medicaid reimbursement claims data. Compared with commonly used prediction models, the proposed model achieved the lowest Mean Absolute Errors and Mean Absolute Percentage Errors on 30-, 60-, 90, 120-, 150-, and 180-day-ahead predictions. In addition, our results showed the statistical significance of considering the triggering mechanism for recurrent opioid overdose events prediction. On average, around 47% of the event recurrence were explained by the triggering mechanism. </div>
Cited by 0Open access
Hönle, Nicola, Grossmann, Matthias, Nicklas, Daniela et al. · Fachbereich Informatik (University of Stuttgart) · 2008
\n We present the design and implementation of a component for the\n preprocessing of position data taken from moving objects. The\n movement of mobile objects is represented by piecewise functions\n over time that approximate the real object movement and\n significantly reduce the initial data volume such that efficient\n storage and analysis of object trajectories can be achieved. The\n maximal acceptable deviation---an input parameter of our\n algorithms---of the approximations also includes the uncertainty of\n the position sensor measurements. We analyze and compare five\n different lossy preprocessing methods. Our results clearly indicate\n that even with simple approaches, a more than sufficient overall\n performance can be achieved.\n
Cited by 1
Taehyeong Kim (8991383), Kyoungmin Lee (7207943), Mookyung Cheon (68352) et al. · Figshare · 2024
<div><p>Understanding time-series interplay of genes is essential for diagnosis and treatment of disease. Spatio-temporally enriched NGS data contain important underlying regulatory mechanisms of biological processes. Generative adversarial networks (GANs) have been used to augment biological data to describe hidden intermediate time-series gene expression profiles during specific biological processes. Developing a pipeline that uses augmented time-series gene expression profiles is needed to provide an unbiased systemic-level map of biological processes and test for the statistical significance of the generated dataset, leading to the discovery of hidden intermediate regulators. Two analytical methods, GAN-WGCNA (weighted gene co-expression network analysis) and rDEG (rescued differentially expressed gene), interpreted spatiotemporal information and screened intermediate genes during cocaine addiction. GAN-WGCNA enables correlation calculations between phenotype and gene expression profiles and visualizes time-series gene module interplay. We analyzed a transcriptome dataset of two weeks of cocaine self-administration in C57BL/6J mice. Utilizing GAN-WGCNA, two genes (Alcam and Celf4) were selected as missed intermediate significant genes that showed high correlation with addiction behavior. Their correlation with addictive behavior was observed to be notably significant in aspect of statistics, and their expression and co-regulation were comprehensively mapped in terms of time, brain region, and biological process.</p></div>
Cited by 0Open access
Ubair Aziz (21485738), Ishrat Jabeen (1284885) · Figshare · 2025
In 1<sup>st</sup> step, 16 available were evaluated based upon the number of compounds being tested (Table. 2.) Most of these studies were observed to be limited to a low number of substances that inhibit the Transforming Growth Factor β/Smad3 pathway either through unknown mechanisms or by inhibiting the TGFβ-Receptor. In 2<sup>nd</sup> step, the active compounds from the selected study (Pubchem ID: 630) were extracted. This data was then preprocessed as 3<sup>rd</sup> step by i) removing possible artifacts reported in the experiment ii) compounds with no Compound ID removed iii) small fragment with molecular weight below 200 removed iv) racemic compounds removed v) only structures with dimeric symmetry retained. Compounds that passed through each of these filters are mentioned in this step. In 4<sup>th</sup> step these 33 dimers were docked to Smad3 at Smad3-FoxH1 binding interface using MolDock algorithm. (TIFF)
Cited by 0Open access
Aditya Singh, Navneet Kaur · 2022
<p> This study begins with an overview of data preprocessing, focusing on real-world data challenges. Before any data analysis method begins, these are the first problems that have got to be understood and resolved. In this work, the author discusses data preprocessing, like standardization and normalization including feature scaling to more readily accomplish the data classification. Finding the most informative collection of features is the goal of preprocessing to boost the classifier's performance. Include standardization is for the most part expected to take out the impact of a few quantitative highlights estimated on various scales. Besides, feature scaling is used to normalize all different numeric numbers to properly scaled numbers. The point of this part is to help analysts in picking a fitting preprocessing procedure for information investigation. The basic preprocessing methods used for the characterization of information are then addressed in this section. Fitting Python features to various information applications will be shown as concrete examples at the end of each session. </p>
Cited by 0Open access