Aman Rawat, Mr.Ritesh Kumar · Zenodo (CERN European Organization for Nuclear Research) · 2025
Data preprocessing is a fundamental step in any machine learning pipeline, as the quality of input data directly influences the reliability and accuracy of predictive models. Real-world datasets often contain missing values, duplicate entries, inconsistent formats, outliers, and non-standardized features, making manual preprocessing time-consuming, error-prone, and difficult to reproduce. To address these challenges, this research presents an Automated Data Cleaning and Preprocessing System designed to streamline the transformation of raw data into structured, analysis-ready form with minimal user intervention. The system integrates automated detection of missingness, statistical and algorithmic imputation, outlier identification, categorical encoding, normalization, and comprehensive summary reporting through a modular pipeline architecture. Experiments conducted on diverse datasets from multiple domains demonstrate improvements in data consistency, distributional stability, and downstream machine learning performance. The results highlight that automated preprocessing not only reduces human effort but also ensures reproducibility, scalability, and enhanced model accuracy, making it suitable for academic research, industrial applications, and production-level data workflows.
Cited by 0Open access
Engy Fouda · Apress eBooks · 2020
We mentioned in the preface that in this book we try to balance between using the integrated development environment (IDE) and the code. In Chapters 5 and 6, we focus on the code. This chapter shows some advanced data cleaning and querying methods, such as using PROC SQL, SELECT, WHERE, WHEN, and DO loops.
Cited by 0
Xiangying Li · 2013
Cited by 37
K. Khurshid, K. Staenz, Lixin Sun et al. · Canadian Journal of Remote Sensing · 2006
A procedure for processing hyperspectral data acquired with Hyperion has been developed with an aim to correct for sensor artifacts and atmospheric and geometric effects. Advances in preprocessing of hyperspectral remote sensing data have enabled more accurate atmospheric correction and have led to the development of new information extraction techniques in the areas of agriculture, forestry, geosciences, and environmental monitoring. These processing and analysis tools have been incorporated into Imaging Spectrometer Data Analysis Systems (ISDAS), a software package developed at the Canada Centre for Remote Sensing (CCRS). The procedure, as applied for Hyperion data, begins with geometric corrections to the short-wave infrared (SWIR) component to register the SWIR and visible near-infrared (VNIR) data spatially. This is followed by the removal of stripes and pixel (column) dropouts and noise reduction, using recently developed automated software tools. The data cube is subsequently analyzed using keystone and spectral smile detection software to characterize these distortions. Included in the smile detection procedure is an optional gain and offset correction technique. The radiance data are converted to reflectance using a MODTRAN-based atmospheric correction procedure. Only at this point are the data corrected for smile effects. Any artifacts still remaining after these corrections are removed by post-processing.
Cited by 60
Arshi Shamsi, Rahul Nayak, Pankaj Pratap Singh et al. · 2012
Now a day’s many web sites are growing and which indirectly leads to increase the complexity of web site designing. To simplify this we have to understand how web sites are being used, how navigation is done, and how many users use it and how much time user spends on pages. This is done using Web Usage Mining. Data sources to usage mining are client side cookies, web server log, software agent etc. This paper presents, how web server log data is preprocesses, which includes data cleaning, user identification and Sessionization, path completion. Once the data is preprocessed it is used for discovering some useful patterns. In this paper, I have cover only the case of a Web server (HTTP server) data. Following fig. 2 shows the sample web server log data recorded by the WebLog Expert lite tool.
Cited by 4
Ayhan Yuksel (733525), Tamer Olmez (733526) · Figshare · 2016
Cited by 0Open access
Allison Jevitt (8729008), Deeptiman Chatterjee (8770931), Gengqiang Xie (167638) et al. · Figshare · 2020
<p>(A) Schematic for the scRNA-Seq analysis pipeline. (B) Violin plots for nGene, nUMI, and percent.mito for preprocessed data set (14,825 cells) and the final data set (7,053 cells). (C) Feature counts were log-normalized and scaled. Pre- and post-normalization plots are shown for total nUMIs and sum of gene expression counts. (D) Elbow Plot to show the ranking of the PCs based on the percentage of variance explained by each; 100 PCs have been computed for the final data set, 29 were selected for clustering, and 75 were selected for visualizing on the final UMAP shown in <a href="http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.3000538#pbio.3000538.g001" target="_blank">Fig 1B</a>. Raw sequence files available from SRA repository (SRX7814226). Processed files using Cell Ranger, Seurat, and Monocle are available through GEO database (GSE146040). nGene, number of genes; nUMI, number of unique molecular identifiers; percent.mito, percentage of mitochondrial gene expression; PC, principal component; scRNA-Seq, single-cell RNA sequencing; UMAP, Uniform Manifold Approximation Projection.</p> <p>(TIF)</p>
Cited by 0Open access
van Leeuwen, Paul · Socio-Environmental Systems Modeling · 2020
Cited by 0Open access
Tomas Pompa · Zenodo (CERN European Organization for Nuclear Research) · 2026
Complete R code for data preprocessing.
Cited by 0Open access
Hongjun Lu, Sam Yuan, Sung Ying Lu · 1996
Classification is probably the most well studied data mining problem. Most recent work in database community focuses on searching efficient classification algorithms. This paper addresses the importance of preprocessing data before applying classification algorithms. In particular, the technique of feature selection that eliminates attributes contributing little to classification is discussed. A feature selection algorithm based on conflict analysis is described, and the related implementation issues are analyzed. Results of experiments conducted to highlight the effectiveness of feature selection and related issues are also presented. 1 Introduction Classification, one of data mining problems [AgIS93], is the process of finding the common properties among different entities and classifying those entities into classes. It has been widely studied by researchers in the AI field [WeKu91]. Recently, researchers in the database community re-examined the problem in the context of large data...
Cited by 0
Lydia R Lucchesi · 2023
Data practitioners regularly use the 'R' and 'Python' programming languages to prepare data for analyses. Thus, they encode important data preprocessing decisions in 'R' and 'Python' code. The 'smallsets' package subsequently decodes these decisions into a Smallset Timeline, a static, compact visualisation of data preprocessing decisions (Lucchesi et al. (2022) <<a href="https://doi.org/10.1145%2F3531146.3533175" target="_top">doi:10.1145/3531146.3533175</a>>). The visualisation consists of small data snapshots of different preprocessing steps. The 'smallsets' package builds this visualisation from a user's dataset and preprocessing code located in an 'R', 'R Markdown', 'Python', or 'Jupyter Notebook' file. Users simply add structured comments with snapshot instructions to the preprocessing code. One optional feature in 'smallsets' requires installation of the 'Gurobi' optimisation software and 'gurobi' 'R' package, available from <<a href="https://www.gurobi.com" target="_top">https://www.gurobi.com</a>>. More information regarding the optional feature and 'gurobi' installation can be found in the 'smallsets' vignette.
Cited by 0Open access
Sotiras, Aristeidis · Figshare · 2017
Talk given during the "Harmonise This! Analyzing Diverse Neuroimaging Datasets" workshop at the 2015 Organization for Human Brain Mapping (OHBM) conference in Hawaii, 14-18 June.
Cited by 0Open access
Rachel Cummings, David Durfee · Society for Industrial and Applied Mathematics eBooks · 2019
The sensitivity metric in differential privacy, which is informally defined as the largest marginal change in output between neighboring databases, is of substantial significance in determining the accuracy of private data analyses. Techniques for improving accuracy when the average sensitivity is much smaller than the worst-case sensitivity have been developed within the differential privacy literature, including tools such as smooth sensitivity, Sample-and-Aggregate, Propose-Test-Release, and Lipschitz extensions. In this work, we provide a new and general Sensitivity-Preprocessing framework for reducing sensitivity, where efficient application gives state-of-the-art accuracy for privately outputting the important statistical metrics median and mean when no underlying assumptions are made about the database. In particular, our framework compares favorably to smooth sensitivity for privately outputting median, in terms of both running time and accuracy. Furthermore, because our framework is a preprocessing step, it can also be complementary to smooth sensitivity and any other private mechanism, where applying both can achieve further gains in accuracy. We additionally introduce a new notion of individual sensitivity and show that it is an important metric in the variant definition of personalized differential privacy. We show that our algorithm can extend to this context and serve as a useful tool for this variant definition and its applications in markets for privacy. Given the effectiveness of our framework in these important statistical metrics, we further investigate its properties and show that: (1) Our construction is conducive to efficient implementation with strong accuracy guarantees, evidenced by an O(n) implementation for median (with presorted data), and O(n2) implementation for more complicated functions such as mean, α-trimmed mean, and variance. (2) Our construction is both NP-hard and also optimal in the general setting (3) Our construction can be extended to higher dimensions, although it incurs accuracy loss that is linear in the dimension.
Cited by 3Open access
Li Li, Qingyun Yu, Kuo-Yi Lin et al. · Advanced and intelligent manufacturing in China · 2023
Cited by 0
Li-Ching Wu (362121), Hsin-Hao Chen (362122), Jorng-Tzong Horng (1749) et al. · Figshare · 2015
<p>The data after using Hilbert Huang transformation without modification. The high frequency noises are removed.</p>
Cited by 0Open access
Ting Qin (1931218), Pingqiang Wei (20805442), Yuanyuan Xie (662418) · Figshare · 2025
<div> Background A superior family background often provides children with more educational resources and a better learning environment, while poor or educationally deficient families may put children at a disadvantage on the starting line of education. This difference is not only related to the fate of individuals, but also to the fairness and justice of the whole society. Therefore, this study explores the influence mechanism of family background on children’s education level and the moderating effect of socioeconomic status. Methods Based on the data of China General Social Survey in 2021, a total of 4592 samples were selected. Principal component analysis model, hierarchical regression model and hierarchical linear model were used to verify the relationship between family background, socioeconomic status and children’s education level. Results (1) Family background has a significant positive impact on children’s education level (<i>β</i>=1.266; <i>p</i> < 0.001). (2) Socioeconomic status positively regulates the influence of family background on children’s education level (<i>β</i>=0.274; <i>p</i> < 0.001). Specifically, even if the parents’ education level is low, families with high socioeconomic status can make up for their disadvantages through economic investment, or obtain scarce educational resources through social relations. On the contrary, even if parents are highly educated, they are limited by economic constraints or lack of social capital, and it is difficult to translate family cultural advantages into actual educational investment (such as inability to bear the cost of extracurricular training or inability to obtain admission to prestigious schools), which limits the improvement of children ‘s education level. (3) Regional and urban-rural differences significantly affect the allocation of educational resources. Conclusion Family background affects children’s education level and is regulated by socioeconomic status. At the same time, regional and urban-rural differences affect the distribution of educational resources. We call for attention to the construction of family background and encourage families to create a good educational environment for their children. Increase investment in education for families and regions with lower socioeconomic status, and promote the balanced distribution of educational resources in different regions and between urban and rural areas to promote educational equity. </div>
Cited by 0Open access
strempfel, teja · Figshare · 2025
The code and data from fNIRS and EEG. Preprocessing was conducted in MATLAB, using EEGLAB and FieldTrip.
Cited by 0Open access
Shuying Sun, Yi‐Wen Huang, Pearlly S. Yan et al. · BioData Mining · 2011
Cited by 16Open access
Koch, Marcel · DSpace VŠB-TUO (VŠB-TUO) · 2004
Cited by 0
Aditya Pratap Singh, Navneet Kaur · 2022
<p> This study begins with an overview of data preprocessing, focusing on real-world data challenges. Before any data analysis method begins, these are the first problems that have got to be understood and resolved. In this work, the author discusses data preprocessing, like standardization and normalization including feature scaling to more readily accomplish the data classification. Finding the most informative collection of features is the goal of preprocessing to boost the classifier's performance. Include standardization is for the most part expected to take out the impact of a few quantitative highlights estimated on various scales. Besides, feature scaling is used to normalize all different numeric numbers to properly scaled numbers. The point of this part is to help analysts in picking a fitting preprocessing procedure for information investigation. The basic preprocessing methods used for the characterization of information are then addressed in this section. Fitting Python features to various information applications will be shown as concrete examples at the end of each session. </p>
Cited by 0Open access