Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “data preprocessing”

DATA WRANGLING AND PREPROCESSING FOR DATA SCIENCE

Anushka Raj Yadav, Partha Chanda, Mohammad Abrar et al. · 2025

Data preprocessing forms the critical foundation of effective data science workflows, transforming raw, unstructured data into reliable inputs for analysis and modeling. This chapter emphasizes the pivotal role of preprocessing in addressing pervasive data quality challenges such as missing values, outliers, and inconsistent formatting, which collectively impact over 80% of real-world datasets [1]. Key techniques include robust missing value imputation strategies (mean, median, and advanced methods like MICE), outlier detection using inter quartile ranges (IQR) and clustering algorithms, and feature engineering to derive meaningful predictors. Practical implementation is demonstrated through industry-standard tools: Python’s Pandas for automated data cleaning, R’s dplyr for structured transformations, and Open Refine for non-programmatic data wrangling. These tools enable reproducible preprocessing pipelines that maintain data integrity while optimizing datasets for machine learning applications. The chapter also highlights how systematic preprocessing reduces computational overhead by 30- 50% and improves model accuracy by addressing biases inherent in raw data [2]. By integrating theoretical principles with handson examples, this section equips practitioners to handle heterogeneous data sources, ensure compatibility across analytical platforms, and build trustworthy data products.

Cited by 0Open access

Data Preprocessing Strategies in Imbalanced Data Classification

Radovan Haluska · Digital Repository (National Repository of Grey Literature) · 2022

Learning from imbalanced data has been a research topic studied for many years. There are two main approaches used today - data-level and algorithm- level methods. We set out to study resampling methods which belong to the category of data-level methods. These methods modify the training part of a dataset as opposed to algorithm-level methods, which modify a classifier itself. Resampling methods are further divided into oversampling and un- dersampling methods. It is challenging to know which group of methods performs better and which algorithms stand out the most. We conducted an experiment of unseen scale. We systematically and robustly compared sixteen preprocessing methods over eighteen imbalanced datasets and sum- marised the results in this thesis. The results show that oversampling meth- ods outperformed most undersampling methods in both performance and preprocessing time. 1

Cited by 0

Data Preprocessing Technology Used in Data Mining

Yue Liu · 2003

This paper expounds the preprocessing technology for data integration and its necessity, and discusses on the methods of cleaning up the incomplete data, noise data and inconsistent data.

Cited by 0

New data preprocessing trends based on ensemble of multiple preprocessing techniques

Puneet Mishra, Alessandra Biancolillo, Jean‐Michel Roger et al. · TrAC Trends in Analytical Chemistry · 2020

Data generated by analytical instruments, such as spectrometers, may contain unwanted variation due to measurement mode, sample state and other external physical, chemical and environmental factors. Preprocessing is required so that the property of interest can be predicted correctly. Different correction methods may remove specific types of artefacts while still leaving some effects behind. Using multiple preprocessing in a complementary way can remove the artefacts that would be left behind by using only one technique. This article summarizes the recent developments in new data preprocessing strategies and specifically reviews the emerging ensemble approaches to preprocessing fusion in chemometrics. A demonstration case is also presented. In summary, ensemble preprocessing allows the selection of several techniques and their combinations that, in a complementary way, lead to improved models. Ensemble approaches are not limited to spectral data but can be used in all cases where preprocessing is needed and identification of a single best option is not easily done.

Cited by 452Open access

Pivot Based Data Preprocessing System for Data Optimization

Ravichandra H, Amutha S · Zenodo (CERN European Organization for Nuclear Research) · 2019

<em>Due to the fastest growth in the data size from various Internet sources DATA extraction, transformation and processing of big data becomes very much important in academics as well as industry. Big data processing platforms like Hadoop adopt MapReduce programming model to perform data processing of applications. Functionality of Hadoop first collects data and stores them in different data node with replication policy, later computation of these data is done using the map reduce operations. The process of collection and analysis of data requires more number of IO operations hence computing resources are not utilized in an efficient way. This paper proposes a preprocessing system based on Hadoop Platform on the idea of preprocessing system to reduce the IO operation on the redundant data collected and is stored in the data nodes leading to the reduced data size on the data nodes and IO operations and thus processing of these data takes less time. Experiments are conducted on the dataset generated which contains the redundant integer data.</em>

Cited by 0Open access

Preprocessing Electrophysiological Data: EEG, iEEG, and MEG Data

Kristin K. Sellers, Joline M. Fan, Leighton B. Hinkley et al. · 2024

The measurement and understanding of brain activity time series is integral to epilepsy care and research. Brain signals may be acquired using external sensors, as with scalp electroencephalography (EEG) and magnetoencephalography (MEG), or using implanted internal sensors, as with intracranial EEG (iEEG). These techniques can provide high temporal resolution recordings of electrical or electromagnetic activity relevant to epilepsy, cognitive processing, and many other functions, but require data processing and cleaning ahead of analysis. We first describe how EEG and iEEG are acquired, discuss similarities and differences, and elaborate on fundamental data preprocessing steps. We next provide example MATLAB functions for the most common preprocessing steps. Subsequently, we provide an overview of why MEG recordings may be beneficial, discuss acquisition considerations, and review preprocessing steps. Finally, we comment on popular software toolboxes.

Cited by 0

A SURVEY TO DATA PREPROCESSING IN DATA MINING

Xu Yu · 2003

Data mining has been one of the heated fields in database and decision support system in recent years. The main role of data preprocessing is discussed and the components and methods of data preprocessing are introduced.

Cited by 2

Data Preprocessing Technology in Chemical Process Data Mining

Feifei Yang · SHILAP Revista de lepidopterología · 2017

This paper studies the data preprocessing technology in chemical process data mining. It mainly studies the real-time chemical process data in-depth from the perspective of software development and explores various kinds of real-time data preprocessing methods. The enterprise pre-processes real-time data based on the MES data mining system and other systems associated with real-time data to develop a real-time data preprocessing system based on the theory. In addition to acquisition and storage of real-time data, this system corrects false data and fills missing data to provide accurate and credible data for data mining tools. Finally, the real-time data preprocessing system integration is applied to data mining systems and other information integration systems associated with real-time data.

Cited by 0Open access

Imbalanced data preprocessing techniques utilizing local data characteristics

Michał Koziarski · arXiv (Cornell University) · 2021

Data imbalance, that is the disproportion between the number of training observations coming from different classes, remains one of the most significant challenges affecting contemporary machine learning. The negative impact of data imbalance on traditional classification algorithms can be reduced by the data preprocessing techniques, methods that manipulate the training data to artificially reduce the degree of imbalance. However, the existing data preprocessing techniques, in particular SMOTE and its derivatives, which constitute the most prevalent paradigm of imbalanced data preprocessing, tend to be susceptible to various data difficulty factors. This is in part due to the fact that the original SMOTE algorithm does not utilize the information about majority class observations. The focus of this thesis is development of novel data resampling strategies natively utilizing the information about the distribution of both minority and majority class. The thesis summarizes the content of 12 research papers focused on the proposed binary data resampling strategies, their translation to the multi-class setting, and the practical application to the problem of histopathological data classification.

Cited by 0Open access

Data preprocessing for autonomous navigation systems

Igor Kopetschke, Petr Kretschmer, Josef Novák · 2017

This paper focuses on issues of device navigation in a specific environment with the possibility of environment recognition, in which the device is currently located. The purpose of the paper is presentation of the system which could be a solution for autonomous navigation system of discussed types of devices for ground and aerial traffic. We are aware of the fact that similar products are available on the market. The aim of our work, which contains below described data preprocessing, is creation of such a distributed application that will be able to process publically available cartographical data and will enable orientation of the device in real 3D environment in different modes of operation.

Cited by 2

Preprocessing Techniques of Solar Irradiation Data

Madderla Chiranjeevi, Skandha Karlamangal, Tukaram Moger et al. · 2023

Solar energy being abundant, non-exhaustive, environmentally friendly attracts the people attention towards the alternate renewable energy. High-quality time series data is essential for producing an accurate estimate of solar power generation. In most cases, the plethora of information hidden in time series data cannot be accessed. Common issues with time series include outliers, noise, missing data, and a lack of order in the timestamps itself that impair forecasting accuracy. So, preprocessing of the input data is a mandate in order to achieve a precise and dependable forecast. This study proposes various pre-processing techniques to improve the performance of the forecasting accuracy. The different ways to handle the missing values and outliers detection by sliding window method and box plots are presented in this study. The solar irradiation data collected from solar center Alice Springs, Australia used for validation of the preprocessing results. The efficacy of the proposed method in detecting the missing values and outliers is effective from the obtained results.

Cited by 2

Preprocessing of Single Cell Sequencing Data

Jiangli Zhao (2308330), Qingyuan Sun (188592), Xuchen Liu (13258842) et al. · Figshare · 2025

(A) Effect and quality of sequencing shown by vlnplots. (B) Correlation between nCount RNA and percent mt or nFeature RNA. (C) PCA plot and vlnplot of embeddings value before harmonization. (D) PCA plot and vlnplot of embeddings value after harmonization. (E) Brain cells used were divided into 36 clusters shown by tSNE plot. (F) Expression of conventional markers in each type of cells. (PNG)

Cited by 0Open access