Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “data preprocessing”

Automated Pipeline for Data Preprocessing

Schimmel, David, Yazdi, M. Amin · Zenodo (CERN European Organization for Nuclear Research) · 2019

Data preprocessing describes steps that are often necessary when applying algorithms with the purpose of extracting information from them. Good data quality is a prerequisite for good results. This thesis presents an overview over typical problems regarding data quality and challenges from combining data from multiple sources. Typical approaches to these challenges are specifed. From these insights, requirements for a united tool enabling reliable, repeatable and extendable data preprocessing pipelines are formulated. Subsequently, a prototypical application attempting to cope for these challenges is being introduced. Finally, a real world scenario and its coverage by the application are being described.

Cited by 0Open access

Synchronized Preprocessing of Sensor Data

Amal Tawakuli, Daniel Kaiser, Thomas Engel · 2020

Sensor data whether collected for machine learning, deep learning or other applications must be preprocessed to fit input requirements or improve performance and accuracy. Data preparation is an expensive, resource consuming and complex phase often performed centrally on raw data for a specific application. The dataflow between the edge and the cloud can be enhanced in terms of efficiency, reliability and lineage by preprocessing the datasets closer to their data sources. We propose a dedicated data preprocessing framework that distributes preprocessing tasks between a cloud stage and two edge stages to create a dataflow with progressively improving quality. The framework handles heterogenous data and dynamic preprocessing plans simultaneously targeting diverse applications and use cases from different domains. Each stage autonomously executes sensor specific preprocessing plans in parallel while synchronizing the progressive execution and dynamic updates of the preprocessing plans with the other stages. Our approach minimizes the workload on central infrastructures and reduces the resources used for transferring raw data from the edge. We also demonstrate that preprocessing data can be sensor specific rather than application specific and thus can be performed prior to knowing a specific application.

Cited by 10

Flow of the data preprocessing.

Shuntaro Yada (12501633), Tomohiro Nishiyama (19563061), Shoko Wakamiya (4286599) et al. · Figshare · 2024

<p>Based on event IDs, we aligned patients and events between the two sources, TABLE and TEXT.</p>

Cited by 0Open access

Data acquisition and preprocessing process.

Liang Chen (73736), You Wen (20646648), Huaquan Su (22656301) et al. · Figshare · 2025

<div> Partial discharge (PD) detection in power transformers is critical for preventing insulation failures in modern power grids, yet remains challenging due to signal complexity and environmental noise. Existing methods struggle with accurate PD classification under strong electromagnetic interference and varying load conditions. This study proposes a novel hybrid Acoustic-VMD and CNN-LSTM model featuring: (1) sample entropy-optimized variational mode decomposition (automatically determining modes and penalty factor), (2) parallel 1D-CNN (5 layers and bidirectional LSTM (2 layers, 256 units) branches, and (3) hierarchical attention mechanisms (8 heads) for dynamic feature fusion. Experimental results demonstrate superior performance with 96.2% classification accuracy for multi-source defects (38% improvement over wavelet methods), 5.8mm mean absolute localization error (53% better than TDOA), and consistent 4.2<sup>°</sup> angular accuracy under high noise, while maintaining practical 0.8s processing time. The research conclusively establishes that synergistic integration of adaptive signal processing and attention-based deep learning significantly advances PD diagnostics, achieving both computational efficiency and robust performance in complex operational environments. </div>

Cited by 0Open access

The result of data preprocessing.

Hongna Dai (21533498), Jingwen Wang (219000), Haixia Feng (8131044) et al. · Figshare · 2025

<div> To enhance the monitoring accuracy of agglomerate fog on expressways, this paper takes the frequently occurring agglomerate fog data on Shandong’s expressways as an example. Based on the analysis of the spatiotemporal distribution characteristics of agglomerate fog, from the spatial perspective, it employs Geographic Weighted Regression (GWR) and Multi-scale Geographic Weighted Regression (MGWR) models to analyze the influence and scale of factors including Digital Elevation Model (DEM), DEM difference, water system density, Normalized Difference Vegetation Index (NDVI), Land Surface Temperature (LST) difference, and precipitation on agglomerate fog. The main research conclusions are as follows: agglomerate fog frequently occurred in the early morning during autumn and winter when the temperature difference is large. Three concentration centers of agglomerate fog-prone road segments were identified along Shandong’s expressways, located near Jiaozhou Bay, within intermountain basins of the central region, and across the northern plain of Mount Tai (where the Yellow River traverses the concentration center). The impacts of various influencing factors on agglomerate fog are ranked as follows: DEM > DEM difference > LST difference > water system density > NDVI > precipitation, among which DEM difference and LST difference mainly promote fog formation, whereas other factors generally exhibit inhibitory effect. The influence range (adaptive scale) of precipitation is the largest, at 673 meters, followed by the water system with an influence range of 599 meters, and NDVI shows the smallest influence range at only 44 meters. It holds significant importance for reducing the accident rate on expressways. </div>

Cited by 0Open access

Data preparation and preprocessing diagram.

Colin D. Middleton (17118077), Kim Boynton (17118080), David Lewis (245511) et al. · Figshare · 2023

&lt;p&gt;Initial data was provided by Avista Utilities, the City of Spokane Utilities department (CoS Utilities), and the City of Spokane Community Management Information System that tracks homelessness (CoS CMIS). The authors received the data as two file groups, Monthly Billing Records and Account-People Records, after de-identification and initial matching was performed by the Avista data team.&lt;/p&gt;

Cited by 0Open access

Preprocessing and data analysis scripts

Graf-Vlachy, Lorenz, Wagner, Stefan, Graziotin, Daniel · Figshare · 2022

Preprocessing and data analysis scripts for: Text and Team: What Article Metadata Characteristics Drive Citations in Software Engineering? L. Graf-Vlachy, D. Graziotin, S. Wagner 25th International Conference on Evaluation and Assessment in Software Engineering (EASE) https://doi.org/10.1145/3530019.3530022

Cited by 0Open access

Data Preprocessing for Advanced Analytics

McKay, Andrew Robertson, Murphy, Patrick Dennis · Digital WPI · 2015

The goal of this project is to improve attribute selection in data preprocessing. This is done using two techniques, attribute combination and clustering. Combination generates new attributes by combining pairs of numeric attributes with arithmetic operations. Attribute clustering discovers groups of categorical attributes based on similarity via Minimum Description Length. The combinations generated frequently have increased correlation to the target attribute compared to those of the original attributes. The clusters let analysts select a subset of the attributes in the original dataset producing about the same classification accuracy as the full set of attributes while reducing the size of the dataset. Both techniques provide data analysts additional insight into a dataset.

Cited by 0Open access

Preprocessing of NMR metabolomics data

Leslie R. Euceda, Guro F. Giskeødegård, Tone F. Bathen · Scandinavian Journal of Clinical and Laboratory Investigation · 2015

Metabolomics involves the large scale analysis of metabolites and thus, provides information regarding cellular processes in a biological sample. Independently of the analytical technique used, a vast amount of data is always acquired when carrying out metabolomics studies; this results in complex datasets with large amounts of variables. This type of data requires multivariate statistical analysis for its proper biological interpretation. Prior to multivariate analysis, preprocessing of the data must be carried out to remove unwanted variation such as instrumental or experimental artifacts. This review aims to outline the steps in the preprocessing of NMR metabolomics data and describe some of the methods to perform these. Since using different preprocessing methods may produce different results, it is important that an appropriate pipeline exists for the selection of the optimal combination of methods in the preprocessing workflow.

Cited by 45Open access

Preprocessing of Airborne Pyranometer Data

Lutz Bannehr, Vince Glover · UCAR/NCAR · 1991

Eppley pyranometers are employed to measure the up- and downwelling short wavelength radiation fluxes in the atmosphere. The quality of the data used to investigate the radiation budget and extinction of solar irradiance depends strongly on the calibration and correction applied to the radiation data, accounting for (1) cosine response of the instruments, (2) ambient temperature effects, (3) leveling of the radiometers, and (4) aircraft maneuvers during the observations. This report makes suggestions about how the quality of the radiation data can be enhanced by a more accurate calibration procedure and correction scheme. By considering the recommendations, data from the upward facing pyranometer can be improved by about 20% under clear sky conditions.

Cited by 19Open access

Preprocessing Tools for Data Preparation

Osval A. Montesinos‐López, Abelardo Montesinos‐López, José Crossa · 2022

Abstract This data preparation chapter is of paramount importance for implementing statistical machine learning methods for genomic selection. We present the basic linear mixed model that gives rise to BLUE and BLUP and explain how to decide when to use fixed or random effects that give rise to best linear unbiased estimates (BLUE or BLUEs) and best linear unbiased predictors (BLUP or BLUPs). The R codes for fitting linear mixed model for the data are given in small examples. We emphasize tools for computing BLUEs and BLUPs for many linear combinations of interest in genomic-enabled prediction and plant breeding. We present tools for cleaning, imputing, and detecting minor and major allele frequency computation, marker recodification, frequency of heterogeneous, frequency of NAs, and three methods for computing the genomic relationship matrix. In addition, scaling and data compression of inputs are important in statistical machine learning. For a more extensive description of linear mixed models, see Chap. 10.1007/978-3-030-89010-0_5 .

Cited by 1Open access

Illustration of data preprocessing process.

Yusuke Suita (14381481), Hardy Bright Jr (22008830), Yuan Pu (3509042) et al. · Figshare · 2025

The epigenetic feature values per gene are extracted, formed into matrices, and standardized for model input. Each gene’s observed RNA-seq value is extracted and transformed creating the target variable for the model. (TIFF)

Cited by 0Open access