Search 250 million research papers

Data Preprocessing

Mehdi Ghayoumi · 2023

Cited by 0

Data Preprocessing

Magy Seif El‐Nasr, Truong Huy Nguyen Dinh, Alessandro Canossa et al. · 2021

Abstract This chapter focuses on the process of cleaning data and preparing it for further processing. Specifically, the chapter discusses various techniques that you will use, including preprocessing, outlier identification, data consistency, and the normalization or standardization process, used to normalize your data. The chapter further discusses different measurement types and what methods can be used for which types. The chapter also discusses ways to deal with issues you may encounter with inconsistent or dirty data. The chapter takes a more practical approach by integrating several labs with actual game data to demonstrate how you can perform these steps on real game data.

Cited by 2

Data preprocessing

Guillaume Coqueret, Tony Guida · 2023

Cited by 0

Data preprocessing.

Nikola Milojevic-Dupont (9749755), Nicolai Hans (9749758), Lynn H. Kaack (9749761) et al. · Figshare · 2020

<p>In this appendix, we summarize how the data preprocessing has been carried out. The whole workflow for preprocessing, feature engineering and machine learning is written in Python and executed on the high performance computing cluster of the Potsdam Institute for Climate Impact Research.</p> <p>(PDF)</p>

Cited by 0Open access

Data preprocessing.

Andriana Manousidaki (17583389), Anna Little (18693063), Yuying Xie (542735) · Figshare · 2024

<div><p>Recent advances in single-cell technologies have enabled high-resolution characterization of tissue and cancer compositions. Although numerous tools for dimension reduction and clustering are available for single-cell data analyses, these methods often fail to simultaneously preserve local cluster structure and global data geometry. To address these challenges, we developed a novel analyses framework, <u>S</u>ingle-<u>C</u>ell <u>P</u>ath <u>M</u>etrics <u>P</u>rofiling (scPMP), using power-weighted path metrics, which measure distances between cells in a data-driven way. Unlike Euclidean distance and other commonly used distance metrics, path metrics are density sensitive and respect the underlying data geometry. By combining path metrics with multidimensional scaling, a low dimensional embedding of the data is obtained which preserves both the global data geometry and cluster structure. We evaluate the method both for clustering quality and geometric fidelity, and it outperforms current scRNAseq clustering algorithms on a wide range of benchmarking data sets.</p></div>

Cited by 0Open access

Data preprocessing.

Lars Raufer (21078320), Jasper Wiedey (21078323), Malte Mueller (21078326) et al. · Figshare · 2025

<div> Cucumbers play a significant role as a greenhouse crop globally. In numerous countries, they are fundamental to dietary practices, contributing significantly to the nutritional patterns of various populations. Due to unfavorable environmental conditions, they are highly vulnerable to various diseases. Therefore the accurate detection of cucumber diseases is essential for maintaining crop quality and ensuring food security. Traditional methods, reliant on human inspection, are prone to errors, especially in the early stages of disease progression. Based on a VGG19 architecture, this paper uses an innovative transfer learning approach for detecting and classifying cucumber diseases, showing the applicability of artificial intelligence in this area. The model effectively distinguishes between healthy and diseased cucumber images, including <i>Anthracnose, Bacterial Wilt, Belly Rot, Downy Mildew, Fresh Cucumber, Fresh Leaf, Pythium Fruit Rot</i>, and <i>Gummy Stem Blight</i>. Using this novel approach, a balanced accuracy of 97.66% on unseen test data is achieved, compared to a balanced accuracy of 93.87% obtained with the conventional transfer learning approach, where fine-tuning is employed. This result sets a new benchmark within the dataset, highlighting the potential of deep learning techniques in agricultural disease detection. By enabling early disease diagnosis and informed agricultural management, this research contributes to enhancing crop productivity and sustainability. </div>

Cited by 0Open access

Data Preprocessing Toolkit : An Approach to Automate Data Preprocessing

Deepak Varma, Alwala Nehansh, P Swathy · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · 2023

-Data Pre-processing transforms the data into a format that is more easily and effectively processed in data mining, machine learning, and other data science tasks. The Data Pre-processing techniques are generally used at the earlier stages of the machine learning and AI development pipeline to ensure accurate results. Data must first go through several pre-processing steps before the machine learning model can use it. We intend to build a GUI that allows users to input the data with inconsistencies and various options are provided to pre- process. After the data is processed, we provide the users an option to analyze the data which gives them clarity on the further aspects of preprocessing. Pre-processing consists of three main phases i.e. Data Cleaning, Data Transformation, and Data Reduction. Data cleaning involves correcting or removing inaccurate, corrupted, improperly formatted, duplicate, or incomplete data from a dataset. Data transformation is a technique used to convert data into a suitable format that eases data mining and retrieves strategic information. Data reduction is the process of shrinking the size of the original data so that it can be represented in a much smaller volume. Preprocessing requires a lot of work, what if we could automate it? With just one click, this web application is capable of transforming inconsistent data into consistent data that could be used further. The GUI is built using the Streamlit library in python and in the backend, we intend to use the AutoClean library along with many other libraries based on different formats of data.. Key Words: Data Cleaning, AutoClean, Data Transformation, Data Reduction, Streamlit.

Cited by 11Open access

Data and Preprocessing

Latesh Malik, Sandhya Arora, Urmila Shrawankar · 2025

Cited by 0

Preprocessing Satellite Data

P. Krishna Rao, Susan J. Holmes, Ralph K. Anderson et al. · American Meteorological Society eBooks · 1990

Data received from a satellite require special handling before they can be used. Data from differennt instruments are sampled at different rates, multiplexed (combined) into a single bit-stream, and recorded or transmitted to a receiving station in a compact format. Such unprocessed "raw" data must be transformed into a more manageable and useful format with the addition of important ancillary data, such as calibration and Earth location parameters. The result is known as "conditioned" data. This transformation from raw to conditioned data, which is called "reprocessing," includes the following ordered steps: Data broadcast from the satellite are "ingested" or received into the preprocessing computer. Different types of data are separated into individual data sets according to the originating satellite and instrument. Instrument data are "calibrated" or described in forms that can be related to physical quantities. "Earth location" information is added so that each measurement can be directly related to a location on the Earth. The performance of the instrument annd datatransmission are regularly monitored. The final conditioned data are stored in a data base accessible by the product processing programs and in a format compatible for archiving.

Cited by 0

Mathematical Data Preprocessing

James Duckworth · Agronomy monograph/Agronomy · 2004

Samples that are measured using diffuse reflectance often exhibit significant differences in the spectra because of the nonhomogeneous distribution of the particles. Light scattering off the particles causes every spectrum, even of remeasurements of the same sample, to look a little bit different because of the particle size distribution and alignment with the incident beam of light. The correction method assumes that the wavelength dependency of the light scattering is different from that of the constituent absorption. The chapter presents the list of baseline correction methods. There are many ways of autocorrecting the spectrum baseline as a chemometric preprocessing step. Due to the multivariate nature of factor-based chemometric models, the direct relationship between the spectral response and the constituent concentration (univariate) is not very important. There are some mathematical enhancements that can be applied to data that are to be used in a multivariate model and would be useless for a univariate model.

Cited by 41

SumatraTT-Data Preprocessing.

Aubrecht, Petr, CVUT Praha, Praha (Czech Republic) · OpenGrey (Institut de l'Information Scientifique et Technique) · 2001

Cited by 0

Data preprocessing procedures.

Yangsong He (13909371), Zheng-Jian Bai (22574779), Wai-Ki Ching (4685404) et al. · Figshare · 2025

<div> <b>Motivation:</b> Epithelial-mesenchymal transition (EMT) plays a key role in cancer metastasis by promoting changes in adhesion and motility. RNA-binding proteins (RBPs) regulate alternative splicing (AS) during EMT, enabling a single gene to produce multiple protein isoforms that affect tumor progression. Disruption of RBP-AS interactions may disrupt the progress of diseases like cancer. Despite the importance of RBP-AS relationships in EMT, few computational methods predict these associations. Existing models struggle in sparse settings with limited known associations. To improve performance, we incorporate both sparsity constraints and heterogeneous biological data to infer RBP–AS associations. <b>Result:</b> We propose a new method based on <u>A</u>ccelerated <u>P</u>roximal <u>DC</u> <u>A</u>lgorithm (APDCA) for predicting RBP–AS associations. In particular, APDCA combines sparse low-rank matrix factorization with a Difference-of-Convex (DC) optimization framework and uses extrapolation to improve convergence. A key feature of APDCA is the use of a sparsity constraint, which filters out noise and highlights key associations. In addition, integrating multiple related data sources with direct or indirect relationships can help in reaching a more comprehensive view of RBPs and AS events and to reduce the impact of false positives associated with individual data sources. we prove that our proposed algorithm is convergent under some conditions and the experimental results have illustrated that APDCA outperforms six baseline methods in both AUC and AUPR. A case study on the RBP QKI shows that the top predictions are verified by the OncoSplicing database. Thus, APDCA provides a fast, interpretable, and scalable tool for discovering post-transcriptional regulatory interactions. </div>

Cited by 0Open access

Preprocessing data table.

Ling Sun (301114) · Figshare · 2021

Cited by 0Open access

Data preprocessing workflow.

Young Sin Ko (14265516), Yoo Mi Choi (14265519), Mujin Kim (14265522) et al. · Figshare · 2022

&lt;p&gt;The workflow for generating patch images during the data preprocessing step is shown. This workflow involves a series of processes for generating patch images based on the annotation data with WSIs loaded into Openslide’s library, which were processed differently according to class. Abbreviations: WSI (whole slide image).&lt;/p&gt;

Cited by 0Open access

Data preprocessing steps.

Jan Odstrcilik (5487032), Radim Kolar (3729133), Folkert Horn (10766259) et al. · Figshare · 2023

&lt;p&gt;a) an original mean image calculated from a video sequence after frame-to-frame registration, b) binary mask of segmented blood vessels, c) binary mask of ONH.&lt;/p&gt;

Cited by 0Open access

Preprocessing data chart.

Jia Xue (133314), Junxiang Chen (1724476), Chen Chen (6544) et al. · Figshare · 2020

Cited by 0Open access

Anatomical data preprocessing.

Jürgen Fuchshuber (10066525), Karl Koschutnig (160765), Andreas Fink (160741) et al. · Figshare · 2025

<div> Background Insecure adult attachment has previously been linked to more severe psychopathology and to alterations within neuronal connectivity on a structural as well as functional level. Little is known about the resting state functional connectivity (rs-FC) of the attachment system in patients suffering from poly-drug use disorder (PUD). Methods The present study investigated rs-FC at two measuring points (t1: ROI-to-ROI; t2: seed-to-voxel) in a sample of PUD patients (n = 33; Age: M = 30y; SD = 8y; Female = 15%). Adult attachment was measured with the German version of the Experiences in Close Relationships Scale (ECR-RD8). Furthermore, insecure attachment was correlated with depressive symptoms (ADS), trait anxiety (STAI) and general psychopathology (BSI-53). Results More insecure attachment was associated with increased trait anxiety, depressive and general psychiatric symptom burden in patients. Furthermore, we observed time-stable links between insecure adult attachment and increased rs-FC between the left lateral parietal default mode network (DMN LP) and bilateral parts of the salience network, as well as decreased rs-FC between DMN LP and medial parts of the DMN. Discussion Implications of the present study are highlighting the association between attachment security and brain areas related to affect regulation. </div>

Cited by 0Open access

LANDSAT data preprocessing

W. W. Austin · NASA STI Repository (National Aeronautics and Space Administration)

The effect on LANDSAT data of a Sun angle correction, an intersatellite LANDSAT-2 and LANDSAT-3 data range adjustment, and the atmospheric correction algorithm was evaluated. Fourteen 1978 crop year LACIE sites were used as the site data set. The preprocessing techniques were applied to multispectral scanner channel data and transformed data were plotted and used to analyze the effectiveness of the preprocessing techniques. Ratio transformations effectively reduce the need for preprocessing techniques to be applied directly to the data. Subtractive transformations are more sensitive to Sun angle and atmospheric corrections than ratios. Preprocessing techniques, other than those applied at the Goddard Space Flight Center, should only be applied as an option of the user. While performed on LANDSAT data the study results are also applicable to meteorological satellite data.

Cited by 0Open access

Data preprocessing steps.

Saima Farhan (22127216), Yasin Ul Haq (22127219), Momina Abdul Khaliq (22127222) et al. · Figshare · 2025

<div> Alzheimer’s Disease poses a significant challenge as a progressive and irreversible neurological condition striking the elderly population. Its incurable nature correlates with a significant rise in death rates. However, early detection can slow its progression and facilitate prompt intervention, thereby mitigating mortality risks. Functional Magnetic Resonance Imaging (fMRI) provides valuable insights into the functional changes within distinct brain regions associated with the disease. The recent research efforts have extracted functional connectivity measures for the classification. These handcrafted functional connectivity features are usually not robust and are computationally intensive. To address the issue, this study introduces an integrated deep-learning framework based on CNN and LSTM networks. This framework autonomously learns both intra-volume and inter-volume features critical for classification tasks. CNNs facilitate feature extraction, while LSTM networks govern the selection of significant features for classification. The key aim of this study is to classify Alzheimer’s disease and its prodromal stage, Mild Cognitive Impairment (MCI). MCI is further categorized as early MCI (EMCI) and late MCI (LMCI). We have evaluated the framework in three dimensions, binary classification, multi-class classification with 3-classes, and multi-class classification with 4-classes. For each dimension, multiple classifications were performed. The results depict the proposed CNN-LSTM framework to attain 99% accuracy and 100% average area under the curve for the majority of the classification. </div>

Cited by 0Open access

EEG data preprocessing

Francesco Grassi · OSF Preprints (OSF Preprints) · 2023

MATLAB code for preprocessing of the EEG data collected in learning and test phase

Cited by 0