Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “data preprocessing”

Research on Data Preprocessing in Supermarket Customers Data Mining

Jianping Wei · 2010

This paper focuses on the research of data preprocessing in data mining. Experiments are carried out on the transaction data of customs in some medium-sized supermarket. During the process of data analyzing and processing, correlation analysis method is used in order to identify which attributes could be integrated. In this process, data scale is reduced enormously and data with relatively small absolute value increases in the integration. Consequently, optimization of analysis result is reached and analysis difficulty has been reduced greatly.

Cited by 12

Data, data collection, and preprocessing for solar photovoltaics forecasting

Lennard Visser, Wilfried van Sark · 2025

As a result of the increased worldwide deployment of photovoltaic (PV) solar power, accurate power forecasting is becoming increasingly important to guarantee a reliable electricity grid. The key variable in PV power forecasting is solar irradiance, with ambient temperature as an important second variable. Hence, accurate weather forecasting at appropriate spatiotemporal timescales is necessary. Many forecasting methods are developed as evident in this book, and their accuracies are critically dependent on the input data used for developing and operating these methods. Therefore, methods for retrieving, cleaning, and utilizing data are essential to provide reliable datasets for the development and operation of forecasting methods. This chapter describes various pre- and postprocessing steps that can be applied to increase the value of the input data.

Cited by 0

A data envelopment analysis-based approach for data preprocessing

Parag C. Pendharkar · IEEE Transactions on Knowledge and Data Engineering · 2005

In this paper, we show how the data envelopment analysis (DEA) model might be useful to screen training data so a subset of examples that satisfy monotonicity property can be identified. Using real-world health care and software engineering data, managerial monotonicity assumption, and artificial neural network (ANN) as a forecasting model, we illustrate that DEA-based data screening of training data improves forecasting accuracy of an ANN.

Cited by 49

Preprocessing of unstructured medical data: the impact of each preprocessing stage on classification

Mariya Kashina, Iuliia Lenivtceva, Georgy Kopanitsa · Procedia Computer Science · 2020

Nowadays, it is still important to develop methods for processing data, in particular medical texts, in Russian. In this paper, we checked how each stage of text pre-processing affects the result of the classifier. The paper analyzed 269923 records of allergic anamnesis of patients, 11670 of which were placed for further processing. We consider the main stages of pre-processing: tokenization, deletion of stop words, error correction, document cropping, normalization, class harmonization, and vectorization. To vectorize the data, we have selected the Bag-of-Words. The method of logistic regression was chosen for classification, since it has easy reproducibility and interpretation. Precision, recall and F-measure were selected as evaluation metrics. The results (F = 88.12%) showed that the most effective was the stage of normalization and error correction.

Cited by 25Open access

Imbalanced data preprocessing techniques utilizing local data\n characteristics

Michał Koziarski · arXiv (Cornell University) · 2021

Data imbalance, that is the disproportion between the number of training\nobservations coming from different classes, remains one of the most significant\nchallenges affecting contemporary machine learning. The negative impact of data\nimbalance on traditional classification algorithms can be reduced by the data\npreprocessing techniques, methods that manipulate the training data to\nartificially reduce the degree of imbalance. However, the existing data\npreprocessing techniques, in particular SMOTE and its derivatives, which\nconstitute the most prevalent paradigm of imbalanced data preprocessing, tend\nto be susceptible to various data difficulty factors. This is in part due to\nthe fact that the original SMOTE algorithm does not utilize the information\nabout majority class observations. The focus of this thesis is development of\nnovel data resampling strategies natively utilizing the information about the\ndistribution of both minority and majority class. The thesis summarizes the\ncontent of 12 research papers focused on the proposed binary data resampling\nstrategies, their translation to the multi-class setting, and the practical\napplication to the problem of histopathological data classification.\n

Cited by 1Open access

An Entropy-based Data Reduction Method for Data Preprocessing

Rocco Cassandro, Quing Li, Zhaojun Li · 2023

The primary task in data mining is to find potential patterns or to discover hidden and useful knowledge from given data sets. However, with the increasing data quantity and exploding complexity, the capabilities of dealing with massive data becomes very crucial. The data preprocessing module is an integral part of data mining procedure, which aims to optimize input data usability for subsequent tasks such as classification, clustering, association analysis as well as other data mining algorithms. In general, data preprocessing procedures can effectively reduce the computational complexity while as possible to ensure accuracy and efficiency of prediction or classification, but meanwhile it even can assist to extract some unknown knowledge before applying more advanced data mining algorithms. This research proposes a three-patterns feature variables technique and an entropy-based data reduction (EBDR) algorithm for data preprocessing based on information theory. The goal is to explore high-purity subsets in which the values of an attribute are directly linked to specific class labels. The results of experiments demonstrate the efficacy of EBDR algorithm on datasets of varying sizes.

Cited by 1

Data Preprocessing: A preliminary step for web data mining

Huma Jamshed, M. Sadiq Ali Khan, Muhammad Khurram et al. · 3C Tecnología_Glosas de innovación aplicadas a la pyme · 2019

In recent years immense growth of data i.e. big data is observed resulting in a brighter and more optimized future. Big Data demands large computational infrastructure with high-performance processing capabilities. Preparing big data for mining and analysis is a challenging task and requires data to be preprocessed to improve the quality of raw data. The data instance representation and quality are foremost. Data preprocessing is preliminary data mining practice in which raw data is transformed into a format suitable for another processing procedure. Data preprocessing improves the data quality by cleaning, normalizing, transforming and extracting relevant feature from raw data. Data preprocessing significantly improve the performance of machine learning algorithms which in turn leads to accurate data mining. Knowledge discovery from noisy, irrelevant and redundant data is a difficult task therefore precise identification of extreme values and outlier, filling up missing values poses challenges. This paper discusses various big data pre-processing techniques in order to prepare it for mining and analysis tasks.

Cited by 28Open access

The impact of data preprocessing on data mining outcomes

Bhargavi Konda · World Journal of Advanced Research and Reviews · 2022

Data preprocessing is a vital initial step during knowledge discovery because it determines the success of data mining projects. A dataset's quality and representation stand as the primary element because any presence of redundant, irrelevant, too noisy, or unreliable information will severely disrupt the knowledge discovery process. The preprocessing phase first converts unstructured data into an analytical format alongside solutions for data inconsistencies, errors, and missing values to maintain data mining result integrity. The preprocessing corrects data quality problems and arranges data properly, improving data mining model accuracy, efficiency, and interpretability. The data mining pipeline requires data preprocessing as its essential foundation since it provides multiple techniques to convert raw data into an effective analytical format. Data mining depends heavily on preprocessing operations because they guarantee proper analysis results through accurate correction of errors and optimal data structure development and absent data point management.

Cited by 4Open access

Data Expedition: Travel Through Data Preprocessing, EDA And PCA

Kaushik Paul · 2024

Pre-processing is essential in order to improve the quality of the data and make it more suitable for specific tasks like data mining. It describes the steps taken to prepare data for analysis, such as cleaning, converting, and integrating it. This chapter focuses on the comprehensive analysis of the data collection process through web scraping techniques, preprocessed using some Python methods, and finally analyzed with the help of exploratory data analysis (EDA). Initially, different data collection methods are outlined, followed by preprocessing steps including statistical information, which determines the overall structure of the dataset considered. To build a machine learning (ML) model, some data pre-processing schemes are considered, such as handling missing or null values (N-V), outlier detection, and removing duplicates. Exploratory Data Analysis (EDA) is conducted at various levels, including univariate, bivariate, and multivariate analysis, to understand the relationships within the dataset. A dataset may contain a large number of feature variables, which can be merged into a smaller number of variables using principal component analysis (PCA). PCA reduces the complexity of the model that will be built using ML algorithms such as logistic regression, linear regression, etc. This chapter provides insights into the entire process of data analysis, from data collection to model evaluation, demonstrating the effectiveness of web scraping in extracting valuable information for predictive modeling. Subsequently, both logistic regression and linear regression models are constructed to predict target variables. Feature selection techniques are employed to identify the most influential variables, and principal component analysis (PCA) is utilized for dimensionality reduction. Finally, model performance is evaluated using confusion matrices for the logistic regression model and root-mean-squarederror for the linear regression model. In this work,the Python language is considered, which is an object-oriented, interpreted, and interactive programming language. It is open source with rich sets of libraries like Pandas, Numpy, Matplotlib, Seaborn, etc. For executing the Python code, JUPYTER NOTEBOOK is used, which provides a web-based application process and a rich media representation of the object.

Cited by 0Open access

Preprocessing fMRI data under correct Rice conditions

L. Lauwers, Kurt Barbé, Wendy Van Moer · 2013

Functional Magnetic Resonance Imaging (fMRI) data consist of relatively weak signals with a complicated noise structure. To reduce the effects of noise arising from both instrumental and physiological sources, a series of standard preprocessing steps is performed. Nevertheless, fMRI signals will show an undesired offset due to the measurement setup. Prior to fMRI data analysis, this offset component needs to be removed in an additional preprocessing step. Classically, one assumes the data to be Gaussian distributed which eases this preprocessing step. However, this assumption is only valid for high signal-to-noise ratios (SNRs). For low SNRs, it is known that fMRI data follow a Rice distribution. Hence, to perform a proper data preprocessing, we need to take into account the correct characteristics of the Rice distributed data.

Cited by 0

Sampling data preprocessing subsystem for integrated protection

Chunyan Wu, Jinghan He, Zhiqian Bo et al. · 2011

This paper designs a novel sampling data preprocessing subsystem in integrated protection based on IEC61850. This subsystem will be used to solve the problems of data redundancy processing and bad value processing. A framework structure of integrated protection software system based on the concept of module distributed is firstly presented, and then the details of preprocessing subsystem are given. This subsystem is the most important foundation of integrated protection. It can provide a platform to analyze and validate the protection arithmetic. Meanwhile, the subsystem will reach the requirement for security, real-time and reliability.

Cited by 0

Multi-modal neuroimaging pipelines for data preprocessing

Mohammadi-Nejad, Ali-Reza, Pszczolkowski, Stefan, Auer, Dorothee et al. · Zenodo (CERN European Organization for Nuclear Research) · 2023

A new option (<code>--dki</code>) is added to the diffusion pipeline to produce microstructural maps using the diffusion kurtosis model. A new option (<code>--wmti</code>) is added to the diffusion pipeline to generate microstructural maps using the white matter tract integrity model. A new option (<code>-mppca</code>) is added to the diffusion pipeline, performing denoising using Marcenko-Pastur PCA. A new option (<code>--use_topup</code>) is added to the diffusion pipeline, ignoring the TOPUP step and using the previously provided files for the TOPUP step.

Cited by 1Open access

PREPROCESSING OF EXPERIMENTAL DATA IN KORELIA SOFTWARE

Kaloyan Yankov · 2010

This work presents a system for primary processing and analysis of experimental data acquired from measuring devices. The main parameters of time series are defined. Data operations to extract parameters from the sequences, produce a similar sequence with particular enhanced or eliminated, encode or compress the sequence, etc are described. Manipulations of data are divided into three classes: manipulations of the independent variable, amplitude transformations and complex operations. Preprocessed time series are recorder in a few most popular file formats.

Cited by 14