Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “data preprocessing”

Data acquisition and preprocessing.

Dorian Arifi (14886050), Bernd Resch (3111675), Jan Kinne (10515040) et al. · Figshare · 2023

<div><p>This paper seeks to unveil how (geospatial) connection strategies associated with business innovation, differ between geolocated social media and hyperlink company networks. Thereby, we provide a first step towards understanding connection strategies of innovative companies on social media platforms. For this purpose, we build a hyperlink and Twitter follower network for 11,892 companies in the information technology (IT) sector and compare them along four dimensions. First, underlying network structures were assessed. Second, we asserted information flow patterns between companies with the help of centrality measures. Third, geographic and cognitive proximities of companies were compared. Fourth, the influence of company characteristics was examined through linear and logistic regression models. This comparison revealed, that on a general level the basic connection patterns of the hyperlink and Twitter network differ. Nevertheless, the geospatial dimension (geographic proximity) and the knowledge base of a company (cognitive proximity) appear to have a similar influence on the decision to connect with other companies on Twitter and via hyperlinks. Further, the results suggest that innovative companies most likely align their connection strategies across hyperlink and Twitter networks. Thus, business innovation might effect connection strategies across online company networks in a comparable manner.</p></div>

Cited by 0Open access

Data extraction and preprocessing.

Ron Mahabir (5946257), Arie Croitoru (2600983), Andrew Crooks (5946260) et al. · Figshare · 2018

<p>In the data extraction step, data on OSM, Wikipedia, Google News, and Google Trends is collected from their various online platforms. The multiscale analysis step involves a progressive refinement approach: Data is first examined at the monthly level and the strongest extremum points on the awareness curve are identified. These points are then assigned a score that is comprised of its magnitude and a weight that is proportional to the duration of consistent trend along the awareness curve prior to the detected extremum point. A search time window is then defined on the awareness peak with the highest score and used for working with data at the weekly analysis level. A similar approach is used to identify a search window for working with data at the next (finer) temporal granular analysis level.</p>

Cited by 0Open access

Transforming IoT Data Preprocessing

Amal Tawakuli, Daniel Kaiser, Thomas Engel · 2022

Data preprocessing is an integral part of Artificial Intelligence (AI) pipelines. It transforms raw data into input data that fulfill algorithmic criteria and improve prediction accuracy. As the adoption of Internet of Things (IoT) gains more momentum, the data volume generated from the edge is exponentially increasing that far exceeds any expansion of infrastructure. Social responsibilities and regulations (e.g., GDPR) must also be adhered when handling IoT data. In addition, we are currently witnessing a shift towards distributing AI to the edge. The aforementioned reasons render the distribution of data preprocessing to the edge an urgent requirement. In this paper, we introduce a modern data preprocessing framework that consists of two main parts. Part1 is a design tool that reduces the complexity and costs of the data preprocessing phase for AI via generalization and normalization. The design tool is a standard template that maps specific techniques into abstract categories and highlights dependencies between them. In addition, it presents a holistic notion of data preprocessing that is not limited to data cleaning. The second part is an IoT tool that adopts the edge-cloud collaboration model to progressively improve the quality of the data. It includes a synchronization mechanism that ensures adaptation to changes in data characteristics and a coordination mechanism that ensures correct and complete execution of preprocessing plans between the cloud and the edge. The paper includes an empirical analysis of the framework using a developed prototype and an automotive use-case. Our results demonstrate reductions in resource consumption (e.g., energy, bandwidth) while maintaining the value and integrity of the data.

Cited by 9Open access

Preprocessing of Spectral Data

Alexis Weber · 2025

In analytical chemistry, preprocessing spectral data is a critical step in ensuring the accuracy and reliability of chemometric models used for quantitative and qualitative analysis. This chapter explores various preprocessing techniques, including normalization, baseline correction, smoothing, derivatives, and instrument-specific tools to remove artifacts. Raman, mid-infrared, and near-infrared spectroscopy will be the focus of these preprocessing techniques. By addressing common artifacts like noise, baseline shifts, missing data, and peak variability, preprocessing enhances the quality of spectral datasets while minimizing the risk of information loss. Key approaches, best practices for data preprocessing, and examples of their practical application in chemical analyses are discussed. The chapter also emphasizes the importance of judiciously selecting and combining preprocessing techniques.

Cited by 0

Preprocessing of minitrack data /

Watkins, Edward R., Jr., Goddard Space Flight Center., United States. National Aeronautics and Space Administration. · The Journal of the Abraham Lincoln Association

Cited by 0Open access

D3.2 Data Preprocessing Toolbox

The EXA4MIND Consortium · Zenodo (CERN European Organization for Nuclear Research) · 2025

This deliverable introduces the “Toolboxes” module of the EXA4MIND Platform, extending and covering the idea of a Data Preprocessing Toolbox developed within Work Package 3 (WP3, Extreme Data Analytics and Processing) of the EXA4MIND project. The Preprocessing Toolbox contains a set of generic and application-specific preprocessing tools that enable data cleaning, transformation, fusion, and harmonisation across heterogeneous data sources. Processing tools focusing on validation have been made into an own Validation Toolbox submodule, and an Analytics & Artificial Intelligence (AI) Toolbox is being compiled as well. With a uniform approach to command line interfacing and curated code, our Toolboxes provide examples that users can use or adapt for their own Extreme Data applications. While such applications are too individual to cater for them with a reasonably limited set of tools, the idea of our Toolboxes (Preprocessing, Validation and Analytics & AI) is thus to enable the users to construct their own processing steps within advanced data-driven workflows. Preprocessing steps in particular ensure consistency and quality of input data, and thus lay the foundation for effective querying and analytics services across the Extreme Data platform. The Preprocessing Toolbox has been designed in close alignment with WP2 (Data Spaces Management), enabling seamless integration between distributed data spaces and advanced analytics capabilities. In particular, it is easy to employ the Toolboxes and individual tools on data from different data sources, constructing data-driven workflows within an instance of the EXA4MIND Advanced Query and Indexing System (AQIS, from WP3). Including a fairness-check tool and validation mechanisms, the Toolboxes support the overall project objective of enabling trustworthy, green, and fair AI. This document complements the Open Source code repositories which are the main part of the deliverable. It briefly presents motivation and design principles, and gives pointers to the actual implementations and usage instructions.

Cited by 0Open access

Results of data preprocessing.

Xinyun Jiang (12481005) · Figshare · 2023

<div><p>With the continuous decline of water resources due to population growth and rapid economic development, precipitation prediction plays an important role in the rational allocation of global water resources. To address the non-linearity and non-stationarity of monthly precipitation, a combined prediction method based on complementary ensemble empirical mode decomposition (CEEMD) and a modified long short-term memory (LSTM) neural network was proposed. Firstly, the CEEMD method was used to decompose the monthly precipitation series into a set of relatively stationary sub-sequence components, which can better reflect the local characteristics of the sequence and further understand the nonlinear dynamic characteristics of the sequence. Then, improved LSTM neural networks were employed to predict each sub-sequence. The proposed improvement method optimized the hyper-parameters of LSTM neural networks using particle swarm optimization algorithm, which avoided the randomness of artificial parameter selection. Finally, the predicted results of each component were superimposed to obtain the final prediction result. The proposed method was validated by taking the monthly precipitation data from 1961 to 2020 in Changde City, Hunan Province as an example. The results of the case study show that, compared with other traditional prediction methods, the proposed method can better reflect the trend of precipitation changes and has higher prediction accuracy.</p></div>

Cited by 0Open access

Data preprocessing for data mining systems

Václav Falc · Digital Repository (National Repository of Grey Literature) · 2012

Main target of this graduation thesis was creating system for data preparation. System was created using programing languages C#, SQL and partly in XML and HTML.

Cited by 0

Data Diversification Analysis on Data Preprocessing

Taeyoon Kim, Chanho Park, Heelim Hong et al. · Zenodo (CERN European Organization for Nuclear Research) · 2023

A statistical analysis to examine the diversity distribution resulting from two different approaches: The first one, the standard approach, is a baseline augmentation approach where a random augmentation is applied to each sample in each epoch independently; The second one, the random batch approach, is another new augmentation approach designed where a random augmentation is applied to each tiny-batch in each epoch independently, and which samples are in the same tiny-batch is random and independent across all epochs.

Cited by 0Open access

Data Preprocessing and Intelligent Data Analysis

A. Fazel Famili, Wei‐Min Shen, Richard W. Weber et al. · Intelligent Data Analysis · 1997

This paper first provides an overview of data preprocessing, focusing on problems of real world data. These are primarily problems that have to be carefully understood and solved before any data analysis process can start. The paper discusses in detail two main reasons for performing data preproces sing: (i) problems with the data and (ii) preparation for data analysis. The paper continues with details of data preprocessing techniques achieving each of the above mentioned objectives. A total of 14 techniques are discussed. Two examples of data preprocessing applications from two of the most data rich domains are given at the end. The applications are related to semiconductor manufacturing and aerospace domains where large amounts of data are available, and they are fairly reliable. Future directions and some challenges are discussed at the end.

Cited by 252

Data preprocessing of spatial data mining

Hanning Yuan, Shuliang Wang · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2005

Data preprocessing is essential in spatial data mining. This paper presents some issues on data preprocessing. It pays more attention on incomplete data, inaccurate data, repetitive data, inconsistent data, and image data. Finally, a case is studied, and come with a satisfactory result.

Cited by 0

Data preprocessing in predictive data mining

Stamatios-Aggelos N. Alexandropoulos, Sotiris Kotsiantis, Michael N. Vrahatis · The Knowledge Engineering Review · 2019

Abstract A large variety of issues influence the success of data mining on a given problem. Two primary and important issues are the representation and the quality of the dataset. Specifically, if much redundant and unrelated or noisy and unreliable information is presented, then knowledge discovery becomes a very difficult problem. It is well-known that data preparation steps require significant processing time in machine learning tasks. It would be very helpful and quite useful if there were various preprocessing algorithms with the same reliable and effective performance across all datasets, but this is impossible. To this end, we present the most well-known and widely used up-to-date algorithms for each step of data preprocessing in the framework of predictive data mining.

Cited by 195