David Gorsich, Robert E. Karlsen, Grant R. Gerhart · 2000
Abstract : Support vector machines are classification algorithms based on quadratic programming that have been found to give excellent classification results on problems such as discriminating targets versus backgrounds. A key capability of these algorithms is that they need not require a preprocessing step to find feature vectors, yet preprocessing is still typically used. Preprocessing is still an important step in the classification process. We discuss what type of preprocessing steps are useful in improving the classifications results of support vector machines. We first give a short introduction to support vector machines. Then the algorithm is applied to the MSTAR radar data set. Several methods to preprocess the data are used before being sent to the support vector machine. The effect on the classification rate of the algorithm is then determined.
Cited by 0
Sudipta Hazra, Arindam Mondal · 2025
The general structure for data curation was proposed in this chapter. It covers the many stages of preparation and preprocessing for data. Many other datasets can be fitted by the overall framework that is described. Raw data that have not been cleaned and curated are typically unsuitable for drawing accurate conclusions. Within the topic of data curation and preparation, the most widely used algorithms and strategies are covered in detail in this chapter. The methodology for data curation, imputation, feature extraction, correlation analysis, and real-world implementation of these algorithms is covered in this chapter's framework. We also offered methods that we have created based on our data processing skills. Lastly, we demonstrated with a real-world example how applying various imputation techniques affects support vector machine efficiency and performance. The chapter outlines a process for taking unstructured, unprocessed data and turning it into well-organized data that may be used with sophisticated machine learning algorithms or other advanced data analysis techniques.
Cited by 0
Ujwal Chaudhary · 2025
Cited by 0
Aman Rawat, Mr.Ritesh Kumar · Zenodo (CERN European Organization for Nuclear Research) · 2025
Data preprocessing is a fundamental step in any machine learning pipeline, as the quality of input data directly influences the reliability and accuracy of predictive models. Real-world datasets often contain missing values, duplicate entries, inconsistent formats, outliers, and non-standardized features, making manual preprocessing time-consuming, error-prone, and difficult to reproduce. To address these challenges, this research presents an Automated Data Cleaning and Preprocessing System designed to streamline the transformation of raw data into structured, analysis-ready form with minimal user intervention. The system integrates automated detection of missingness, statistical and algorithmic imputation, outlier identification, categorical encoding, normalization, and comprehensive summary reporting through a modular pipeline architecture. Experiments conducted on diverse datasets from multiple domains demonstrate improvements in data consistency, distributional stability, and downstream machine learning performance. The results highlight that automated preprocessing not only reduces human effort but also ensures reproducibility, scalability, and enhanced model accuracy, making it suitable for academic research, industrial applications, and production-level data workflows.
Cited by 0Open access
Ying Quan, Weimin Zheng · 2010
The software correlator system (SCS) is a key module of Chinese VLBI (Very Long Baseline Interferometry) Network (CVN). The main tasks of SCS include data pre-processing, fringe-searching, correlation, and phase correction signal extraction. In this paper, we propose four approaches for high-speed VLBI data preprocessing on the universal PC-server platform, namely sliding window based time decoding, dynamic lookup-table scheme, squeeze strategy and data synchronization. The experiment results achieved a very small latency especially when the data rate was greater than 256Mbps per station.
Cited by 0
Shreyas Wankhede, Ranjit A. Patil, Sagar Sonawane et al. · 2018
Sentiment analysis is an important research area that identifies the people's sentiments, opinions and emotions underlying a text. As the use of social media is increasing day by day, it plays an essential role in communication through technology. Twitter, which is one of the popular and largely used social media platforms for communication has more than 200 million tweets per day. Tweets are short in length and due to limited size of tweets people generally commit some mistakes while tweeting so pre-processing is necessary. The use of modern emoticons which are known as emojis that is largely used in social media communications that conveys variety of emotions. The purpose of this paper is to use N -gram method and Hidden Markov Model for Spell-Checking and Correction of tweets and also Emoji Sentiment Ranking method which is used to evaluate sentiment mapping of emojis by using sentiment polarity such as negative, neutral, or positive.
Cited by 12
Tie‐Yan Liu · 2011
Cited by 1
Huijie Zeng, Jiansheng Guo, Jian Wang et al. · 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) · 2019
The system grabs data packets from campus network routers, divides the collected basic data into seven basic databases, preprocesses all the basic data and preprocesses the text. Through the statistics and calculation of the data, it extracts the frequency of user browsing web pages, the frequency of using application categories, the time period of surfing the Internet, the time of surfing the Internet, the total flow of surfing the Internet, the amount of Posts and the mail. Based on the emotional dictionary, the key words of emotional features and topic features are extracted from the text information such as the content of browsing web pages and the content of posting, and the user's emotional orientation values are obtained. The obtained behavioral, emotional and topic features are stored in the feature database. Finally, through the functional test and performance test, the system can extract the daily characteristic quantity according to the daily data, get the characteristic data table, and use the characteristic data to judge the depression tendency of college students by the depression recognition and warning model.
Cited by 2
郑为民 权莹 · 2010
Cited by 0
S. Hrushikesava Raju, T. Swarna Latha · 2015
Paper presents external sorting using data preprocessing. Generally, huge data of any organization possess data redundancy, noise and data inconsistency. To eliminate, Data preprocessing should be performed on raw data, then sorting technique is applied on it. Data preprocessing includes many methods such as data cleaning, data integration, data transformation and data reduction. Depending on the complexity of given data, these methods are taken and applied on raw data in order to produce quality of data. Then, external sorting is applied. The external sorting now takes the number of passes less than actual passes log B (N/M) + 1 for B – way external merge sorting, and number of Input / Outputs less than 2*N * (log B (N/M) + 1) of Input / Outputs and also involve least number of runs compared to actual basic external sorting.
Cited by 0
Salvador García, Sergio Ramírez‐Gallego, Julián Luengo et al. · Big Data Analytics · 2016
The massive growth in the scale of data has been observed in recent years being a key factor of the Big Data scenario. Big Data can be defined as high volume, velocity and variety of data that require a new high-performance processing. Addressing big data is a challenging and time-demanding task that requires a large computational infrastructure to ensure successful data processing and analysis. The presence of data preprocessing methods for data mining in big data is reviewed in this paper. The definition, characteristics, and categorization of data preprocessing approaches in big data are introduced. The connection between big data and data preprocessing throughout all families of methods and big data technologies are also examined, including a review of the state-of-the-art. In addition, research challenges are discussed, with focus on developments on different big data framework, such as Hadoop, Spark and Flink and the encouragement in devoting substantial research efforts in some families of data preprocessing methods and applications on new big data learning paradigms.
Cited by 677Open access
The MIT Press eBooks · 2004
Cited by 2
Fiona Draxler · Zenodo (CERN European Organization for Nuclear Research) · 2025
This presentation presents examples of data matching in practice. As a first example, it presents a case study on the effectiveness of COVID-19 vaccinations. It covers technical and privacy considerations for matching spatial data with surveys, challenges and potential applications. It also discusses how traditional data matching is changing with larger volumes of (unstructured) data (Big Data Perspective), and Language Models for data matching. This presentation is the fourth part of a set of presentations on the topic of "Tools and Workflows (Linked Data)". Presentation available after free registration: https://elearning.gesis.org/course/section.php?id=1274
Cited by 0Open access
Jafari R., Jafari R. · Repository of Samara University (Samara National Research University) · 2022
"Hands-On Data Preprocessing" is a practical guide to mastering data cleaning, integration, reduction, and transformation techniques using Python. Written by an expert, it covers data collection, handling missing values, outliers, and preparing data for analytics. With hands-on examples, readers learn to optimize data for better insights and decision-making. Ideal for data analysts, BI professionals, and data enthusiasts, the book requires basic Python and analytics knowledge. It equips readers to preprocess data effectively, ensuring it's ready for advanced analysis.
Cited by 0
王智威(zhiwei wang) · China National GeneBank DataBase · 2026
Cited by 0Open access
Josefine Hebisch, Ayelet Arazi, Joshua Calder-Travis et al. · OSF Preprints (OSF Preprints) · 2025
Preprocessing pipeline for pupil size data
Cited by 0Open access
Hernández Alonso, Manuel Andrés · Dipòsit Digital de la Universitat de Barcelona (Universitat de Barcelona) · 2023
Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2023, Director: Ignasi Cos Aguilera i Michael DePass
Cited by 0Open access
Mudhafar Jalil Jassim Ghrabat (22177655), Arkan A. Ghaib (22177658), Auhood Al-Hossenat (22177661) et al. · Figshare · 2025
<div> Breast cancer is highlighted in recent research as one of the most prevalent types of cancer. Timely identification is essential for enhancing patient results and decreasing fatality rates. Utilizing computer-assisted detection and diagnosis early on may greatly improve the chances of recovery by accurately predicting outcomes and developing suitable treatment plans. Grading breast cancer properly, especially evaluating nuclear atypia, is difficult owing to faults and inconsistencies in slide preparation and the intricate nature of tissue patterns. This work explores the capability of deep learning to extract characteristics from histopathology photos of breast cancer. The research introduces a new method called SMOTE-based Convolutional Neural Network (CNN) technology to detect areas impacted by Invasive Ductal Carcinoma (IDC) in whole slide pictures. The trials used a dataset of 162 individuals with IDC, split into training (113 photos) and testing (49 images) groups. Every model was subjected to individual testing. The SMO_CNN model we developed demonstrated exceptional testing and training accuracies of 98.95% and 99.20% respectively, surpassing CNN, VGG19, and ResNet50 models. The results highlight the effectiveness of the created model in properly detecting IDC-affected tissue areas, showing great promise for improving breast cancer diagnosis and treatment planning. We surpassing other models as such, CNN, VGG19, ResNet50. </div>
Cited by 0Open access
Bo Jiang (76119), Junjiao Hu (9585087), Xiaofan Chen (140368) et al. · Figshare · 2025
Image Preprocessing and Data Allocation Flowchart.
Cited by 0Open access
Sotiras, Aristeidis · Figshare · 2017
Talk given during the "Harmonise This! Analyzing Diverse Neuroimaging Datasets" workshop at the 2015 Organization for Human Brain Mapping (OHBM) conference in Hawaii, 14-18 June.
Cited by 0Open access