Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 4,565 papers for “parallel data mining”

Parallel Algorithm for Spatial Data Mining Using CUDA

Byoung-Woo Oh · JOURNAL OF ADVANCED INFORMATION TECHNOLOGY AND CONVERGENCE · 2019

Recently, there is an increasing demand for applications utilizing maps and locations such as autonomous vehicles and location-based services. Since these applications are developed based on spatial data, interest in spatial data processing is increasing and various studies are being conducted. In this paper, I propose a parallel mining algorithm using the CUDA library to efficiently analyze large spatial data. Spatial data includes both geometric (spatial) and non-spatial (aspatial) attributes. The proposed parallel spatial data mining algorithm analyzes both the geometric and non-spatial relationships between two layers. The experiment was performed on graphics cards containing CUDA cores based on TIGER/Line data, which is the actual spatial data for the US census. Experimental results show that the proposed parallel algorithm using CUDA greatly improves spatial data mining performance.

Cited by 9

Scaling data mining in massively parallel dataflow systems

Sebastian Schelter · 2014

The demand for mining large datasets using shared-nothing clusters is steadily on the rise. Despite the availability of parallel processing paradigms such as MapReduce, scalable data mining is still a tough problem. Naïve ports of existing algorithms to platforms like Hadoop exhibit various scalability bottlenecks, which prevent their application to large real-world datasets. These bottlenecks arise from various pitfalls that have to be overcome, including the scalability of the mathematical operations of the algorithm, the performance of the system when executing iterative computations, as well as its ability to efficiently execute meta learning techniques such as cross-validation and ensemble learning.

Cited by 0

Parallel research of sequential pattern data mining algorithm

Wang, Jinlin, Chen et al. · 2008

Sequential pattern has important applications in many areas and a large number of data and patterns need efficient and scalable parallel algorithm. In light of the widespread problem in current sequential pattern data mining algorithm and on the basis of researching the data mining algorithm of serial sequential pattern, a data mining algorithm of parallel sequential pattern is presented in this paper. Through theoretical analysis and experimental verification, the parallel data mining algorithm can well improve the efficiency of data mining in massive data circumstances. © 2008 IEEE. (11 refs.)

Cited by 0

An Efficient, Scalable, Parallel Classifier for Data Mining

Anurag Srivastava Dept, Anurag Srivastava, Vineet Singh · 1997

One of the important problems in data mining is classification. Recently there has been a lot of interest in classifying disk resident datasets. Experiments show that classifying large datasets leads to improvement in accuracy. Most of the current algorithms are suited for small datasets and are inefficient for large disk resident data sets. SPRINT is a state of the art decision tree based algorithm for disk resident data. We propose a new decision-tree-based algorithm which achieves essentially similar accuracy as SPRINT and gives much better performance. Our algorithm has absolutely no memory restrictions compared to SPRINT which has some memory restrictions. Our algorithm also does significantly less I/O than SPRINT. We also present a parallel formulation of our serial algorithm which is inherently more scalable than parallel-SPRINT. An analysis of the cost of computation and communication of our parallel formulation is done. Our analysis and experimental results on IBM SP2 show tha...

Cited by 0

Towards a Cost-Effective Parallel Data Mining Approach

Zoltan Jarai, Aashu Virmani, Liviu Iftode · 1998

Massive rule induction has recently emerged as one of the powerful data mining techniques. The problem is known to be exponential in the size of the attributes, and given its ever increasing use, can greatly benefit from parallelization. In this paper, we study cost-effective approaches to parallelize rule generation algorithms. In particular, we consider the propositional rule generation algorithm of the Discovery Board system, and present our design and implementation of a parallel algorithm for the same task. We then present some early performance results of our parallelization scheme on hardware and software distributed shared memory multiprocessors. 1 Introduction Massive rule induction has recently emerged as one of the powerful data mining techniques. However, the technique is well known to suffer from a combinatorial explosion, and the number of rules generated can often be an order of magnitude larger than data itself. The algorithms for this problem must explore all combina...

Cited by 0

PaDDMAS: parallel and distributed data mining application suite

Omer Rana, David Walker, Maozhen Li et al. · 2002

Discovering complex associations, anomalies and patterns in distributed data sets is gaining popularity in a range of scientific, medical and business applications. Various algorithms are employed to perform data analysis within a domain, and range from statistical to machine learning and AI based techniques. Several issues need to be addressed however to scale such approaches to large data sets, particularly when these are applied to data distributed at various sites. As new analysis techniques are identified, the core tool set must enable easy integration of such analytical components. Similarly, results from an analysis engines must be sharable, to enable storage, visualisation or further analysis of results. We describe the architecture of PaDDMAS, a component based system for developing distributed data mining applications. PaDDMAS provides a tool set for combining pre-developed or custom components using a dataflow approach, with components performing analysis, data extraction or data management and translation. Each component is wrapped as a Java/CORBA object, and has an interface defined in XML. Components can be serial or parallel objects, and may be binary or contain a more complex internal structure. We demonstrate a prototype using a neural network analysis algorithm.

Cited by 24

An Efficient, Scalable, Parallel Classifier for Data Mining

Anurag Srivastava, Vineet Singh · 1997

One of the important problems in data mining is classification. Recently there has been a lot of interest in classifying disk resident datasets. Experiments show that classifying large datasets leads to improvement in accuracy. Most of the current algorithms are suited for small datasets and are inefficient for large disk resident data sets. SPRINT is a state of the art decision tree based algorithm for disk resident data. We propose a new decision-tree-based algorithm which achieves essentially similar accuracy as SPRINT and gives much better performance. Our algorithm has absolutely no memory restrictions compared to SPRINT which has some memory restrictions. Our algorithm also does significantly less I/O than SPRINT. We also present a parallel formulation of our serial algorithm which is inherently more scalable than parallel-SPRINT. An analysis of the cost of computation and communication of our parallel formulation is done. Our analysis and experimental results on IBM SP2 show tha...

Cited by 0

WEKA-G: Parallel data mining on computational grids

Amanda Aparecida Pimenta, P. VALENTIM, Danilo França Conceição dos Santos et al. · SHILAP Revista de lepidopterología · 2009

Data mining is a technology that can extract useful information from large amounts of data. However, mining a database often requires a high computational power. To resolve this problem, this paper presents a tool (Weka-G), which runs in parallel algorithms used in the mining process data. As the environment for doing so, we use a computational grid by adding several features within a WAN.

Cited by 0Open access

Unstructured Big Data Threat Intelligence Parallel Mining Algorithm

Zhihua Li, Xinye Yu, Tao Wei et al. · Big Data Mining and Analytics · 2024

To efficiently mine threat intelligence from the vast array of open-source cybersecurity analysis reports on the web, we have developed the Parallel Deep Forest-based Multi-Label Classification (PDFMLC) algorithm. Initially, open-source cybersecurity analysis reports are collected and converted into a standardized text format. Subsequently, five tactics category labels are annotated, creating a multi-label dataset for tactics classification. Addressing the limitations of low execution efficiency and scalability in the sequential deep forest algorithm, our PDFMLC algorithm employs broadcast variables and the Lempel-Ziv-Welch (LZW) algorithm, significantly enhancing its acceleration ratio. Furthermore, our proposed PDFMLC algorithm incorporates label mutual information from the established dataset as input features. This captures latent label associations, significantly improving classification accuracy. Finally, we present the PDFMLC-based Threat Intelligence Mining (PDFMLC-TIM) method. Experimental results demonstrate that the PDFMLC algorithm exhibits exceptional node scalability and execution efficiency. Simultaneously, the PDFMLC-TIM method proficiently conducts text classification on cybersecurity analysis reports, extracting tactics entities to construct comprehensive threat intelligence. As a result, successfully formatted STIX2.1 threat intelligence is established.

Cited by 6Open access

Parallel Data Mining from Multicore to Cloudy Grids

Geoffrey Fox, Seunghee Bae, Jaliya Ekanayake et al. · Advances in parallel computing · 2009

We describe a suite of data mining tools that cover clustering, information retrieval and the mapping of high dimensional data to low dimensions for visualization. Preliminary applications are given to particle physics, bioinformatics and medical informatics. The data vary in dimension from low (2-20), high (thousands) to undefined (sequences with dissimilarities but not vectors defined). We use deterministic annealing to provide more robust algorithms that are relatively insensitive to local minima. We discuss the algorithm structure and their mapping to parallel architectures of different types and look at the performance of the algorithms on three classes of system; multicore, cluster and Grid using a MapReduce style algorithm. Each approach is suitable in different application scenarios. We stress that data analysis/mining of large datasets can be a supercomputer application.

Cited by 37

A Middleware for Developing Parallel Data Mining Applications

Ruoming Jin, Gagan Agrawal · 2001

1 Introduction Data mining is an interdisciplinary field, having applications in diverse areas like bioinformatics, medical informatics, scientific data analysis, financial analysis, consumer profiling, etc. In each of these application domains, the amount of data available for analysis has exploded in recent years, making the scalability of data mining implementations a critical factor. To this end, parallel versions of most of the well-known data mining techniques have been developed in recent years. However, the expertise and effort currently required in implementing, maintaining, and performance tuning a parallel data mining application is a severe impediment in the wide use of parallel computers for scalable data mining.

Cited by 38

A middleware for developing parallel data mining implementations

Ruoming Jin, Gagan Agrawal · 2001

Data mining is an interdisciplinary field, having applications in diverse areas like bioinformatics, medical informatics, scientific data analysis, financial analysis, consumer profiling, etc. In each of these application domains, the amount of data available for analysis has exploded in recent years, making the scalability of data

Cited by 0

Research on Parallel Data Mining Based on Spark

Jiali Shen · 2022

In the current era of big data, the rapid development of network technology and hardware equipment leads to exponential data growth. However, under the challenge of massive data, there are still some problems in the field of data mining, such as low efficiency of algorithm execution, insufficient parallel optimization of algorithms and poor usability of data mining platforms. This paper focuses on parallel data mining algorithms and parallel data mining tools. Based on Spark as a programming model and processing engine, a distributed parallel data mining scheduling framework is designed and implemented based on Hadoop and Spark, which can meet the needs of users for mining and analyzing large data sets. The scheduling system implements common data mining algorithms such as classification, prediction, clustering and data preprocessing, and can complete data mining modeling by visual drag and drop algorithm program.

Cited by 1

A sampling-based framework for parallel data mining

Shengnan Cong, Jiawei Han, Jay Hoeflinger et al. · 2005

The goal of data mining algorithm is to discover useful information embedded in large databases. Frequent itemset mining and sequential pattern mining are two important data mining problems with broad applications. Perhaps the most efficient way to solve these problems sequentially is to apply a pattern-growth algorithm, which is a divide-and-conquer algorithm [9, 10]. In this paper, we present a framework for parallel mining frequent itemsets and sequential patterns based on the divide-and-conquer strategy of pattern growth. Then, we discuss the load balancing problem and introduce a sampling technique, called selective sampling, to address this problem. We implemented parallel versions of both frequent itemsets and sequential pattern mining algorithms following our framework. The experimental results show that our parallel algorithms usually achieve excellent speedups.

Cited by 42

High performance data mining using data cubes on parallel computers

Sanjay Goil, Alok Choudhary · 2002

Online analytical processing techniques are used for data analysis and decision support systems. The multidimensionality of the underlying data is well represented by multidimensional databases. For data mining in knowledge discovery, OLAP calculations can be effectively used. For these, high performance parallel systems are required to provide interactive analysis. Precomputed aggregate calculations in a data cube can provide efficient query processing for OLAP applications. We present parallel data cube construction on distributed-memory parallel computers from a relational database. The data cube is used for data mining of associations using attribute focusing. Results are presented for these on the IBM-SP2, which show that our algorithms and techniques are scalable to a large number of processors, providing a high performance platform for such applications.

Cited by 12