Leslie S. Perkins, Phil Andrews, Dhabaleswar K. Panda et al. · 2006
Over the past decade, computing power has increased extraordinarily, enabling researchers to tackle large complex problems -- a great step forward in our ability to model the complex world. This increased capability resulted in massive amounts of data being produced and used by increasingly detailed simulations. It is the location, viewing, manipulation, storage, movement, sharing, and interpretation of this data -- now in hundreds of terabytes and growing -- that is causing major performance bottleneck. Data Intensive Computing has emerged as a topic to address the problem. The Advanced Virtual Engine Test Cell (AVETeC) Inc. and Computer Sciences Corporation (CSC) are under contract to establish, integrate and manage a test bed located at Department of Defense Major Shared Resource Center, Wright-Patterson Air Force Base, OH, at AVETeC in Springfield, OH collaborating with Department of Energy, and NASA Goddard, Greenbelt, MD. This test bed environment will actively evaluate emerging technologies to improve data management.
Cited by 8
Rajkumar Buyya, Christian Vecchiola, S. Thamarai Selvi et al. · Elsevier eBooks · 2013
Cited by 0
Ian Gorton, Ian Gorton, Ian Gorton et al. · Cambridge University Press eBooks · 2012
The world is awash with digital data from social networks, blogs, business, science and engineering. Data-intensive computing facilitates understanding of complex problems that must process massive amounts of data. Through the development of new classes of software, algorithms and hardware, data-intensive applications can provide timely and meaningful analytical results in response to exponentially growing data complexity and associated analysis requirements. This emerging area brings many challenges that are different from traditional high-performance computing. This reference for computing professionals and researchers describes the dimensions of the field, the key challenges, the state of the art and the characteristics of likely approaches that future data-intensive problems will require. Chapters cover general principles and methods for designing such systems and for managing and analyzing the big data sets of today that live in the cloud and describe example applications in bioinformatics and cybersecurity that illustrate these principles in practice.
Cited by 14
Judy Qiu, Jaliya Ekanayake, Thilina Gunarathne et al. · 2010
Cited by 0
Antouny, Luke · Macquarie University · 2018
Applications are continually growing in both size and scale. As a result of this growth the need for distributed processing has become increasingly apparent. Cloud computing offers an effective method of hiring machines to facilitate the execution of specific applications. Users of cloud services have the flexibility to assemble configurations of instances that satisfy their specific computational requirements. Cloud computing also adopts a pay as you go nature ensuring that users only pay for the services that they use. Due to its high cost effectiveness and elasticity, cloud computing has become a desirable platform for many different applications and services. In this thesis we attempt to develop scheduling solutions for a scientific application called Montage. There is an existing cloud native workflow execution framework that has been tested in Amazon EC2.
Cited by 0Open access
Antouny, Luke · Figshare · 2018
Applications are continually growing in both size and scale. As a result of this growth the need for distributed processing has become increasingly apparent. Cloud computing offers an effective method of hiring machines to facilitate the execution of specific applications. Users of cloud services have the flexibility to assemble configurations of instances that satisfy their specific computational requirements. Cloud computing also adopts a pay as you go nature ensuring that users only pay for the services that they use. Due to its high cost effectiveness and elasticity, cloud computing has become a desirable platform for many different applications and services. In this thesis we attempt to develop scheduling solutions for a scientific application called Montage. There is an existing cloud native workflow execution framework that has been tested in Amazon EC2.
Cited by 0Open access
Alexander S. Szalay · Computing in Science & Engineering · 2011
Scientific computing increasingly involves massive data; in astronomy, observations and numerical simulations are on the verge of generating petabytes. This new, data-centric computing requires a new look at computing architectures and strategies. Using Amdahl's law to characterize architectures and workloads, it's possible to use existing commodity parts to build systems that approach an ideal Amdahl machine.
Cited by 51
Ian Gorton, Adam Wynne, Justin Almquist et al. · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2017
MeDICi is an evolving middleware platform for building complex, high performance analytical applications. These applications typically comprise a pipeline of software components, each of which performs some analysis on incoming data and passes on its results to the next step in the pipeline.
Cited by 0Open access
Anuj Shah, Joshua Adkins, Douglas Baxter et al. · Advances in computers · 2010
Cited by 2
Thomas Fahringer · 2008
No abstract available.
Cited by 0
Mário A. R. Dantas · 2020
This work presents an introduction to the Data Intensive Scalable Computing (DISC) approach. This paradigm represents a valuable effort to tackle the large amount of data produced by several ordinary applications. Therefore, subjects such as characterization of big data and storage approaches, in addition to brief comparison between HPC and DISC are differentiated highlight.
Cited by 0Open access
Judy Qiu, Jaliya Ekanayake, Thilina Gunarathne et al. · Advances in systems analysis, software engineering, and high performance computing book series · 2012
Data intensive computing, cloud computing, and multicore computing are converging as frontiers to address massive data problems with hybrid programming models and/or runtimes including MapReduce, MPI, and parallel threading on multicore platforms. A major challenge is to utilize these technologies and large-scale computing resources effectively to advance fundamental science discoveries such as those in Life Sciences. The recently developed next-generation sequencers have enabled large-scale genome sequencing in areas such as environmental sample sequencing leading to metagenomic studies of collections of genes. Metagenomic research is just one of the areas that present a significant computational challenge because of the amount and complexity of data to be processed. This chapter discusses the use of innovative data-mining algorithms and new programming models for several Life Sciences applications. The authors particularly focus on methods that are applicable to large data sets coming from high throughput devices of steadily increasing power. They show results for both clustering and dimension reduction algorithms, and the use of MapReduce on modest size problems. They identify two key areas where further research is essential, and propose to develop new O(NlogN) complexity algorithms suitable for the analysis of millions of sequences. They suggest Iterative MapReduce as a promising programming model combining the best features of MapReduce with those of high performance environments such as MPI.
Cited by 4
Jong Hoon Ahnn · eScholarship (California Digital Library) · 2015
Mobile computing, where mobile users continuously gather, process, and share sensor or application-specific data, is emerging as a new computing and network paradigm of data sharing in seamless manner. The key enablers are the smartphones (e.g., iPhones and Android phones) equipped with onboard sensors (e.g., cameras, accelerometer, compass, GPS) and various wireless devices (e.g., WiFi, 3G/4G-LTE, and other network standards). However, despite of all the advances in recent years, mobile devices have limited resources for computation, memory, network, and battery. Mobile cloud computing (MCC) is a promising practical approach to relax such constraints in mobile devices for computational intensive applications, where computation offloading to virtually unlimited resources in clouds can help them to save the energy consumption. Existing offloading algorithms and corresponding MCC systems offload computationally intensive applications to remote cloud servers in a distributed manner.However, they fail to address challenges in modern data-intensive applications, where sensor data is continuously harvested and application-specific data among users is shared to process or personalize applications. For instance, modern machine learning (ML)-based applications require the large amount of training data to train a model, often require periodic retraining or adaptation based on gathered user's specific profiles and context data. For such applications, offloading is not only computationally intensive but data intensive, and it requires a seamless interface to upload/fetch data to/from cloud storage.A comprehensive survey provides 15 different perspectives in ten major MCC systems from the viewpoint of data-intensive computation offloading. Typical MCC surveys often constitute network protocol, code rewrite requirement, offloading granularity, profiling, resource monitoring, cost model, and software preinstallation requirement. Our survey differs from typical MCC surveys mainly in several ways: scalable big data and computing support and fault-tolerance of MCC systems, providing features such as parallel offloading, multi-cloud support, disconnected operation, reliable message delivery, cloud sever scalability, and cloud storage access.The process of computation offloading is complex. In this thesis, we tackle it as several subsystem problems: First, existing MCC systems fail to address multiple cloud settings in computation offloading scenarios. The application partitioning and offloading to multiple clouds are formally formulated and solved as an integer linear programming (ILP) problem. The objective function of the partitioning problem makes a set of offloading decision in terms of tradeoffs between mobile execution and cloud execution with respect to energy costs and time cost under the assumption of multi-clouds; Second, existing MCC systems fail to address a wireless network instability problem. In order to achieve fault-tolerant communication between mobile devices and cloud servers, we propose and implement a mechanism that can tolerate unstable network conditions by asynchronous implementation of network binding; Third, existing MCC systems fail to achieve reliable offloading message delivery, where millions of mobile devices submit computation offloading requests and data related requests to clouds. To achieve such scalability and reliability, we propose a producer/consumer-based message queue, where producers for delivering offloading requests send offloading requests over the network to the cluster, which in turn serves them up to consumers. The multiple consumers are in charge of submitting the offloaded tasks to cloud resources. Eventual reliability is achieved by message partitioning and replication in the message queue system while scalable message delivery is achieved by horizontal scaling of producers/consumers and reconfiguring of message partition factor.Fourth, cloud resource management is another challenging area, where researchers put little attention to data intensive application offloading. Existing MCC systems are not scalable when dealing with millions of offloading requests as well as data requests. For the data intensive application offloading, we consider integration to one of the conventional big data systems where a thin mobile client is to access data and to process it from the cloud. We propose the fuse of Hadoop-based parallel computation offloading to multiple clouds by developing seamless interfaces. The integration enables our MCC system to be scalable and reliable in scheduling offloading tasks and in serving data-related operations on top of Hadoop.We provide a couple of evaluation results in terms of time saving and energy saving based on two types of benchmarks: computationally-intensive applications and data-intensive applications. For the former, we consider chess game, puzzle game, cryptographic algorithm, and Huffman compression algorithm, for the latter, we consider the state-of-the-art speech recognition application. The evaluation extends a single offloading to parallel offloading. The further evaluation of data-intensive computational cloud is performed in several aspects: performance comparison of Hadoop version 1 and version 2, Hadoop overhead, benefit of file compression, and performance of cloud storage to support small/large files. Finally, the performance of the proposed message queue is presented.
Cited by 1Open access
Estes, Erin · ISU Red - Research and eData (Illinois State University) · 2020
Cited by 0
Mittal, M, Balas, V E, Hemanth, D J · CERN Document Server (European Organization for Nuclear Research) · 2018
Cited by 0
P. Loncar · Annals of DAAAM for ... & proceedings of the ... International DAAAM Symposium · 2018
The development of ICT has led to enormous growth and data accumulation, and thus created the need for proper storage and processing of large data, known as Big Data. The number of data sources like mobile telephones and applications, social networks, digital television, data of various Internet objects and sensors has increased due to the development of technology and the emergence of the IoT, the evolutionary step in the development of the Internet. Analysis and proper interpretations that take place on the latest distributed platforms are key to data-intensive systems from which feedback can be gained in areas such as industry, finance, healthcare, science and education. Distributed computing paradigms are a fundamental component of research and innovation for e-infrastructures with the intent of providing advanced computing, storage resources and network connectivity necessary for modern and multidisciplinary science and society. The aim of this paper is to provide a systematic overview and address challenges of distributed computing paradigms that marked and brought revolution into computing science and are able to store and process large amounts of data. As an example of a big data source, the world's largest particle detector, CERN's LHC is analyzed.
Cited by 7Open access
Indira Ms.K., T. Ramya · Advances in parallel computing · 2018
The massively parallel architecture of GPUs, coming from its graphics heritage, is now delivering transformative results for scientists and researchers all over the world. For some of the world's most challenging problems in medical research, drug discovery, weather modeling, and seismic exploration – computation is the ultimate tool. Without it, research would still be confined to trial and error-based physical experiments and observation. Parallel computing: use of multiple computers or processors working together on a common task. Each processor works on its section of the problem. Processors can exchange information. In this we discuss about the parallel architecture, parallel programming models, MPI and OpenMP and CUDA programming.
Cited by 1
Cited by 0
Yanhui Wu, Guoqing Li, Lizhe Wang et al. · 2014
Abstract—Data intensive computing is a common research problem in science, industry and computer academia. In recent twenty years, the explosive growth of science data has appeared all over the world. Typical data intensive computing applications include Internet text data processing, scientific research data processing, large scale graph computing, inverse and perspective problems. Data intensive computing research faces challenges of scalability of massive data management and processing, integrated data processing technology, system management, new transaction demand, unstructured data processing, service mode, good fault tolerance and high availability. Appropriate data intensive computing programming model needs to be suitable for large scale data sets parallel computing, multiple virtual machine task scheduling and constructing new data intensive computing applications. As a typical data intensive application, aerosol inverse computing I/O data volume is analyzed. Although industry and academia has brought some methods for data intensive applications, scientific discovery problems with both data intensive and computation intensive features are still no appropriate solution at present. Keywords- data intensive computing, Big data, parallel processing
Cited by 0
Li, Xiaolin, Qiu, Judy · CERN Document Server (European Organization for Nuclear Research) · 2014
Cited by 0