Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 2,804 papers for “distributed query processing”

Efficient query processing in distributed search engines

Simon Jonassen · ACM SIGIR Forum · 2012

Web search engines have to deal with a rapidly increasing amount of information, high query loads and tight performance constraints. The success of a search engine depends on the speed with which it answers queries (efficiency) and the quality of its answers (effectiveness). These two metrics have a large impact on the operational costs of the search engine and the overall user satisfaction, which determine the revenue of the search engine. In this context, any improvement in query processing efficiency can reduce the operational costs and improve user satisfaction, hence improve the overall benefit. In this thesis, we elaborate on query processing efficiency, address several problems within partitioned query processing, pruning and caching and propose several novel techniques: First, we look at term-wise partitioned indexes and address the main limitations of the state-of-the-art query processing methods. Our first approach combines the advantage of pipelined and traditional (non-pipelined) query processing. This approach assumes one disk access per posting list and traditional term-at-a-time processing. For the second approach, we follow an alternative direction and look at document-at-a-time processing of sub-queries and skipping. Subsequently, we present several skipping extensions to pipelined query processing, which as we show can improve the query processing performance and/or the quality of results. Then, we extend one of these methods with intra-query parallelism, which as we show can improve the performance at low query loads. Second, we look at skipping and pruning optimizations designed for a monolithic index. We present an efficient self-skipping inverted index designed for modern index compression methods and several query processing optimizations. We show that these optimizations can provide a significant speed-up compared to a full (non-pruned) evaluation and reduce the performance gap between disjunctive (OR) and conjunctive (AND) queries. We also propose a linear programming optimization that can further improve the I/O, decompression and computation efficiency of Max-Score. Third, we elaborate on caching in Web search engines in two independent contributions. First, we present an analytical model that finds the optimal split in a static memory-based two-level cache. Second, we present several strategies for selecting, ordering and scheduling prefetch queries and demonstrate that these can improve the efficiency and effectiveness of Web search engines. We carefully evaluate our ideas either using a real implementation or by simulation using real-world text collections and query logs. Most of the proposed techniques are found to improve the state-of-the-art in the conducted empirical studies. However, the implications and applicability of these techniques in practice need further evaluation in real-life settings. This dissertation was completed at the Department of Computer and Information Science at the Norwegian University of Science and Technology (NTNU) under advise of Prof. Svein Erik Bratsberg, Dr. Øystein Torbjørnsen and Dr. Magnus Lie Hetland. Some of the work was done in collaboration with Yahoo! Research Barcelona and mentored by Prof. Ricardo Baeza-Yates and Dr. B. Barla Cambazoglu. Prof. Alistair Moffat (University of Melbourne), Dr. Christina Lioma (University of Copenhagen) and Prof. Kjell Bratsbergsengen (NTNU) served as dissertation committee member. Available online at: http://www.idi.ntnu.no/research/doctor_theses/simonj.pdf.

Cited by 1

Distributed Data Summaries for Approximate Query Processing

Katja Hose, Daniel Klan, Kai-uwe Sattler · 2006

Evolving from heterogeneous database systems one of the main problems in Peer Data Management Systems (PDMS) is distributed query processing. With the absence of global knowledge such strategies have to focus on routing the query efficiently to only those peers that are most likely to contribute to the final result. Using routing indexes is one possibility to achieve this. Since data may change over time these structures have to be updated and maintained which can be very expensive. In this paper, we present a novel kind of routing indexes that enables efficient query routing. Furthermore, we propose a threshold based update strategy that can help to reduce maintenance costs by far. We exemplify the benefit of these indexes using a distributed skyline strategy as an example. Finally, we show how relaxing exactness requirements, that are usually posed on results, can compensate the use of slightly outdated index information. 1

Cited by 0

Distributed Spatiotemporal Trajectory Query Processing in SQL

Mohamed Bakli, Mahmoud Sakr, Esteban Zimányi · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · 2020

Nowadays, the collection of moving object data is significantly increasing due to the ubiquity of GPS-enabled devices. Managing and analyzing this kind of data is crucial in many application domains, including social mobility, pandemics, and transportation. In previous work, we have proposed the MobilityDB moving object database system. It is a production-ready system, that is built on top of PostgreSQL and PostGIS. It accepts SQL queries and offers most of the common spatiotemporal types and operations. In this paper, to address the scalability requirement of big data, we provide an architecture and an implementation of a distributed moving object database system based on MobilityDB. More specifically, we define: (1) an architecture for deploying a distributed MobilityDB database on a cluster using readily available tools, (2) two alternative trajectory data partitioning and index partitioning methods, and (3) a query optimizer that is capable of distributing spatiotemporal SQL queries over multiple MobilityDB instances. The overall outcome is that the cluster is managed in SQL at the run-time and that the user queries are transparently distributed and executed. This is validated with experiments using a real dataset, which also compares MobilityDB with other relevant systems.

Cited by 15Open access

Distributed complex event processing with query rewriting

Nicholas Poul Schultz-Møller, Matteo Migliavacca, Peter Pietzuch · 2009

The nature of data in enterprises and on the Internet is changing. Data used to be stored in a database first and queried later. Today timely processing of new data, represented as events, is increasingly valuable. In many domains, complex event processing (CEP) systems detect patterns of events for decision making. Examples include processing of environmental sensor data, trades in financial markets and RSS web feeds. Unlike conventional database systems, most current CEP systems pay little attention to query optimisation. They do not rewrite queries to more efficient representations or make decisions about operator distribution, limiting their overall scalability.

Cited by 188

Declarative networking with distributed recursive query processing

Boon Thau Loo, Tyson Condie, Minos Garofalakis et al. · 2006

There have been recent proposals in the networking and distributed systems literature on declarative networking, where network protocols are declaratively specified using a recursive query language. This represents a significant new application area for recursive query processing technologies from databases. In this paper, we extend upon these recent proposals in the following ways. First, we motivate and formally define the NDlog language for declarative network specifications. We introduce the concept of link-restricted rules, which can be syntactically guaranteed to be executable via single-node derivations and message passing on an underlying network graph. Second, we introduce and prove correct relaxed versions of the traditional semi-naive execution technique that overcome fundamental problems of traditional semi-naive evaluation in an asynchronous distributed setting. Third, we consider the dynamics of network state, and formalize the “eventual consistency” of our programs even when bursts of updates can arrive in the midst of query execution. Fourth, we present a number of query optimization opportunities that arise in the declarative networking context, including applications of traditional techniques and new optimizations. Last, we present evaluation results based on an implementation of the above ideas in the P2 declarative networking system, running on 100 machines over the Emulab network testbed.

Cited by 6

SPARQL Distributed Query Processing over Linked Data

Abdoul Macina · theses.fr (ABES) · 2018

De plus en plus de sources de données liées sont publiées à travers le Web en s'appuyant sur les technologies du Web sémantique, formant ainsi un large réseau de données distribuées. Cependant il est difficile pour les consommateurs de données de profiter de la richesse de ces données, compte tenu de leur distribution, de l'augmentation de leur volume et de l'autonomie des sources de données. Les moteurs fédérateurs de données permettent d'interroger ces sources de données en utilisant des techniques de traitement de requêtes distribuées. Cependant, une mise en œuvre naïve de ces techniques peut générer un nombre considérable de requêtes distantes et de nombreux résultats intermédiaires entraînant ainsi un long temps de traitement des requêtes et des communications réseau coûteuse. Par ailleurs, la sémantique des requêtes distribuées est souvent ignorée. L'expressivité des requêtes, le partitionnement des données et leur réplication sont d'autres défis auxquels doivent faire face les moteurs de requêtes. Pour répondre à ces défis, nous avons d'abord proposé une sémantique des requêtes distribuées compatible avec les standards SPARQL et RDF qui préserve l’expressivité de SPARQL. Nous avons ensuite présenté plusieurs stratégies d'optimisation pour un moteur de requêtes fédérées qui interroge de manière transparente des sources de données distribuées. La performance de ces optimisations est évaluée sur une implémentation d’un moteur de requêtes distribuées SPARQL

Cited by 1Open access

Query Decomposition for Distributed Complex Event Processing

Akili, Samira · edoc Publication server (Humboldt University of Berlin) · 2025

Complex Event Processing (CEP) Systeme evaluieren Abfragen über hochfrequenten Ereignisströmen, um relevante Muster darin zu erkennen. In Szenarien mit verteilten Ereignisquellen, etwa in Sensornetzwerken oder IoT-Anwendungen, gewinnt CEP zunehmend an Bedeutung. Bestehende Ansätze nutzen das Optimierungspotenzial in solchen Netzwerken nur begrenzt aus. Diese Dissertation stellt Techniken zur Optimierung verteilter Mustererkennung vor, die darauf abzielen, die Netzwerkkosten, also die Anzahl der im Netzwerk ausgetauschten Nachrichten, zu minimieren. Dazu nutzen wir sowohl die Semantik der Abfragesprache als auch Netzwerkeigenschaften aus. Insbesondere führen wir eine Dekompositionstechnik von Mustern ein, mit der korrekte Teilmuster spezifiziert und für die verteilte Erkennung genutzt werden können. Darauf aufbauend führen wir Modelle für die verteilte Mustererkennung in Netzwerken ein: MuSE- und INEv-Graphen. Während MuSE-Graphen eine vollständige Netzwerktopologie voraussetzen, erlauben INEv-Graphen beliebige Netzwerktopologien und spezifizieren die effizienten Weiterleitung von Ereignissen. Zusätzlich untersuchen wir die Ereignisübertragung. Während Ereignisse üblicherweise sofort nach ihrer Generierung versandt werden, können Kommunikationskosten reduziert werden, indem die Übertragung erst durch eine Anfrage geschieht. Hierzu führen wir PrePP-Pläne ein, die die Anfrage-basierte Übertragung von Ereignissen mit Prädikat-basierter Filterung modellieren. Schließlich übertragen wir unsere Dekompositionstechniken auf die parallele Mustererkennung. Dabei analysieren wir Durchsatzengpässe bei Rechenknoten und entwickeln ein Kostenmodell zur Musterdetektion in Clusterumgebungen. Die resultierenden DecoPa-Pläne ermöglichen eine durchsatzoptimierte, parallele Mustererkennung. Für alle Modelle definieren wir formale Eigenschaften, beweisen Korrektheit und geben Konstruktionsalgorithmen an. Experimente zeigen die Überlegenheit unserer Ansätze gegenüber bestehenden Verfahren.

Cited by 0Open access

Efficient query processing in distributed database systems

Αλέξανδρος Πανάρετος · 2015

Στο πλαίσιο της παρούσας Διδακτορικής Διατριβής χρησιμοποιήθηκε εκτενώς η Κατανεμημένη Δομή ART (Autonomous Range Tree) η οποία αποτέλεσε την βάση για την υλοποίηση μίας σειράς αποδοτικών συστημάτων κατανεμημένης διαχείρισης δεδομένων σε ποικίλα πεδία εφαρμογών, όπως: εξομοιωτές μέτρησης της απόδοσης διαφορετικών αλγορίθμων κατανεμημένης δόμησης (P2P Infrastructures or overlays), διαχείριση ψηφιακών δικαιωμάτων σε υποδομές ανταλλαγής προστατευμένου ψηφιακού υλικού, διαχείριση δεδομένων που αναπαριστούν επίπεδα ενέργειας σε δίκτυα αισθητήρων και διαχείριση χώρο-χρονικών δεδομένων που αναπαριστούν τροχιές κινούμενων αντικειμένων στο επίπεδο.

Cited by 0Open access

Distributed Query Processing for Mobile Surveillance ABSTRACT

Stewart Greenhill, Svetha Venkatesh · 2008

Addressing core issues in mobile surveillance, we present an architecture for querying and retrieving distributed, semipermanent multi-modal data through challenged networks with limited connectivity. The system provides a rich set of queries for spatio-temporal querying in a surveillance context, and uses the network availability to provide best quality of service. It incrementally and adaptively refines the query, using data already retrieved that exists on static platforms and on-demand data that it requests from mobile platforms. We demonstrate the system using a real surveillance system on a mobile 20 bus transport network coupled with static bus depot infrastructure. In addition, we show the robustness of the system in handling different conditions in the underlying infrastructure by running simulations on a real, but historic dataset collected in an offline manner.

Cited by 0