2012
This chapter explores the evolving analytics discipline, examining the goals of business analytics in the current environment, as well as the strategies and techniques being deployed to extract, transform, load and analyze unstructured data. Success in getting business value from unstructured data analytics is being driven by a race of businesses to survive. The chapter also discusses the importance of text analytics in today's business environment, which should help companies do some analysis that they could not do before, such as content categorization, sentiment analysis, open-ended questions analysis, e-mails and voice of customer analysis, to enrich predictive models. There has been a scramble for semantic search that is becoming more and more predominant for innovative organizations. The pace of technical improvement is rapid, in keeping with the very high value that text analytics is delivering to enterprise users. Text analytics solutions have proved capable of addressing diverse challenges and with business intelligence integration and embedding lines of business applications, text analytics is poised for a broad market adoption.
Cited by 0
Morey J. Haber · Apress eBooks · 2020
Not so long ago, it was much easier to protect your data. Perimeter defenses were in place and meant something, and there were limited pathways to access your organization’s data. Data came in from IT-approved, enterprise-controlled devices and applications. It lived on your servers and in storage arrays. It was protected by walling off the outsiders and trusting your insiders. But IT environments have changed in a big way. Now, data is increasingly collected from applications, users, devices, cloud services, and connected hardware, with dwindling amounts of it under enterprise control. New forms of doing business demand easy access from the outside world. With the emergence of the cloud, your data, users, and applications may not even be on the inside anymore. And “insiders” with access to your data increasingly include third parties who don’t work for your organization at all. The approach to managing the granularity of access to this unstructured data at the file or application layer can be done with privileged access management.
Cited by 0
Ernesto Priego · 2017
¿Para qué poetas en tiempos de unstructured data?
Cited by 0Open access
Paul Geertsema · 2023
Cited by 1
2021
Cited by 0
C WOOTTON · Elsevier eBooks · 2007
Cited by 0
2019
This chapter looks in more detail at the differences between structured and unstructured data. This difference in type of data often drives the selection of certain classes of algorithms for Machine Learning. The chapter explains what makes unstructured data different and why it needs particular attention to handle it properly. It explores common types of unstructured data like images, videos, and text. The chapter describes how images are represented as pixel intensity arrays and how, using computer vision techniques, the data and extract information can be cleansed. It also explores how the same methods can be extended to video, which is a timeseries of images. The chapter also explains how to handle text data using natural language processing and extract information. It presents an example of analyzing audio data using frequency analysis. These methods can be used on their own to extract valuable information from unstructured data.
Cited by 0
Rajeev Motwani, Shubha U. Nabar · ArXiv.org · 2008
In this paper we consider the problem of anonymizing datasets in which each individual is associated with a set of items that constitute private information about the individual. Illustrative datasets include market-basket datasets and search engine query logs. We formalize the notion of k-anonymity for set-valued data as a variant of the k-anonymity model for traditional relational datasets. We define an optimization problem that arises from this definition of anonymity and provide O(klogk) and O(1)-approximation algorithms for the same. We demonstrate applicability of our algorithms to the America Online query log dataset.
Cited by 23Open access
2013
Cited by 0
Nasfi, Rihem, Bronselaer, Antoon · Zenodo (CERN European Organization for Nuclear Research) · 2024
In this project, we work on repairing three datasets: Trials design: This dataset was obtained from the European Union Drug Regulating Authorities Clinical Trials Database (EudraCT) register and the ground truth was created from external registries. In the dataset, multiple countries, identified by the attribute country_protocol_code, conduct the same clinical trials which is identified by eudract_number. Each clinical trial has a title that can help find informative details about the design of the trial. Trials population: This dataset delineates the demographic origins of participants in clinical trials primarily conducted across European countries. This dataset include structured attributes indicating whether the trial pertains to a specific gender, age group or healthy volunteers. Each of these categories is labeled as (`1') or (`0') respectively denoting whether it is included in the trials or not. It is important to note that the population category should remain consistent across all countries conducting the same clinical trial identified by an eudract_number. The ground truth samples in the dataset were established by aligning information about the trial populations provided by external registries, specifically the CT.gov database and the German Trials database. Additionally, the dataset comprises other unstructured attributes that categorize the inclusion criteria for trial participants such as inclusion. Allergens: This dataset contains information about products and their allergens. The data was collected from the German version of the `Alnatura' (Access date: 24 November, 2020), a free database of food products from around the world `Open Food Facts', and the websites: `Migipedia', 'Piccantino', and `Das Ist Drin'. There may be overlapping products across these websites. Each product in the dataset is identified by a unique code. Samples with the same code represent the same product but are extracted from a differentb source. The allergens are indicated by (‘2’) if present, or (‘1’) if there are traces of it, and (‘0’) if it is absent in a product. The dataset also includes information on ingredients in the products. Overall, the dataset comprises categorical structured data describing the presence, trace, or absence of specific allergens, and unstructured text describing ingredients. N.B: Each '.zip' file contains a set of 5 '.csv' files which are part of the afro-mentioned datasets: "{dataset_name}_train.csv": samples used for the ML-model training. (e.g "allergens_train.csv") "{dataset_name}_test.csv": samples used to test the the ML-model performance. (e.g "allergens_test.csv") "{dataset_name}_golden_standard.csv": samples represent the ground truth of the test samples. (e.g "allergens_golden_standard.csv") "{dataset_name}_parker_train.csv": samples repaired using Parker Engine used for the ML-model training. (e.g "allergens_parker_train.csv") "{dataset_name}_parker_train.csv": samples repaired using Parker Engine used to test the the ML-model performance. (e.g "allergens_parker_test.csv")
Cited by 0Open access
Nasfi, Rihem, Bronselaer, Antoon · Zenodo (CERN European Organization for Nuclear Research) · 2024
In this project, we work on repairing three datasets: Trials design: This dataset was obtained from the European Union Drug Regulating Authorities Clinical Trials Database (EudraCT) register and the ground truth was created from external registries. In the dataset, multiple countries, identified by the attribute country_protocol_code, conduct the same clinical trials which is identified by eudract_number. Each clinical trial has a title that can help find informative details about the design of the trial. Trials population: This dataset delineates the demographic origins of participants in clinical trials primarily conducted across European countries. This dataset include structured attributes indicating whether the trial pertains to a specific gender, age group or healthy volunteers. Each of these categories is labeled as (`1') or (`0') respectively denoting whether it is included in the trials or not. It is important to note that the population category should remain consistent across all countries conducting the same clinical trial identified by an eudract_number. The ground truth samples in the dataset were established by aligning information about the trial populations provided by external registries, specifically the CT.gov database and the German Trials database. Additionally, the dataset comprises other unstructured attributes that categorize the inclusion criteria for trial participants such as inclusion. Allergens: This dataset contains information about products and their allergens. The data was collected from the German version of the `Alnatura' (Access date: 24 November, 2020), a free database of food products from around the world `Open Food Facts', and the websites: `Migipedia', 'Piccantino', and `Das Ist Drin'. There may be overlapping products across these websites. Each product in the dataset is identified by a unique code. Samples with the same code represent the same product but are extracted from a differentb source. The allergens are indicated by (‘2’) if present, or (‘1’) if there are traces of it, and (‘0’) if it is absent in a product. The dataset also includes information on ingredients in the products. Overall, the dataset comprises categorical structured data describing the presence, trace, or absence of specific allergens, and unstructured text describing ingredients. N.B: Each '.zip' file contains a set of 5 '.csv' files which are part of the afro-mentioned datasets: "{dataset_name}_train.csv": samples used for the ML-model training. (e.g "allergens_train.csv") "{dataset_name}_test.csv": samples used to test the the ML-model performance. (e.g "allergens_test.csv") "{dataset_name}_golden_standard.csv": samples represent the ground truth of the test samples. (e.g "allergens_golden_standard.csv") "{dataset_name}_parker_train.csv": samples repaired using Parker Engine used for the ML-model training. (e.g "allergens_parker_train.csv") "{dataset_name}_parker_train.csv": samples repaired using Parker Engine used to test the the ML-model performance. (e.g "allergens_parker_test.csv")
Cited by 0Open access
Nasfi, Rihem, Bronselaer, Antoon · Zenodo (CERN European Organization for Nuclear Research) · 2024
In this project, we work on repairing three datasets: Trials design: This dataset was obtained from the European Union Drug Regulating Authorities Clinical Trials Database (EudraCT) register and the ground truth was created from external registries. In the dataset, multiple countries, identified by the attribute country_protocol_code, conduct the same clinical trials which is identified by eudract_number. Each clinical trial has a title that can help find informative details about the design of the trial. Trials population: This dataset delineates the demographic origins of participants in clinical trials primarily conducted across European countries. This dataset include structured attributes indicating whether the trial pertains to a specific gender, age group or healthy volunteers. Each of these categories is labeled as (`1') or (`0') respectively denoting whether it is included in the trials or not. It is important to note that the population category should remain consistent across all countries conducting the same clinical trial identified by an eudract_number. The ground truth samples in the dataset were established by aligning information about the trial populations provided by external registries, specifically the CT.gov database and the German Trials database. Additionally, the dataset comprises other unstructured attributes that categorize the inclusion criteria for trial participants such as inclusion. Allergens: This dataset contains information about products and their allergens. The data was collected from the German version of the `Alnatura' (Access date: 24 November, 2020), a free database of food products from around the world `Open Food Facts', and the websites: `Migipedia', 'Piccantino', and `Das Ist Drin'. There may be overlapping products across these websites. Each product in the dataset is identified by a unique code. Samples with the same code represent the same product but are extracted from a differentb source. The allergens are indicated by (‘2’) if present, or (‘1’) if there are traces of it, and (‘0’) if it is absent in a product. The dataset also includes information on ingredients in the products. Overall, the dataset comprises categorical structured data describing the presence, trace, or absence of specific allergens, and unstructured text describing ingredients. N.B: Each '.zip' file contains a set of 5 '.csv' files which are part of the afro-mentioned datasets: "{dataset_name}_train.csv": samples used for the ML-model training. (e.g "allergens_train.csv") "{dataset_name}_test.csv": samples used to test the the ML-model performance. (e.g "allergens_test.csv") "{dataset_name}_golden_standard.csv": samples represent the ground truth of the test samples. (e.g "allergens_golden_standard.csv") "{dataset_name}_parker_train.csv": samples repaired using Parker Engine used for the ML-model training. (e.g "allergens_parker_train.csv") "{dataset_name}_parker_train.csv": samples repaired using Parker Engine used to test the the ML-model performance. (e.g "allergens_parker_test.csv")
Cited by 0Open access
W.H. Inmon, Daniel Linstedt, Mary Levins · Elsevier eBooks · 2019
Cited by 0
W.H. Inmon, Daniel Linstedt · Elsevier eBooks · 2015
Cited by 0
Jacques Péré-Laperne · theses.fr (ABES) · 2019
Cette thèse traite de la restructuration des documents déstructurés de type PDF contenant des éléments graphiques tels que les schémas, les plans et les dessins, dans l’objectif de les restructurer. En nous appuyant sur la méthode KDD (Knowledge Discovery in Database) pour la restructuration des données, nous introduisons la méthode (A)KDD (Antropocentric Knowledge Discovery in Database) que nous avons développé et qui est dérivée de la méthode KDD en ajoutant l’aspect incrémental et l’aspect centré sur l’utilisateur. Nous présentons, en particulier, une technique fondée sur le principe du tri par paquet pour extraire efficacement les symboles graphiques contenus dans un document PDF. Elle est comparée aux résultats de Puglissi sur les chaînes de caractères. Puis, nous formulons l’hypothèse selon laquelle la prise en compte de l’ordre chronologique présent dans les fichiers PDF dans le processus incrémental améliore la restructuration des documents. Nous montrons la validité de cette hypothèse sur un certain nombre d’exemples. Enfin, nous montrons l’efficacité du processus pour identifier les symboles en même temps que les équipotentielles. Le mémoire se conclut en montrant les avancées et les limites de la solution de la méthode (A)KDD et nous proposons des perspectives.
Cited by 0Open access
Alberto Bacchelli, Lanza, Michele · reroDoc Digital Library · 2013
Our thesis is that the analysis of unstructured data supports software understanding and evolution analysis, and complements the data mined from structured sources. To this aim, we implemented the necessary toolset and investigated methods for exploring, exposing, and exploiting unstructured data.To validate our thesis, we focused on development email data. We found two main challenges in using it to support program comprehension and software development: The disconnection between emails and code artifacts and the noisy and mixed-language nature of email content. We tackle these challenges proposing novel approaches. First, we devise lightweight techniques for linking email data to code artifacts. We use these techniques for creating a tool to support program comprehension with email data, and to create a new set of email based metrics to improve existing defect prediction approaches. Subsequently, we devise techniques for giving a structure to the content of email and we use this structure to conduct novel software analyses to support program comprehension. In this dissertation we show that unstructured data, in the form of development emails, is a valuable addition to structured data and, if correctly mined, can be used successfully to support software engineering activities.
Cited by 1
Rizwan, Muhammad · 2016
Cited by 1Open access
Yang Lu, Xing Fang, Justin Zhan · 2014
When time or computational resources is a constraint, dealing with large amount of data can be painful for many organizations. In this paper, we proposed a new concept called Data Readiness Level(DRL). It will measures the readiness of a file very quickly. We define readiness as ready for immediate analytical purposes. DRL is pair-wised measure, one is PKSS value, another is PVI value. One can see PKSS as the degree of similarity to the objective. PVI exhibits how much valuable information the file has. In the real word, not every data contains valuable information. Even if it is, it is not guaranteed that it will be relevant to the objective. With the aid of DRL, users can simply analyze those data with higher DRL values when time is a constraint. We collected 40,000 PDF files by hand from IEEE Xplore digital library and spriger. The experimental result was quite encouraging.
Cited by 0
Neil McNaughton · Oil IT Journal. · 2017
Open source software’s field day at UK upstream linguistic analysis testbed.
Cited by 0
W.H. Inmon, Anthony Nesavich · 2007
“The authors, the best minds on the topic, are breaking new ground. They show how every organization can realize the benefits of a system that can search and present complex ideas or data from what has been a mostly untapped source of raw data.”--Randy Chalfant, CTO, Sun MicrosystemsThe Definitive Guide to Unstructured Data Management and Analysis--From the World's Leading Information Management ExpertA wealth of invaluable information exists in unstructured textual form, but organizations have found it difficult or impossible to access and utilize it. This is changing rapidly: new approaches finally make it possible to glean useful knowledge from virtually any collection of unstructured data.William H. Inmon--the father of data warehousing--and Anthony Nesavich introduce the next data revolution: unstructured data management. Inmon and Nesavich cover all you need to know to make unstructured data work for your organization. You'll learn how to bring it into your existing structured data environment, leverage existing analytical infrastructure, and implement textual analytic processing technologies to solve new problems and uncover new opportunities. Inmon and Nesavich introduce breakthrough techniques covered in no other book--including the powerful role of textual integration, new ways to integrate textual data into data warehouses, and new SQL techniques for reading and analyzing text. They also present five chapter-length, real-world case studies--demonstrating unstructured data at work in medical research, insurance, chemical manufacturing, contracting, and beyond.This book will be indispensable to every business and technical professional trying to make sense of a large body of unstructured text: managers, database designers, data modelers, DBAs, researchers, and end users alike.Coverage includes What unstructured data is, and how it differs from structured data First generation technology for handling unstructured data, from search engines to ECM--and its limitations Integrating text so it can be analyzed with a common, colloquial vocabulary: integration engines, ontologies, glossaries, and taxonomies Processing semistructured data: uncovering patterns, words, identifiers, and conflicts Novel processing opportunities that arise when text is freed from context Architecture and unstructured data: Data Warehousing 2.0 Building unstructured relational databases and linking them to structured data Visualizations and Self-Organizing Maps (SOMs), including Compudigm and Raptor solutions Capturing knowledge from spreadsheet data and email Implementing and managing metadata: data models, data quality, and more William H. Inmon is founder, president, and CTO of Inmon Data Systems. He is the father of the data warehouse concept, the corporate information factory, and the government information factory. Inmon has written 47 books on data warehouse, database, and information technology management; as well as more than 750 articles for trade journals such as Data Management Review, Byte, Datamation, and ComputerWorld. His b-eye-network.com newsletter currently reaches 55,000 people.Anthony Nesavich worked at Inmon Data Systems, where he developed multiple reports that successfully query unstructured data.Preface xvii 1i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ Unstructured Textual Data in the Organization 1 2i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ The Environments of Structured Data and Unstructured Data 15 3i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ First Generation Textual Analytics 33 4i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ Integrating Unstructured Text into the Structured Environment 47 5i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ Semistructured Data 73 6i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ Architecture and Textual Analytics 83 7i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ The Unstructured Database 95 8i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ Analyzing a Combination of Unstructured Data and Structured Data 113 9i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ i¾ Analyzing Text Through Visualization 12710i¾ i¾ i¾ i¾ i¾ i¾ i¾ Spreadsheets and Email 13511i¾ i¾ i¾ i¾ i¾ i¾ i¾ Metadata in Unstructured Data 14712i¾ i¾ i¾ i¾ i¾ i¾ i¾ A Methodology for Textual Analytics 16313i¾ i¾ i¾ i¾ i¾ i¾ i¾ Merging Unstructured Databases into the Data Warehouse 17514i¾ i¾ i¾ i¾ i¾ i¾ i¾ Using SQL to Analyze Text 18515i¾ i¾ i¾ i¾ i¾ i¾ i¾ Case Study--Textual Analytics in Medical Research 19516i¾ i¾ i¾ i¾ i¾ i¾ i¾ Case Study--A Database for Harmful Chemicals 20317i¾ i¾ i¾ i¾ i¾ i¾ i¾ Case Study--Managing Contracts Through an Unstructured Database 20918i¾ i¾ i¾ i¾ i¾ i¾ i¾ Case Study--Creating a Corporate Taxonomy (Glossary) 21519i¾ i¾ i¾ i¾ i¾ i¾ i¾ Case Study--Insurance Claims 219Glossary 227Index 233
Cited by 40