Pavel Češka · Digital Repository (National Repository of Grey Literature) · 2006
The bachelor thesis focuses on basic pre-processing (tokenization and segmentation) of Czech texts, mainly for purposes of Czech internet corpus. The texts for this corpus will be automatically obtained from the world wide web, therefore the segmentation is preceeded by character encoding recognition, cleaning and language identification. We performed experiments with two methods of language identification and present their results. The first method is based on comparison of the most frequent n-grams (substrings of length n) extracted from an unknown document and a large Czech corpus. The second one employs a model estimating word probabilities by conditional probabilities of trigrams estimated on the same corpus. For wider usage, we developed a module for tokenization and identification of sentences boundaries by a decision tree analysis of the nearest context of potential sentence boundaries and utilizing extensive lists of Czech abbreviations. The decision tree was trained on a set of manually processed data. Its evaluation was based on independent human judgements and results are presented in the work.
Cited by 0
Taeho Jo · Studies in big data · 2024
Cited by 0
SpringerReference · 2011
Cited by 0
Matthew J. Carroll · Open MIND · 2020
Cited by 0
Pavel Češka · Digital Repository (National Repository of Grey Literature) · 2006
The bachelor thesis focuses on basic pre-processing (tokenization and segmentation) of Czech texts, mainly for purposes of Czech internet corpus. The texts for this corpus will be automatically obtained from the world wide web, therefore the segmentation is preceeded by character encoding recognition, cleaning and language identification. We performed experiments with two methods of language identification and present their results. The first method is based on comparison of the most frequent n-grams (substrings of length n) extracted from an unknown document and a large Czech corpus. The second one employs a model estimating word probabilities by conditional probabilities of trigrams estimated on the same corpus. For wider usage, we developed a module for tokenization and identification of sentences boundaries by a decision tree analysis of the nearest context of potential sentence boundaries and utilizing extensive lists of Czech abbreviations. The decision tree was trained on a set of manually processed data. Its evaluation was based on independent human judgements and results are presented in the work.
Cited by 0
Taeho Jo · Studies in big data · 2018
Cited by 0
Natural Language Processing · 2013
Cited by 0
Rosario Torres-Díaz, Raysil Mosquera Suárez, Marlis Ontivero Ortega et al. · PsycTESTS Dataset · 2020
Cited by 0
Christopher Fournier · uO Research (University of Ottawa) · 2013
This thesis investigates the evaluation of automatic and manual text segmentation. Text segmentation is the process of placing boundaries within text to create segments according to some task-dependent criterion. An example of text segmentation is topical segmentation, which aims to segment a text according to the subjective definition of what constitutes a topic. A number of automatic segmenters have been created to perform this task, and the question that this thesis answers is how to select the best automatic segmenter for such a task. This requires choosing an appropriate segmentation evaluation metric, confirming the reliability of a manual solution, and then finally employing an evaluation methodology that can select the automatic segmenter that best approximates human performance. A variety of comparison methods and metrics exist for comparing segmentations (e.g., WindowDiff, Pk), and all save a few are able to award partial credit for nearly missing a boundary. Those comparison methods that can award partial credit unfortunately lack consistency, symmetricity, intuition, and a host of other desirable qualities. This work proposes a new comparison method named boundary similarity (B) which is based upon a new minimal boundary edit distance to compare two segmentations. Near misses are frequent, even among manual segmenters (as is exemplified by the low inter-coder agreement reported by many segmentation studies). This work adapts some inter-coder agreement coefficients to award partial credit for near misses using the new metric proposed herein, B. The methodologies employed by many works introducing automatic segmenters evaluate them simply in terms of a comparison of their output to one manual segmentation of a text, and often only by presenting nothing other than a series of mean performance values (along with no standard deviation, standard error, or little if any statistical hypothesis testing). This work asserts that one segmentation of a text cannot constitute a “true” segmentation; specifically, one manual segmentation is simply one sample of the population of all possible segmentations of a text and of that subset of desirable segmentations. This work further asserts that an adapted inter-coder agreement statistics proposed herein should be used to determine the reproducibility and reliability of a coding scheme and set of manual codings, and then statistical hypothesis testing using the specific comparison methods and methodologies demonstrated herein should be used to select the best automatic segmenter. This work proposes new segmentation evaluation metrics, adapted inter-coder agreement coefficients, and methodologies. Most important, this work experimentally compares the state-or-the-art comparison methods to those proposed herein upon artificial data that simulates a variety of scenarios and chooses the best one (B). The ability of adapted inter-coder agreement coefficients, based upon B, to discern between various levels of agreement in artificial and natural data sets is then demonstrated. Finally, a contextual evaluation of three automatic segmenters is performed using the state-of-the art comparison methods and B using the methodology proposed herein to demonstrate the benefits and versatility of B as opposed to its counterparts.
Cited by 5Open access
Robin Cabeza Ruiz · Sistemas y Telemática · 2016
There are two approaches for text segmentation by language: first, assuming that language changes happen in the “border” between sentences (never within a sentence); second, assuming that language changes can happen anyplace in the text. This work presents methods for both types of text’s segmentation by languages. On the first proposal, the text is initially segmented by sentence, then the language of each sentence is obtained; the second proposal is an adaptation of hidden Markov model to this task. Both cases, according to results obtained in experimental proofs, exceed the state of art.
Cited by 2Open access
Jing Shi · 2006
Text segmentation is very important in information retrieval,automatic summarization,discourse analysis,and many other fields.Static written text,speech text and dynamic text can be segmented.The granularity of segmentation is varied for different purpose.Direct and indirect evaluations are applied to assess algorithms.The current work on segmentation approaches and direct evaluation methods are generalized on the basis of lots of literatures.The paper presents the status of text segmentation,points out the problems and future research.
Cited by 0
Boris Dadachev, Alexander Balinsky, Helen Balinsky · 2014
Automatic text segmentation, which is the task of breaking a text into topically-consistent segments, is a fundamental problem in Natural Language Processing, Document Classification and Information Retrieval. Text segmentation can significantly improve the performance of various text mining algorithms, by splitting heterogeneous documents into homogeneous fragments and thus facilitating subsequent processing. Applications range from screening of radio communication transcripts to document summarization, from automatic document classification to information visualization, from automatic filtering to security policy enforcement - all rely on, or can largely benefit from, automatic document segmentation. In this article, a novel approach for automatic text and data stream segmentation is presented and studied. The proposed automatic segmentation algorithm takes advantage of feature extraction and unusual behaviour detection algorithms developed in [4, 5]. It is entirely unsupervised and flexible to allow segmentation at different scales, such as short paragraphs and large sections. We also briefly review the most popular and important algorithms for automatic text segmentation and present detailed comparisons of our approach with several of those state-of-the-art algorithms.
Cited by 6
Robin Cabeza Ruiz · Greater South Information System · 2016
There are two approaches for text segmentation by language: first, assuming that language changes happen in the "border" between sentences (never within a sentence); second, assuming that language changes can happen anyplace in the text. This work presents methods for both types of text's segmentation by languages. On the first proposal, the text is initially segmented by sentence, then the language of each sentence is obtained; the second proposal is an adaptation of hidden Markov model to this task. Both cases, according to results obtained in experimental proofs, exceed the state of art.
Cited by 0Open access
Эбраль Александр Владимирович, Ebral Aleksandr · Research Repository Saint Petersburg State University (Saint Petersburg State University) · 2018
В работе рассматривается проблема сегментации текста на сложном фоне. Предлагается подход к предобработке изображений, содержащих текст, а также предлагаются и тестируются подходы к сегментации на основе сверточных нейронных сетей.
Cited by 0Open access
Qin Wenfeng, Wu Yanyi · 2014
Cited by 7
Haiyang Yu, Xiaocong Wang, Ke Niu et al. · 2023
Text segmentation is a crucial aspect of various text-related tasks, including text erasing, text editing, and font style transfer. In recent years, multiple text segmentation datasets, such as TextSeg focusing on Latin text segmentation and BTS on bilingual text segmentation, have been proposed. However, existing methods either disregard the annotations of text location or directly use pre-trained text detectors. In general, these methods cannot fully utilize the annotations of text location in the datasets. To explicitly incorporate text location information to guide text segmentation, we propose an end-to-end text-focused segmentation framework, where text detection and segmentation are jointly optimized. In the proposed framework, we first extract multi-level global visual features through residual convolution blocks and then predict the mask of text areas using a text detection head. Subsequently, we develop a text-focused module that compels the model to pay more attention to text areas. Specifically, we introduce two types of attention masks to extract corresponding features: text-aware and instance-aware features. Finally, we employ hierarchical Transformer encoders to fuse multi-level features and predict the text mask with a text segmentation head. To evaluate the effectiveness of our method, we conduct experiments on six text segmentation benchmarks. The experimental results demonstrate that the proposed method outperforms the previous state-of-the-art (SOTA) methods by a clear margin in most cases. The code and supplementary materials are available at https://github.com/FudanVI/FudanOCR/tree/main/text-focused-Transformers https://github.com/FudanVI/FudanOCR/tree/main/text-focused-Transformers.
Cited by 8
Thien, Nguyen · JAIST Repository · 2012
Cited by 0
Chengdong Zhang, Guee-Sang Lee · 2011
In this paper, text line segmentation based on 2-D Tensor Voting is proposed, 2-D tensor voting is used to remove outliers and located center points of connected components. The saliency values and direction of normal vectors are represented information of tensors for segmentation. The text images obtained from HIT-MW dataset.
Cited by 3
이종혁 · Open Access System for Information Sharing (Pohang University of Science and Technology) · 2018
Cited by 0
Ekaterina Mayatskaya · 2024
In text document classification, researchers often deal with long documents, which creates difficulties in efficient text processing. When processing long sequences such as long documents, BERT and similar transformers face a limitation on the amount of input data. This paper addresses the problem on handling long texts that arises when working with transformer models. Most often, researchers perform text truncation when dealing with long documents. In this paper, we segment the input text into several fragments and feed each of them into the base model. Then we merge the resulting vector representations for each chunk of text. To handle long vectors efficiently, we segment the texts into batches based on the length of the sequence and create a mask before feeding the data into the recurrent neural network. We performed a performance analysis and demonstrated that our method achieved competitive results on two medical datasets, showing a significant reduction in model training time, compared to baseline methods.
Cited by 0