Piotr Malak · Advances in human and social aspects of technology book series · 2018
Digital humanities and information visualization rely on huge sets of digital data. Those data are mostly delivered in the text form. Although computational linguistics provides a lot of valuable tools for text processing, the initial phase (text preprocessing) is very involved and time-consuming. The problems arise due to a human factor – they are not always errors; there is also inconsistency in forms, affecting data quality. In this chapter, the author describes and discusses the main issues that arise during the preprocessing phase of textual data gathering for InfoVis. Chosen examples of InfoVis applications are presented. Except for problems with raw, original data, solutions are also referred. Canonical approaches used in text preprocessing and common issues affecting the process and ways to prevent them are also presented. The quality of data from different sources is also discussed. The content of this chapter is a result of a few years of practical experience in natural language processing gained during realization of different projects and evaluation campaigns.
Cited by 2
Murugan Anandarajan, Chelsey Hill, Thomas Nolan · Advances in analytics and data science · 2018
Cited by 69
Linzbach, Stephan · GESIS Methods Hub · 2025
Cited by 0Open access
Ya-Han Hu (3522998), Jeng-Hsiu Hung (522277), Li-Yu Hu (398469) et al. · Figshare · 2023
<div><p>Background</p><p>The prevalence of violence in acute psychiatric wards is a critical concern. According to a meta-analysis investigating violence in psychiatric inpatient units, researchers estimated that approximately 17% of inpatients commit one or more acts of violence during their stay. Inpatient violence negatively affects health-care providers and patients and may contribute to high staff turnover. Therefore, predicting which psychiatric inpatients will commit violence is of considerable clinical significance.</p><p>Objective</p><p>The present study aimed to estimate the violence rate for psychiatric inpatients and establish a predictive model for violence in psychiatric inpatients.</p><p>Methods</p><p>We collected the structured and unstructured data from Chinese nursing electronic medical records (EMRs) for the violence prediction. The data was obtained from the psychiatry department of a regional hospital in southern Taiwan, covering the period between January 2008 and December 2018. Several text mining and machine learning techniques were employed to analyze the data.</p><p>Results</p><p>The results demonstrated that the rate of violence in psychiatric inpatients is 19.7%. The patients with violence in psychiatric wards were generally younger, had a more violent history, and were more likely to be unmarried. Furthermore, our study supported the feasibility of predicting aggressive incidents in psychiatric wards by using nursing EMRs and the proposed method can be incorporated into routine clinical practice to enable early prediction of inpatient violence.</p><p>Conclusions</p><p>Our findings may provide clinicians with a new basis for judgment of the risk of violence in psychiatric wards.</p></div>
Cited by 0Open access
Vasyl Starko, Andriy Rysin, Maria Shvedova · 2021 IEEE 16th International Conference on Computer Sciences and Information Technologies (CSIT) · 2021
GRAC is a large reference corpus of Ukrainian. Some texts come from OCR-ed documents and contain errors that must be fixed before inclusion in the corpus. Ukrainian texts also contain foreign-language (mostly Russian) fragments that need to be identified and either excised or left in place, depending on their length. To address these key issues, the GRAC team has developed algorithms and tools and applied them to corpus texts. A discussion of the specifics is provided and potential for further development is outlined. The developed tools have been made available online to the NLP and CL community.
Cited by 5
Hongli Niu (11478268), Qiaoying Pan (17457893), Kunliang Xu (17457896) · Figshare · 2023
<div><p>The prediction of stock prices has long been a captivating subject in academic research. This study aims to forecast the prices of prominent stocks in five key industries of the Chinese A-share market by leveraging the synergistic power of deep learning techniques and investor sentiment analysis. To achieve this, a sentiment multi-classification dataset is for the first time constructed for China’s stock market, based on four types of sentiments in modern psychology. The significant heterogeneity of sentiment changes in the sectors’ leading stock markets is trained and mined using the Bi-LSTM-ATT model. The impact of multi-classification investor sentiment on stock price prediction was analyzed using the CNN-Bi-LSTM-ATT model. It finds that integrating sentiment indicators into the prediction of industry leading stock prices can enhance the accuracy of the model. Drawing upon four fundamental sentiment types derived from modern psychology, our dataset provides a comprehensive framework for analyzing investor sentiment and its impact on forecasting the stock prices of China’s A-share market.</p></div>
Cited by 0Open access
Цимбалюк, Г., Tsymbaliuk, G. · ELARTU (Ternopil National Technical University) · 2025
Cited by 0
2010
Cited by 1
Anela Rački · ODRAZ (University of Zagreb Faculty of Humanities and SocialSciences) · 2024
Pretprocesiranje teksta važna je tehnika koja se koristi za čišćenje i transformiranje nestrukturiranih tekstualnih podatka i njihovu pripremu za daljnju analizu. Standardni postupak pretprocesiranja teksta uključuje uklanjanje interpunkcije, transformiranje velikih u mala slova, uklanjanje stop riječi i HTML oznaka, korjenovanje, tokenizaciju i lematizaciju. Cilj ovog diplomskog rada je izgradnja aplikacije koja će skratiti proces pretprocesiranja teksta i omogućiti njenim korisnicima bolji fokus na istraživanje i obradu prirodnog jezika. Rad se sastoji od teorijskog i praktičnog djela. Teorijski dio fokusiran je na važnost pretprocesiranja teksta i tehnike korištene za pretprocesiranje. U praktičnom djelu prikazana je metodologija izgradnje aplikacije u programskom jeziku Python. Zaključno je provedeno testiranje i evaluacija aplikacije.
Cited by 0
Hongli Niu (11478268), Qiaoying Pan (17457893), Kunliang Xu (17457896) · Figshare · 2023
<div><p>The prediction of stock prices has long been a captivating subject in academic research. This study aims to forecast the prices of prominent stocks in five key industries of the Chinese A-share market by leveraging the synergistic power of deep learning techniques and investor sentiment analysis. To achieve this, a sentiment multi-classification dataset is for the first time constructed for China’s stock market, based on four types of sentiments in modern psychology. The significant heterogeneity of sentiment changes in the sectors’ leading stock markets is trained and mined using the Bi-LSTM-ATT model. The impact of multi-classification investor sentiment on stock price prediction was analyzed using the CNN-Bi-LSTM-ATT model. It finds that integrating sentiment indicators into the prediction of industry leading stock prices can enhance the accuracy of the model. Drawing upon four fundamental sentiment types derived from modern psychology, our dataset provides a comprehensive framework for analyzing investor sentiment and its impact on forecasting the stock prices of China’s A-share market.</p></div>
Cited by 0Open access
Uwe D.Reichel, Hartmut R. Pfitzinger · 2006
In this paper we describe our text preprocessing modules for English text-to-speech synthesis. These modules comprise rule-based text normalization subsuming sentence segmentation and normalization of non-standard words, statistical part-of-speech tagging, and statistical syllabification, grapheme-to-phoneme conversion, and word stress assignment relying in parts on rule-based morphological analysis.
Cited by 0
Hongli Niu (11478268), Qiaoying Pan (17457893), Kunliang Xu (17457896) · Figshare · 2023
<div><p>The prediction of stock prices has long been a captivating subject in academic research. This study aims to forecast the prices of prominent stocks in five key industries of the Chinese A-share market by leveraging the synergistic power of deep learning techniques and investor sentiment analysis. To achieve this, a sentiment multi-classification dataset is for the first time constructed for China’s stock market, based on four types of sentiments in modern psychology. The significant heterogeneity of sentiment changes in the sectors’ leading stock markets is trained and mined using the Bi-LSTM-ATT model. The impact of multi-classification investor sentiment on stock price prediction was analyzed using the CNN-Bi-LSTM-ATT model. It finds that integrating sentiment indicators into the prediction of industry leading stock prices can enhance the accuracy of the model. Drawing upon four fundamental sentiment types derived from modern psychology, our dataset provides a comprehensive framework for analyzing investor sentiment and its impact on forecasting the stock prices of China’s A-share market.</p></div>
Cited by 0Open access
Hongli Niu (11478268), Qiaoying Pan (17457893), Kunliang Xu (17457896) · Figshare · 2023
<div><p>The prediction of stock prices has long been a captivating subject in academic research. This study aims to forecast the prices of prominent stocks in five key industries of the Chinese A-share market by leveraging the synergistic power of deep learning techniques and investor sentiment analysis. To achieve this, a sentiment multi-classification dataset is for the first time constructed for China’s stock market, based on four types of sentiments in modern psychology. The significant heterogeneity of sentiment changes in the sectors’ leading stock markets is trained and mined using the Bi-LSTM-ATT model. The impact of multi-classification investor sentiment on stock price prediction was analyzed using the CNN-Bi-LSTM-ATT model. It finds that integrating sentiment indicators into the prediction of industry leading stock prices can enhance the accuracy of the model. Drawing upon four fundamental sentiment types derived from modern psychology, our dataset provides a comprehensive framework for analyzing investor sentiment and its impact on forecasting the stock prices of China’s A-share market.</p></div>
Cited by 0Open access
Zhu, Lin · Figshare · 2024
This code for MD&A text preprocessing
Cited by 0Open access
Abd Rahman, Suraya, Sainin, Mohd Shamrie · Universiti Utara Malaysia Institutional Repository (Universiti Utara Malaysia) · 2006
This study deals with the document preprocessing step of Text Categorization System (TCS) done by previous researcher. Previously, the document used for TCS was entered manually by key-in the abstract of the document into database.So it will burden if it involves a large volume of articles.In this study, the extraction based approach was applied in order to automate the document preprocessing. One module was added into the prototype of text categorization system that is used to add document into database.
Cited by 0Open access
T. H. Bini Krishna, P. Ezudheen · 2015
Cited by 0
Mohamed Osman Hegazi, Yasser Mubarak Aldossari, Abdullah Al-Yahy et al. · Heliyon · 2021
<b>C</b>urrently, social media plays an important role in daily life and routine. Millions of people use social media for different purposes. Large amounts of data flow through online networks every second, and these data contain valuable information that can be extracted if the data are properly processed and analyzed. However, most of the processing results are affected by preprocessing difficulties. This paper presents an approach to extract information from social media Arabic text. It provides an integrated solution for the challenges in preprocessing Arabic text on social media in four stages: data collection, cleaning, enrichment, and availability. The preprocessed Arabic text is stored in structured database tables to provide a useful corpus to which, information extraction and data analysis algorithms can be applied. The experiment in this study reveals that the implementation of the proposed approach yields a useful and full-featured dataset and valuable information. The resultant dataset presented the Arabic text in three structured levels with more than 20 features. Additionally, the experiment provides valuable information and processed results such as topic classification and sentiment analysis.
Cited by 78Open access
Zhou, Shengshi, Xu, Lei, Liu, Xia et al. · Figshare · 2025
The MD&A text preprocessing code in the paper of a dataset on time value propositions.
Cited by 0Open access
Tian, Jinfang, Cheng, Qian, Xue, Rui et al. · Figshare · 2023
This code is mainly used for preprocessing of MD&A text.
Cited by 0Open access
Zhu, Lin · Figshare · 2024
This code for MD&A text preprocessing
Cited by 0Open access