Search 250 million research papers

Word Embedding

Takeshi Okadome · 2025

Cited by 0

Word embedding predictions

Adrianna C Jenkins, Ming Hsu, Pierre Karashchuk · OSF Preprints (OSF Preprints) · 2017

Cited by 0

Attention Word Embedding

International Committee for Computational Linguistics 2020, Baraniuk, Richard G., Sonkar, Shashank et al. · Underline Science Inc. · 2020

Word embedding models learn semantically rich vector representations of words and are widely used to initialize natural processing language (NLP) models. The popular continuous bag-of words (CBOW) model of word2vec learns a vector embedding by masking a given word in a sentence and then using the other words as a context to predict it. A limitation of CBOW is that it equally weights the context words when making a prediction, which is inefficient, since some words have higher predictive value than others. We tackle this inefficiency by introducing the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. We also propose AWE-S, which incorporates subword information. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models.

Cited by 0Open access

BVerfG - Word Embedding

Lüders, Kilian · Zenodo (CERN European Organization for Nuclear Research) · 2024

Information The word embedding model provided here has been trained exclusively on the case law of the German Federal Constitutional Court. The model has been implemented using Gensim. To ensure that there are no compatibility problems, please make sure that you are using the correct version of Gensim (the model has been tested with Gensim 4.3.1).The model has been trained using the following configuration: min_count=10 sg=1 window=12 vector_size=200 alpha=0.03 negative=5 sample=1e-3 epochs = 200 The model is trained on tokenised lower case text. Contact kilian.lueders@hu-berlin.de Data Sources Wendel, Luisa, & Möllers, Christoph (2023). Korpus der Entscheidungen des Bundesverfassungsgerichts (2.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10369205 Wendel, Luisa (2023). Metadaten zu Entscheidungen des Bundesverfassungsgerichts (2.7.1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10378324 Project Leibniz Linguistic Research into Constitutional Lawhttps://www.lehrstuhl-moellers.de/llcon Funding The creation of this model has been realized in the L.L.Con project graciously funded in the years 2017-2023 by means of the Gottfried-Wilhelm Leibniz prize awarded to Prof. Dr. Christoph Möllers.

Cited by 0Open access

Musical Word Embedding

SeungHeonDoh · Zenodo (CERN European Organization for Nuclear Research) · 2022

Musical Word Embedding for Music Tagging and Retrieval IEEE Transactions on Audio, Speech and Language Processing (submitted) - SeungHeon Doh, Jongpil Lee, Dasaem Jeong, Juhan NamDEMO: https://seungheondoh.github.io/musical_word_embedding_demo/ Word embedding has become an essential means for text-based information retrieval. Typically, word embeddings are learned from large quantities of general and unstructured text data. However, in the domain of music, the word embedding may have difficulty understanding musical contexts or recognizing music-related entities like artists and tracks. To address this issue, we propose a new approach called Musical Word Embedding (MWE), which involves learning from various types of texts, including both everyday and music-related vocabulary. We integrate MWE into an audio-word joint representation framework for tagging and retrieving music, using words like tag, artist, and track that have different levels of musical specificity. Our experiments show that using a more specific musical word like track results in better retrieval performance, while using a less specific term like tag leads to better tagging performance. To balance this compromise, we suggest multi-prototype training that uses words with different levels of musical specificity jointly. We evaluate both word embedding and audio-word joint embedding on four tasks (tag rank prediction, music tagging, query-by-tag, and query-by-track) across two datasets (Million Song Dataset and MTG-Jamendo). Our findings show that the suggested MWE is more efficient and robust than the conventional word embedding.

Cited by 0Open access

word embedding pdf

odoaqbiggv · Zenodo (CERN European Organization for Nuclear Research) · 2024

<pre><code>\n<p><strong>word embedding pdf</strong><br></p>\n<p>Rating: 4.8 / 5 (1720 votes)<br></p>\n<p>Downloads: 19010<br><br></p>\n <p>= = = = = \n<strong><a href="https://calendario2023.es/21Nr9y?keyword=word embedding pdf" target="_blank">CLICK HERE TO DOWNLOAD</a></strong>\n = = = = = <br><br></p>\n<p><br><br><br><br></p>\n<p><br><br><br><br></p>\n<p><br><br>In this study, we use the TREC. Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. We develop a systematic framework and metrics to analyze word embeddings trained over y of text corpora A solution which normalizes the word vectors on a hypersphere and constrains the linear transform as an orthogonal transform and can offer better performance on a word similarity task and an English-toSpanish word translation task is proposed. Motivated by the unitary-invariance of word embedding, we pro-pose the , · A Comparative Study on Word Embeddings in Deep Learning for Text Classification. Similarly, the co occurrence statistics used by GloVe rely on har monic series where words at distance d from the focus word are assigned a weight1 d. First, we introduce popular word embedding models and discuss desired properties of word models and eval-uation methods (or evaluators) method we count the number of times each word appears inside a window of a particular size around the word of interest. Abstract. By leveraging large corpora of unlabeled text, such continuous space representations can be computed for Tags In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Let our corpus contain just three sentences and the window size beUsing Word-Word Co-occurrence Matrix: •Generate jVjj Vjco-occurrence Word embedding is one such technique. These vectors are obviously conceptually and practically different to those of the 'vector space model' well-known to political scientists. standard dataset. We calculate this count for all the words in corpus. effects of word embedding and the GRU on text classification to address this difficulty. Vector dimension = number of words in vocabulary (e.g.,,) Abstract—Extensive evaluation on a large number of word embedding models for language processing applications is con-ducted in this work. For one thing, word embedding vectors are, per their name, for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space , · Our study presents a unified approach t o examine the. the embeddings of analogy "woman is to queen as man is to king" approximately describe a paral lelogram We use the word embeddings as a quantitative lens through which to study histor-ical trends—specifically trends in the gender and ethnic stereo-types in theth andst centuries in the United States. Word embed-ding, or the encoding of words as vectors, has received much interest as a feature learning technique for natural language processing in recent times. Representing words as discrete symbols. Second, the number of parameters for a word embedding or a model that builds on word embeddings window =P (wjjwi;window)P (window) =L (L j j ij+ 1) where window is the actual window size (fromto L) sampled by the algorithm. In traditional NLP, we regard words as discrete symbols: hotel, conference, motel — a localist representation Words can be represented by one-hot vectors: one 1, the rest 0's. We will start by conventional word embeddings (e.g., Word2Vec and GloVe) and then move to other types of embeddings, such as sense-specific and graph alternatives Word embedding is a feature learning technique which aims at mapping words from a vocabulary into vectors of real numbers in a low-dimensional space. The vector space model has each document as a vector the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. RApproach: Learn Word Embedding Space •An embedding space represents a finite number of words, ided in training •A word embeddingis represented as a vector indicating its context •The dimensionality of all word embeddings in an embedding space match •What is the dimensionality for the shown example? Word embedding has been found to be highly powerful to translate words from one language to another by a Introduction. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models This tutorial will provide a high-level synthesis of the main embedding techniques in NLP, in the broad sense. Carl Allen1Timothy Hospedales. P(y=∣x) =σ(θ⊺w+b) w(1) w(2) sunny don't. ember DOI: Conference: NLPIR First, it directly impacts the quality of word vectorsa word embedding with a small dimensionality is typically not expressive enough to capture all possible word relations, whereas one with a very large dimensionality suffers from over-fitting. This review presents a better way of understanding and working with word embeddings The vector for a given word can be called a word embedding. We also propose AWE-S, which incorporates subword information. GloVe is an unsupervised learning algorithm for obtaining vector representations for words. We display an example below. For example, with the window size 3, the context Analogies Explained: Towards Understanding Word Embeddings.</p></code></pre>

Cited by 0Open access

Word-Embedding Benchmarking

Philipp Behnen, Felix Kruse, Jorge Marx Gómez · 2020

Die automatisierte Verarbeitung natürlicher Sprache bietet großes Potenzial, um neues Wissen aus unstrukturierten Daten zu gewinnen. Für die Bewältigung dieser Herausforderung haben sich in den vergangenen Jahren Word-Embedding-Verfahren bewährt, da diese semantische Ähnlichkeiten abbilden können. Durch die intensive Forschung entstehen stetig neue Verfahren. Je nach Aufgabe ist dabei aber nicht immer eindeutig, welches Verfahren für den Einzelfall empfehlenswert ist. Um dieses Problem zu adressieren, liefert dieser Beitrag einen Word-Embedding Benchmark, der die NLP-Tasks Ähnlichkeit, Analogie und (Multiclass-/Multilabel-) Klassifikation für einheitlich trainierte Word-Embeddings betrachtet und neben der Genauigkeit auch auf Kriterien wie die Trainingsdauer und die Hardwareressourcen eingeht. Zudem werden die Standardparametereinstellungen der Word-Embeddings modifiziert, um die Ergebnisse zu validieren. Die besten Ergebnisse lieferten die Word-Embeddings FastText und ELMo. Mit diesem Benchmark werden Data Scientisten bei der Auswahl eines Word-Embeddings für den Einsatz in entsprechenden NLPAufgaben unterstützt.

Cited by 0

BVerfG - Word Embedding

Lüders, Kilian · Zenodo (CERN European Organization for Nuclear Research) · 2024

Information The word embedding model provided here has been trained exclusively on the case law of the German Federal Constitutional Court. The model has been implemented using Gensim. To ensure that there are no compatibility problems, please make sure that you are using the correct version of Gensim (the model has been tested with Gensim 4.3.1).The model has been trained using the following configuration: min_count=10 sg=1 window=12 vector_size=200 alpha=0.03 negative=5 sample=1e-3 epochs = 200 The model is trained on tokenised lower case text. Contact kilian.lueders@hu-berlin.de Data Sources Wendel, Luisa, & Möllers, Christoph (2023). Korpus der Entscheidungen des Bundesverfassungsgerichts (2.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10369205 Wendel, Luisa (2023). Metadaten zu Entscheidungen des Bundesverfassungsgerichts (2.7.1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10378324 Project Leibniz Linguistic Research into Constitutional Lawhttps://www.lehrstuhl-moellers.de/llcon Funding The creation of this model has been realized in the L.L.Con project graciously funded in the years 2017-2023 by means of the Gottfried-Wilhelm Leibniz prize awarded to Prof. Dr. Christoph Möllers.

Cited by 1Open access

Attention Word Embedding

Shashank Sonkar, Andrew E. Waters, Richard G. Baraniuk · 2020

Word embedding models learn semantically rich vector representations of words and are widely used to initialize natural processing language (NLP) models. The popular continuous bag-ofwords (CBOW) model of word2vec learns a vector embedding by masking a given word in a sentence and then using the other words as a context to predict it. A limitation of CBOW is that it equally weights the context words when making a prediction, which is inefficient, since some words have higher predictive value than others. We tackle this inefficiency by introducing the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. We also propose AWE-S, which incorporates subword information. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models.

Cited by 0Open access

Word embedding predictions

Hsu, Ming, Karashchuk, Pierre, Jenkins, Adrianna C · Open Science Framework · 2017

Cited by 0Open access

Word Embedding using word position information

Hyunsun Hwang, Changki Lee, Hyunki Jang et al. · 2017

Cited by 0

Morpheme Level Word Embedding

Ruslan Galinsky, Tatiana Kovalenko, Julia Yakovleva et al. · Communications in computer and information science · 2017

Cited by 3

Relevance-based Word Embedding

Hamed Zamani, W. Bruce Croft · 2017

Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, has recently attracted much attention in natural language processing and information retrieval tasks. The embedding vectors are typically learned based on term proximity in a large corpus. This means that the objective in well-known word embedding algorithms, e.g., word2vec, is to accurately predict adjacent word(s) for a given word or context. However, this objective is not necessarily equivalent to the goal of many information retrieval (IR) tasks. The primary objective in various IR tasks is to capture relevance instead of term proximity, syntactic, or even semantic similarity. This is the motivation for developing unsupervised relevance-based word embedding models that learn word representations based on query-document relevance information. In this paper, we propose two learning models with different objective functions; one learns a relevance distribution over the vocabulary set for each query, and the other classifies each term as belonging to the relevant or non-relevant class for each query. To train our models, we used over six million unique queries and the top ranked documents retrieved in response to each query, which are assumed to be relevant to the query. We extrinsically evaluate our learned word representation models using two IR tasks: query expansion and query classification. Both query expansion experiments on four TREC collections and query classification experiments on the KDD Cup 2005 dataset suggest that the relevance-based word embedding models significantly outperform state-of-the-art proximity-based embedding models, such as word2vec and GloVe.

Cited by 148

Word Embedding-Based Automatic

Hiroshi Echizen‐ya, Kenji Araki, Eduard Hovy · 2019

Hiroshi Echizen’ya, Kenji Araki, Eduard Hovy. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

Cited by 9Open access

Materials Word Embedding Models

2026

Cited by 0

Relevance-based Word Embedding

Hamed Zamani, W. Bruce Croft · arXiv (Cornell University) · 2017

Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, has recently attracted much attention in natural language processing and information retrieval tasks. The embedding vectors are typically learned based on term proximity in a large corpus. This means that the objective in well-known word embedding algorithms, e.g., word2vec, is to accurately predict adjacent word(s) for a given word or context. However, this objective is not necessarily equivalent to the goal of many information retrieval (IR) tasks. The primary objective in various IR tasks is to capture relevance instead of term proximity, syntactic, or even semantic similarity. This is the motivation for developing unsupervised relevance-based word embedding models that learn word representations based on query-document relevance information. In this paper, we propose two learning models with different objective functions; one learns a relevance distribution over the vocabulary set for each query, and the other classifies each term as belonging to the relevant or non-relevant class for each query. To train our models, we used over six million unique queries and the top ranked documents retrieved in response to each query, which are assumed to be relevant to the query. We extrinsically evaluate our learned word representation models using two IR tasks: query expansion and query classification. Both query expansion experiments on four TREC collections and query classification experiments on the KDD Cup 2005 dataset suggest that the relevance-based word embedding models significantly outperform state-of-the-art proximity-based embedding models, such as word2vec and GloVe.

Cited by 20Open access

Sentiment Specific Word Embedding

Tang · Zenodo (CERN European Organization for Nuclear Research) · 2018

sentiment specific word embedding learned based on the approach described in the following paper: D. Tang, et al., Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification, ACL 2014

Cited by 0Open access

Word Mover’s Embedding: From Word2Vec to Document Embedding

Lingfei Wu, Ian En-Hsu Yen, Kun Xu et al. · 2018

Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, Michael J. Witbrock. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

Cited by 99Open access

Word Sense Disambiguation by Refining Target Word Embedding

Xuefeng Zhang, Richong Zhang, Xiaoyang Li et al. · 2023

Word Sense Disambiguation (WSD) which aims to identify the correct sense of a target word appearing in a specific context is essential for web text analysis. The use of glosses has been explored as a means for WSD. However, only a few works model the correlation between the target context and gloss. We add to the body of literature by presenting a model that employs a multi-head attention mechanism on deep contextual features of the target word and candidate glosses to refine the target word embedding. Furthermore, to encourage the model to learn the relevant part of target features that align with the correct gloss, we recursively alternate attention on target word features and that of candidate glosses to gradually extract the relevant contextual features of the target word, refining its representation and strengthening the final disambiguation results. Empirical studies on the five most commonly used benchmark datasets show that our proposed model is effective and achieves state-of-the-art results.

Cited by 4

Word co-occurrence network analysis using Word Embedding

Laura Vanessa Cruz Quispe · 2024

Recent studies in the literature demonstrate that human language can be effectively modeled as a complex network, commonly referred to as a word co-occurrence network.These networks exhibit characteristics typical of scale-free and small-world networks, aligning them with fundamental concepts in network theory.The use of word co-occurrence networks in text classification has shown notable success, primarily due to their ability to capture the structural and syntactic properties of a text, without relying on parsers that require a deeper linguistic knowledge of the language.However, the increasing use of word embeddings across various applications highlights the importance of integrating contextual and semantic information, which co-occurrence networks, in their traditional form, may lack.In this research, we propose to extend the modeling of word co-occurrence networks by incorporating word embedding data to generate virtual edges, thereby unifying syntactic, semantic, and contextual elements within the same network.This approach aims to improve several aspects of text classification, particularly in terms of quality, robustness, and adaptability to short texts, which often present unique challenges.Due to the generalizability of the proposed model and the flexible nature of embeddings, we believe that these networks can further our understanding of how word embeddings operate within complex network structures.The results of our experiments reveal that the use of virtual edges generated from embeddings such as GloVe, Word2Vec, and FastText enhances the discriminative power of the network, significantly improving text classification performance.Additionally, we discovered that the most optimized results are achieved when stop-words are retained and a simple global thresholding strategy is applied to establish virtual edges.Moreover, incorporating word embeddings in these networks not only enhances but also maintains a high level of informativeness, allowing the network to better distinguish between human texts and nonsensical texts over short and long texts.Finally, the combination of word embeddings with stop-word filtering, provides the network with semantic richness, giving the ability to capture semantical and context information of texts.However, maintaining word embedding without stop-word filtering, retains its ability to capture the underlying syntactic structure, making it possible to identify the linguistic properties of different languages.This approach adds robustness to word co-occurrence networks, preserving their initial syntactic capabilities without being compromised by the addition of virtual edges.

Cited by 0Open access