Distributed Representations of Sentences and Documents
read the original abstract
Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperform bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Adaptive Computation Time for Recurrent Neural Networks
ACT lets RNNs dynamically adapt computation depth per input via a differentiable halting unit, yielding large gains on synthetic tasks and structural insights on language data.
-
Language Models as Knowledge Bases?
BERT stores relational knowledge extractable via cloze queries without fine-tuning and matches supervised baselines on open-domain QA tasks.
-
Larch: Learned Query Optimization for Semantic Predicates
Larch uses a GNN-MDP formulation and a selectivity predictor plus dynamic programming to reorder semantic filter evaluation, cutting token usage 3x-19x versus prior systems on real and synthetic workloads.
-
ConcernBERT: Learning Responsibilities Using Class Membership
ConcernBERT is a BERT embedding model trained with triplet loss on class membership to encode concern-level semantics in Java entities, evaluated by recovering original classes from merged unlabeled groups on a new da...
-
Medical Concept Representation Learning from Claims Data and Application to Health Plan Payment Risk Adjustment
Embedding models trained on medical codes outperform a commercial linear regression risk adjustment model for prospective risk score prediction.
-
OCC: A Smart Reply System for Efficient In-App Communications
Uber's OCC system uses unsupervised embeddings plus nearest-neighbor intent detection followed by historical reply retrieval, reporting 76% intent accuracy and 71% production usage in English-speaking countries.
-
Detecting Speculative Language in Biomedical Texts using Recurrent Neural Tensor Networks
RNTN achieves F1 of 0.885 vs 0.881 for linear bigram SVM in speculative language detection in biomedical texts, with Paragraph Vector at 0.368.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.