What is latent semantic analysis?

In machine learning, latent semantic analysis (LSA) is an approach to topic modeling. LSA uses dimensionality reduction to create structured data from unstructured text in order to aid text classification and retrieval. LSA is one of two principal topic modeling techniques, the other being latent Dirichlet allocation (LDA).

Topic modeling is a natural language processing (NLP) technique that applies unsupervised learning on large text datasets in order to produce a summary set of terms derived from those documents. These terms are meant to represent the collection’s overall primary set of topics. Topic models aim to uncover the latent topics or themes characterizing a number of documents.¹

Users can generate LSA topic models using scikit-learn’s (commonly referred to as sklearn) natural language toolkit (NLTK) and gensim in Python. The topic models and lsa packages in R also contain functions for generating LSA topic models.

Latent semantic analysis is associated with latent semantic indexing (LSI) which is an information retrieval technique. In information retrieval systems, LSI uses the same mathematical procedure underlying LSA to map user queries to documents based on word co-occurrence. If a user queries a system for waltz and foxtrot, they might be interested in documents that don't contain either of those terms but do contain terms that often co-occur with their query terms. For instance tango and bolero may frequently co-occur with the query terms and should indicate documents about the same topic. LSI indexes documents according to latent semantic word groups consisting of commonly co-occurring words. In this way, it can improve search engine results. LSA applies the same mathematical procedure as LSI in order to capture the hidden semantic structure underlaying large collections of documents.²

LSA begins with the document-term matrix or sometimes a term-document matrix. This displays the number of occurrences for each word across all documents. In Python (to offer one example), users can construct these matrices using a pandas dataframe. Here is an example document-term matrix using the three text strings as individual documents:

d1: My love is like red, red roses

d2: Roses are red, violets are blue

d3: Moses supposes his toes-es are roses

This matrix shows the word frequency of each word across all three documents following tokenization and stopword removal. Each column corresponds to a document, while each row corresponds to a specific word found across the whole text corpus. The values in the matrix signify the number of times a given term appears in a given document. If term w occurs n times within document d, then [w,d] = n. So, for example, document 1 uses 'red' twice, and so [red, d1] = 2.

From the document-term matrix, LSA produces a document-document matrix and term-term matrix. If the document-term matrix dimensions are defined as d documents times w words, then the document-document matrix is d times d and the term-term matrix w times w. Each value in the document-document matrix indicates the number of words each document has in common. Each value in the term-term matrix indicates the number of documents in which two terms co-occur.³

Data sparsity, which leads to model overfitting, is when a majority of data values in a given dataset are null (that is, empty). This happens regularly when constructing document-term matrices, for which each individual word is a separate row and vector space dimension, as one document will regularly lack a majority of the words that are more frequent in other documents. Indeed, the example document-term matrix here used contains numerous uses for words such as Moses, violets and blue that appear in only one document. Of course, text preprocessing techniques, such as stopword removal, stemming and lemmatization, can help reduce sparsity. LSA offers a more targeted approach however.

LSA deploys a dimensionality reduction technique known as singular value decomposition (SVD) to reduce sparsity in the document-term matrix. SVD powers many other dimension reduction approaches such as principal component analysis. SVD helps alleviate problems resulting from polysemy, single words that have multiple meanings, and synonymy, different words with  similar meaning.

Using the matrices calculated from the terms across document-document and term-term matrices, the LSA algorithm performs SVD on the initial term-document matrix. This produces new special matrices of eigenvectors that break down the original term-document relationships into linearly independent factors. Most important of these is the diagonal matrix of singular values produced from the square roots of the document-document matrix’s eigenvalues. In this diagonal matrix, often represented as Σ, values are always positive and arranged in decreasing order down the matrix diagonal:

As shown in this example Σ matrix, many of the lower values are near zero. The developer determines a cut-off value appropriate for their situation and reduces all singular values in Σ below that threshold to zero. This effectively means removing all of the rows and columns entirely occupied by zeros. In turn, we remove rows and columns from our other original matrices until they have the same number of rows and columns as Σ. This reduces the model’s dimensions.⁴

Once model dimensions have been reduced through SVD, the LSA algorithm compares documents in a lower dimensional semantic space using cosine similarity. The first step in this comparison stage involves mapping documents in vector space. Here, LSA treats texts as a bag of words model. The algorithm plots each text from the corpus or corpora as document vector, with individual words from the reduced matrix as the dimensions of that vector. Plotting ignores word order and context, focusing instead on how often words occur and how often they co-occur across documents.⁵

With standard bag of words models, semantically irrelevant words (for example, words such as the and some, and other similar words) can have the highest term frequency, and so greatest weight in a model. Term frequency-inverse document frequency (TF-IDF) is one technique to correct for this. It does this by accounting for a word’s prevalence throughout every document in a text set and weighting words in each document according to the word’s prevalence throughout the corpus.⁶

Once documents are plotted in vector space, the LSA algorithm uses the cosine similarity metric to compare them. Cosine similarity signifies the measurement of the angle between two vectors in vector space. It can be any value between -1 and 1. The higher the cosine score, the more alike the two documents are considered. Cosine similarity is represented by this formula, where a and b signify two document vectors:⁷

There are many use cases for topic models, from literary criticism⁸ to bioinformatics⁹ to hate speech detection in social media.¹⁰ As with many NLP tasks, a significant proportion of topic modeling research through the years concerns English and other Latin-script languages. More recently, however, research has explored topic modeling approaches for Arabic and other non-Latin languages.¹¹ Research has also turned to how large language models (LLMs) might advance and improve topic models. For instance, one study argues that LLMs provide an automated method for resolving longstanding problems in topic modeling, namely how to determine the appropriate number of topics and how to evaluate generated topics.¹²