Genism
1. How to compare document similarity?
A commonly used approach to match similar documents is based on counting the maximum number of common words between the documents.
But this approach has an inherent flaw. That is, as the size of the document increases, the number of common words tend to increase even if the documents talk about different topics.
The cosine similarity helps overcome this fundamental flaw in the ‘count-the-common-words’ or Euclidean distance approach.
2. What is Cosine Similarity and why is it advantageous?
Cosine similarity is a metric used to determine how similar the documents are irrespective of their size.
Mathematically, it measures the cosine of the angle between two vectors projected in a multi-dimensional space. In this context, the two vectors I am talking about are arrays containing the word counts of two documents.
As a similarity metric, how does cosine similarity differ from the number of common words?
When plotted on a multi-dimensional space, where each dimension corresponds to a word in the document, the cosine similarity captures the orientation (the angle) of the documents and not the magnitude. If you want the magnitude, compute the Euclidean distance instead.
The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance because of the size (like, the word ‘cricket’ appeared 50 times in one document and 10 times in another) they could still have a smaller angle between them. Smaller the angle, higher the similarity.
Documents in a 3-dimensional space
3. What is soft cosine similarity?
Soft cosine similarity is similar to cosine similarity but in addition considers the semantic relationship between the words through its vector representation.
