spark技术分享Latent Dirichlet Allocation (LDA)
|
Note
|
Information here are based almost exclusively from the blog post Topic modeling with LDA: MLlib meets GraphX. |
Topic modeling is a type of model that can be very useful in identifying hidden thematic structure in documents. Broadly speaking, it aims to find structure within an unstructured collection of documents. Once the structure is “discovered”, you may answer questions like:
-
What is document X about?
-
How similar are documents X and Y?
-
If I am interested in topic Z, which documents should I read first?
Spark MLlib offers out-of-the-box support for Latent Dirichlet Allocation (LDA) which is the first MLlib algorithm built upon GraphX.
Topic models automatically infer the topics discussed in a collection of documents.
Example
|
Caution
|
FIXME Use Tokenizer, StopWordsRemover, CountVectorizer, and finally LDA in a pipeline. |
