Tutorial (Japanese Topic Modeling)

Run topic modeling using the Livedoor News Corpus

This tutorial provides an example of running topic modeling on Japanese news articles by using nagisa for tokenization and scikit-learn’s LDA (Latent Dirichlet Allocation).

Install python libraries

Before we get started, please run the following commands to install the libraries used in this tutorial.

pip install nagisa
pip install scikit-learn
pip install requests
pip install tqdm

Run the script

Run the following command to download the corpus and extract topics. The Livedoor News Corpus (~9 MB) is downloaded automatically on the first run into the ldcc/ directory. By default, the model is built from the title of each article (TEXT_TYPE = "title"). To use the article body instead, see Use the article body.

python tutorial_topic_model.py
tutorial_topic_model.py
  1import io
  2import os
  3import tarfile
  4
  5import nagisa
  6import requests
  7from sklearn.decomposition import LatentDirichletAllocation
  8from sklearn.feature_extraction.text import CountVectorizer
  9from tqdm import tqdm
 10
 11CORPUS_URL = "https://www.rondhuit.com/download/ldcc-20140209.tar.gz"
 12CORPUS_DIR = "ldcc"
 13N_TOPICS = 9
 14N_TOP_WORDS = 10
 15MAX_ITER = 20
 16TEXT_TYPE = "title"  # or body
 17
 18
 19def download_corpus():
 20    if os.path.isdir(CORPUS_DIR):
 21        return CORPUS_DIR
 22
 23    resp = requests.get(CORPUS_URL, timeout=120)
 24    resp.raise_for_status()
 25
 26    with tarfile.open(fileobj=io.BytesIO(resp.content), mode="r:*") as tar:
 27        # The filter argument is not available in older Python versions.
 28        if hasattr(tarfile, "data_filter"):
 29            tar.extractall(path=".", filter="data")
 30        else:
 31            tar.extractall(path=".")
 32
 33    extracted = "text"
 34    os.rename(extracted, CORPUS_DIR)
 35    return CORPUS_DIR
 36
 37
 38def load_documents(corpus_dir, text_type=TEXT_TYPE):
 39    documents = []
 40    for category in sorted(os.listdir(corpus_dir)):
 41        cat_path = os.path.join(corpus_dir, category)
 42        if not os.path.isdir(cat_path):
 43            continue
 44        for fname in sorted(os.listdir(cat_path)):
 45            if not fname.endswith(".txt") or fname.startswith("LICENSE"):
 46                continue
 47            fpath = os.path.join(cat_path, fname)
 48            with open(fpath, encoding="utf-8") as f:
 49                lines = f.readlines()
 50
 51            # line 1: URL, line 2: date, line 3: title, line 4-: body
 52            if text_type == "title":
 53                text = lines[2].strip() if len(lines) > 2 else ""
 54            else:
 55                text = "".join(lines[3:]).strip()
 56
 57            if text:
 58                documents.append(text)
 59    return documents
 60
 61
 62def tokenize_documents(documents):
 63    stopwords = nagisa.stopwords
 64    tokenized = []
 65    for doc in tqdm(documents):
 66        # nagisa works on a sentence, so tokenize the text line by line.
 67        words = []
 68        for line in doc.splitlines():
 69            tokens = nagisa.tagging(line)
 70            words += [w for w in tokens.words if len(w) > 1 and w not in stopwords]
 71        tokenized.append(" ".join(words))
 72    return tokenized
 73
 74
 75def run_lda(tokenized_docs):
 76    vectorizer = CountVectorizer(max_df=0.85, min_df=5)
 77    dtm = vectorizer.fit_transform(tokenized_docs)
 78    feature_names = vectorizer.get_feature_names_out()
 79
 80    lda = LatentDirichletAllocation(
 81        n_components=N_TOPICS,
 82        max_iter=MAX_ITER,
 83        learning_method="online",
 84        random_state=1234,
 85    )
 86    lda.fit(dtm)
 87
 88    for topic_idx, topic in enumerate(lda.components_):
 89        top_indices = topic.argsort()[-N_TOP_WORDS:][::-1]
 90        top_words = [feature_names[i] for i in top_indices]
 91        print(f"\nTopic {topic_idx + 1}:")
 92        print(f"\t{', '.join(top_words)}")
 93
 94
 95def main():
 96    corpus_dir = download_corpus()
 97    documents = load_documents(corpus_dir)
 98    tokenized_documents = tokenize_documents(documents)
 99    run_lda(tokenized_documents)
100
101
102if __name__ == "__main__":
103    main()

This is an example of the output. The results may differ with other versions of the libraries (we used nagisa 0.3.0 and scikit-learn 1.9.1).

Topic 1:
    女子, 世界, 発言, オトナ, 理由, あなた, 恋愛, 時代, 美人, 独女

Topic 2:
    発売, 発表, 終了, プレゼント, 登場, 代表, デジ通, ビデオ, xperia, 生活

Topic 3:
    さん, 写真, 知っ, 虎の巻, すぎる, 結婚, 語る, ススメ, 説教, 自分

Topic 4:
    映画, アプリ, vol, レビュー, 特集, 独女, 選手, 年収, ゴルフ, 部屋

Topic 5:
    watch, sports, 日本, 韓国, ネット, vol, 批判, by, 東京, cafe

Topic 6:
    android, チェック, 対応, ドコモ, スマホ, 売れ筋, 提供, 搭載, ニュース, 向け

Topic 7:
    話題, ntt, スマート, フォン, テレビ, インタビュー, ドコモ, 問題, 利用, 社長

Topic 8:
    iphone, 公開, ランキング, 動画, 決定, 女性, 開催, 殺到, 出演, ゲーム

Topic 9:
    開始, レポート, 監督, モデル, 人気, サービス, google, キャンペーン, akb, 好き

Article titles are short, so many topics mix the words of several categories. Some topics have a clear subject. For example, Topic 6 (android, チェック, 対応, ドコモ, スマホ) is about smartphones. Topic 5 (watch, sports, 日本, 韓国) is about sports, but “watch” and “sports” come from the tag 【Sports Watch】 at the beginning of many titles in the sports-watch category.

Use the article body

Set TEXT_TYPE = "body" and run the script again. Tokenizing the article bodies takes several minutes. This is an example of the output with the article body.

Topic 1:
    紹介, 記事, 写真, 画面, アプリ, チェック, 使っ, facebook, 表示, 便利

Topic 2:
    さん, 自分, 女性, いい, 結婚, 仕事, 男性, 思っ, 思い, どう

Topic 3:
    ネット, 番組, 放送, テレビ, 関連, 記事, 選手, 掲示, 話題, 情報

Topic 4:
    プレゼント, 料理, ゴルフ, 東京, 女性, アイテム, 商品, クリスマス, 効果, さん

Topic 5:
    映画, 公開, 作品, 世界, 監督, 本作, 映像, 撮影, 日本, ドラマ

Topic 6:
    女子, 人気, さん, 彼女, 女性, ファッション, cm, 登場, モデル, ちゃん

Topic 7:
    日本, 韓国, 代表, 写真, 世界, akb, チーム, 関連, 監督, 話題

Topic 8:
    スマート, フォン, android, アプリ, 更新, max, 利用, サービス, 機能, ソフトウェア

Topic 9:
    発売, 搭載, 対応, モデル, カメラ, 発表, 製品, 機能, バッテリー, サイズ

The table shows, for the model built from the article body, how many articles have each topic as their most probable topic (the argmax of lda.transform(dtm)) and the most common categories among them. The script does not print these counts.

Topic

Articles

Most common categories (share)

Top 5 words

Topic 1

342

it-life-hack (48%)

紹介, 記事, 写真, 画面, アプリ

Topic 2

1,377

dokujo-tsushin (50%)

さん, 自分, 女性, いい, 結婚

Topic 3

1,092

topic-news (44%), sports-watch (38%)

ネット, 番組, 放送, テレビ, 関連

Topic 4

749

peachy (58%)

プレゼント, 料理, ゴルフ, 東京, 女性

Topic 5

984

movie-enter (77%)

映画, 公開, 作品, 世界, 監督

Topic 6

218

peachy (31%)

女子, 人気, さん, 彼女, 女性

Topic 7

673

sports-watch (49%)

日本, 韓国, 代表, 写真, 世界

Topic 8

1,166

smax (53%)

スマート, フォン, android, アプリ, 更新

Topic 9

766

kaden-channel (37%)

発売, 搭載, 対応, モデル, カメラ

Some topics match a category well, such as movie-enter for Topic 5 (77%) and smax for Topic 8 (“max” comes from “S-MAX”). Topics 3, 6 and 9 mix several categories.