Tutorial (Japanese Text Classification)

Classify Japanese news articles with nagisa and scikit-learn

This tutorial provides an example of classifying Japanese news articles into categories by using nagisa for tokenization and scikit-learn for training a classifier. You convert the tokenized articles into TF-IDF vectors, train a logistic regression classifier, evaluate it and classify new texts.

Install python libraries

Before we get started, please run the following commands to install the libraries used in this tutorial.

pip install nagisa
pip install scikit-learn
pip install tqdm

Download the dataset

This tutorial uses the Livedoor News Corpus, which contains 7,367 Japanese news articles in 9 categories. The corpus (~9 MB) is downloaded automatically into the ldcc/ directory on the first run. Each document is the title and the body of an article, and its label is the category (the news site of the article).

Tokenize Japanese text with nagisa

tokenize() keeps the content words (nouns, verbs, adjectives, etc.) using nagisa.extract(), lowercases them and removes the stopwords in nagisa.stopwords. nagisa works on a sentence, so the text is tokenized line by line.

def tokenize(text):
    for name in SOURCE_NAMES:
        text = text.replace(name, " ")

    # nagisa works on a sentence, so tokenize the text line by line.
    words = []
    for line in text.splitlines():
        tokens = nagisa.extract(line, extract_postags=TARGET_POSTAGS)
        words += [word.lower() for word in tokens.words]
    return [word for word in words if word not in nagisa.stopwords]

Before splitting the text into words, tokenize() removes the names of the news sites in SOURCE_NAMES. Many articles contain the name of their site (e.g., “S-MAX” appears in 863 of the 870 articles of the smax category). Such words tell the category directly, so a classifier would learn the site names instead of the content of the articles.

Train and evaluate a classifier

The script splits the corpus into the train set (80%) and the test set (20%), keeping the ratio of the categories (stratify=labels). The vectorizer and the classifier are fitted on the train set only. TF-IDF gives a high value to a word that appears frequently in a document but rarely in the other documents. TfidfVectorizer receives the tokenized documents as they are (analyzer=identity) and ignores the words that appear in only one document (min_df=2). LogisticRegression learns a weight for each word and each category.

Run the following command. Tokenizing all articles takes several minutes.

python tutorial_text_classification.py
tutorial_text_classification.py
  1import io
  2import os
  3import tarfile
  4import urllib.request
  5
  6import nagisa
  7import numpy as np
  8from sklearn.feature_extraction.text import TfidfVectorizer
  9from sklearn.linear_model import LogisticRegression
 10from sklearn.metrics import accuracy_score, classification_report
 11from sklearn.model_selection import train_test_split
 12from tqdm import tqdm
 13
 14CORPUS_URL = "https://www.rondhuit.com/download/ldcc-20140209.tar.gz"
 15CORPUS_DIR = "ldcc"
 16
 17# Content words used as features
 18TARGET_POSTAGS = ["名詞", "動詞", "形容詞", "形状詞", "英単語"]
 19
 20# News site names, which tell the category directly
 21SOURCE_NAMES = ["独女通信", "ITライフハック", "家電チャンネル", "livedoor HOMME", "MOVIE ENTER",
 22                "Peachy", "S-MAX", "エスマックス", "smaxjp", "Sports Watch"]
 23
 24
 25def download_corpus():
 26    if os.path.isdir(CORPUS_DIR):
 27        return
 28
 29    with urllib.request.urlopen(CORPUS_URL, timeout=120) as resp:
 30        data = resp.read()
 31
 32    with tarfile.open(fileobj=io.BytesIO(data), mode="r:*") as tar:
 33        # The filter argument is not available in older Python versions.
 34        if hasattr(tarfile, "data_filter"):
 35            tar.extractall(path=".", filter="data")
 36        else:
 37            tar.extractall(path=".")
 38    os.rename("text", CORPUS_DIR)
 39
 40
 41def load_corpus():
 42    texts = []
 43    labels = []
 44    for category in sorted(os.listdir(CORPUS_DIR)):
 45        cat_path = os.path.join(CORPUS_DIR, category)
 46        if not os.path.isdir(cat_path):
 47            continue
 48        for fname in sorted(os.listdir(cat_path)):
 49            if not fname.endswith(".txt") or fname.startswith("LICENSE"):
 50                continue
 51            with open(os.path.join(cat_path, fname), encoding="utf-8") as f:
 52                lines = f.read().splitlines()
 53
 54            # line 1: URL, line 2: date, line 3: title, line 4-: body
 55            texts.append("\n".join(lines[2:]))
 56            labels.append(category)
 57    return texts, labels
 58
 59
 60def tokenize(text):
 61    for name in SOURCE_NAMES:
 62        text = text.replace(name, " ")
 63
 64    # nagisa works on a sentence, so tokenize the text line by line.
 65    words = []
 66    for line in text.splitlines():
 67        tokens = nagisa.extract(line, extract_postags=TARGET_POSTAGS)
 68        words += [word.lower() for word in tokens.words]
 69    return [word for word in words if word not in nagisa.stopwords]
 70
 71
 72def identity(tokens):
 73    # The documents are already tokenized.
 74    return tokens
 75
 76
 77def main():
 78    download_corpus()
 79    texts, labels = load_corpus()
 80    docs = [tokenize(text) for text in tqdm(texts)]
 81
 82    # Split the corpus into the train (80%) and test (20%) sets
 83    train_docs, test_docs, train_labels, test_labels = train_test_split(
 84        docs, labels, test_size=0.2, stratify=labels, random_state=1234)
 85
 86    # Convert the documents into TF-IDF vectors
 87    vectorizer = TfidfVectorizer(analyzer=identity, min_df=2)
 88    train_vecs = vectorizer.fit_transform(train_docs)
 89    test_vecs = vectorizer.transform(test_docs)
 90    print("train: {}, test: {}, features: {}".format(
 91        len(train_docs), len(test_docs), len(vectorizer.vocabulary_)))
 92
 93    # Train a classifier
 94    model = LogisticRegression(max_iter=1000)
 95    model.fit(train_vecs, train_labels)
 96
 97    # Evaluate the classifier on the test set
 98    pred_labels = model.predict(test_vecs)
 99    print("accuracy: {:.4f}".format(accuracy_score(test_labels, pred_labels)))
100    print(classification_report(test_labels, pred_labels, digits=3))
101
102    # Words with the largest weights for each category
103    words = vectorizer.get_feature_names_out()
104    for category, coef in zip(model.classes_, model.coef_):
105        top_words = [words[i] for i in np.argsort(-coef)[:8]]
106        print("{:<15} {}".format(category, ", ".join(top_words)))
107
108    # Classify new texts
109    new_texts = [
110        "新しいAndroidスマートフォンが発売され、アプリの動作も快適になった。",
111        "日本代表は昨日の試合で2対1で勝利した。",
112        "監督が新作映画の見どころを語った。",
113    ]
114    new_vecs = vectorizer.transform([tokenize(text) for text in new_texts])
115    for text, probs in zip(new_texts, model.predict_proba(new_vecs)):
116        best = np.argmax(probs)
117        print("\n{}\n  => {} ({:.2f})".format(text, model.classes_[best], probs[best]))
118
119
120if __name__ == "__main__":
121    main()

This is the output of tutorial_text_classification.py. The results may differ with other versions of the libraries (we used nagisa 0.3.0 and scikit-learn 1.9.1).

train: 5893, test: 1474, features: 34307
accuracy: 0.8996
                precision    recall  f1-score   support

dokujo-tsushin      0.855     0.845     0.850       174
  it-life-hack      0.918     0.897     0.907       174
 kaden-channel      0.948     0.942     0.945       173
livedoor-homme      0.934     0.696     0.798       102
   movie-enter      0.893     0.960     0.925       174
        peachy      0.785     0.822     0.803       169
          smax      0.923     0.971     0.947       174
  sports-watch      0.927     0.983     0.954       180
    topic-news      0.938     0.890     0.913       154

      accuracy                          0.900      1474
     macro avg      0.902     0.890     0.894      1474
  weighted avg      0.901     0.900     0.899      1474

dokujo-tsushin  独女, 女性, 女, 人, 結婚, 自分, 3, 男性
it-life-hack    孫, pc, 見る, 紺子, ロゴ, 製品, クリック, ter
kaden-channel   話題, 売れ筋, ビデオ, salon, チェック, 発売, パナソニック, hulu
livedoor-homme  ゴルフ, 転職, 求人, クルマ, ビジネス, 0, 年収, ドバイ
movie-enter     映画, 公開, 本作, 作品, 劇場, 演じる, 映像, dvd
peachy          女子, 女性, プレゼント, 恋愛, クリスマス, 事, 肌, 応募
smax            アプリ, on, android, 執筆, タップ, フォン, 画面, twitter
sports-watch    選手, 試合, 代表, サッカー, 日本, 野球, 五輪, ダルビッシュ
topic-news      掲示, ネット, 声, 韓国, 関連, 批判, 記事, 報じ

新しいAndroidスマートフォンが発売され、アプリの動作も快適になった。
  => smax (0.78)

日本代表は昨日の試合で2対1で勝利した。
  => sports-watch (0.88)

監督が新作映画の見どころを語った。
  => movie-enter (0.70)

livedoor-homme has the lowest recall (0.696). It has the fewest articles (511) and covers many topics such as cars, golf and jobs. peachy and dokujo-tsushin, which are both about women’s lifestyles, also have low F1 scores (0.803 and 0.850).

The words with the largest weights are mostly content words, such as 映画 and 公開 for movie-enter, and 選手 and 試合 for sports-watch. Some site-specific words remain: 独女 (a word used by dokujo-tsushin), 紺子 (a column character in it-life-hack), and 執筆 and on (from “記事執筆：…” and “… smaxjp on Twitter” in the smax articles).

Without removing SOURCE_NAMES, the accuracy is 0.904, but site names such as peachy, max, sports and watch appear in the top words.