Tutorial (Japanese Named Entity Recognition)
Train a Japanese NER model for KWDLC
This tutorial provides an example of training a Japanese NER model by using Kyoto University Web Document Leads Corpus(KWDLC).
Download the dataset
KWDLC is distributed on GitHub (https://github.com/ku-nlp/KWDLC). Clone the repository to the working directory. This tutorial uses version 1.1.1 of KWDLC.
git clone --depth 1 --branch v1.1.1 https://github.com/ku-nlp/KWDLC
Install python libraries
Before we get started, please run the following command to install the libraries used in this tutorial.
pip install nagisa
pip install seqeval
Preprocess the dataset
First, convert the downloaded data to the input data format for nagisa. The input data format of the train/dev/test files is the tsv format. The Each line is word and tag and one line is represented by word \t tag. Note that you put EOS between sentences.
In the KNP format files of KWDLC (KWDLC/knp/), a named entity is annotated
with a tag such as <NE:ORGANIZATION:京都大学> on a basic phrase line (a line starting with +).
The code below converts these annotations into word-level tags in the IOB2 format (B-TYPE, I-TYPE and O).
It also splits the corpus into the train/dev/test sets by using the official split of the document IDs (KWDLC/id/split_for_pas/).
After running the code, kwdlc.train, kwdlc.dev and kwdlc.test are output to the working directory.
python tutorial_preprocess_kwdlc.py
1import os
2import re
3
4
5# e.g., <NE:ORGANIZATION:京都大学>
6NE_PATTERN = re.compile(r"<NE:([A-Z]+):([^>]+)>")
7
8
9def find_ne_span(words, ne, phrase_start, phrase_end):
10 # An NE tag is attached to the basic phrase that contains the last word of the NE,
11 # so search for the words that end in the basic phrase and match the NE string.
12 for end in range(phrase_start + 1, phrase_end + 1):
13 for start in range(end):
14 if "".join(words[start:end]) == ne:
15 return start, end
16 return None
17
18
19def convert_to_iob2_tags(words, phrase_starts, nes):
20 tags = ["O"] * len(words)
21 phrase_ends = phrase_starts[1:] + [len(words)]
22
23 for netype, ne, phrase_id in nes:
24 span = find_ne_span(words, ne, phrase_starts[phrase_id], phrase_ends[phrase_id])
25
26 # Skip an NE that is a part of a word (e.g., 英 in 英語).
27 if span is None:
28 continue
29
30 start, end = span
31 tags[start] = "B-"+netype
32 for i in range(start + 1, end):
33 tags[i] = "I-"+netype
34
35 return tags
36
37
38def load_kwdlc(filename):
39 data = []
40
41 words = []
42 phrase_starts = []
43 nes = []
44
45 with open(filename, "r", encoding="utf-8") as f:
46 for line in f:
47 line = line.rstrip("\n")
48
49 if line.startswith("+ "):
50 # A basic phrase line may have an NE tag.
51 phrase_starts.append(len(words))
52 match = NE_PATTERN.search(line)
53 if match:
54 netype, ne = match.groups()
55 nes.append([netype, ne, len(phrase_starts) - 1])
56
57 elif line.startswith("# ") or line.startswith("* "):
58 continue
59
60 elif line == "EOS":
61 tags = convert_to_iob2_tags(words, phrase_starts, nes)
62 data.append([words, tags])
63
64 # reset
65 words = []
66 phrase_starts = []
67 nes = []
68
69 else:
70 # The first field of a morpheme line is the surface form.
71 surface = line.split(" ")[0]
72 words.append(surface)
73
74 return data
75
76
77def write_file(filename, data):
78 with open(filename, "w", encoding="utf-8") as f:
79 for words, tags in data:
80 for word, tag in zip(words, tags):
81 f.write("\t".join([word, tag])+"\n")
82 f.write("EOS\n")
83
84
85def main():
86 kwdlc_dir = "./KWDLC"
87
88 for split in ["train", "dev", "test"]:
89 # Use the official train/dev/test split of the documents.
90 fn_id = os.path.join(kwdlc_dir, "id", "split_for_pas", split+".id")
91 with open(fn_id, "r", encoding="utf-8") as f:
92 doc_ids = [line.strip() for line in f if line.strip()]
93
94 data = []
95 for doc_id in doc_ids:
96 # e.g., ./KWDLC/knp/w201106-00000/w201106-0000060050.knp
97 fn_knp = os.path.join(kwdlc_dir, "knp", doc_id[:13], doc_id+".knp")
98 data += load_kwdlc(fn_knp)
99
100 fn_out = "kwdlc."+split
101 write_file(fn_out, data)
102
103
104if __name__ == "__main__":
105 main()
This is an example of the converted data.
フランコ B-PERSON
・ I-PERSON
モスキーノ I-PERSON
は O
１９５０ B-DATE
年 I-DATE
イタリア B-LOCATION
に O
生まれる O
。 O
EOS
Train a model
Next, you train a NER model by using the nagisa.fit() function.
After finish training the model, save the three model files (kwdlc_ner_model.vocabs, kwdlc_ner_model.params, kwdlc_ner_model.hp) in the current directory.
python tutorial_train_kwdlc.py
1import nagisa
2
3
4def main():
5 fn_in_train = "kwdlc.train"
6 fn_in_dev = "kwdlc.dev"
7 fn_in_test = "kwdlc.test"
8
9 # start training
10 fn_out_model = "kwdlc_ner_model"
11 nagisa.fit(
12 train_file=fn_in_train,
13 dev_file=fn_in_dev,
14 test_file=fn_in_test,
15 model_name=fn_out_model
16 )
17
18
19if __name__ == "__main__":
20 main()
This is a log of the training process.
[nagisa] LAYERS: 1
[nagisa] THRESHOLD: 2
[nagisa] DECAY: 1
[nagisa] EPOCH: 10
[nagisa] WINDOW_SIZE: 3
[nagisa] DIM_UNI: 32
[nagisa] DIM_BI: 16
[nagisa] DIM_WORD: 16
[nagisa] DIM_CTYPE: 8
[nagisa] DIM_TAGEMB: 16
[nagisa] DIM_HIDDEN: 100
[nagisa] LEARNING_RATE: 0.1
[nagisa] DROPOUT_RATE: 0.3
[nagisa] SEED: 1234
[nagisa] TRAINSET: kwdlc.train
[nagisa] TESTSET: kwdlc.test
[nagisa] DEVSET: kwdlc.dev
[nagisa] DICTIONARY: None
[nagisa] EMBEDDING: None
[nagisa] HYPERPARAMS: kwdlc_ner_model.hp
[nagisa] MODEL: kwdlc_ner_model.params
[nagisa] VOCAB: kwdlc_ner_model.vocabs
[nagisa] EPOCH_MODEL: kwdlc_ner_model_epoch.params
[nagisa] NUM_TRAIN: 12271
[nagisa] NUM_TEST: 2195
[nagisa] NUM_DEV: 1585
[nagisa] VOCAB_SIZE_UNI: 2295
[nagisa] VOCAB_SIZE_BI: 26344
[nagisa] VOCAB_SIZE_WORD: 9748
[nagisa] VOCAB_SIZE_POSTAG: 20
Epoch LR Loss Time_m DevWS_f1 DevPOS_f1 TestWS_f1 TestPOS_f1
1 0.100 6.536 1.596 93.42 88.54 95.56 92.06
2 0.100 3.711 1.610 94.39 89.64 96.08 92.70
3 0.100 2.912 1.615 94.62 90.26 96.46 93.52
4 0.100 2.431 1.676 94.79 90.80 96.47 93.66
5 0.100 2.081 1.692 94.83 90.77 96.52 93.82
6 0.100 1.768 1.672 95.10 91.14 96.51 93.68
7 0.100 1.589 1.892 95.18 91.31 96.58 93.90
8 0.050 1.399 1.758 95.00 90.95 96.58 93.90
9 0.050 1.014 1.869 95.45 91.41 96.70 93.98
10 0.025 0.865 1.591 95.26 91.22 96.70 93.98
Predict
You can build the tagger only by loading the three trained model files (kwdlc_ner_model.vocabs, kwdlc_ner_model.params, kwdlc_ner_model.hp) to set arguments in nagisa.Tagger().
python tutorial_predict_kwdlc.py
1import nagisa
2
3
4def main():
5 # build the tagger for kwdlc
6 ner_tagger = nagisa.Tagger(vocabs='kwdlc_ner_model.vocabs',
7 params='kwdlc_ner_model.params',
8 hp='kwdlc_ner_model.hp')
9
10 # predict
11 text = "鈴木花子さんは2020年に京都大学を卒業しました。"
12 tokens = ner_tagger.tagging(text)
13 print(tokens)
14 #> 鈴木/B-PERSON 花子/I-PERSON さん/O は/O 2020/B-DATE 年/I-DATE に/O 京都/B-ORGANIZATION 大学/I-ORGANIZATION を/O 卒業/O し/O ました/O 。/O
15
16
17if __name__ == "__main__":
18 main()
Error analysis
By checking tag-level accuracy/entity-level micro-f1/classification_report, you can see what the model is wrong with.
python tutorial_error_analysis_kwdlc.py
1import nagisa
2
3from seqeval.metrics import f1_score
4from seqeval.metrics import accuracy_score
5from seqeval.metrics import classification_report
6
7
8def main():
9 # load the testset
10 test_X, test_Y = nagisa.utils.load_file("kwdlc.test")
11
12 # build the tagger for kwdlc
13 ner_tagger = nagisa.Tagger(vocabs='kwdlc_ner_model.vocabs',
14 params='kwdlc_ner_model.params',
15 hp='kwdlc_ner_model.hp')
16
17 # predict
18 true_Y = []
19 pred_Y = []
20 for words, true_y in zip(test_X, test_Y):
21 pred_y= ner_tagger.decode(words)
22
23 _pred_y = []
24 _true_y = []
25 for word, pred, true in zip(words, pred_y, true_y):
26 _pred_y.append(pred)
27 _true_y.append(true)
28 true_Y.append(_true_y)
29 pred_Y.append(_pred_y)
30
31 # evaluate
32 accuracy = accuracy_score(true_Y, pred_Y)
33 print("accuracy: {}".format(accuracy))
34 f1 = f1_score(true_Y, pred_Y)
35 print("micro-f1: {}".format(f1))
36 report = classification_report(true_Y, pred_Y)
37 print(report)
38
39
40if __name__ == "__main__":
41 main()
42
This is the result of the error analysis on the test set.
accuracy: 0.9671861495999331
micro-f1: 0.5798764342453663
precision recall f1-score support
ARTIFACT 0.24 0.27 0.26 114
DATE 0.79 0.79 0.79 245
LOCATION 0.69 0.69 0.69 399
MONEY 0.90 0.90 0.90 20
OPTIONAL 0.25 0.09 0.13 35
ORGANIZATION 0.33 0.40 0.36 168
PERCENT 0.75 0.75 0.75 8
PERSON 0.50 0.53 0.51 118
TIME 0.25 0.12 0.17 16
micro avg 0.57 0.59 0.58 1123
macro avg 0.52 0.50 0.51 1123
weighted avg 0.58 0.59 0.58 1123