Main changes

• Add a Wasm demo #115
• Handle locale on the Wasm demo #119
• Add bi-gram feature info generator for MeCab models #121
• Embed a magic number into a model #129

Precompiled model files

We provide precompiled models for Vibrato, allowing you to get started with tokenization easily. You can download them from Assets in this release. The licenses are contained in each file.

All models were compiled and modified in the manners described in compile.md and map.md. We trained the mappings of connection ids using CORE data in BCCWJ v1.1 (except the PN category).

Note that all the models are compressed in zstd format. You can directly input them to Vibrato CLIs, but if using vibrato APIs, you need to extract them outside the APIs (see README).

Models trained using Vibrato

The three variants are trained using BCCWJ v1.1 (except the PN category) and UniDic v3.1.1.

• bccwj-suw+unidic-cwj-3_1_1: A standard version.
• bccwj-suw+unidic-cwj-3_1_1+compact: A smaller version that compresses the connection matrix in the manner described in small-dic.md.
• bccwj-suw+unidic-cwj-3_1_1-extracted+compact: A further smaller version that contains only POS and pronunciation features.

These models were trained with L1-regularization.

Models converted from publicly-available resources

• ipadic-mecab-2_7_0 from IPADIC v2.7.0
• jumandic-mecab-7_0 from mecab-jumandic-utf8 v7.0
• naist-jdic-mecab-0_6_3b from NAIST Japanese Dictionary v0.6.3b
• unidic-mecab-2_1_2 from UniDic v2.1.2
• unidic-cwj-3_1_1 from UniDic v3.1.1
• unidic-cwj-3_1_1+compact from UniDic v3.1.1, whose connection matrix is compressed in a manner of mecab_smalldic.

Statistics for compressed UniDic models

The following table shows UniDic model sizes in the two versions: without and with +compact (not in zstd format).