pypi sentencepiece 0.1.8
v0.1.8

latest releases: 0.2.0, 0.1.99, 0.1.98...
5 years ago

Feature: Get rid of the dependency to external protobuf
Feature: added (Encode|Decode)AsSerializedProto interface so Python module can get full access to the SentencePieceText proto including the byte offsets/aligments
Feature: added --treat_whitespace_as_suffix option to make _ be a suffix of word.
Feature: Added normalization rules to remove control characters in the default nmt_* normalizers
Minor fix: simplify the error messager
Minor fix: do not emit full source path in LOG(INFO)

For more detail: v0.1.7...v0.1.8

Don't miss a new sentencepiece release

NewReleases is sending notifications on new releases.