✨ New features and improvements
- Allow thinc v10, the continuation of thinc v8.3.x.
- Support SudachiPy 0.7 for the Japanese tokenizer.
registry.<name>.get_all()now lists spaCy's registered functions and factories without having to load a pipeline first.
🔴 Bug fixes
- Fix the rule-based lemmatizer preferring the last matching rule over the first, which since v3.8.8 gave wrong lemmas for many English verbs, e.g. "rides" → "rid", "hoped" → "hop", "sites" → "sit" (#13860, #3052). The fix for hyphenated words from #13816 is kept ("co-authored" → "co-author").
- Keep German gender-inclusive colon forms like "Kund:innen", "Ärzt:in" and "eine:n" as single tokens (#14010). Other colons between letters are still split, e.g. "Achtung:Bitte". Packaged pipelines keep the tokenizer rules they were trained with, so this applies to blank German pipelines and newly trained ones.
- Show the mecab-ko install message when natto can't load MeCab.
- Handle
Morpheme.split()returning an empty list in the Japanese tokenizer.