spacy 1.7.0

v1.7.0: New 50 MB model, CLI, better downloads and lots of bug fixes

on Python PyPI

✨ Major features and improvements

• NEW: Improved English model.
• NEW: Additional smaller English model (50 MB, only 2% less accurate than larger model).
• NEW: Command line interface to download and link models, view debugging info and print Markdown info for easy copy-pasting to GitHub.
• NEW: Alpha support for Finnish and Bengali.
• Updated language data for Swedish and French.
• Simplified import of lemmatizer data to make it easier to add lemmatization for other languages.

Improved model download and installation

To increase transparency and make it easier to use spaCy with your own models, all data is now available as direct downloads, organised in individual releases. spaCy v1.7 also supports installing and loading models as Python packages. You can now choose how and where you want to keep the data files, and set up "shortcut links" to load models by name from within spaCy. For more info on this, see the new models documentation.

# out-of-the-box: download best-matching default model
python -m spacy download en

# download best-matching version of specific model for your spaCy installation
python -m spacy download en_core_web_md

# pip install .tar.gz archive from path or URL
pip install /Users/you/en_core_web_md-1.2.0.tar.gz
pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-1.2.0/en_core_web_md-1.2.0.tar.gz

# set up shortcut link to load installed package as "en_default"
python -m spacy link en_core_web_md en_default

# set up shortcut link to load local model as "my_amazing_model"
python -m spacy link /Users/you/data my_amazing_model

nlp1 = spacy.load('en')
nlp2 = spacy.load('en_core_web_md')
nlp3 = spacy.load('my_amazing_model')

⚠️ Backwards incompatibilities

• IMPORTANT: Due to fixes to the lemmatizer, the previous English model (v1.1.0) is not compatible with spaCy v1.7.0. When upgrading to this version, you need to download the new model (en_core_web_md v1.2.0). The German model is still valid and will be linked to the de shortcut automatically.
• spaCy's package manger sputnik is now deprecated. For now, we will keep maintaining our download server to support the python -m spacy.{en|de}.download all command in older versions, but it will soon re-route to download the models from GitHub instead.
• English lemmatizer data is now stored in Python files in spacy/en and the WordNet data previously stored in corpora/en has been removed. This should not affect your code, unless you have added functionality that relies on these data files.

This will be the last major release before v2.0, which will introduce a few breaking changes to allow native deep learning integration. If you're using spaCy in production, don't forget to pin your dependencies:

# requirements.txt
spacy>=1.7.0,<2.0.0

# setup.py
install_requires=['spacy>=1.7.0,<2.0.0']

🔴 Bug fixes

• Fix issue #401: Contractions with 's are now lemmatized correctly.
• Fix issue #507, #711, #798: Models are now available as direct downloads.
• Fix issue #669: Span class now has lower_ and upper_ properties.
• Fix issue #686: Pronouns now lemmatize to -PRON-.
• Fix issue #704: Sentence boundary detection improved with new English model.
• Fix issue #717: Contracted verbs now have the correct lemma.
• Fix issue #730, #763, #880, #890: A smaller English model (en_core_web_sm) is now available.
• Fix issue #755: Add missing import to prevent exception using --force.
• Fix issue #759: All available NUM_WORDS are now recognised correctly as like_number.
• Fix issue #766: Add operator to matcher and make sure open patterns are closed at doc end.
• Fix issue #768: Allow zero-width infix tokens in tokenizer exceptions.
• Fix issue #771: Version numbers for ujson and plac are now specified correctly.
• Fix issue #775: "Shell" and "shell" are now excluded from English tokenizer exceptions.
• Fix issue #778: spaCy is now available on conda via conda-forge.
• Fix issue #781: Lemmatizer is now correctly applied to OOV words.
• Fix issue #791: Environment variables are now passed to subprocess calls in cythonize.
• Fix issue #792: Trailing whitespace is now handled correctly by the tokenizer.
• Fix issue #801: Update global infix rules to prevent attached punctuation in complex cases.
• Fix issue #805: Swedish tokenizer exceptions are now imported correctly.
• Fix issue #834: load_vectors() now accepts arbitrary space characters as word tokens.
• Fix issue #840: Use better regex for matching URL patterns in tokenizer exceptions.
• Fix issue #847: "Shed" and "shed" are now excluded from English tokenizer exceptions.
• Fix issue #856: Vocab now adds missing words when importing vectors.
• Fix issue #859: Prevent extra spaces from being added after applying token_match regex.
• Fix issue #868: Model data can now be downloaded to any directory.
• Fix issue #886: token.idx now matches original index when text contains newlines.

📖 Documentation and examples

• NEW: Models usage documentation.
• NEW: Command line interface documentation.
• NEW: spacy-models, including the latest model releases.
• NEW: Linear model feature scheme API docs.
• Add help wanted (easy) issue label for contribution requests suitable for beginners.
• Update installation docs with more details on model download and building spaCy from source.
• Fix various typos and inconsistencies.

👥 Contributors

This release is brought to you by @honnibal and @ines. Thanks to @magnusburton, @jktong, @JasonKessler, @sudowork, @oiwah, @raphael0202, @latkins, @ematvey, @Tpt, @wallinm1, @knub, @wehlutyk, @vaulttech, @nycmonkey, @jondoughty, @aniruddha-adhikary, @badbye, @shuvanon, @rappdw, @ericzhao28, @juanmirocks and @rominf for the pull requests!