VectorChord-bm25 0.2 Release Notes

We are excited to announce the release of VectorChord-bm25 version 0.2. This release introduces significant improvements to the tokenizer functionality, enhancing flexibility, customization, and multi-language support. Below are the key updates in this release.

🚀 New Features and Improvements

Decoupled tokenizer

The tokenizer-related API has been extracted into a dedicated repository: pg_tokenizer.rs. This change introduces several key improvements:

1. Enhanced Customization

   • The tokenizer now offers more flexible configuration options to suit a wide range of use cases.
   • Users can now customize stopwords, synonyms, and stemmers to better align with their specific requirements.

2. Multi-Language Support

   • Support for multiple languages has been significantly enhanced, making the extension more versatile for global applications.

🔧 Upgrade Guide

• Review the Updated API: Users are encouraged to review the updated API in the pg_tokenizer.rs repository for detailed information on the new features and customization options.

What's Changed

• refactor: optimize block_wand by @silver-ymz in #43
• chore: Update README by @gaocegege in #44
• feat: add iso stopwords by @silver-ymz in #45
• docs: update readme about tokenizer by @silver-ymz in #46
• fix: set max token length to 2600 to follow btree index limit by @silver-ymz in #47
• fix(README): Reload the conf to make it work by @gaocegege in #48
• Adding german stopwords by @jenspapenhagen in #50
• chore: update dependencies by @silver-ymz in #51
• bump vchord to latest by @lukemarsden in #52
• docs: add missing agplv3 license by @silver-ymz in #55
• feat: cast int[] to bm25vector by @silver-ymz in #53
• !feat: remove tokenizer api by @silver-ymz in #56
• ci: update check pipeline by @silver-ymz in #57
• docs: update tokenizer in readme by @silver-ymz in #60
• ci: update release pipeline for 0.2.0 by @silver-ymz in #58

New Contributors

• @jenspapenhagen made their first contribution in #50
• @lukemarsden made their first contribution in #52

Full Changelog: 0.1.1...0.2.0