This release adds new ANN backends for RaBitQ and Zvec Sparse, native support for ONNX vectors along with a bonanza of improvements and bug fixes
See below for full details on the new features, improvements and bug fixes.
New Features
- Add ONNX vectors backend (#1211, #1212) @joseph-bergin
- Add RaBitQ as an ANN backend (#1244, #1251) @pma1999
- Add Zvec Sparse ANN (#1280, #1307) @Ni0ri
- Require Python 3.11 (#1273)
Improvements
- Update vectors package to standardize fallback logic (#1221)
- Support pooling_mode with models trained with recent Sentence Transformers (#1292)
- Refactor ANN unit tests (#1305)
- Support trust_remote_code in associated AutoModels (#1342)
Bug Fixes
- Latest version of LiteLLM is broken with Python 3.10+ (#1193)
- Fix bash tool hanging on commands that read stdin (#1209, #1210) @pacocartones
- fix(api): parse the graph query parameter as a bool (#1213) @serhiizghama
- Load the Milvus collection when opening a saved index (#1214) @morgan-coded
- Ignore padding rows in LateEncoder scores (#1215) @morgan-coded
- Resolve label filters by label id (#1216) @morgan-coded
- Improve late pooling configuration errors (#1217) @morgan-coded
- Keep count() accurate after repeated deletes in GGML and IVFSparse (#1218) @morgan-coded
- Delete every index entry for an id in Terms (#1219) @morgan-coded
- fix(vectors): remove the sparse spool file when checkpointing is disabled (#1220) @serhiizghama
- Keep the archive working directory alive when saving a loaded index (#1222) @morgan-coded
- Key the vectors model cache by configuration (#1223) @morgan-coded
- Save empty keyword and content indexes without errors (#1224) @morgan-coded
- fix(workflow): add request timeout to ServiceTask (#1225, #1226) @pacocartones
- Select the top documents in LateEncoder by the document axis (#1227) @morgan-coded
- Fix ANN delete on quantized Torch stores and tiny GGML indexes (#1228) @morgan-coded
- Fix Transform.stream() yielding a None text value into the encoder (#1231) @AmirF194
- fix(ann): honor quantize: false in the Faiss and SQLite backends (#1234) @serhiizghama
- Fix quantize: false silently enabling Q8_0 quantization in the GGML backend (#1235, #1236) @AmirF194
- Fix Cluster combining avg() as an unweighted mean of shard averages (#1237) @AmirF194
- fix(database): don't drop rows when sorting or grouping by a JSON field (#1238) @serhiizghama
- Latest version of Transformers has breaking change for Text to Audio pipeline (#1239)
- Save IVFSparse indexes after deleting with NumPy ids (#1241) @morgan-coded
- Set the HNSW delete counter when loading an index (#1242) @morgan-coded
- Keep community nodes that match none of the topic terms (#1243) @linhongyu510
- fix(api): forward zero weights and url-encode parameters in Cluster search (#1245) @serhiizghama
- Honor the nfeatures setting in IVFSparse (#1246, #1250) @liugj0710
- Keep deleted keyword results excluded after common-term scoring (#1247, #1248) @LimbC-C
- Keep SQLite ANN data when saving a modified index to a new path (#1249) @Irahan2
- Fix DuckDB.formatargs dropping repeated named bind parameters (#1252, #1253) @AmirF194
- Preserve local filenames in RetrieveTask (#1255, #1256) @dingpuyu
- Support explicit LIKE escapes in wildcard term queries (#1258, #1259) @LimbC-C
- Preserve falsy TemplateTask rule matches (#1261, #1262) @HuaTNA
- Label QA spans without the answer with the CLS token (#1263, #1264) @huiq777
- Fix training with pandas DataFrames (#1265, #1266) @huiq777
- Preserve reusable RagTask inputs (#1267, #1268) @HuaTNA
- Pack rows into chunks up to and including maxlength (#1269, #1270) @huiq777
- Reject iterator inputs for multi-action tasks without buffering (#1271, #1272) @LimbC-C
- Preserve keyword document ID types across save/load (#1274, #1275) @HuaTNA
- Fall back to root tokenizers for ONNX and LiteRT Hub subdirectory models (#1277) @morgan-coded
- Ignore negative deletion IDs in NumPy and Torch ANN indexes (#1278, #1279) @LimbC-C
- Normalize SQLite binary ANN similarity scores (#1281, #1282) @LimbC-C
- Format nested HTML lists without duplicating items (#1284, #1286) @MohammadHijjawi97
- Keep uppercase tokens in alphanumeric tokenization with lowercase=False (#1285, #1287) @MohammadHijjawi97
- Open lazy SQLite connections before saving indexes (#1288, #1290) @LimbC-C
- Include offset rows in similar() candidates (#1289, #1291) @devYRPauli
- Pad batched inputs in the Summary pipeline (#1294, #1295) @Bdysj
- Store only the quantized index in the RaBitQ backend (#1297) @pma1999
- Preserve integer document IDs and numeric column types in Tabular (#1298, #1299) @LimbC-C
- Return OpenAI-compliant index, created and finish_reason from chat completions (#1300, #1301) @Bdysj
- Delete the requested rows when ANN ids are a NumPy array (#1302, #1303) @kartsan03
- Support append and delete in the RaBitQ ivf mode (#1304) @pma1999
- Fix ordinary streaming output buffering with stripthink enabled (#1310, #1311) @tuanzirwar
- Keep graph categories aligned with topics after a delete (#1313, #1315) @devYRPauli
- Parse true and false as boolean literals in SQL queries (#1314, #1316) @devYRPauli
- Keep all rows for native vstack merges with list inputs (#1319, #1320) @Zhuoxi2000
- Cache document token vectors in Reranker (#1321) @morgan-coded
- Keep 0 and False in concat merge outputs (#1322, #1323) @Lesereingrape
- Fix DuckDB bind parameters followed by a non-word character not being converted (#1324, #1325) @Lesereingrape
- Decode bash tool output as UTF-8 instead of locale encoding (#1326, #1327) @manobhisriram
- Skip NULL aggregate values from shards with no matching rows (#1330, #1331) @gauravch-code
- Respect explicitly selected falsy ID column names in Tabular (#1332, #1333) @HuaTNA
- Keep node id 0 in graph path search results (#1334, #1335) @MohammadHijjawi97
- docs: correct the workflow input tuple from (2, 2) to (2, 8) (#1336, #1337) @shabeeth2
- Keep DuckDB bind values in query order when a parameter repeats (#1338) @serhiizghama
New Contributors
- @pacocartones made their first contribution in #1210
- @joseph-bergin made their first contribution in #1212
- @Irahan2 made their first contribution in #1249
- @dingpuyu made their first contribution in #1256
- @linhongyu510 made their first contribution in #1243
- @LimbC-C made their first contribution in #1248
- @liugj0710 made their first contribution in #1250
- @HuaTNA made their first contribution in #1262
- @huiq777 made their first contribution in #1270
- @pma1999 made their first contribution in #1251
- @MohammadHijjawi97 made their first contribution in #1287
- @Bdysj made their first contribution in #1295
- @kartsan03 made their first contribution in #1303
- @Ni0ri made their first contribution in #1307
- @Zhuoxi2000 made their first contribution in #1320
- @Lesereingrape made their first contribution in #1325
- @manobhisriram made their first contribution in #1327
- @tuanzirwar made their first contribution in #1311
- @gauravch-code made their first contribution in #1331
- @shabeeth2 made their first contribution in #1337
Full Changelog: v9.13.0...v9.14.0