weaviate/weaviate v1.38.0

v1.38.0 - HFresh, Namespaces, Nested Object Filtering, Alter Schema Reindex property

on GitHub

Breaking Changes

none

HFresh (GA)

Extensive improvements including numerous performance optimizations reducing memory usage, disk writes, and allocations.

• hfresh: normalize query vector before rescoring by @asdine in #11097
• hfresh: use asymmetric distance computation by @asdine in #11098
• hfresh: guard distToNode against nil distancer during async init by @SAY-5 in #11087
• hfresh: fix metrics reporting by @asdine in #11140
• hfresh: retry transient errors during reassign by @asdine in #11124
• hfresh: dequeue analyze after ressignments by @asdine in #11177
• hfresh: backport rescoring fix by @asdine in #11205
• Backport hfresh task priorities by @asdine in #11408
• hfresh: reduce posting map memory usage by @asdine in #11358

Namespaces (Preview)

Namespaces add control-plane and data isolation between users on a shared cluster.

• Namespaces: Add RAFT entity, handlers and settings by @dirkkul in #11128
• Namespace: Add to principal and wire up db user by @dirkkul in #11143
• Namespaces: Add support for creating collections and many endpoints by @dirkkul in #11167
• Namespaces: Make RBAC casbin aware by @dirkkul in #11178
• Namespaces: Wire up OIDC and redefine build-in roles by @dirkkul in #11200
• Relax namespace naming regexp to allow UUIDs by @dirkkul in #11207
• Namespaces: Add support for delete and update collection by @dirkkul in #11213
• Namespaces: Wire up remaining schema endpoints by @dirkkul in #11216
• Wire up namespacing for tenant operations by @dirkkul in #11227
• Namespaces: Fix auto schema by @dirkkul in #11229
• Namespacs: Add namespace info to audit log by @dirkkul in #11250
• Namespaces: Add support for multi node apis by @dirkkul in #11249
• Namespaces: Add support for filters and enable batch delete by @dirkkul in #11264
• Add support for namespaces in SSB by @dirkkul in #11271
• Namespaces: Add cascading delete by @dirkkul in #11224
• Namespaces: Validate namespaces provided by global operator by @dirkkul in #11268
• Add namespacing to update shard by @dirkkul in #11286
• Namespaces: Wire up remaining alias endpoints by @dirkkul in #11285
• Namespace: Fix getting roles for namespace users by @dirkkul in #11287
• Add per-namespace restrictions for number of collections by @dirkkul in #11283
• Namespace deprecate incompatible endpoints by @dirkkul in #11288
• Namespaces: Wire up MCP by @dirkkul in #11344
• feat(schema): add validation for reserved property name suffixes by @amourao in #11122
• Namespace: object limit per namespace by @dirkkul in #11342
• Namespaces: Response stripping by @dirkkul in #11357
• Namespaces error stripping by @dirkkul in #11386
• Namespaces: Add support for references in schema by @dirkkul in #11374
• Namespaces: Strip query_profiling reply by @dirkkul in #11404
• feat: cross-ref namespace strip by @amourao in #11311
• Namespace: Cascading cleanup fixes by @dirkkul in #11423
• Namespace: Auto-schema and reindex bug fixes by @dirkkul in #11473
• Namespaces: Require RBAC to be enabled by @dirkkul in #11412
• Namespace flaky test by @dirkkul in #11476
• fix: enable ns xref filtersexp paths by @amourao in #11484
• Namespaces: Fix stripping in batch delete by @dirkkul in #11497
• Namespaces: Close test gaps by @dirkkul in #11499
• Namespace: Round of bugfixes by @dirkkul in #11505
• Namespaces: Add user management by @dirkkul in #11501
• fix: handle transient errors during namespaced user creation by @amourao in #11522
• Fix: namespace response strip by @amourao in #11498
• fix: move strip to outside ref loop by @amourao in #11528
• Concurrent namespace test by @dirkkul in #11604
• test: add retry logic for shard status retrieval in namespace tests by @amourao in #11628

Nested Object Filtering (Preview)

Adds ability to perform search within indexed JSON properties.

• Nested object filtering — Part 1: position encoding and assignment by @aliszka in #10954
• Nested object filtering — Part 2: write path analysis by @aliszka in #10955
• Nested object filtering — Part 3: LSM storage, write path, and bucket management by @aliszka in #10956
• Nested object filtering — Part 4: batched writes, delete path, and prefix-bounded reads by @aliszka in #10957
• Nested object filtering — Part 5: filter validation, execution, and integration tests by @aliszka in #10974
• Nested object filtering — Part 6: correlated AND resolution plan and executor by @aliszka in #10975
• Nested object filtering — Part 7: correlated AND resolution by @aliszka in #10976
• Nested object filtering — Part 8: IsNull support and arr[N] positional filtering by @aliszka in #11010
• Nested object filtering — Part 9: flat execution plan and arr[N] cross-index fix by @aliszka in #11048
• Nested object filtering — Part 10: pool-backed BitmapOps and lifecycle testing by @aliszka in #11084
• Nested object filtering — Part 11: path consolidation, delete cleanup, UUID and IsNull support by @aliszka in #11315
• Nested object filtering — Part 12: recursive plan/executor and same-element-at-LCA semantics by @aliszka in #11316
• Nested object filtering — Part 13: DB-level test consolidation and same-K-different-parent fix by @aliszka in #11317
• Nested object filtering — Part 14: regression baseline test sweep by @aliszka in #11338
• Nested object filtering — Part 15: position-level evaluation across operator subtrees by @aliszka in #11339
• Nested object filtering — Part 16: scope-aware NOT and IsNull by @aliszka in #11340
• Nested object filtering — Part 17: Contains* operators as first-class on nested paths by @aliszka in #11341
• Nested object filtering — Part 18: gRPC ingress for nested filter paths + Python e2e suite by @aliszka in #11365
• Nested object filtering — Part 19: Python e2e coverage for single-OBJECT intermediates by @aliszka in #11369
• Nested object filtering — Part 20: preview gate, GraphQL ingress, and pre-release correctness by @aliszka in #11387
• Nested object filtering — Part 21: 96-bit path hash, IsNull in operator subtrees, write-path allocation cuts by @aliszka in #11448
• chore: bumps sroar to v0.0.14 by @aliszka in #11626

Alter Schema - Reindex property (Preview)

Adds support for changing property's index types.

• Runtime property reindex (v1.38 Preview) by @etiennedi in #11326
• chore(reindex): post-merge cleanups — CI matrix split, configure_api helper extract, docs by @etiennedi in #11350
• [Reindex v1.38 Preview] Two-phase RAFT swap barrier for semantic reindex migrations by @etiennedi in #11328
• fix(reindex): compose multi-task progress into per-unit envelope by @etiennedi in #11346
• fix(cluster): cascade-delete DTM task records on collection drop (closes weaviate/0-weaviate-issues#231) by @etiennedi in #11345
• fix(reindex): graceful-restart resilience for in-flight units by @etiennedi in #11370
• test+fix(reindex/distributedtask): recovery-convergence matrices + #244/#246 fixes by @etiennedi in #11415
• fix(distributedtask): close #240 orphaned-unit class (throttler retry + RecordUnitCompletion NodeID) by @etiennedi in #11428
• [Reindex v1.38 Preview] Backup × runtime-reindex fixes by @etiennedi in #11327
• chore(reindex): fix linter errors for reindex handler by @antas-marcin in #11490
• fix(reindex): promote MapToBlockmax to a semantic migration (closes weaviate/0-weaviate-issues#254 finding 3) by @etiennedi in #11500
• test(reindex-multinode): use atomic Container.Restart for rolling-restart cycles by @etiennedi in #11506
• fix(reindex-tests): plant torn sentinels across restart boundary by @etiennedi in #11507
• test(reindex-backup): poll for orphan sidecar dir removal (Mode B flake) by @etiennedi in #11514
• test(reindex-backup): fix BackupRefusedDuringInFlightMigration flake by @etiennedi in #11512
• fix(reindex): atomic tokenization overlay swap by @etiennedi in #11513
• test(distributedtask): fix two reactive-firing test-harness flakes by @etiennedi in #11518
• test(reindex-singlenode): poll DeleteClass past the in-flight migration guard (APIValidation flake) by @etiennedi in #11515
• test(reindex): replace time.Sleep with dynamic polling, cap poll intervals at 50ms by @etiennedi in #11495

Secure MCP Server (Preview)

Secure Built-In Model Context Protocol (MCP) server interface. This allows AI agents (such as Claude, IDEs, etc.) to natively read and write to Weaviate without requiring custom code. The implementation shifts Weaviate from a passive retrieval engine to an active "long-term memory" for agentic workflows, supporting hybrid search, RAG, and multi-tenancy out of the box.

• [MCP] Refactor permissions by @g-despot in #11037
• [MCP] Make MCP env vars runtime-configurable by @g-despot in #11176
• [MCP] Add Prometheus metrics for the MCP server by @g-despot in #11284
• [MCP] Align hybrid default alpha with GraphQL/gRPC (0.5 → 0.75) by @g-despot in #11537

Async Replication Prod Readiness

Async replication optimizations and fixes.

• fix: optimize zstd encoder usage in replication client by @jeroiraz in #11078
• test: set timeout unit for replication client in HashTreeLevel tests by @jeroiraz in #11158
• feat: per-level discriminants for hashtree peer comparison by @jeroiraz in #11204
• feat: async replication scheduler by @jeroiraz in #11155
• fix: prevent race condition in async replication scheduler's dispatch logic by @jeroiraz in #11266
• feat: async replication using CompareDigests by @jeroiraz in #11223
• fix: update async replication frequency handling by @jeroiraz in #11297
• feat: Implement async checkpoint functionality by @jeroiraz in #10926
• feat: async replication enabled by default by @jeroiraz in #11214
• feat: add async replication compatibility layer by @jeroiraz in #11502
• chore: lower async replication scheduler defaults by @jeroiraz in #11630

Boost API

Soft-ranking support.

• Add Boost API for soft-ranking by @trengrj in #11103
• Boost scoring performance improvements by @trengrj in #11446

Backup reliability improvements

Improves backup reliability for large collections, includes inactive tenants, avoids halting compactions and introduces incremental backups.

• fix(backup-gcs): use delimiter listing in AllBackups to avoid full object scan by @gkampitakis in #11234
• fix(backup): narrow shardCreateLocks scope to unblock queries during snapshot (#234) by @etiennedi in #11349
• Cherry pick fix to backups blocking during compactions by @tsmith023 in #11359
• fix(backup-gcs): skip not found errors on AllBackups by @gkampitakis in #11389
• Add root only baseBackupId return by @dirkkul in #11417
• Reduce size of backup to speed up test in ci by @tsmith023 in #11479
• Add more unit tests for backups by @dirkkul in #10769

Production-ready Replica Movement

• fix(replication): use writable shards in OverwriteObjects() by @moogacs in #11002
• refact(replication): introduce replication error pkg and unify erroring by @moogacs in #10026
• [feat] Introduce change capture log in replica movement by @tsmith023 in #11201
• Remove additionalWriteReplicas from routing behaviour by @tsmith023 in #11590
• Harden replica movement gating before CCL sealing by @tsmith023 in #11631
• Ensure that CCL is started when local schema is up-to-date with leader by @tsmith023 in #11640

Modules & Integrations

• Update gemini model path by @erika-shorten in #11118
• modules: remove EOL Anthropic models from generative-aws e2e tests by @antas-marcin in #11195
• (fix): Adjust MaxObjectsPerBatch from 150 to 100 for google embeddings by @jpsoultanis-kapa in #11277
• add digitalocean text2vec module by @dillonledoux in #11298
• chore: adjust text2vec-google batch logic by @antas-marcin in #11279
• modules: new text2vec-digitalocean module by @antas-marcin in #11318
• feat(modules): introduce rate limiter logic in batch simple logic by @antas-marcin in #11637

Performance & Optimization

• startup: speedup startup by checking db status more often by @asdine in #11064
• Improve compressed vector index cache by @trengrj in #11059
• Fix startup race in object TTL by @dirkkul in #11095
• Switch ObjectTTL log level to debug by @dirkkul in #11096
• Reorder timeout checks for raft ready and leadership elections in attempt to avoid flaking in ci by @tsmith023 in #11105
• Collection export: Speed up snapshotting with concurrent tenant de-activation by @dirkkul in #11031
• compactv2: migrate all compactv2 files during startup by @asdine in #11150
• Add default vector index setting by @trengrj in #11042
• async indexing: retry forever when disk is full by @asdine in #11168
• Ignore none for default quantization by @trengrj in #11175
• queue: promote partial chunk in PrepareForBackup to prevent S3 BadDigest by @rlmanrique in #11211
• feat: dont read class name from objects bucket by @amourao in #11199
• feat: add support for skipping class name in object storage by @amourao in #11240
• fix(lsmkv): remove flakiness in bucket threshold tests via synctest by @aliszka in #11343
• Support named vectors in default vector index by @trengrj in #11364
• queue: fix trigger behavior by @asdine in #11379
• feat(restrictions): split searchable.rebuild from algorithm switch (closes weaviate/0-weaviate-issues#227) by @etiennedi in #11383
• Increase SSB memlimit threshold from 80% to 90% by @tsmith023 in #10660
• gh-11440 parse BM25 property boosts as floats by @msnandhis in #11471
• Restrict user assignment permissions to roles the user has permission to by @dirkkul in #11493
• Add deterministic tie-breaking for equal-distance vector search results by @abdelr in #11632
• Use local HuggingFace cache for ann-datasets to avoid 429 by @asdine in #11649

Bug Fixes

• Fix access-check race in multi-node Azure backup Initialize by @jfrancoa in #11026
• fix: handle secondary index size accumulation for varying key counts by @amourao in #11060
• fix: refactor bucket creation to use options for sec indices by @amourao in #11067
• Set export default path to empty by default by @dirkkul in #11068
• fix(ttl): guarantee tenant re-deactivation when TTL context is canceled mid-deletion by @jfrancoa in #11063
• Fix 'succesfully' typo in benchmark/remote/run.sh by @MukundaKatta in #11046
• fix: support old stopword index config on endpoint by @amourao in #11079
• fix: flush memtables after queue drain in HaltForTransfer by @rlmanrique in #10991
• fix(cluster): prevent shutdown hangs by buffering close signal channels by @moogacs in #11114
• fix: normalize negative zero in LexicographicallySortableFloat64 to fix inverted index filters by @anishesg in #11120
• docs: fix TextAnalyzerConfig descriptions for immutabiity by @amourao in #11127
• MMR Fix for Multi-Node by @trengrj in #11041
• hnsw: fix snapshotFileName for compactv2 .sorted commit logs by @asdine in #11147
• Fix concurrent sends and dropped reports in SSB by @tsmith023 in #11145
• Fix recursive RAFT commands in func (s *SchemaManager) UpdateShardStatus path by @tsmith023 in #11186
• chore(test_usage): relax vector storage bytes check to reduce flakiness by @antas-marcin in #11270
• fix: drain dispatcher workCh correctly by @jeroiraz in #11293
• fix(offload-s3): build per-call cli.App to avoid concurrent freeze panic by @moogacs in #11289
• Fix flaky TestMakeUnboundedChannel_CapShrink by @asdine in #11380
• fix: duplicated words in commit_log_combiner, queue.go, and path.go comments by @oab24413gmai in #11299
• fix: initialize dateAggregator max sentinel to MinInt64 to handle pre-1970 dates by @anishesg in #11130
• fix: skip stale doc IDs in HFRESH rescore and flat search after deletion by @anishesg in #11132
• [FIX] - Check DB user identifier collisions in auth index by @kedar49 in #11381
• fix(inverted): wrap child error in sequential nested AND/OR by @SAY-5 in #11363
• Reject duplicate static API keys by @3em0 in #11393
• fix: deflake TestReplicationAbort/Error and /DecodeResponse by @moogacs in #11419
• fix: deflake TestReplicationAbort/Error and /DecodeResponse by @moogacs in #11421
• fix(test): bump CancelClearsAcrossReplicas backup deadline 60s → 120s by @etiennedi in #11453
• Fix/hnsw findnewentrypoint panic by @abdelr in #11151
• fix(compression): skip readonly bucket write-back in recoverCompresse… by @trengrj in #11458
• fix(lsmkv): make compaction abort immediately on cancel by @etiennedi in #11461
• fix(db): sync-rename + async RemoveAll on collection / shard delete by @etiennedi in #11462
• fix(db): make async-delete drop idempotent for a missing source by @etiennedi in #11477
• Fix dangling-alias batch insert silently re-creating the deleted target by @jfrancoa in #11489
• fix(lsmkv): avoid recursive bucketAccessLock.RLock and harden Pause/ResumeObjectBucketCompaction concurrency by @etiennedi in #11486
• Improve assertions and log verbosity in TestReplicationReplicateWithLazyShardLoading by @tsmith023 in #11481
• fix(rbac): pass principal to prettyPermissionsResources (main is red) by @etiennedi in #11524
• test: deflake TestReplicationDigestObjectsInRange/TruncatedBinaryRecord by @jeroiraz in #11511
• test: drop flaky RegisterAssignsMonotoneSeq subtest by @jeroiraz in #11532
• test: update tests to ensure input beacons are echoed on failed rows by @amourao in #11531
• use the same update time for batch references everywhere by @dirkkul in #11553
• refactor(schema): extract tokenize logic into shared usecase by @mpartipilo in #11180
• Revert "refactor(schema): extract tokenize logic into shared usecase" by @antas-marcin in #11558
• fix: add support for strfmt.URI in reference aggregation and strip NS by @amourao in #11526
• fix(inverted): index references provided as untyped JSON beacons by @moogacs in #11551
• Fix race in dynamic users with concurrent pointer access by @dirkkul in #11573
• Fix failing linter + tests due to merge forward divergence by @tsmith023 in #11591
• Interpret empty string as empty for slices in dynamic runtime config by @dirkkul in #11589
• Remove target from additionalWriteReplicas to avoid races between the CCL catch-up and hot-path writes by @tsmith023 in #11588
• fix(resource): don't auto-recover shards read-only for a non-resource reason by @amourao in #11581
• test: fix data race in TestAsyncRepairMultiTenancyColdTenantConfigUpdate by @jeroiraz in #11594
• test: deflake TestReplicationDigestObjectsInRange/TruncatedBinaryRecord (backport #11511 to v1.36) by @jeroiraz in #11595
• test: deflake replication client tests via generous default timeoutUnit by @jeroiraz in #11601
• Fix flaky HNSW search test by @dirkkul in #11579
• Fix flaky usage test by @dirkkul in #11617
• Remove flaky dynamic index latency assertion by @asdine in #11623

Observability

• chore: disable cloud info if telemetry is disabled by @antas-marcin in #11202
• Reduce log spam waiting for DB by @dirkkul in #11215
• Usage limits: server-side guardrails for objects/collections/tenants/shards by @etiennedi in #11212
• feat: vector index + compression allow-lists by @etiennedi in #11313
• Add more info to RBAC denied log by @dirkkul in #11517
• implements telemetry usage for X-Weaviate-Client-Integration by @dudanogueira in #10535
• Remove deprecation notice for weaviate.json config file by @dirkkul in #11615

Refactoring & Code Quality

• chore: change main Weaviate version to v1.38.0-dev by @antas-marcin in #11065
• Disable compactions in backup test by @dirkkul in #11082
• Exclude auto-generated files in orca by @dirkkul in #11107
• chore(test): fix flakey tests TestRaftEndpoints TestSnapshotRestoreSchemaOnly by @moogacs in #11116
• test: automate QA'ing pull requests by @antas-marcin in #11232
• chore: pass current user as assignee in qa_pr.sh script by @antas-marcin in #11269
• feat: add possibility to pass e2e and chaos branch names to qa_pr.sh script by @antas-marcin in #11290
• chore: add support for multiarch PR docker images in qa_pr.sh script by @antas-marcin in #11291
• feat: create simple create_release.sh script by @antas-marcin in #11301
• release: add QA option to create_release.sh script by @antas-marcin in #11314
• tools/dev: add release.sh release driver by @mpartipilo in #11241
• chore: remove release.sh script by @antas-marcin in #11367
• chore: modify create_release.sh script output by @antas-marcin in #11368
• chore: don't log issue link in qa_pr.sh script by @antas-marcin in #11504
• Dynamic runtime list parsing by @dirkkul in #11605
• Use flat index to assert brute force search by @dirkkul in #11608
• fix: increase retry wait time and reduce timeout for Docker Hub login by @amourao in #11610
• Speed up rbac authz tests by @dirkkul in #11607
• Ensure test errors are correctly logged to aid debugging by @tsmith023 in #11611
• Reuse testcontainers in tests for fast group 3 by @dirkkul in #11612
• More test speedups by reusing containers by @dirkkul in #11619

Testing

• test(acceptance): make Weaviate host/port configurable via env vars by @mpartipilo in #11153
• test/docker: per-network subnet with retry to fix concurrent network overlap by @etiennedi in #11520

CI/CD & Infrastructure

• ci: delegate SHA-pinning to native policy, expose unicode scan as composite by @mpartipilo in #11093
• Fix linter errors after CI update by @dirkkul in #11188
• ci: switch docker image PR builds and docker report CI jobs to use Ubicloud runners by @antas-marcin in #11258
• fix(ci): disable job reruns for dependabot PR's by @antas-marcin in #11259
• fix(ci): use github's ubuntu-24.04-arm runner for PR ARM docker image build by @antas-marcin in #11261
• fix: docker login and dowload retry by @amourao in #11599
• ci: introduce fast group 5 tests by @antas-marcin in #11622
• ci(docker): switch docker pipelines to use Ubicloud's premium runners by @antas-marcin in #11641
• ci: switch back multi-arch docker image build pipeline to ubuntu-latest by @antas-marcin in #11646

Security

• cluster(raft): add basic auth support to Raft cluster endpoints by @antas-marcin in #11242

Dependencies

• dependencies(update): bump github.com/buger/jsonparser from 1.1.2 to 1.2.0 by @dependabot[bot] in #11184
• dependencies(update): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.6.3 to 1.6.4 by @dependabot[bot] in #11183
• Bump github/codeql-action from 4.35.1 to 4.35.3 by @dependabot[bot] in #11187
• dependencies(update): bump golang.org/x/net from 0.52.0 to 0.53.0 by @dependabot[bot] in #11181
• build(deps): bump goreleaser/goreleaser-action from 7.0.0 to 7.2.1 by @dependabot[bot] in #11138
• chore(deps): bump actions/cache from 5.0.4 to 5.0.5 by @dependabot[bot] in #11090
• dependencies(update): adjust dependencies is Go sub projects by @antas-marcin in #11189
• chore(deps): bump aws-actions/configure-aws-credentials from 6.1.0 to 6.1.1 by @dependabot[bot] in #11252
• chore(deps): bump github/codeql-action from 4.35.3 to 4.35.4 by @dependabot[bot] in #11251
• Bump docker/setup-buildx-action from 4.0.0 to 4.1.0 by @dependabot[bot] in #11441
• Bump psf/black from 26.3.1 to 26.5.1 by @dependabot[bot] in #11445
• Bump github/codeql-action from 4.35.4 to 4.36.0 by @dependabot[bot] in #11444
• Bump codecov/codecov-action from 6.0.0 to 6.0.1 by @dependabot[bot] in #11443
• Bump goreleaser/goreleaser-action from 7.2.1 to 7.2.2 by @dependabot[bot] in #11442
• chore(deps): adjust go.sum by @antas-marcin in #11483

New Contributors

• @MukundaKatta made their first contribution in #11046
• @SAY-5 made their first contribution in #11087
• @anishesg made their first contribution in #11120
• @jpsoultanis-kapa made their first contribution in #11277
• @dillonledoux made their first contribution in #11298
• @oab24413gmai made their first contribution in #11299
• @kedar49 made their first contribution in #11381
• @3em0 made their first contribution in #11393
• @msnandhis made their first contribution in #11471

Full Changelog: v1.37.0...v1.38.0