Meilisearch v1.2 introduces new filter operators IS NULL
and IS EMPTY
, a new route to delete documents with filters, and significant relevancy and performance updates.
🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.
Some SDKs might not include all new features—consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).
New features and improvements 🔥
Delete documents by filter
Meilisearch v1.2 allows you to use filters to delete documents with the new /documents/delete
route:
curl -X POST http://localhost:7700/indexes/dogs/documents/delete \
-H 'Content-Type: application/json' \
--data-binary '{ "filter": ["doggo = 'bernese mountain'", "face = cute"] }'
Fields must be set as filterable before you can use them as filters.
Meilisearch will return a summarized task object:
{
"taskUid": 242,
"indexUid": "dogs",
"status": "enqueued",
"type": "documentDeletion",
"enqueuedAt": "2023-05-03T11:01:58.721841Z"
}
Use the returned taskUid
with the task route to check the task status.
Done by @dureuill and @irevoire in #3550.
Get documents by filter
You can now use filters in the GET
endpoint of the /documents
route:
curl -X GET 'http://localhost:7700/indexes/dogs/documents?limit=1&filter=doggo=bernese'
You can also use the new /documents/fetch
route to handle complex filters:
curl -X POST http://localhost:7700/indexes/dogs/documents/fetch \
-H 'Content-Type: application/json' \
--data-binary '{ "limit": 1, "filter": "doggo = bernese" }'
/documents/fetch
accepts following parameters: limit
, offset
, fields
, and filter
.
Fields must be set as filterable before you can use them as filters.
Done by @dureuill and @irevoire in #3570.
New filter operators: IS EMPTY
and IS NULL
This release introduces two new filter operators. IS EMPTY
matches existing fields with a valid, but empty value. IS NULL
matches existing fields with an explicit null
value.
Given the following documents:
[{
"id": 0,
"color": []
},
{
"id": 1,
"color": null
},
{
"id": 2,
},
]
color IS EMPTY
matches document 0
.
color IS NULL
matches document 1
.
Both operators can be used together with the NOT
operator: color IS NOT EMPTY
and NOT color IS EMPTY
match document 1
. color IS NOT NULL
and NOT color IS NULL
match document 0
.
Neither operator will match documents missing the specified field.
Done by @Kerollmops in #3571.
Improve relevancy and search performance
The search engine has seen significant refactoring. This brings performance and relevancy improvements, as well as setting the foundations for new features related to search customization. Stay tuned!
If you have any questions or experience unexpected behavior, feel free to reach out.
Done by @loiclec, @dureuill, @ManyTheFish, @Kerollmops and @irevoire in #3542.
Performance improvements
- The fastest 75 percent of queries now consistently answer below 50ms on our test setup
- Complexity of search queries has been limited to limit CPU time and RAM consumption. More specifically:
- A single term is now limited to 150 possible typo matches for queries containing 1 typo, and to 50 possibilities for queries containing 2 typos
- Both single word or multi word queries consider a maximum of 50 synonyms
- The total number of words for all synonyms of a single term cannot exceed 100
- Queries can now contain a maximum of 1000 words. This change only impacts queries containing very long phrases. Search terms (words not inside a phrase, and individual phrases) remain limited to 10 per query
- Geo search improvements:
- Increased performance when geosorting small sets of documents
- Descending sort is now as performant as ascending sort
- Address assorted minor performance issues: #3124, #3378, #2202, #3123
Relevancy improvements
- The
exactness
ranking rule no longer treats synonyms as exact matches. This boosts documents containing the query exactly as typed by the user. Solves #3410 - Meilisearch will always sort results as if the
words
ranking rule were present with a higher priority than theattributes
,exactness
,typo
andproximity
ranking rules. This happens even if thewords
ranking rule has been removed or set with a lower priority. Search behavior when using the specified ranking rules before or withoutwords
was confusing and hurt relevancy - Split words are now also treated as possible digrams. This means the query
whit ehorse
may match documents containingwhite horse
, which improves the relevancy of typo tolerance - N-grams and split words are now ranked lower in comparison to the exact word when processing the
typo
ranking rule. For example, a query forsun flower
will now return a document titledSun Flower
before one titledSunflower
- Ranking rule behavior no longer depends on the number of ranked documents, improving consistency in terms of both relevancy and latency. Solves #3356
Automated task deletion
Maximum number of tasks the task queue can hold now limited to 1M. Fixes issues with enqueuing new tasks when the database is full.
Once the task queue reaches this limit, Meilisearch will try to delete the oldest 100k tasks. If not all 100k tasks can be deleted due to some of them being unfinished, Meilisearch will delete as many tasks as possible. If there are no unfinished tasks, Meilisearch will not delete anything but still enqueue new tasks as usual.
The engine will stop you from adding new tasks once the hard limit of 10GiB of tasks is reached (that’s between 5M and 15M of tasks depending on your workflow). In this case, Meilisearch will try to automatically delete unfinished tasks. If this is unsuccessful, it will not enqueue new tasks, and log a warning notifying the user the engine is not working properly.
Language support improvements
- Split camelCase in Latin segmenter
- Improve Arabic: normalizer and segmentation
Done by @ManyTheFish, @akeamc, @DrAliRagab, @goodhoko, and @mosuka in #3702 and in Charabia v0.7.2
Other improvements
- Introduce experimental feature to reduce the RAM usage. Can be useful if Meilisearch is sometimes killed by your OS. Check out this discussion to get more information and how to use it (#3651) @Kerollmops
- Add boolean support for CSV documents (#3576) @irevoire
- Improve geosort errors during sorting and filtering (#3638 and #3631) @cymruu
- Improve error message when payload is too large (#3739) @cymruu
Fixes 🐞
- Improve the
GET /health
route by ensuring the internal DB is not down (#3647) @irevoire - Handle single quotation mark (
’
) and apostrophe ('
) the same way (#3702) @ManyTheFish - Fix the sort error message: regression seen in v1.1 (#3749 and #3755) @ManyTheFish
Misc
- Dependencies upgrade
- CIs and tests
- Add scheduled tests for all features to Actions (#3510) @jlucktay
- Clean CI file names (#3567) @curquiza
- Enable cache again in test suite CI (#3587) @curquiza
- Remove CLI changes for clippy (#3464) @dureuill
- Improve docker cache in CIs (#3566) @inductor
- Remove Uffizzi because the team no longer uses it (#3694) @curquiza
- Add SDK tests to CI (#3709) @curquiza
- Use URL of our self-hosted bors instance (#3721) @curquiza
- Improve index-scheduler tests (#3639) @Sufflope
- Documentation
- Add newline after the meilisearch version in the issue template (#3636) @bidoubiwa
- Add sprint issue to template issues (#3643) @curquiza
- Update the prototype section in CONTRIBUTING.md (#3649) @curquiza
- Update README to reference new docs website (#3666, #3710, #3718 and #3720) @guimachiavelli @roy9495 @Kerollmops @curquiza
- Move comments above keys in
config.toml
(#3731) @jirutka
- Misc
❤️ Thanks again to our external contributors:
- Meilisearch: @cymruu, @GregoryConrad, @inductor, @jirutka, @jlucktay, @roy9495, and @Sufflope.
- Charabia: @akeamc, @DrAliRagab, @goodhoko, and @mosuka.