v1.41.0 🏈
-
Support dynamic search rules with pinning by @YoEight in https://github.com/meilisearch/meilisearch/pull/6182 Introduce the Dynamic Search Rules (DSR) experimental feature, making it easy to promote the right content for the right searches with flexible, condition-based pinning. In this first version, rules can be triggered by query-based conditions such as empty queries or literal substring matches, as well as time windows. Multiple documents can be pinned in a defined order, and pins still work with filtering, pagination, facet distribution, hybrid search, and federated search.
-
Default to
useNetwork: truein sharded instances by @dureuill in https://github.com/meilisearch/meilisearch/pull/6278This release contains breaking changes for users of the
networkexperimental feature.§ Breaking changes
When
network.leaderis set in the instance,useNetworknow defaults totruein search requests when omitted.So, when a network is configured, the following:
Search request without an explicit useNetwork
is now equivalent to a federated search over multiples remotes so that all shards in the network are covered exactly once.
Equivalent federated search request
// POST /multi-search { "federation": { "limit": 5, "facetsByIndex": { "movies": [ "genres" ] }, "merge": {} }, // assuming 3 remotes 0, 1, 2 and 3 shards a, b, c such that // 0 owns a, b // 1 owns b, c // 2 owns c, a "queries": [ { "indexUid": "movies", "federationOptions": { "remote": "0" }, "q": "Batman dark knight returns 1", "filter": ["genres IN [Action, Adventure]", "_shard = a"] }, { "indexUid": "movies", "federationOptions": { "remote": "0" // useNetwork picked remote 0 for both shard a and b }, "q": "Batman dark knight returns 1", "filter": ["genres IN [Action, Adventure]", "_shard = b"] }, { "indexUid": "movies", "federationOptions": { "remote": "2" }, "q": "Batman dark knight returns 1", "filter": ["genres IN [Action, Adventure]", "_shard = c"] } ] }
Responding with documents from all the shards:
Response
{ "hits": [ { "id": 123025, "title": "Batman: The Dark Knight Returns, Part 1", "overview": "Batman has not been seen for ten years. A new breed of criminal ravages Gotham City, forcing 55-year-old Bruce Wayne back into the cape and cowl. But, does he still have what it takes to fight crime in a new era?", "genres": [ "Action", "Animation", "Mystery" ], "poster": "https://image.tmdb.org/t/p/w500/kkjTbwV1Xnj8wBL52PjOcXzTbnb.jpg", "release_date": 1345507200, "_federation": { "indexUid": "mieli", "queriesPosition": 2, "weightedRankingScore": 0.9894586894586894, "remote": "2" } }, { "id": 142061, "title": "Batman: The Dark Knight Returns, Part 2", "overview": "Batman has stopped the reign of terror that The Mutants had cast upon his city. Now an old foe wants a reunion and the government wants The Man of Steel to put a stop to Batman.", "genres": [ "Action", "Animation", "Mystery" ], "poster": "https://image.tmdb.org/t/p/w500/arEZYd6uMOFTILne9Ux0A8qctMe.jpg", "release_date": 1357171200, "_federation": { "indexUid": "mieli", "queriesPosition": 1, "weightedRankingScore": 0.9894558963186414, "remote": "0" } }, { "id": 16234, "title": "Batman Beyond: Return of the Joker", "overview": "The Joker is back with a vengeance, and Gotham's newest Dark Knight needs answers as he stands alone to face Gotham's most infamous Clown Prince of Crime.", "genres": [ "Animation", "Family", "Action", "Science Fiction" ], "poster": "https://image.tmdb.org/t/p/w500/7RlBs0An83fqAuKfwH5gKMcqgMc.jpg", "release_date": 976579200, "_federation": { "indexUid": "mieli", "queriesPosition": 1, "weightedRankingScore": 0.9427964918160996, "remote": "0" } }, { "id": 155, "title": "The Dark Knight", "overview": "Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker.", "genres": [ "Drama", "Action", "Crime", "Thriller" ], "poster": "https://image.tmdb.org/t/p/w500/qJ2tW6WMUDux911r6m7haRef0WH.jpg", "release_date": 1216166400, "_federation": { "indexUid": "mieli", "queriesPosition": 1, "weightedRankingScore": 0.5784178187403994, "remote": "0" } }, { "id": 49026, "title": "The Dark Knight Rises", "overview": "Following the death of District Attorney Harvey Dent, Batman assumes responsibility for Dent's crimes to protect the late attorney's reputation and is subsequently hunted by the Gotham City Police Department. Eight years later, Batman encounters the mysterious Selina Kyle and the villainous Bane, a new terrorist leader who overwhelms Gotham's finest. The Dark Knight resurfaces to protect a city that has branded him an enemy.", "genres": [ "Action", "Crime", "Drama", "Thriller" ], "poster": "https://image.tmdb.org/t/p/w500/vzvKcPQ4o7TjWeGIn0aGC9FeVNu.jpg", "release_date": 1342396800, "_federation": { "indexUid": "mieli", "queriesPosition": 2, "weightedRankingScore": 0.5772657450076805, "remote": "2" } } ], "query": "Batman dark knight returns 1", "processingTimeMs": 173, "limit": 5, "offset": 0, "estimatedTotalHits": 47, "facetDistribution": { "genres": { "Action": 46, "Adventure": 15, "Animation": 34, "Comedy": 3, "Crime": 14, "Drama": 6, "Family": 15, "Fantasy": 8, "Horror": 1, "Mystery": 4, "Romance": 1, "Science Fiction": 14, "TV Movie": 4, "Thriller": 4, "Western": 1 } }, "facetStats": {}, "requestUid": "019bbcf4-a609-7701-8d82-d370611adfb3", "remoteErrors": {} }This change allows to naturally query all the documents in a sharded context. When
network.leaderis not present (in particular, when nonetworkwas defined), then the behavior is identical to previous versions. -
useNetworkpicks local instance if it owns the shard by @dureuill in https://github.com/meilisearch/meilisearch/pull/6287 To prevent unnecessary network activity, when deciding which remote to ask for a shard in a search over the network, Meilisearch will now always pick the local instance if it owns the shard. -
More efficient FST building by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6279 It improves the construction of the word FST (word dictionary) by removing the need for a full scan of the word docids database. This should drastically improve the database's performance when inserting a large number of documents, even when inserting only a few.
- Handle empty VERSION file as missing by @themavik in https://github.com/meilisearch/meilisearch/pull/6275
- Remap post processing targets for better benchmark readability by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6285
- Update the description of the get task's document payload route by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6284
- @themavik made their first contribution in https://github.com/meilisearch/meilisearch/pull/6275
v1.40.0 🦈
This release introduced support for the distinct attribute in federated search, enabling cross-index distinct attributes with facet distribution support. Additionally, significant performance improvements were delivered, including faster federated search (approximately 100ms faster), optimized JSON document generation for better handling of large documents and a much better memory usage for large workloads.
-
Support
distinctin federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6214The
distinctattribute can now be passed to thefederationobject in federated search to apply a global, cross-index and cross-remote distinct computation to the results.Example of a federated search request with distinct
{ "federation": { "distinct": "genres", // ✨ NEW "facetsByIndex": { // recovering facet distribution is also supported with distinct "comics": [ "genres" ], "movies": [ "genres" ] }, "mergeFacets": {} // merging facet distributions is also supported with distinct }, "queries": [ { "indexUid": "comics", "q": "batman", "attributesToRetrieve": ["title", "genres"], "useNetwork": true // distinct is also supported with network queries }, { "indexUid": "movies", "q": "superman", "attributesToRetrieve": ["title", "genres"], "useNetwork": true } ] }Sample response to a federated search request with distinct
{ "hits": [ { "title": "Batman", "genres": [ "Family", "Adventure", "Comedy", "Science Fiction", "Crime" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 1.0, "remote": "ms2" } }, { "title": "Batman", "genres": [ "Fantasy", "Action" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 1.0, "remote": "ms1" } }, { "title": "Batman & Bill", "genres": [ "Documentary" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 0.9848484848484848, "remote": "ms1" } }, { "title": "Superman: Red Son", "genres": [], "_federation": { "indexUid": "movies", "queriesPosition": 1, "weightedRankingScore": 0.9848484848484849, "remote": "ms0" } }, { "title": "Superman, Spider-Man or Batman", "genres": [ "Drama" ], "_federation": { "indexUid": "movies", "queriesPosition": 1, "weightedRankingScore": 0.9848484848484849, "remote": "ms0" } } ], "processingTimeMs": 15, "limit": 5, "offset": 0, "estimatedTotalHits": 11, "facetDistribution": { "genres": { "Action": 1, "Adventure": 1, "Comedy": 3, "Crime": 2, "Documentary": 1, "Drama": 1, "Family": 1, "Fantasy": 1, "Horror": 2, "Romance": 1, "Science Fiction": 1, "Thriller": 1, "Western": 1 } }, "facetStats": {}, "requestUid": "019d05c7-ea65-77a1-8274-22a8ba9e26db", "remoteErrors": {} }Note the following to apply the distinct attribute at the federation level:
- Applying
distinctat the query level at the same time as the federation level is disallowed and will return a HTTP 400 error. - The chosen distinct field will apply to all remotes and indexes, so it must be a filterable attribute for all participating remotes and indexes.
- While Meilisearch attempts to compute the most accurate facet distribution, in distributed contexts this cannot be guaranteed as the distinct algorithm is not applied on all of the remote documents.
- Applying
-
Improve performance of federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6229
Improves performance of federated search: about 100ms faster for all requests. Improves reliability of the HTTP server: the server will no longer be blocked when too many federated search requests are being processed.
-
Optimize the generation of JSON documents by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6257
Addresses performance issues that users might encounter when requesting large documents. Additionally, performance is enhanced when users request only a small subset of fields from large documents.
-
Use the latest version of mimalloc to improve memory usage by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6201
Updates mimalloc from v2 to v3, improving memory sharing between threads and significantly reducing memory usage on large workloads. It also overrides the allocator to use mimalloc at linking time, allowing LMDB, Meilisearch, and other C libraries to share their allocations for better overall memory efficiency. @Kerollmops wrote a blog post about the story behind this improvement.
-
Add
POST /tasks/compactfor task queue compaction by @YoEight in https://github.com/meilisearch/meilisearch/pull/6193Compacts the task queue database and reclaim space so new tasks can keep being enqueued, without deleting existing tasks. This feature is behind the
taskQueueCompactionRouteexperimental feature flag.
[!WARNING]
Once task queue compaction completes, all write operations are blocked until the server is restarted.
- Bump rustls-webpki from 0.103.8 to 0.103.10 in https://github.com/meilisearch/meilisearch/pull/6273
- Bump tar from 0.4.44 to 0.4.45 in https://github.com/meilisearch/meilisearch/pull/6271
- Bump actix-web-lab from 0.24.3 to 0.26.0 in https://github.com/meilisearch/meilisearch/pull/6218
- Bump quinn-proto from 0.11.13 to 0.11.14 in https://github.com/meilisearch/meilisearch/pull/6217
- Check char boundaries when hiding secrets by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6262
- Network fixes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6255
- Override "default" source base url with the config one to support AWS bedrock OpenAI API by @dureuill in https://github.com/meilisearch/meilisearch/pull/5905
- Fix panic on virtiofs filesystem by @itsamine27 in https://github.com/meilisearch/meilisearch/pull/6261
- Add
X-Accel-Buffering: nofor/logs/streamroute by @YoEight in https://github.com/meilisearch/meilisearch/pull/6268 - Add link to documentation on
--experimental-allowed-ip-networksparameter by @paulden in https://github.com/meilisearch/meilisearch/pull/6263 - Update sysinfo version to
0.38.4by @GuillaumeGomez in https://github.com/meilisearch/meilisearch/pull/6211 - Add a no-agent [AGENTS.md](http://agents.md/) file to prevent excessive AI-generated PRs by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6256
- @GuillaumeGomez made their first contribution in https://github.com/meilisearch/meilisearch/pull/6211
- @itsamine27 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6261
v1.39.0 🐳
Add a new ForeignKeys setting allowing to hydrate documents with documents coming from other indexes.
📓 Note: This implementation doesn't support a remote sharding environment
A new experimental feature, foreignKeys, has been added to the /experimental-feature route that must be set to true to activate the hydration.
curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
-H 'Content-Type: application/json' \
--data-binary '{"foreignKeys": true}'
A new index setting, foreignKeys, has been added to the /indexes/{index_uid}/settings:
// new setting, an array of foreign keys that allows multiple foreign relationships between indexes
"foreignKeys": [
{
// the path in the JSON document containing foreign document ids
"fieldName": "actors",
// the UID of the foreign index containing the documents to fetch during hydration
"foreignIndexUid": "actors"
}
]
With this new feature, a document shaped as follows:
{
"id": 1,
"title": "Forrest Gump",
// Document IDs in foreign index
"actors": [
1
]
}
Will be hydrated as follows in a search response:
{
"id": 1,
"title": "Forrest Gump",
"actors": [
{
"id": 1,
"name": "Tom",
"familyName": "Hanks",
"birthDate": "1956-07-09"
}
]
}
By @ManyTheFish in #6047
Add X-Accel-Buffering: no on POST /chats/{workspace_uid}/chat/completions when the streaming mode is activated. By @YoEight in #6228
-
Fix a memory leak in the indexation pipeline by @Kerollmops in #6212
We fixed an important memory leak caused by an invalid use of bumpalo. If you've seen Meilisearch using more and more memory over time, this issue is no longer visible in the latest version. If you want to read more about the bumpalo-trap we felt in, here is a good summary. It looks like this leak was introduced in v1.12, so approximately a year ago...
-
Avoid losing tasks by applying deltas on the wtxn view by @Kerollmops in #6222
We addressed a small performance regression introduced in v1.38.1 that affected sending updates to the engine during task deletion. We've restored the task deletion performance to match v1.38.0 levels and ensured no race conditions occur while still allowing updates during deletion.
- Ignore flaky test on windows by @dureuill in #6226
- Fix small documentation mistakes by @ManyTheFish in #6205
- Improve messaging when field is missing in the settings indexing pipeline by @Kerollmops in #6213
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.38.0...v1.39.0
v1.38.1 🐍
Meilisearch v1.38.1 fixes a regression we have seen in task processing, which caused the engine to stop processing tasks.
- Revert, optimize, and fix task deletion. By @Mubelotix and @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6219
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.38.0...v1.38.1
v1.38.0 🐍
Meilisearch v1.38 drastically improves indexing performance for embeddings, cleans up the task queue of orphan tasks and batches, ensures they are correctly deleted now, fixes issues users may encounter with OpenAI or VoyageAI when generating embeddings, and improves the tooling for generating routes in the documentation.
- Embeddings: Use the latest version of hannoy with improved performances by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6184 We drastically reduced embedding indexing time by eliminating full database scans. It is more efficient to add a couple of embeddings to a large database now.
- Tasks: Optimize and fix task deletion by @Mubelotix and @kerollmops in https://github.com/meilisearch/meilisearch/pull/5828
- Embedders: Fix the intermittent
connection reset by peererrors when embedding documents or search queries using a remote embedder. By @dureuill in https://github.com/meilisearch/meilisearch/pull/6191
- Enforce that routes appear in the reference documentation by @dureuill and @manythefish in https://github.com/meilisearch/meilisearch/pull/6200 The reference API is now generated from the source code, but it used to miss some actix-declared routes. It is now mandatory to attach a route when using the newly-provided
routes::routesandroutes::pathmacros, which ensures the route is added to the reference. Please refer to the documentation of the individual macros and the many examples throughout the codebase for more details. - Chore: remove verbose debug log on list documents by @StephaneRob in https://github.com/meilisearch/meilisearch/pull/6198
- Add guide to CONTRIBUTING.md to test doc locally by @curquiza in https://github.com/meilisearch/meilisearch/pull/6180
- Docs: capitalize LangChain and MCP in README by @24f1000637 in https://github.com/meilisearch/meilisearch/pull/6183
- Docs: improve documentation of the parameters of the search route by @curquiza in https://github.com/meilisearch/meilisearch/pull/6194
- @StephaneRob made their first contribution in https://github.com/meilisearch/meilisearch/pull/6198
- @24f1000637 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6183
v1.37.0
[!IMPORTANT]
This release contains breaking changes for users of thenetworkexperimental feature.
Meilisearch v1.37 introduces replicated sharding, removes the vectorStoreSetting experimental feature, stabilizes our new vector store for best performance, adds a security fix and miscellaneous improvements.
§ Replicated sharding
[!NOTE] Replicated sharding requires Meilisearch Enterprise Edition (EE).
- Users of Meilisearch Cloud, please contact support if you need replicated sharding.
- Users of the Community Edition, please contact the sales if you want to use replicated sharding in production.
§ Breaking changes
networkobjects sent to thePATCH /networkroute must now contain at least oneshardobject containing at least one remote whenleaderis notnull.
Existing databases will be migrated automatically when upgraded with --experimental-dumpless-upgrade when leader is not null, such that for each remote:
- A shard with the same name as the remote is created
- This shard has exactly one remote in its
remoteslist: the remote with the same name as the shard.
This change will not cause any document to be resharded.
To be able to upgrade without resharding, the migration uses the same name for remotes and for shards. However, in new configurations, we recommend using different names for shards and remotes.
Example of migration
For instance, the following network object:
{
"leader": "ms-00",
"self": "ms-01",
"remotes": {
"ms-00": { /* .. */ },
"ms-01": { /* .. */ }
}
}
is converted to:
{
"leader": "ms-00",
"self": "ms-01",
"remotes": {
"ms-00": { /* .. */ },
"ms-01": { /* .. */ }
},
"shards": { // ✨ NEW
"ms-00": { // shard named like the remote
"remotes": ["ms-00"] // is owned by the remote
},
"ms-01": {
"remotes": ["ms-01"]
}
}
}
The network object for routes PATCH /network and GET /network now contains the new field shards, which is an object whose values are shard objects, and keys the name of each shard.
Each shard object contains a single field remotes, which is an array of strings, each string representing the name of an existing remote.
The shard objects in PATCH /network contain the additional fields addRemotes and removeRemotes meant for convenience:
- pass an array of remote names to
shard.addRemotesto add these remotes to the list of remotes of a shard. - pass an array of remote names to
shard.removeRemotesto remove these remotes from the list of remotes of a shard. - if present and non-
null,shard.remoteswill completely override the existing list of remotes for a shard. - if several of these options are present and non-
null, then the order of application isshard.remotes, thenshard.addRemotes, thenshard.removeShards.
Adding a new shard with some remotes
// PATCH /network
{
// assuming that remotes `ms-0`, `ms-1`, `ms-2` where sent in a previous call to PATCH /network
"shards": {
"s-a": { // new shard
"remotes": ["ms-0", "ms-1"]
}
}
}
Remotes ms-0 and ms-1 own the new shard s-a.
Fully overriding the list of remotes owning a shard
// PATCH /network
{
// assuming remotes `ms-0`, `ms-1`, `ms-2`
// assuming shard `s-a`, owned by `ms-0` and `ms-1`
"shards": {
"s-a": {
"remotes": ["ms-2"]
}
}
}
ms-2 is now the sole owner of s-a, replacing ms-0 and ms-1.
Adding a remote without overriding the list of remotes owning a shard
// PATCH /network
{
// assuming remotes `ms-0`, `ms-1`, `ms-2`
// assuming shard `s-a`, owned by `ms-2`
"shards": {
"s-a": {
"addRemotes": ["ms-0"]
}
}
}
ms-0 and ms-2 are now the owners of s-a.
Removing a remote without overriding the list of remotes owning a shard
// PATCH /network
{
// assuming remotes `ms-0`, `ms-1`, `ms-2`
// assuming shard `s-a`, owned by `ms-0` and `ms-2`
"shards": {
"s-a": {
"removeRemotes": ["ms-2"]
}
}
}
ms-0 is now the sole owner of s-a.
Entirely removing a shard from the list of shards
Set the shard to null:
// PATCH /network
{
"shards": {
"s-a": null
}
}
Or set its remotes list to the empty list:
// PATCH /network
{
"shards": {
"s-a": {
"remotes": []
}
}
}
When network.leader is not null, each shard object in network.shards must:
- Only contain
remotesthat exist in the list ofremotes. - Contain at least one remote.
Additionally, network.shards must contain at least one shard.
Failure to meet any of these conditions will cause the PATCH /network route to respond with 400 invalid_network_shards.
Documents are now sharded according to the list of shards declared in the network rather than the list of remotes. All remotes owning a shard will process the documents that belong to this shard, allowing for replication.
Example of replication
The following configuration defines 3 remotes 0, 1 and 2, and 3 shards A, B, C, such that each remote owns two shards, achieving replication (losing one remote does not lose any document).
{
"leader": "0",
"self": "0",
"remotes": {
"0": { /* .. */ },
"1": { /* .. */ },
"2": { /* .. */ }
},
"shards": {
"A": {
"remotes": ["0", "1"]
},
"B": {
"remotes": ["1", "2"]
},
"C": {
"remotes": ["2", "0"]
}
}
}
- Full replication is supported by having all remotes own all the shards.
- Unbalanced replication is supported by having some remotes own more shards than other remotes.
- "Watcher" remotes are supported by having remotes that own no shards. Watcher remotes are not very useful in this release, and might be upgraded in a future release, so that they keep all documents without indexing them, allowing to "respawn" shards for other remotes.
When useNetwork: true is passed to a search query, it is expanded to multiple queries such that each shard declared in network.shards appears exactly once, associated with a remote that owns that shard.
This ensures that there is no missing or duplicate documents in the results.
When the network experimental feature is enabled, then it becomes possible to filter documents depending on the shard they belong to.
Given s-a and s-b the names of two shards declared in network.shards, then:
_shard = "s-a"in afilterparameter to the search or documents fetch will return the documents that belong tos-a._shard != "s-a"will return the documents that do not belong tos-a_shard IN ["s-a", "s-b"]will return the documents that belong tos-aor tos-b.
You can use these new filters in manual remote federated search to create a partitioning over all shards in the network.
[!IMPORTANT] To avoid duplicate or missing documents in results, for manually crafted remote federated search requests, all shards should appear in exactly one query.
[!TIP] Search requests built with
useNetwork: truealready build a correct partitioning over shards. They should be preferred to manually crafted remote federated search requests in replicated sharding scenarios.
When updating your Meilisearch network using dumpless upgrade, please observe the following guidelines:
- Do not call the
PATCH /networkroute until all remotes of the network are finished updating - If using the search routes with
useNetwork: true, call them on un-updated remotes. Calling it on already updated remotes will cause un-updated remotes to fail the search as they don't know about the_shardfilters.
By @dureuill in https://github.com/meilisearch/meilisearch/pull/6128
§ Remove vectorStoreSetting experimental feature
The new HNSW vector store (hannoy) has been stabilized and is now the only supported vector store in Meilisearch.
As a result, updating to v1.37.0 will migrate all remaining legacy vector store indexes (using arroy) to hannoy, and the vectorStoreSetting experimental feature is no longer available.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6176
We removed a computationally expensive step from vector indexing.
On a DB with 20M documents, this removes 300s per indexing batch of 1100s.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6175
§ 🔒 Security
- Bump mini-dashboard (local web interface) which
- now stores API key in RAM instead of in the
localStorage - bumps dependencies with potential security vulnerabilities
- now stores API key in RAM instead of in the
By @Strift and @curquiza in https://github.com/meilisearch/meilisearch/pull/6186 and https://github.com/meilisearch/meilisearch/pull/6172
§ 🔩 Miscellaneous
- Mark Cargo.lock as not linguist-generated by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6181
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.36.0...v1.37.0
v1.36.0 🐙
Version v1.36.0 introduces an exciting update to the ranking rules to improve the engine's relevance. It's actually the first time we've made such a change since v1.0, and we're really happy about this improvement!
-
Introduce the
attributeRankandwordPositioncriteria by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6154, https://github.com/meilisearch/meilisearch/pull/6155, and https://github.com/meilisearch/meilisearch/pull/6164We released two new ranking rules that Meilisearch had already been using internally for the
attributeone, which is basically both ranking rules applied one after the other:attributeRank: A document is considered better if the query words match in a higher searchable attribute. It ignores the position of the query words in this attribute.wordPosition: A document is considered better if the query words match closer to the beginning of an attribute. The attribute rank is ignored by this rule.
We continue our policy of migrating everyone to use a homemade HNSW by introducing a new dumpless upgrade step that migrates index uses the old annoy vector store to the new Hannoy one. Changing the vector store backend affects the ranking score. This step can take a couple of minutes when the number of embeddings is high, and we recommend changing the vector store backend beforehand to gain more control if needed. To do so, you must enable the
vectorStoreSettingexperimental feature and set thevectorStoreroot setting toexperimental.
- Fix search failing when using
attributesToSearchOnon an empty index by @zen-zap in https://github.com/meilisearch/meilisearch/pull/5944
- Bump bytes from 1.11.0 to 1.11.1 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6142
- Bump jsonwebtoken from 9.3.1 to 10.3.0 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6141
- Bump time from 0.3.44 to 0.3.47 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6144
- Bump rsa from 0.9.9 to 0.9.10 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6149
- Improve OpenAPI docs and CIs to improve documentation experience by @curquiza in https://github.com/meilisearch/meilisearch/pull/6145
[!WARNING] Breaking change: the
meilisearch-openapi-mintlify.jsonfile will not be available in the release assets anymore. If you were using it, please refer to the one that is now available in our public documentation repository.
- GenAI contributing and PR template changes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6140
- Fix "test with almost all features" by @dureuill in https://github.com/meilisearch/meilisearch/pull/6139 and https://github.com/meilisearch/meilisearch/pull/6148
- Allow any IP for SDK tests by @Strift in https://github.com/meilisearch/meilisearch/pull/6151
- Improve output of the declarative tests, also in https://github.com/meilisearch/meilisearch/pull/6164
❤️ Thanks to @zen-zap for contributing to this release!
v1.35.1
Meilisearch v1.35.1 fixes a possible task database corruption issue that would occur when using the S3 streaming snapshot method.
- Users of the Community Edition do not need to upgrade as this feature is Enterprise Edition only
- Affected users of the Cloud have been contacted.
While snapshotting the task database, sending a task to Meilisearch could corrupt it. Index DBs were not affected, so manually rebuilding the task database fixes the corruption.
By @dureuill in #6160
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.35.0...v1.35.1
v1.35.0
The search routes accept a new field named showPerformanceDetails. When set to ù true`, the search response contains a performance trace, allowing the user to better understand what takes time during a search query.
impacted routes:
POST /indexes/<index_uid>/searchGET /indexes/<index_uid>/searchPOST /multi-searchPOST /indexes/<index_uid>/similarGET /indexes/<index_uid>/similar
routes: POST /indexes/<index_uid>/search and GET /indexes/<index_uid>/search
new request parameters:
showPerformanceDetails:true/false(boolean)
example
{
"q": "glass",
"showPerformanceDetails": true
}
new response field:
performanceDetails:{"<span>": "<human_duration>", .. }(map)
example
{
"hits": [
// hits ..
],
"query": "glass",
"processingTimeMs": 5,
"limit": 20,
"offset": 0,
"estimatedTotalHits": 1,
"requestUid": "<uuid>",
"performanceDetails": {
"wait for permit": "295.29µs",
"search > tokenize": "436.67µs",
"search > resolve universe": "649.00µs",
"search > keyword search": "515.71µs",
"search > format": "288.54µs",
"search": "3.56ms"
}
}
route: /multi-search
new request parameters:
queries.showPerformanceDetails:true/false(boolean)
example
"queries": [
{
"indexUid": "<index_uid>",
"q": "glass",
"showPerformanceDetails": true
}
]
new response field:
results.performanceDetails:{"<span>": "<human_duration>", .. }(map)
example
{
"results": [
{
"indexUid": "<index_uid>",
"hits": [
// hits ..
],
"query": "glass",
"processingTimeMs": 5,
"limit": 20,
"offset": 0,
"estimatedTotalHits": 1,
"requestUid": "<uuid>",
"performanceDetails": {
"wait for permit": "295.29µs",
"search > tokenize": "436.67µs",
"search > resolve universe": "649.00µs",
"search > keyword search": "515.71µs",
"search > format": "288.54µs",
"search": "3.56ms"
}
}
]
}
route: /multi-search
new request parameters:
federation.showPerformanceDetails:true/false(boolean)
example
{
"federation": { "showPerformanceDetails": true },
"queries": [
{
"indexUid": "<index_uid>",
"q": "glass"
}
]
}
new response field:
performanceDetails:{"<span>": "<human_duration>", .. }(map)
example
{
"hits": [
// hits ..
],
"query": "glass",
"processingTimeMs": 5,
"limit": 20,
"offset": 0,
"estimatedTotalHits": 1,
"requestUid": "<uuid>",
"performanceDetails": {
"wait for permit": "213.83µs",
"search > tokenize": "171.67µs",
"search > resolve universe": "257.63µs",
"search > keyword search": "577.71µs",
"search > format": "114.96µs",
"search > federation > wait for remote results": "62.71µs",
"search > federation > merge results": "120.04µs",
"search > federation > merge facets": "53.42µs",
"search > federation": "237.04µs",
"search": "289.08ms"
}
}
route: /indexes/<index_uid>/similar
new request parameters:
showPerformanceDetails:true/false(boolean)
example
{
"id": 143,
"embedder": "manual",
"showPerformanceDetails": true
}
new response field:
performanceDetails:{"<span>": "<human_duration>", .. }(map)
example
{
"hits": [
// hits ..
],
"id": "143",
"processingTimeMs": "[duration]",
"limit": 20,
"offset": 0,
"estimatedTotalHits": 4,
"performanceDetails": {
"search > format": "244.92µs",
"search": "1.25ms"
}
}
By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6132
- Add AI usage disclosure guidelines to CONTRIBUTING.md by @meili-bot in https://github.com/meilisearch/meilisearch/pull/6114
- fix: expose SettingsDelta and related types for external crate usage by @MkDev11 in https://github.com/meilisearch/meilisearch/pull/6119
- Return Pagination fields in /fields route by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6126
- Stabilize the post-processing multithreading by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6130
- Bump mini-dashboard version to fix display of image in local UI interface by @Strift in https://github.com/meilisearch/meilisearch/pull/6129
- Fix mTLS compatibility with Go by @curquiza in https://github.com/meilisearch/meilisearch/pull/6133
- @MkDev11 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6119
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.34.3...v1.35.0