v1.34.3
- Increase the auth store database size to 2 GiB so that we can index more keys by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6134
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.34.2...v1.34.3
v1.34.2
This patch fixes an accidental breaking change in v1.34.1 where Meilisearch would not start with a configuration file if experimental_allowed_ip_networks was not defined.
- Meilisearch Cloud users do not need to update as they were not affected by this regression
- We recommend that OSS users upgrade to v1.34.2
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.34.1...v1.34.2
v1.34.0
The search query object passed in the body of POST /indexes/{:indexUid}/search now accepts an optional boolean useNetwork. When present and set to true, the search is executed "as-if" it was a remote federated search over all remotes in the network.
That is, the following:
Search request
Is executed by Meilisearch as if it was the following, assuming a network of 3 Meilisearch instances with names "0", "1" and "2":
Equivalent multi-search request
// POST /multi-search
{
"federation": {
"limit": 5,
"facetsByIndex": {
"movies": [
"genres"
]
},
"merge": {}
},
"queries": [
{
"indexUid": "movies",
"federationOptions": {
"remote": "0"
},
"q": "Batman dark knight returns 1",
"filter": "genres IN [Action, Adventure]"
},
{
"indexUid": "movies",
"federationOptions": {
"remote": "1"
},
"q": "Batman dark knight returns 1",
"filter": "genres IN [Action, Adventure]"
},
{
"indexUid": "movies",
"federationOptions": {
"remote": "2"
},
"q": "Batman dark knight returns 1",
"filter": "genres IN [Action, Adventure]"
}
]
}
Resulting in:
Search Response
{
"hits": [
{
"id": 123025,
"title": "Batman: The Dark Knight Returns, Part 1",
"overview": "Batman has not been seen for ten years. A new breed of criminal ravages Gotham City, forcing 55-year-old Bruce Wayne back into the cape and cowl. But, does he still have what it takes to fight crime in a new era?",
"genres": [
"Action",
"Animation",
"Mystery"
],
"poster": "https://image.tmdb.org/t/p/w500/kkjTbwV1Xnj8wBL52PjOcXzTbnb.jpg",
"release_date": 1345507200,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 2,
"weightedRankingScore": 0.9894586894586894,
"remote": "2"
}
},
{
"id": 142061,
"title": "Batman: The Dark Knight Returns, Part 2",
"overview": "Batman has stopped the reign of terror that The Mutants had cast upon his city. Now an old foe wants a reunion and the government wants The Man of Steel to put a stop to Batman.",
"genres": [
"Action",
"Animation",
"Mystery"
],
"poster": "https://image.tmdb.org/t/p/w500/arEZYd6uMOFTILne9Ux0A8qctMe.jpg",
"release_date": 1357171200,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 1,
"weightedRankingScore": 0.9894558963186414,
"remote": "1"
}
},
{
"id": 16234,
"title": "Batman Beyond: Return of the Joker",
"overview": "The Joker is back with a vengeance, and Gotham's newest Dark Knight needs answers as he stands alone to face Gotham's most infamous Clown Prince of Crime.",
"genres": [
"Animation",
"Family",
"Action",
"Science Fiction"
],
"poster": "https://image.tmdb.org/t/p/w500/7RlBs0An83fqAuKfwH5gKMcqgMc.jpg",
"release_date": 976579200,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 1,
"weightedRankingScore": 0.9427964918160996,
"remote": "1"
}
},
{
"id": 155,
"title": "The Dark Knight",
"overview": "Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker.",
"genres": [
"Drama",
"Action",
"Crime",
"Thriller"
],
"poster": "https://image.tmdb.org/t/p/w500/qJ2tW6WMUDux911r6m7haRef0WH.jpg",
"release_date": 1216166400,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 1,
"weightedRankingScore": 0.5784178187403994,
"remote": "1"
}
},
{
"id": 49026,
"title": "The Dark Knight Rises",
"overview": "Following the death of District Attorney Harvey Dent, Batman assumes responsibility for Dent's crimes to protect the late attorney's reputation and is subsequently hunted by the Gotham City Police Department. Eight years later, Batman encounters the mysterious Selina Kyle and the villainous Bane, a new terrorist leader who overwhelms Gotham's finest. The Dark Knight resurfaces to protect a city that has branded him an enemy.",
"genres": [
"Action",
"Crime",
"Drama",
"Thriller"
],
"poster": "https://image.tmdb.org/t/p/w500/vzvKcPQ4o7TjWeGIn0aGC9FeVNu.jpg",
"release_date": 1342396800,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 2,
"weightedRankingScore": 0.5772657450076805,
"remote": "2"
}
}
],
"query": "Batman dark knight returns 1",
"processingTimeMs": 173,
"limit": 5,
"offset": 0,
"estimatedTotalHits": 47,
"facetDistribution": {
"genres": {
"Action": 46,
"Adventure": 15,
"Animation": 34,
"Comedy": 3,
"Crime": 14,
"Drama": 6,
"Family": 15,
"Fantasy": 8,
"Horror": 1,
"Mystery": 4,
"Romance": 1,
"Science Fiction": 14,
"TV Movie": 4,
"Thriller": 4,
"Western": 1
}
},
"facetStats": {},
"requestUid": "019bbcf4-a609-7701-8d82-d370611adfb3",
"remoteErrors": {}
}
useNetwork requires the network experimental feature to be enabled.
Passing useNetwork=true as a query parameter to GET /indexes/{:indexUid}/search has the same effect as passing useNetwork: true as a field parameter to POST /indexes/{:indexUid}/search
useNetworkcan also be passed as a field of the individual queries inside of amulti-searchrequest.- When used on a query in a non-federated search request, it has the same effect as on
POST /indexes/{:indexUid}/searchfor that query - When used on a query in a federated search request, the request is executed "as-if" one query per remote of the network had been performed.
Federated search example:
Multi-search request
{
"federation": {
"limit": 5
},
"queries": [
{
"q": "Batman returns",
"indexUid": "mieli",
"useNetwork": true
},
{
"q": "Superman returns",
"indexUid": "mieli",
"useNetwork": true
}
]
}
Multi-search response
{
"hits": [
{
"id": 364,
"title": "Batman Returns",
"overview": "While Batman deals with a deformed man calling himself the Penguin, an employee of a corrupt businessman transforms into the Catwoman.",
"genres": [
"Action",
"Fantasy"
],
"poster": "https://image.tmdb.org/t/p/w500/jKBjeXM7iBBV9UkUcOXx3m7FSHY.jpg",
"release_date": 708912000,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 0,
"weightedRankingScore": 1.0,
"remote": "1"
}
},
{
"id": 1452,
"title": "Superman Returns",
"overview": "Superman returns to discover his 5-year absence has allowed Lex Luthor to walk free, and that those he was closest to felt abandoned and have moved on. Luthor plots his ultimate revenge that could see millions killed and change the face of the planet forever, as well as ridding himself of the Man of Steel.",
"genres": [
"Science Fiction",
"Action",
"Adventure"
],
"poster": "https://image.tmdb.org/t/p/w500/qIegbn6DSUYmggfwxOBNOVS35q.jpg",
"release_date": 1151452800,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 1,
"weightedRankingScore": 1.0,
"remote": "0"
}
},
{
"id": 324249,
"title": "Requiem for Krypton: Making 'Superman Returns'",
"overview": "A detailed behind-the-scenes documentary on the making of Superman Returns.",
"genres": [
"Documentary"
],
"poster": "https://image.tmdb.org/t/p/w500/eC1XKswKSoyDyJXXZszLTuwUHli.jpg",
"release_date": 1164672000,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 1,
"weightedRankingScore": 0.9907407407407408,
"remote": "1"
}
},
{
"id": 294294,
"title": "Saltwater",
"overview": "This American Indie drama follows several endearing characters as they wade through life seeking happiness, peace and ultimately, love. Will (Ronnie Kerr, Vampire Boys 2, Shut Up and Kiss Me) leaves the Navy after many years, soon reunites old friends and begins to start his new civilian life. His friend Rich (Bruce L Hart) tries to set him up with ruggedly handsome Josh (Ian Roberts-a former Australian professional rugby player, actor and model-Cedar Boys, Superman Returns, Little Fish). While there is immense chemistry between the two, timing and certain ideals never seem to align. When a shocking tragedy happens the two are paired up to pick up the pieces and sort through the after effects. Saltwater is a story about men of all ages, finding love, losing friends, navigating their way through life and knowing it's the journey rather then the destination that's important.",
"genres": [
"Romance",
"Drama"
],
"poster": "https://image.tmdb.org/t/p/w500/bDnLdYqpH9abHo4ASMPKiInx8dm.jpg",
"release_date": 1342310400,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 1,
"weightedRankingScore": 0.966931216931217,
"remote": "1"
}
},
{
"id": 142061,
"title": "Batman: The Dark Knight Returns, Part 2",
"overview": "Batman has stopped the reign of terror that The Mutants had cast upon his city. Now an old foe wants a reunion and the government wants The Man of Steel to put a stop to Batman.",
"genres": [
"Action",
"Animation",
"Mystery"
],
"poster": "https://image.tmdb.org/t/p/w500/arEZYd6uMOFTILne9Ux0A8qctMe.jpg",
"release_date": 1357171200,
"_federation": {
"indexUid": "mieli",
"queriesPosition": 0,
"weightedRankingScore": 0.8697089947089947,
"remote": "1"
}
}
],
"processingTimeMs": 247,
"limit": 5,
"offset": 0,
"estimatedTotalHits": 97,
"requestUid": "019bbd3a-5106-70e0-94fc-f58b2f0c28c8",
"remoteErrors": {}
}
- Facet search referencing
useNetworkare not supported - The chat route cannot use
useNetworkat the moment: doing so is not trivial implementation-wise, because chat route expects to be able to open the index (to fetch chat configs), but federated search only opens the indexes once during a short critical section.
By @dureuill in https://github.com/meilisearch/meilisearch/pull/6101
Pass federation.page and federation.hitsPerPage with the same meaning as in a regular search request to use exhaustive pagination in the federated search
By @dureuill in https://github.com/meilisearch/meilisearch/pull/6101
The settings indexer is more efficient when users are removing searchable attributes from the searchable fields.
By @VedantMadane in https://github.com/meilisearch/meilisearch/pull/6109
Solves a low-severity timing attack vulnerability on key comparison by using constant-time comparison
By @curquiza in https://github.com/meilisearch/meilisearch/pull/6077
- Remove openssl from deps by @dureuill in https://github.com/meilisearch/meilisearch/pull/6117
- Fix geo-types version to avoid multiple rstar deps by @dureuill in https://github.com/meilisearch/meilisearch/pull/6118
❤️ Thanks again @VedantMadane for the contribution to this release!
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.33.1...v1.34.0
v1.33.1 🐞
In Meilisearch v1.33.1, we released a security fix that affected the dump import, improvements to vector store relevance in massive databases, and a fix related to ranking scores.
- Cloud users: you don't need to do anything. We found no evidence of exploitation of this vulnerability on Meilisearch Cloud.
- Open-source users: if you allow importing dumps from an untrusted source, we recommend you update to v1.33.1
All versions of Meilisearch before v1.33.0 are vulnerable to a path traversal vulnerability involving the dump import functionality.
Importing a specially crafted dump could result in giving access to the Meilisearch instance to arbitrary, specifically formatted files, present on the file system of the Meilisearch instance.
- We updated the vector store to trigger linear scanning even on bigger databases, leading to improved performance and so better result quality when the search cutoff is reached. This applies in particular when the number of filtered candidates is small relative to the number of documents in the index by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6113.
- We fixed a bug where only the first non-blocking buckets were taken for the non-final ranking rules. This improves the quality of search results when the search cutoff triggers, especially when vector search and a sort are involved by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6113.
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.33.0...v1.33.1
v1.33.0 🐞
-
Add
/fieldsroute to get all the fields of an index by @YoEight in https://github.com/meilisearch/meilisearch/pull/6082 Adds a new POST /indexes/{indexUid}/fields endpoint that returns detailed metadata about all fields in an index. This endpoint provides comprehensive information about each field's configuration, including display, search, filtering, and localization settings. -
Implement parallel cleanup of old field IDs by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6100 We reduce the time required to perform the dumpless upgrade for instances before v1.32.0 by multi-threading database fetches. By doing that, we noticed improvements from 2 hours and 50 minutes to a bit less than 7 minutes.
- Meilisearch can now rescale when the index reaches 2 TB by @Vipul-045 in https://github.com/meilisearch/meilisearch/pull/6095
- Bump hannoy to 0.1.4-nested-rtxns by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6103 We updated our internal vector store to speed up the dumpless upgrade by improving the graph rebuilding and improving the speed and relevance of the search by using the explore factor as a limit to stop document searches rather than the query limit.
- Upgrade mini-dashboard to benefit security vulnerability fixes by @curquiza in https://github.com/meilisearch/meilisearch/pull/6091
- Fix tests by @dureuill in https://github.com/meilisearch/meilisearch/pull/6090
- Add verbose OpenAPI schema descriptions for documentation by @qdequele in https://github.com/meilisearch/meilisearch/pull/6071
- Add code samples to the generated openAPI file by @curquiza in https://github.com/meilisearch/meilisearch/pull/6028
Thank you @Vipul-045 for your first contribution ❤️
v1.32.2 🐟
- Do not panic when doing a dumpless upgrade on empty indexes with configured embeddings by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6102
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.32.1...v1.32.2
v1.32.1
Introduce a MEILI_EXPERIMENTAL_DISABLE_FID_BASED_DATABASES_CLEANUP env var to opt out of the field ID-based database cleanup when upgrading a Meilisearch <1.32.0.
by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6096
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.32.0...v1.32.1
v1.32.0
Introduces comprehensive progress tracking and logging for search operations in Meilisearch. It adds detailed timing information for each step of the search process, enabling better observability and performance analysis.
by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6072
We accelerate document indexing by processing a large number of tasks in batches or a large number of records in parallel. We expedited the preparation of the payloads by extracting the various changes and assigning internal IDs in parallel. We achieved a 7x speedup on a four-million-document insertion using four CPUs, and the performance scales with the number of CPUs.
The indexedDocuments field in tasks using skipCreation no longer precisely reflects the number of document operations performed, specifically for POST and PUT operations. This count may be higher than the actual number of operations, but it doesn't affect the computation; only the reported count is impacted. We prioritize speed over perfect accuracy here, and the documents are still correctly indexed as before.
by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6080
Fixed vector sort bucketing so documents with identical similarity scores are grouped together, ensuring subsequent ranking rules are applied correctly.
by @dureuill in https://github.com/meilisearch/meilisearch/pull/6081
Fixes a bug where changing searchableAttributes from ["*"] to a subset of fields left orphaned data in fid-based databases, causing corruption and warnings during search.
by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6076
Bumps hannoy to v0.1.3-nested-rtxns, which fixes graph-related recall issues and adds a method to rebuild graph links to recover previously malformed graphs. Also fixes a minor issue in the dumpless upgrade flow where the upgrade description was not displayed correctly and related operations were not properly associated with the upgrade.
by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6055
Updated the JavaScript SDK tests to use pnpm instead of yarn in CI workflows, switching the package manager across test configurations to ensure the SDK test suite runs correctly and consistently with the current tooling.
by @Strift in https://github.com/meilisearch/meilisearch/pull/6075
Updated the SDK tests CI workflow for the JavaScript SDKs
by @curquiza in https://github.com/meilisearch/meilisearch/pull/6050
Fix Stacked Borrows violation in IterMut.
by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6087
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.31.0...v1.32.0
v1.31.0 🦃
The recently introduced S3-streaming snapshots feature is now available as an Enterprise Edition feature. From now on, you'll need a license to use this feature when using Meilisearch as a self-hosted solution. Note that this version introduces a breaking change regarding the S3-streaming snapshot feature, which is no longer available in the Community Edition; however, on-disk snapshots remain accessible. You can read our BUSL license for contact information, if you like. Note that if you are using the Community Edition of Meilisearch between version 1.25 and this release, you can freely use the S3 Streaming feature without an Enterprise Edition license.
by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6057
We just introduced support for IRSA authentication to do snapshots on AWS. IRSA allows the use of short-lived access and secret keys to upload snapshots through S3. This feature is available under the Enterprise Edition and can be accessed through two experimental CLI parameters.
by @paulden in https://github.com/meilisearch/meilisearch/pull/6044
Adds an optional skipCreation boolean query parameter to POST and PUT on /indexes/{index}/documents. When set to true, documents that don't exist in the index are silently ignored rather than created. Default is false, preserving existing behavior.
by @YoEight in https://github.com/meilisearch/meilisearch/pull/6053
- Fix actix payload error handling by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6052
- Bump hannoy to v0.1.2 by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6063
- Run windows in the PR CI by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6058
- Make CI test titles more human friendly by @curquiza in https://github.com/meilisearch/meilisearch/pull/6062