• For REST/OpenAI/ollama autoembedders users: Retry if deserialization of remote response failed by @dureuill in https://github.com/meilisearch/meilisearch/pull/5058

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.11.2...v1.11.3

• Add timeout on read and write operations. by @dureuill in https://github.com/meilisearch/meilisearch/pull/5051

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.11.1...v1.11.2

• Add 3s timeout to embedding requests made during search by @dureuill in https://github.com/meilisearch/meilisearch/pull/5039

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.11.0...v1.11.1

Meilisearch v1.11 introduces AI-powered search performance improvements thanks to binary quantization and various usage changes, all of which are steps towards a future stabilization of the feature. We have also improved federated search usage following user feedback.

🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.

Some SDKs might not include all new features. Consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).

This release is Meilisearch's first step towards stabilizing AI-powered search and introduces a few breaking changes to its API. Consult the PRD for full usage details.

Done by @dureuill in #4906, #4920, #4892, and #4938.

• When performing AI-powered searches, hybrid.embedder is now a mandatory parameter in GET and POST /indexes/{:indexUid}/search
• As a consequence, it is now mandatory to pass hybrid even for pure semantic searches
• embedder is now a mandatory parameter in GET and POST /indexes/{:indexUid}/similar
• Meilisearch now ignores semanticRatio and performs a pure semantic search for queries that include vector but not q

• The default model for OpenAI is now text-embedding-3-small instead of text-embedding-ada-002
• This release introduces a new embedder option: documentTemplateMaxBytes. Meilisearch will truncate a document's template text when it goes over the specified limit
• Fields in documentTemplate include a new field.is_searchable property. The default document template now filters out both empty fields and fields not in the searchable attributes list:

v1.11:

{% for field in fields %}
  {% if field.is_searchable and not field.value == nil %}
    {{ field.name }}: {{ field.value }}\n
  {% endif %}
{% endfor %}

v1.10:

{% for field in fields %}
  {{ field.name }}: {{ field.value }}\n
{% endfor %}

Embedders using the v1.10 document template will continue working as before. The new default document template will only work with newly created embedders.

v1.11 introduces a new embedder option, binaryQuantized:

curl \
  -X PATCH 'http://localhost:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "embedders": {
      "image2text": {
        "binaryQuantized": true
      }
    }
  }'

Enable binary quantization to convert embeddings of floating point numbers into embeddings of boolean values. This will negatively impact the relevancy of AI-powered searches but significantly improve performance in large collections with more than 100 dimensions.

In our benchmarks, this reduced the size of the database by a factor of 10 and divided the indexing time by a factor of 6 with little impact on search times.

[!WARNING] Enabling this feature will update all of your vectors to contain only 1s or -1s, significantly impacting relevancy.

You cannot revert this option once you enable it. Before setting binaryQuantized to true, Meilisearch recommends testing it in a smaller or duplicate index in a development environment.

Done by @irevoire in #4941.

This release adds two new federated search options, facetsByIndex and mergeFacets. These allow you to request a federated search for facet distributions and stats data.

To obtain facet distribution and stats for each separate index, use facetsByIndex when querying the POST /multi-search endpoint:

POST /multi-search
{
  "federation": {
    "limit": 20,
    "offset": 0,
	"facetsByIndex": {
	  "movies": ["title", "id"],
	  "comics": ["title"],
	}
  },
  "queries": [
    {
      "q": "Batman",
      "indexUid": "movies"
    },
    {
      "q": "Batman",
      "indexUid": "comics"
    }
  ]
}

The multi-search response will include a new field, facetsByIndex with facet data separated per index:

{
  "hits": […],
  …
  "facetsByIndex": {
      "movies": {
        "distribution": {
          "title": {
            "Batman returns": 1
          },
          "id": {
            "42": 1
          }
        },
        "stats": {
          "id": {
            "min": 42,
            "max": 42
          }
        }
      },
     …
  }
}

To obtain facet distribution and stats for all indexes merged into a single, use both facetsByIndex and mergeFacets when querying the POST /multi-search endpoint:

POST /multi-search
{

  "federation": {
    "limit": 20,
    "offset": 0,
	  "facetsByIndex": {
	    "movies": ["title", "id"],
	    "comics": ["title"],
	  },
	  "mergeFacets": {
	    "maxValuesPerFacet": 10,
	  }
  }
  "queries": [
    {
      "q": "Batman",
      "indexUid": "movies"
    },
    {
      "q": "Batman",
      "indexUid": "comics"
    }
  ]
}

The response includes two new fields, facetDistribution and facetStarts:

{
  "hits": […],
  …
  "facetDistribution": {
    "title": {
      "Batman returns": 1
      "Batman: the killing joke":
    },
    "id": {
      "42": 1
    }
  },
  "facetStats": {
    "id": {
      "min": 42,
      "max": 42
    }
  }
}

Done by @dureuill in #4929.

Enable the experimental feature to use the STARTS WITH filter operator:

curl \
  -X PATCH 'http://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "containsFilter": true
  }'

Use the STARTS WITH operator when filtering:

curl \
  -X POST http://localhost:7700/indexes/movies/search \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "filter": "hero STARTS WITH spider"
  }'

🗣️ This is an experimental feature, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.

Done by @Kerollmops in #4939.

• Language support and localizedAttributes settings by @ManyTheFish in #4937
  • Add ISO-639-1 variants
  • Convert ISO-639-1 into ISO-639-3
• Add a German language tokenizer by @luflow in meilisearch/charabia#303 and in #4945
• Improve Turkish language support by @tkhshtsh0917 in meilisearch/charabia#305 and in #4957
• Upgrade "batch failed" log to error level in #4955 by @dureuill.
• Update the search UI: remove the forced capitalized fields, by @curquiza in #4993

• ⚠️ When using federated search, query.facets was silently ignored at the query level, but should not have been. It now returns the appropriate error. Use federation.facetsByIndex instead if you want facets to be applied during federated search.
• Prometheus /metrics return the route pattern instead of the real route when returning the HTTP requests total by @irevoire in #4839
• Truncate values at the end of a list of facet values when the number of facet values is larger than maxValuesPerFacet. For example, setting maxValuesPerFacet to 2 could result in ["blue", "red", "yellow"], being truncated to ["blue", "yellow"] instead of ["blue", "red"]`. By @dureuill in #4929
• Improve the task cancellation when vectors are used, by @irevoire in #4971
• Swedish support: the characters å, ä, ö are no longer normalized to a and o. By @ManyTheFish in #4945
• Update rhai to fix an internal error when updating documents with a function (experimental) by @irevoire in #4960
• Fix the bad experimental search queue size by @irevoire in #4992
• Do not send empty edit document by function by @irevoire in #5001
• Display vectors when no custom vectors were ever provided by @dureuill in #5008

• Dependencies updates
  • Security dependency upgrade: bump quinn-proto from 0.11.3 to 0.11.8 by @dependabot in #4911
• CIs and tests
  • Make the tests run faster by @irevoire in #4808
• Documentation
  • Fix broken links in README by @iornstein in #4943
• Misc
  • Allow Meilitool to upgrade from v1.9 to v1.10 without a dump in some conditions, by @dureuill in #4912
  • Fix bench by adding embedder by @dureuill in #4954
  • Revamp analytics by @irevoire in #5011

❤️ Thanks again to our external contributors:

• Meilisearch: @iornstein.
• Charabia: @luflow, @tkhshtsh0917.

[!WARNING] Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

• Revamp analytics by @irevoire in https://github.com/meilisearch/meilisearch/pull/5011

[!WARNING] Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

• fix the bad experimental search queue size by @irevoire in https://github.com/meilisearch/meilisearch/pull/4992
• Do not send empty edit document by function by @irevoire in https://github.com/meilisearch/meilisearch/pull/5001
• Display vectors when no custom vectors were ever provided by @dureuill in https://github.com/meilisearch/meilisearch/pull/5008

• Bring back changes from v1.10.3 by @irevoire in https://github.com/meilisearch/meilisearch/pull/5006

See the release note of the v1.10.3 for more information about the two experimental cli flags that are being introduced in this RC.

• Update mini-dashboard by @curquiza in https://github.com/meilisearch/meilisearch/pull/4993

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.11.0-rc.1...v1.11.0-rc.2

This PR lets you configure two behaviors of the engine through experimental cli flags:

• The number of searches Meilisearch can process concurrently per core with the --experimental-nb-searches-per-core cli flag
• After how many seconds Meilisearch can consider a search as irrelevant and drop it straight away without processing it with the --experimental-drop-search-after cli flag

Done by @irevoire in https://github.com/meilisearch/meilisearch/pull/5000

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.10.2...v1.10.3

[!WARNING] Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

• Update rhai by @irevoire in https://github.com/meilisearch/meilisearch/pull/4960 This should fix a bug where Meilisearch was reporting an internal error while modifying documents. If you encounter the same issue, please try the RC and let us know if it fixes the issue!
• Update charabia feature flags and add support for the Turkish language by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/4957
• Update arroy by @irevoire in https://github.com/meilisearch/meilisearch/pull/4971
  • Make the indexing of vectors slightly quicker
  • Improve the cancelling of task containing vectors

• Upgrade "batch failed" log to error level by @dureuill in https://github.com/meilisearch/meilisearch/pull/4955
• Fix bench by adding embedder by @dureuill in https://github.com/meilisearch/meilisearch/pull/4954
• Move the multi arroy index logic to the arroy wrapper by @irevoire in https://github.com/meilisearch/meilisearch/pull/4953
• Improve the error message by @dureuill in https://github.com/meilisearch/meilisearch/pull/4972

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.11.0-rc.0...v1.11.0-rc.1

The Swedish tokenization pipeline were deactivated in the previous versions, now it is activated when specifying the index Language in the settings:

{
  "localizedAttributes": [ { "locales": ["swe"], "attributePatterns": ["*"] } ]
}

related PR: #4949

[!WARNING] Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

Meilisearch v1.11 introduces several changes about hybrid search: usage improvements to prepare a future stabilization of the feature, but also a huge impact on the performance thanks to the binary quantization. Finally, according to your feedback, we improved the federated search usage.

For a future stabilization, multiple changes have been applied to hybrid search features.

📖 More details about the usage in this page.

Done by @dureuill in #4906, #4920, #4892, and #4938.

• When using the semantic or the hybrid search, hybrid.embedder is now a mandatory parameter in GET and POST /indexes/{:indexUid}/search
• As a consequence, it is now mandatory to pass hybrid even for full-vector search (with only vector and not q)
• embedder is now a mandatory parameter in GET and POST /indexes/{:indexUid}/similar
• Ignore non-zero semanticRatio when vector is passed but not q: a semantic search will be performed.

• The default model for OpenAI is now text-embedding-3-small instead of text-embedding-ada-002.
• Limit the maximum length of a rendered document template: when the source of an embedder is set to huggingFace, openAi, rest or ollama, then documentTemplateMaxBytes is now available as an optional parameter. This parameter describes the number of bytes in which the rendered document template text should fit when trying to embed a document. Longer texts are truncated to fit.
• Add the ability to query whether a field is searchable in documentTemplate: use field.is_searchable: true if the field is a searchable attribute, otherwise false.
• The new document template changed to
  • filter out fields that are not searchable
  • filter out fields that are nil or missing for the current document

v1.10 and before:

{% for field in fields %}
{{ field.name }}: {{ field.value }}\n
{% endfor %}

From v1.11:

{% for field in fields %}
{% if field.is_searchable and not field.value == nil %}
{{ field.name }}: {{ field.value }}\n
{% endif %}
{% endfor %}

This last change is backward compatible. Embedders that use the v1.10 document template will keep using it. The new default document template will only be applied to newly added embedders.

The binary quantization converts your embeddings of floating point numbers (using 32 bits) into embeddings of boolean values (using 1 bit). In our benchmarks, it reduced the size of the database by a factor of 10, and divided the indexing time by a factor of 6 without much impact on the search time.

[!WARNING] Enabling this feature will update all of your vectors to contain only 1s or -1s, impacting the relevancy significantly. Once enabled, there is no coming back. Before trying it out on your main index, you should try the feature on a smaller index to ensure it fits your use case.

Although this feature degrades the relevancy of the semantic searches, it showed great results on collections with many dimensions (> 1300) and many points (more than 100000).

Enable the binary quantization by updating the settings of your embedder with:

curl \
  -X PATCH 'http://localhost:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "embedders": {
      "image2text": {
        "binaryQuantized": true
      }
    }
  }'

Done by @irevoire in #4941.

Request facet distributions and facet stats in your federated search, by using federation.facetsByIndex in the POST POST /multi-search route.

By default the distribution and stats are returned for each index that participates in the federated search.

POST /multi-search
{
  "federation": {
    "limit": 20,
    "offset": 0,
	"facetsByIndex": {
	  "movies": ["title", "id"],
	  "comics": ["title"],
	}
  },
  "queries": [
    {
      "q": "Batman",
      "indexUid": "movies"
    },
    {
      "q": "Batman",
      "indexUid": "comics"
    }
  ]
}

Merge the returned facets in a single facet distribution and stats that is global to the entire request, by using federation.mergeFacets in the POST POST /multi-search route.

The number of returned values can be limited for all facets with federation.mergeFacets.maxValuesPerFacet.

POST /multi-search
{

  "federation": {
    "limit": 20,
    "offset": 0,
	  "facetsByIndex": {
	    "movies": ["title", "id"],
	    "comics": ["title"],
	  },
	  "mergeFacets": {
	    "maxValuesPerFacet": 10,
	  }
  }
  "queries": [
    {
      "q": "Batman",
      "indexUid": "movies"
    },
    {
      "q": "Batman",
      "indexUid": "comics"
    }
  ]
}

Done by @dureuill in #4929.

Enable the experimental feature to use the STARTS_WITH filter operator:

curl \
  -X PATCH 'http://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "containsFilter": true
  }'

Use the STARTS_WITH operator when filtering:

curl \
  -X POST http://localhost:7700/indexes/movies/search \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "filter": "hero STARTS_WITH spider"
  }'

🗣️ This is an experimental feature, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.

Done by @Kerollmops in #4939.

• Language support and localizedAttributes settings by @ManyTheFish in #4937
  • Add iso-639-1 variants
  • Convert iso-639-1 into iso-639-3

• ⚠️ When using federated search, query.facets was silently ignored at the query level, but should not have been. It now returns the appropriate error. Use federation.facetsByIndex instead if you want facets to be applied during federated search.
• In prometheus metrics return the route pattern instead of the real route when returning the HTTP requests total by @irevoire in #4839
• Always truncate values at the end of the list of facet values when the number of facet values is larger than the index setting faceting.maxValuesPerFacet. For example, for the facet values ordered alphabetically ["blue", "red", "yellow"], setting faceting.maxValuesPerFacet sometimes resulted in ["blue", "yellow"] instead of ["blue", "red"]`. By @dureuill in #4929
• Fix Swedish support: the characters å/ä/ö are not normalized like the a/o characters anymore. By @ManyTheFish in #4945

• Dependencies updates
  • Security dependency upgrade: bump quinn-proto from 0.11.3 to 0.11.8 by @dependabot in #4911
• CIs and tests
  • Make the tests run faster by @irevoire in #4808
• Documentation
  • Fix broken links in README by @iornstein in #4943
• Misc
  • Allow Meilitool to dumplessly, offline upgrade v1.9 -> v1.10 in some conditions by @dureuill in #4912

❤️ Thanks again to our external contributors:

• Meilisearch: @iornstein.