v1.7: Updates to OpenAI embedder

[gui machiavelli]

v1.7 introduces two minor changes to Meilisearch's OpenAI embedder.

1. Two new supported models: text-embedding-3-small and text-embedding-3-large
2. Users can set the dimensions parameter for the OpenAI embedder (dimensions must be >0 and lower than the model size)

Questions

• Why would a user manually set the dimension? What is the benefit of setting 100 dimensions when the model was built with 1500?

Tasks

• Update OpenAI instructions in /experimental/vector_search

References

• SME: @dureuill
• GH discussion: https://github.com/orgs/meilisearch/discussions/677#discussioncomment-8306384
• engine issue: meilisearch/meilisearch#4394

[Louis Dureuil]

ah I had missed that one, hello there

Why would a user manually set the dimension? What is the benefit of setting 100 dimensions when the model was built with 1500?

this is an "accuracy" vs "storage and compute" trade-off.

See the announcement blogpost from OpenAI, section "Native support for shortening embeddings" (why aren't people putting anchors on all section titles? I don't know...)