[1/n] Add vLLM wrapper support

[noooop]

MTEB testing helps align vllm with the implementation of sentence-transformers, and identifying potential numerical precision issues is becoming increasingly important.

It's great to have MTEB support vLLM as a backend.

Fix #3747

[noooop]

@Samoed

I’m not very familiar with the MTEB framework—I’m unsure where the relevant code should be placed and how the interface should be designed. So for now, I’ve implemented a very, very dirtyversion in order to get feedback as early as possible.

VLLM has many, many parameters. I have highlighted the most commonly used ones for MTEB users and added a lot of comments, which may require optimization.

[Samoed]

Overall, this looks good! I think we can extend and change further if some models would need to update something. Thank you for adding!

[noooop]

How should documents, examples, and tests be organized?

[Samoed]

I don't think you need to add tests, because this is wrapper and for now no models use it.

As for docs, you can keep as docstrings without additional documentation

[Samoed]

Wrapper looks good. You can add vllm as optional dependency to pyproject and add check to VllmWrapperBase like here

mteb/mteb/models/search_encoder_index/search_indexes/faiss_search_index.py

Lines 28 to 33 in d3bc4cc

	requires_package(
	self,
	"faiss",
	"FAISS-based search",
	install_instruction="pip install mteb[faiss-cpu]",
	)

[KennethEnevoldsen]

I don't think you need to add tests, because this is wrapper and for now no models use it.

I think we should at least write one test wrapping a smallish model (even if it is not the official reference implementation)
This test can also act as the example used in the documentation for the vLLM wrapper.

[Samoed]

Yes, but I'm not sure if it would install and run correctly in CI

[KennethEnevoldsen]

Great work on this! I would love to add the tests + documentation.

It would even be great to add a blog post entry on comparing the speed between the sentence transformers implementation and the vLLM implementation, however that is def. optional.

[noooop]

I think we should add these two examples to the documentation. My suggestion would be here:

https://embeddings-benchmark.github.io/mteb/usage/defining_the_model/#using-a-sentence-transformer-model

under the title ## using vLLM

how about docs/advanced_usage/vllm_wrapper.md

I'm not very good at naming things or writing documents (as I'm a non-native English speaker). It would be great if someone could help rename things and optimize the documentation. Make the content of this PR fit naturally with the other parts.

[KennethEnevoldsen]

I can go over the docs once we have settled on content

[noooop]

If the comparison is fair and the methodology is reasonable, the gap between the ST implementation and the VLLM implementation is not significant.

X-axis: Throughput (request/s)
Y-axis: Latency, Time needed for one step (ms) <- logarithmic scale
The curve lower right is better ↘

• The throughput using float16 is approximately four times that of float32. VLLM has almost no optimization for float32 inference.
• VLLM does not use padding, while ST uses padding. For a batch of requests, if there is a significant disparity in sequence lengths, ST's performance will undoubtedly be very poor.

[Samoed]

Great illustration! I think you can add labels to axis and add to docs

[KennethEnevoldsen]

Do we want to add this as a blog post (I think I would prefer that) - so it is more of a "how to compare" rather that "this is how it compares", which is harder to maintain

[Samoed]

blog post

Do you mean as HF post?

[KennethEnevoldsen]

I was thinking of doing a blog post like this:

https://squidfunk.github.io/mkdocs-material/blog/2022/09/12/blog-support-just-landed/

(easy to copy to HF if we want to)

[Samoed]

Oh, nice! we can add this

[KennethEnevoldsen]

Yeah - it is a pretty good way to e.g. announce stuff or showcase an example, while not enforcing us to keep updating it

[KennethEnevoldsen]

Do we want to add this as a blog post (I think I would prefer that) - so it is more of a "how to compare" rather that "this is how it compares", which is harder to maintain

[KennethEnevoldsen]

Note to myself for when I rewrite this:

1. add link to speeding up mteb section
2. split into API docs and usage documentation
3. remove function scope
4. rewrite install section into admonition
5. refactor heading

[noooop]

I’ve submitted anything I found useful. It’s best to just update this PR to get it ready.

[noooop]

I think the content of this PR is pretty solid. Let’s polish it up so it’s ready to merge.

[Samoed]

Too many conflicts in dependencies. Probably we can move out model specific tests to different actions to simplify everything

[noooop]

Too many conflicts in dependencies. Probably we can move out model specific tests to different actions to simplify everything

vLLM has very complex dependencies; that’s why I haven’t added tests to CI.