Massive Text Embedding Benchmark

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for KennethEnevoldsen from gravatar.com KennethEnevoldsen Avatar for Muennighoff from gravatar.com Muennighoff Avatar for nouamanetazi from gravatar.com nouamanetazi Avatar for nreimers from gravatar.com nreimers

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License)
  • Author: MTEB Contributors
  • Tags deep learning , text embeddings , benchmark
  • Requires: Python <3.13, >=3.9
  • Provides-Extra: image , dev , codecarbon , speedtask , peft , leaderboard , flagembedding , jina , flash-attention , openai , model2vec , pylate , bm25s , gritlm , xformers , blip2 , voyageai , voyage-v , cohere , vertexai , ll2vec , timm , open-clip-torch
Classifiers
Report project as malware

Project description

Massive Text Embedding Benchmark

GitHub release GitHub release License Downloads

Installation | Usage | Leaderboard | Documentation | Citing

Installation

pip install mteb

Example Usage

Using a script

import mteb
from sentence_transformers import SentenceTransformer

# Define the sentence-transformers model name
model_name = "average_word_embeddings_komninos"

model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)
tasks = mteb.get_tasks(tasks=["Banking77Classification"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")

Using the CLI

mteb available_tasks # list _all_ available tasks

mteb run -m sentence-transformers/all-MiniLM-L6-v2 \
    -t Banking77Classification  \
    --verbosity 3

# if nothing is specified default to saving the results in the results/{model_name} folder

Note that using multiple GPUs in parallel can be done by just having a custom encode function that distributes the inputs to multiple GPUs like e.g. here or here. See custom models for more information.

Usage Documentation

The following links to the main sections in the usage documentation.

Section
General
Evaluating a Model How to evaluate a model
Evaluating on different Modalities How to evaluate image and image-text tasks
Selecting Tasks
Selecting a benchmark How to select and filter tasks
Task selection How to select and filter tasks
 Selecting Split and Subsets  How to select evaluation splits or subsets
Using a Custom Task How to evaluate on a custom task
Selecting a Model
Using a Pre-defined Model  How to run a pre-defined model
Using a SentenceTransformer Model  How to run a model loaded using sentence-transformers
Using a Custom Model  How to run and implement a custom model
Running Evaluation
Passing Arguments to the model How to pass encode arguments to the model
Running Cross Encoders How to run cross encoders for reranking
Running Late Interaction (ColBERT) How to run late interaction models
Saving Retrieval Predictions How to save prediction for later analysis
Caching Embeddings How to cache and re-use embeddings
Leaderboard
Running the Leaderboard Locally How to run the leaderboard locally
Report Data Contamination How to report data contamination for a model
Fetching Result from the Leaderboard How to fetch the raw results from the leaderboard

Overview

Overview
📈 Leaderboard The interactive leaderboard of the benchmark
📋 Tasks Overview of available tasks
📐 Benchmarks Overview of available benchmarks
Contributing
🤖 Adding a model Information related to how to submit a model to MTEB and to the leaderboard
👩‍🔬 Reproducible workflows Information related to how to create reproducible workflows with MTEB
👩‍💻 Adding a dataset How to add a new task/dataset to MTEB
👩‍💻 Adding a benchmark How to add a new benchmark to MTEB and to the leaderboard
🤝 Contributing How to contribute to MTEB and set it up for development

Citing

MTEB was introduced in "MTEB: Massive Text Embedding Benchmark", and heavily expanded in "MMTEB: Massive Multilingual Text Embedding Benchmark". When using mteb, we recommend that you cite both articles.

Bibtex Citation (click to unfold) 
@article{muennighoff2022mteb,
  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo{\"\i}c and Reimers, Nils},
  title = {MTEB: Massive Text Embedding Benchmark},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2210.07316},
  year = {2022}
  url = {https://arxiv.org/abs/2210.07316},
  doi = {10.48550/ARXIV.2210.07316},
}

@article{enevoldsen2025mmtebmassivemultilingualtext,
  title={MMTEB: Massive Multilingual Text Embedding Benchmark},
  author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2502.13595},
  year={2025},
  url={https://arxiv.org/abs/2502.13595},
  doi = {10.48550/arXiv.2502.13595},
}

If you use any of the specific benchmarks, we also recommend that you cite the authors.

benchmark = mteb.get_benchmark("MTEB(eng, v2)")
benchmark.citation # get citation for a specific benchmark

# you can also create a table of the task for the appendix using:
benchmark.tasks.to_latex()

Some of these amazing publications include (ordered chronologically):

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for KennethEnevoldsen from gravatar.com KennethEnevoldsen Avatar for Muennighoff from gravatar.com Muennighoff Avatar for nouamanetazi from gravatar.com nouamanetazi Avatar for nreimers from gravatar.com nreimers

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License)
  • Author: MTEB Contributors
  • Tags deep learning , text embeddings , benchmark
  • Requires: Python <3.13, >=3.9
  • Provides-Extra: image , dev , codecarbon , speedtask , peft , leaderboard , flagembedding , jina , flash-attention , openai , model2vec , pylate , bm25s , gritlm , xformers , blip2 , voyageai , voyage-v , cohere , vertexai , ll2vec , timm , open-clip-torch
Classifiers

Release history Release notifications | RSS feed

2.12.30

2.12.29

2.12.28

2.12.27

2.12.26

2.12.25

2.12.24

2.12.23

2.12.22

2.12.21

2.12.20

2.12.19

2.12.18

2.12.17

2.12.16

2.12.15

2.12.14

2.12.13

2.12.12

2.12.11

2.12.10

2.12.9

2.12.8

2.12.7

2.12.6

2.12.5

2.12.4

2.12.3

2.12.2

2.12.1

2.12.0

2.11.10

2.11.9

2.11.8

2.11.7

2.11.6

2.11.5

2.11.4

2.11.3

2.11.2

2.11.1

2.11.0

2.10.14

2.10.13

2.10.12

2.10.11

2.10.10

2.10.9

2.10.8

2.10.7

2.10.6

2.10.5

2.10.4

2.10.3

2.10.2

2.10.1

2.10.0

2.9.1

2.9.0

2.8.10

2.8.9

2.8.8

2.8.7

2.8.6

2.8.5

2.8.4

2.8.3

2.8.2

2.8.1

2.7.30

2.7.29

2.7.28

2.7.27

2.7.26

2.7.25

2.7.24

2.7.23

2.7.22

2.7.21

2.7.20

2.7.19

2.7.18

2.7.17

2.7.16

2.7.15

2.7.14

2.7.13

2.7.12

2.7.11

2.7.10

2.7.9

2.7.8

2.7.7

2.7.6

2.7.5

2.7.4

2.7.3

2.7.2

2.7.1

2.7.0

2.6.9

2.6.8

2.6.7

2.6.6

2.6.5

2.6.4

2.6.3

2.6.2

2.6.1

2.6.0

2.5.5

2.5.4

2.5.3

2.5.2

2.5.1

2.5.0

2.4.2

2.4.1

2.3.11

2.3.10

2.3.9

2.3.8

2.3.7

2.3.6

2.3.5

2.3.4

2.3.3

2.3.2

2.3.1

2.3.0

2.2.2

2.2.1

2.2.0

2.1.19

2.1.18

2.1.17

2.1.16

2.1.15

2.1.14

2.1.13

2.1.12

2.1.11

2.1.10

2.1.9

2.1.8

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

1.39.7

1.39.6

1.39.5

1.39.4

1.39.3

1.39.2

1.39.1

1.39.0

1.38.61

1.38.60

1.38.59

1.38.58

1.38.57

1.38.56

1.38.55

1.38.54

1.38.53

1.38.52

1.38.51

1.38.50

1.38.49

1.38.48

1.38.47

1.38.46

1.38.45

1.38.44

1.38.43

1.38.42

1.38.41

1.38.40

1.38.39

1.38.38

1.38.37

1.38.36

1.38.35

1.38.34

1.38.33

1.38.32

1.38.31

1.38.30

1.38.29

1.38.28

1.38.27

1.38.26

1.38.25

1.38.24

1.38.23

1.38.22

1.38.21

1.38.20

1.38.19

1.38.18

1.38.17

1.38.16

1.38.15

1.38.14

1.38.13

1.38.12

1.38.11

1.38.10

1.38.9

1.38.8

1.38.7

1.38.6

1.38.5

1.38.4

1.38.3

1.38.2

1.38.1

1.38.0

1.37.0

1.36.41

1.36.40

1.36.39

1.36.38

1.36.37

1.36.36

This version

1.36.35

1.36.34

1.36.33

1.36.32

1.36.31

1.36.30

1.36.29

1.36.28

1.36.27

1.36.26

1.36.25

1.36.24

1.36.23

1.36.22

1.36.21

1.36.20

1.36.19

1.36.18

1.36.17

1.36.16

1.36.15

1.36.14

1.36.13

1.36.12

1.36.11

1.36.10

1.36.9

1.36.8

1.36.7

1.36.6

1.36.5

1.36.4

1.36.3

1.36.2

1.36.1

1.36.0

1.35.2

1.35.1

1.35.0

1.34.30

1.34.29

1.34.28

1.34.27

1.34.26

1.34.25

1.34.24

1.34.23

1.34.22

1.34.21

1.34.20

1.34.19

1.34.18

1.34.17

1.34.16

1.34.15

1.34.14

1.34.13

1.34.12

1.34.11

1.34.10

1.34.9

1.34.8

1.34.7

1.34.6

1.34.5

1.34.4

1.34.3

1.34.2

1.34.1

1.34.0

1.33.1

1.33.0

1.32.0

1.31.8

1.31.7

1.31.6

1.31.5

1.31.4

1.31.3

1.31.2

1.31.1

1.31.0

1.30.0

1.29.16

1.29.15

1.29.14

1.29.13

1.29.12

1.29.11

1.29.10

1.29.9

1.29.8

1.29.7

1.29.6

1.29.5

1.29.4

1.29.3

1.29.2

1.29.1

1.29.0

1.28.7

1.28.6

1.28.5

1.28.4

1.28.3

1.28.2

1.28.1

1.28.0

1.27.0

1.26.6

1.26.5

1.26.4

1.26.3

1.26.2

1.26.1

1.26.0

1.25.20

1.25.19

1.25.18

1.25.17

1.25.16

1.25.15

1.25.14

1.25.13

1.25.12

1.25.11

1.25.10

1.25.9

1.25.8

1.25.7

1.25.6

1.25.5

1.25.4

1.25.3

1.25.2

1.25.1

1.25.0

1.24.2

1.24.1

1.24.0

1.23.2

1.23.1

1.23.0

1.22.1

1.22.0

1.21.8

1.21.7

1.21.6

1.21.5

1.21.4

1.21.3

1.21.2

1.21.1

1.21.0

1.20.6

1.20.5

1.20.4

1.20.3

1.20.2

1.20.1

1.20.0

1.19.10

1.19.9

1.19.8

1.19.7

1.19.6

1.19.5

1.19.4

1.19.3

1.19.2

1.19.1

1.19.0

1.18.9

1.18.8

1.18.7

1.18.6

1.18.5

1.18.4

1.18.3

1.18.2

1.18.1

1.18.0

1.17.0

1.16.5

1.16.4

1.16.3

1.16.2

1.16.1

1.16.0

1.15.8

1.15.7

1.15.6

1.15.5

1.15.4

1.15.3

1.15.2

1.15.1

1.15.0

1.14.26

1.14.25

1.14.24

1.14.23

1.14.22

1.14.21

1.14.20

1.14.19

1.14.18

1.14.17

1.14.16

1.14.15

1.14.14

1.14.13

1.14.12

1.14.11

1.14.10

1.14.9

1.14.8

1.14.7

1.14.6

1.14.5

1.14.4

1.14.3

1.14.2

1.14.1

1.14.0

1.13.2

1.13.1

1.13.0

1.12.94

1.12.93

1.12.92

1.12.91

1.12.90

1.12.89

1.12.88

1.12.87

1.12.86

1.12.85

1.12.84

1.12.83

1.12.82

1.12.81

1.12.80

1.12.79

1.12.78

1.12.77

1.12.76

1.12.75

1.12.74

1.12.73

1.12.72

1.12.71

1.12.70

1.12.69

1.12.68

1.12.67

1.12.66

1.12.65

1.12.64

1.12.63

1.12.62

1.12.61

1.12.60

1.12.59

1.12.58

1.12.57

1.12.56

1.12.55

1.12.54

1.12.53

1.12.52

1.12.51

1.12.50

1.12.49

1.12.48

1.12.47

1.12.46

1.12.45

1.12.44

1.12.43

1.12.42

1.12.41

1.12.40

1.12.39

1.12.38

1.12.37

1.12.36

1.12.35

1.12.34

1.12.33

1.12.32

1.12.31

1.12.30

1.12.29

1.12.28

1.12.27

1.12.26

1.12.25

1.12.24

1.12.23

1.12.22

1.12.21

1.12.20

1.12.19

1.12.18

1.12.17

1.12.16

1.12.15

1.12.14

1.12.13

1.12.12

1.12.11

1.12.10

1.12.9

1.12.8

1.12.7

1.12.6

1.12.5

1.12.4

1.12.3

1.12.2

1.12.1

1.12.0

1.11.19

1.11.18

1.11.17

1.11.16

1.11.15

1.11.14

1.11.13

1.11.12

1.11.11

1.11.10

1.11.9

1.11.8

1.11.7

1.11.6

1.11.5

1.11.4

1.11.3

1.11.2

1.11.1

1.11.0

1.10.18

1.10.17

1.10.16

1.10.15

1.10.14

1.10.13

1.10.12

1.10.11

1.10.10

1.10.9

1.10.8

1.10.7

1.10.6

1.10.5

1.10.4

1.10.3

1.10.2

1.10.1

1.10.0

1.9.3

1.9.2

1.9.1

1.9.0

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.64

1.7.63

1.7.62

1.7.61

1.7.60

1.7.59

1.7.58

1.7.57

1.7.56

1.7.55

1.7.54

1.7.53

1.7.52

1.7.51

1.7.50

1.7.49

1.7.48

1.7.47

1.7.46

1.7.45

1.7.44

1.7.43

1.7.42

1.7.41

1.7.40

1.7.39

1.7.38

1.7.37

1.7.36

1.7.35

1.7.34

1.7.33

1.7.32

1.7.31

1.7.30

1.7.29

1.7.28

1.7.27

1.7.26

1.7.25

1.7.24

1.7.23

1.7.22

1.7.21

1.7.20

1.7.19

1.7.18

1.7.17

1.7.16

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.38

1.6.37

1.6.36

1.6.35

1.6.34

1.6.33

1.6.32

1.6.31

1.6.30

1.6.29

1.6.28

1.6.27

1.6.26

1.6.25

1.6.24

1.6.23

1.6.22

1.6.21

1.6.20

1.6.19

1.6.18

1.6.17

1.6.16

1.6.15

1.6.14

1.6.13

1.6.12

1.6.11

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.1

1.4.0

1.3.4

1.3.3

1.3.2

1.3.1

1.2.0

1.1.2

1.1.1

1.1.0

1.0.2

1.0.1

1.0.0

0.9.1

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mteb-1.36.35.tar.gz (1.3 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

mteb-1.36.35-py3-none-any.whl (2.0 MB view details)

Uploaded Python 3

File details

Details for the file mteb-1.36.35.tar.gz.

File metadata

  • Download URL: mteb-1.36.35.tar.gz
  • Upload date:
  • Size: 1.3 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for mteb-1.36.35.tar.gz
Algorithm Hash digest
SHA256 5d0a70bbea72a0c5c05db29144ff45ea4cd53a100535e6b10636ce738000ed15
MD5 872fb31ad29883103b1b0c7dd8b9f8f9
BLAKE2b-256 b935fe822f158c436b3ab3f9decf3e204450e52d21c7ff45a1c17f1b9332ca46

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-1.36.35.tar.gz:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file mteb-1.36.35-py3-none-any.whl.

File metadata

  • Download URL: mteb-1.36.35-py3-none-any.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for mteb-1.36.35-py3-none-any.whl
Algorithm Hash digest
SHA256 d37181c17449fc7c3b2d6dccf322feb4bd32c712324f89dd73c8403013a32361
MD5 beb9ddddc319d80ea62d3ed51aedf613
BLAKE2b-256 cdd9d9c6ec097e4e72a6754afd6d1cdbbd57c5d0842f1d188bd0135c36aa72a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-1.36.35-py3-none-any.whl:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page