Massive Text Embedding Benchmark

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for KennethEnevoldsen from gravatar.com KennethEnevoldsen Avatar for Muennighoff from gravatar.com Muennighoff Avatar for nouamanetazi from gravatar.com nouamanetazi Avatar for nreimers from gravatar.com nreimers

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License)
  • Author: MTEB Contributors
  • Tags deep learning , text embeddings , benchmark
  • Requires: Python <3.13, >=3.9
  • Provides-Extra: image , codecarbon , speedtask , peft , leaderboard , flagembedding , jina , jina-v4 , flash-attention , openai , model2vec , pylate , bm25s , gritlm , xformers , blip2 , voyageai , voyage-v , cohere , vertexai , llm2vec , timm , open-clip-torch , xet , ark , colpali-engine , youtu
Classifiers
Report project as malware

Project description

Massive Text Embedding Benchmark

GitHub release GitHub release License Downloads

Installation | Usage | Leaderboard | Documentation | Citing

Installation

pip install mteb

Example Usage

Using a script

import mteb
from sentence_transformers import SentenceTransformer

# Define the sentence-transformers model name
model_name = "average_word_embeddings_komninos"

model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)
tasks = mteb.get_tasks(tasks=["Banking77Classification"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")

Using the CLI

mteb available_tasks # list _all_ available tasks

mteb run -m sentence-transformers/all-MiniLM-L6-v2 \
    -t Banking77Classification  \
    --verbosity 3

# if nothing is specified default to saving the results in the results/{model_name} folder

Note that using multiple GPUs in parallel can be done by just having a custom encode function that distributes the inputs to multiple GPUs like e.g. here or here. See custom models for more information.

Usage Documentation

The following links to the main sections in the usage documentation.

Section
General
Evaluating a Model How to evaluate a model
Evaluating on different Modalities How to evaluate image and image-text tasks
MIEB How to run the Massive Image Embedding Benchmark
Selecting Tasks
Selecting a benchmark How to select benchmarks
Task selection How to select and filter tasks
 Selecting Split and Subsets  How to select evaluation splits or subsets
Using a Custom Task How to evaluate on a custom task
Selecting a Model
Using a Pre-defined Model  How to run a pre-defined model
Using a SentenceTransformer Model  How to run a model loaded using sentence-transformers
Using a Custom Model  How to run and implement a custom model
Running Evaluation
Passing Arguments to the model How to pass encode arguments to the model
Running Cross Encoders How to run cross encoders for reranking
Running Late Interaction (ColBERT) How to run late interaction models
Saving Retrieval Predictions How to save prediction for later analysis
Caching Embeddings How to cache and re-use embeddings
Leaderboard
Running the Leaderboard Locally How to run the leaderboard locally
Report Data Contamination How to report data contamination for a model
Loading and working with Results How to load and working with the raw results from the leaderboard, including making result dataframes

Overview

Overview
📈 Leaderboard The interactive leaderboard of the benchmark
📋 Tasks Overview of available tasks
📐 Benchmarks Overview of available benchmarks
Contributing
🤖 Adding a model How to submit a model to MTEB and to the leaderboard
👩‍🔬 Reproducible workflows How to create reproducible workflows with MTEB
👩‍💻 Adding a dataset How to add a new task/dataset to MTEB
👩‍💻 Adding a benchmark How to add a new benchmark to MTEB and to the leaderboard
🤝 Contributing How to contribute to MTEB and set it up for development

Citing

MTEB was introduced in "MTEB: Massive Text Embedding Benchmark", and heavily expanded in "MMTEB: Massive Multilingual Text Embedding Benchmark". When using mteb, we recommend that you cite both articles.

Bibtex Citation (click to unfold) 
@article{muennighoff2022mteb,
  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo{\"\i}c and Reimers, Nils},
  title = {MTEB: Massive Text Embedding Benchmark},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2210.07316},
  year = {2022}
  url = {https://arxiv.org/abs/2210.07316},
  doi = {10.48550/ARXIV.2210.07316},
}

@article{enevoldsen2025mmtebmassivemultilingualtext,
  title={MMTEB: Massive Multilingual Text Embedding Benchmark},
  author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2502.13595},
  year={2025},
  url={https://arxiv.org/abs/2502.13595},
  doi = {10.48550/arXiv.2502.13595},
}

If you use any of the specific benchmarks, we also recommend that you cite the authors.

benchmark = mteb.get_benchmark("MTEB(eng, v2)")
benchmark.citation # get citation for a specific benchmark

# you can also create a table of the task for the appendix using:
benchmark.tasks.to_latex()

Some of these amazing publications include (ordered chronologically):

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for KennethEnevoldsen from gravatar.com KennethEnevoldsen Avatar for Muennighoff from gravatar.com Muennighoff Avatar for nouamanetazi from gravatar.com nouamanetazi Avatar for nreimers from gravatar.com nreimers

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License)
  • Author: MTEB Contributors
  • Tags deep learning , text embeddings , benchmark
  • Requires: Python <3.13, >=3.9
  • Provides-Extra: image , codecarbon , speedtask , peft , leaderboard , flagembedding , jina , jina-v4 , flash-attention , openai , model2vec , pylate , bm25s , gritlm , xformers , blip2 , voyageai , voyage-v , cohere , vertexai , llm2vec , timm , open-clip-torch , xet , ark , colpali-engine , youtu
Classifiers

Release history Release notifications | RSS feed

2.12.30

2.12.29

2.12.28

2.12.27

2.12.26

2.12.25

2.12.24

2.12.23

2.12.22

2.12.21

2.12.20

2.12.19

2.12.18

2.12.17

2.12.16

2.12.15

2.12.14

2.12.13

2.12.12

2.12.11

2.12.10

2.12.9

2.12.8

2.12.7

2.12.6

2.12.5

2.12.4

2.12.3

2.12.2

2.12.1

2.12.0

2.11.10

2.11.9

2.11.8

2.11.7

2.11.6

2.11.5

2.11.4

2.11.3

2.11.2

2.11.1

2.11.0

2.10.14

2.10.13

2.10.12

2.10.11

2.10.10

2.10.9

2.10.8

2.10.7

2.10.6

2.10.5

2.10.4

2.10.3

2.10.2

2.10.1

2.10.0

2.9.1

2.9.0

2.8.10

2.8.9

2.8.8

2.8.7

2.8.6

2.8.5

2.8.4

2.8.3

2.8.2

2.8.1

2.7.30

2.7.29

2.7.28

2.7.27

2.7.26

2.7.25

2.7.24

2.7.23

2.7.22

2.7.21

2.7.20

2.7.19

2.7.18

2.7.17

2.7.16

2.7.15

2.7.14

2.7.13

2.7.12

2.7.11

2.7.10

2.7.9

2.7.8

2.7.7

2.7.6

2.7.5

2.7.4

2.7.3

2.7.2

2.7.1

2.7.0

2.6.9

2.6.8

2.6.7

2.6.6

2.6.5

2.6.4

2.6.3

2.6.2

2.6.1

2.6.0

2.5.5

2.5.4

2.5.3

2.5.2

2.5.1

2.5.0

2.4.2

2.4.1

2.3.11

2.3.10

2.3.9

2.3.8

2.3.7

2.3.6

2.3.5

2.3.4

2.3.3

2.3.2

2.3.1

2.3.0

2.2.2

2.2.1

2.2.0

2.1.19

2.1.18

2.1.17

2.1.16

2.1.15

2.1.14

2.1.13

2.1.12

2.1.11

2.1.10

2.1.9

2.1.8

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

1.39.7

1.39.6

1.39.5

1.39.4

1.39.3

1.39.2

1.39.1

1.39.0

1.38.61

This version

1.38.60

1.38.59

1.38.58

1.38.57

1.38.56

1.38.55

1.38.54

1.38.53

1.38.52

1.38.51

1.38.50

1.38.49

1.38.48

1.38.47

1.38.46

1.38.45

1.38.44

1.38.43

1.38.42

1.38.41

1.38.40

1.38.39

1.38.38

1.38.37

1.38.36

1.38.35

1.38.34

1.38.33

1.38.32

1.38.31

1.38.30

1.38.29

1.38.28

1.38.27

1.38.26

1.38.25

1.38.24

1.38.23

1.38.22

1.38.21

1.38.20

1.38.19

1.38.18

1.38.17

1.38.16

1.38.15

1.38.14

1.38.13

1.38.12

1.38.11

1.38.10

1.38.9

1.38.8

1.38.7

1.38.6

1.38.5

1.38.4

1.38.3

1.38.2

1.38.1

1.38.0

1.37.0

1.36.41

1.36.40

1.36.39

1.36.38

1.36.37

1.36.36

1.36.35

1.36.34

1.36.33

1.36.32

1.36.31

1.36.30

1.36.29

1.36.28

1.36.27

1.36.26

1.36.25

1.36.24

1.36.23

1.36.22

1.36.21

1.36.20

1.36.19

1.36.18

1.36.17

1.36.16

1.36.15

1.36.14

1.36.13

1.36.12

1.36.11

1.36.10

1.36.9

1.36.8

1.36.7

1.36.6

1.36.5

1.36.4

1.36.3

1.36.2

1.36.1

1.36.0

1.35.2

1.35.1

1.35.0

1.34.30

1.34.29

1.34.28

1.34.27

1.34.26

1.34.25

1.34.24

1.34.23

1.34.22

1.34.21

1.34.20

1.34.19

1.34.18

1.34.17

1.34.16

1.34.15

1.34.14

1.34.13

1.34.12

1.34.11

1.34.10

1.34.9

1.34.8

1.34.7

1.34.6

1.34.5

1.34.4

1.34.3

1.34.2

1.34.1

1.34.0

1.33.1

1.33.0

1.32.0

1.31.8

1.31.7

1.31.6

1.31.5

1.31.4

1.31.3

1.31.2

1.31.1

1.31.0

1.30.0

1.29.16

1.29.15

1.29.14

1.29.13

1.29.12

1.29.11

1.29.10

1.29.9

1.29.8

1.29.7

1.29.6

1.29.5

1.29.4

1.29.3

1.29.2

1.29.1

1.29.0

1.28.7

1.28.6

1.28.5

1.28.4

1.28.3

1.28.2

1.28.1

1.28.0

1.27.0

1.26.6

1.26.5

1.26.4

1.26.3

1.26.2

1.26.1

1.26.0

1.25.20

1.25.19

1.25.18

1.25.17

1.25.16

1.25.15

1.25.14

1.25.13

1.25.12

1.25.11

1.25.10

1.25.9

1.25.8

1.25.7

1.25.6

1.25.5

1.25.4

1.25.3

1.25.2

1.25.1

1.25.0

1.24.2

1.24.1

1.24.0

1.23.2

1.23.1

1.23.0

1.22.1

1.22.0

1.21.8

1.21.7

1.21.6

1.21.5

1.21.4

1.21.3

1.21.2

1.21.1

1.21.0

1.20.6

1.20.5

1.20.4

1.20.3

1.20.2

1.20.1

1.20.0

1.19.10

1.19.9

1.19.8

1.19.7

1.19.6

1.19.5

1.19.4

1.19.3

1.19.2

1.19.1

1.19.0

1.18.9

1.18.8

1.18.7

1.18.6

1.18.5

1.18.4

1.18.3

1.18.2

1.18.1

1.18.0

1.17.0

1.16.5

1.16.4

1.16.3

1.16.2

1.16.1

1.16.0

1.15.8

1.15.7

1.15.6

1.15.5

1.15.4

1.15.3

1.15.2

1.15.1

1.15.0

1.14.26

1.14.25

1.14.24

1.14.23

1.14.22

1.14.21

1.14.20

1.14.19

1.14.18

1.14.17

1.14.16

1.14.15

1.14.14

1.14.13

1.14.12

1.14.11

1.14.10

1.14.9

1.14.8

1.14.7

1.14.6

1.14.5

1.14.4

1.14.3

1.14.2

1.14.1

1.14.0

1.13.2

1.13.1

1.13.0

1.12.94

1.12.93

1.12.92

1.12.91

1.12.90

1.12.89

1.12.88

1.12.87

1.12.86

1.12.85

1.12.84

1.12.83

1.12.82

1.12.81

1.12.80

1.12.79

1.12.78

1.12.77

1.12.76

1.12.75

1.12.74

1.12.73

1.12.72

1.12.71

1.12.70

1.12.69

1.12.68

1.12.67

1.12.66

1.12.65

1.12.64

1.12.63

1.12.62

1.12.61

1.12.60

1.12.59

1.12.58

1.12.57

1.12.56

1.12.55

1.12.54

1.12.53

1.12.52

1.12.51

1.12.50

1.12.49

1.12.48

1.12.47

1.12.46

1.12.45

1.12.44

1.12.43

1.12.42

1.12.41

1.12.40

1.12.39

1.12.38

1.12.37

1.12.36

1.12.35

1.12.34

1.12.33

1.12.32

1.12.31

1.12.30

1.12.29

1.12.28

1.12.27

1.12.26

1.12.25

1.12.24

1.12.23

1.12.22

1.12.21

1.12.20

1.12.19

1.12.18

1.12.17

1.12.16

1.12.15

1.12.14

1.12.13

1.12.12

1.12.11

1.12.10

1.12.9

1.12.8

1.12.7

1.12.6

1.12.5

1.12.4

1.12.3

1.12.2

1.12.1

1.12.0

1.11.19

1.11.18

1.11.17

1.11.16

1.11.15

1.11.14

1.11.13

1.11.12

1.11.11

1.11.10

1.11.9

1.11.8

1.11.7

1.11.6

1.11.5

1.11.4

1.11.3

1.11.2

1.11.1

1.11.0

1.10.18

1.10.17

1.10.16

1.10.15

1.10.14

1.10.13

1.10.12

1.10.11

1.10.10

1.10.9

1.10.8

1.10.7

1.10.6

1.10.5

1.10.4

1.10.3

1.10.2

1.10.1

1.10.0

1.9.3

1.9.2

1.9.1

1.9.0

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.64

1.7.63

1.7.62

1.7.61

1.7.60

1.7.59

1.7.58

1.7.57

1.7.56

1.7.55

1.7.54

1.7.53

1.7.52

1.7.51

1.7.50

1.7.49

1.7.48

1.7.47

1.7.46

1.7.45

1.7.44

1.7.43

1.7.42

1.7.41

1.7.40

1.7.39

1.7.38

1.7.37

1.7.36

1.7.35

1.7.34

1.7.33

1.7.32

1.7.31

1.7.30

1.7.29

1.7.28

1.7.27

1.7.26

1.7.25

1.7.24

1.7.23

1.7.22

1.7.21

1.7.20

1.7.19

1.7.18

1.7.17

1.7.16

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.38

1.6.37

1.6.36

1.6.35

1.6.34

1.6.33

1.6.32

1.6.31

1.6.30

1.6.29

1.6.28

1.6.27

1.6.26

1.6.25

1.6.24

1.6.23

1.6.22

1.6.21

1.6.20

1.6.19

1.6.18

1.6.17

1.6.16

1.6.15

1.6.14

1.6.13

1.6.12

1.6.11

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.1

1.4.0

1.3.4

1.3.3

1.3.2

1.3.1

1.2.0

1.1.2

1.1.1

1.1.0

1.0.2

1.0.1

1.0.0

0.9.1

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mteb-1.38.60.tar.gz (1.1 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

mteb-1.38.60-py3-none-any.whl (1.8 MB view details)

Uploaded Python 3

File details

Details for the file mteb-1.38.60.tar.gz.

File metadata

  • Download URL: mteb-1.38.60.tar.gz
  • Upload date:
  • Size: 1.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for mteb-1.38.60.tar.gz
Algorithm Hash digest
SHA256 100e1d6a99bec546813e163213923c22f4eba0b021fa768eedaa30c1d7954473
MD5 7fa4c308f4ac6a28cc860b07d87fcc23
BLAKE2b-256 f0ffff49387eded0b7f6db2aa5d17459ce9b34352eeb566894e0e96f8356f618

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-1.38.60.tar.gz:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file mteb-1.38.60-py3-none-any.whl.

File metadata

  • Download URL: mteb-1.38.60-py3-none-any.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for mteb-1.38.60-py3-none-any.whl
Algorithm Hash digest
SHA256 d8d4ed1f9c72a62e43474fa3477c004cbd43630b23f89c5d7659ea05e9509425
MD5 daccf67af83afe7885a89b7751c93289
BLAKE2b-256 4323f9b3f716bc7ddef6ce7a8c1ea12b0db87623b259f5f73319e43392ea0c3f

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-1.38.60-py3-none-any.whl:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page