rahulseetharaman/reranker-msmarco-v1.1-bert-uncased_L-8_H-512_A-8-listnet

CrossEncoder based on bansalaman18/bert-uncased_L-8_H-512_A-8

This is a Cross Encoder model finetuned from bansalaman18/bert-uncased_L-8_H-512_A-8 on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: bansalaman18/bert-uncased_L-8_H-512_A-8
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("rahulseetharaman/reranker-msmarco-v1.1-bert-uncased_L-8_H-512_A-8-listnet")
# Get scores for pairs of texts
pairs = [
    ['are phone area codes', 'In the US and its territories, Canada, and the Caribbean, the organization and allocation of telephone numbers is governed by the North American Numbering Plan Administration (NANPA). The NANPA organizes the allocation of area codes and telephone prefixes to various phone companies. The basic format of a phone number in any of these countries is NPA-NXX-XXXX or (NPA) NXX-XXX. NPA codes are more commonly referred to as area codes. Some area codes are reserved for special purpos'],
    ['are phone area codes', 'AreaCode.org is an area codes locator and comprehensive directory of area codes across the United States, Canada, and within the North American Numbering Plan. Find information on each area code, including coverage area, local dialing instructions, area code map and list of cities within each telephone area code. Other relevant telephone area codes information is found in the Area Code Detail section. Use AreaCode.org for area code lookup, area code search, and as an area code finder. You could say that AreaCode.org takes the code out of telephone area code.'],
    ['are phone area codes', 'Area Codes. AreaCode.org is an area code directory with helpful information on the various United States area codes, area code listing, and other area code phone details. AreaCode.org also list international area codes within the North American Numbering Plan. Other relevant telephone area codes information is found in the Area Code Detail section. Use AreaCode.org for area code lookup, area code search, and as an area code finder. You could say that AreaCode.org takes the code out of telephone area code.'],
    ['are phone area codes', 'Search for a phone number, phone code, or place: The UK Telephone Code Locator takes telephone numbers, std codes, area codes, access/short codes, exchange names or the name of a location. Details returned may include: information about where the area is, map location, BT charge information and nearby exchange names. Type in any of the following: 1  A full UK phone number, or just an area code or other dialling code.'],
    ['are phone area codes', 'Country Calling Codes was founded in 2000 as a free resource to findtelephone area codes and is now used by millions of users. We provide fast and accurate information on how to dial internationally from the USA, Canada and every other country in the world. Telephone country codes, international area codes, and reverse lookups are provided for every country in the world, in addition to the world time zone at each location.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'are phone area codes',
    [
        'In the US and its territories, Canada, and the Caribbean, the organization and allocation of telephone numbers is governed by the North American Numbering Plan Administration (NANPA). The NANPA organizes the allocation of area codes and telephone prefixes to various phone companies. The basic format of a phone number in any of these countries is NPA-NXX-XXXX or (NPA) NXX-XXX. NPA codes are more commonly referred to as area codes. Some area codes are reserved for special purpos',
        'AreaCode.org is an area codes locator and comprehensive directory of area codes across the United States, Canada, and within the North American Numbering Plan. Find information on each area code, including coverage area, local dialing instructions, area code map and list of cities within each telephone area code. Other relevant telephone area codes information is found in the Area Code Detail section. Use AreaCode.org for area code lookup, area code search, and as an area code finder. You could say that AreaCode.org takes the code out of telephone area code.',
        'Area Codes. AreaCode.org is an area code directory with helpful information on the various United States area codes, area code listing, and other area code phone details. AreaCode.org also list international area codes within the North American Numbering Plan. Other relevant telephone area codes information is found in the Area Code Detail section. Use AreaCode.org for area code lookup, area code search, and as an area code finder. You could say that AreaCode.org takes the code out of telephone area code.',
        'Search for a phone number, phone code, or place: The UK Telephone Code Locator takes telephone numbers, std codes, area codes, access/short codes, exchange names or the name of a location. Details returned may include: information about where the area is, map location, BT charge information and nearby exchange names. Type in any of the following: 1  A full UK phone number, or just an area code or other dialling code.',
        'Country Calling Codes was founded in 2000 as a free resource to findtelephone area codes and is now used by millions of users. We provide fast and accurate information on how to dial internationally from the USA, Canada and every other country in the world. Telephone country codes, international area codes, and reverse lookups are provided for every country in the world, in addition to the world time zone at each location.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.0787 (-0.4109)	0.2859 (+0.0249)	0.0656 (-0.3540)
mrr@10	0.0533 (-0.4242)	0.4384 (-0.0614)	0.0410 (-0.3857)
ndcg@10	0.0739 (-0.4666)	0.3024 (-0.0227)	0.0646 (-0.4360)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.1434 (-0.2466)
mrr@10	0.1776 (-0.2904)
ndcg@10	0.1470 (-0.3084)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 9 characters mean: 33.39 characters max: 109 characters	min: 2 elements mean: 6.00 elements max: 10 elements	min: 2 elements mean: 6.00 elements max: 10 elements

• Samples:

  query	docs	labels
  how do you oven bake a salmon	['Directions. Preheat the oven to 450 degrees F. Season salmon with salt and pepper. Place salmon, skin side down, on a non-stick baking sheet or in a non-stick pan with an oven-proof handle. Bake until salmon is cooked through, about 12 to 15 minutes. Serve with the Toasted Almond Parsley Salad and squash, if desired. Mince the shallot and add to a small bowl.', 'Preheat the oven to 350 degrees Fahrenheit. Lay the salmon, skin side up, on the baking sheet. Drizzle the steaks or fillets with olive oil, then sprinkle with Kosher salt and pepper. Flip the salmon over so the skin side is down. Drizzle the flesh side with olive oil and season with salt and pepper.', 'Directions. Preheat oven to 475 degrees. Place butter on a rimmed baking sheet and season with salt and pepper. Place sheet in oven until butter melts, about 1 minute. Carefully remove sheet from oven and place salmon on top of butter, skin side down; season with salt and pepper. Return sheet to oven. Roast until salmon is jus...	[1, 0, 0, 0, 0, ...]
  how much does it cost for a new central heating boiler	["Taking the increased boiler cost and additional tank installation an LPG central heating system can typically cost anything from £2,500 - £6,000 depending on the size of property. Ongoing running costs are higher than natural mains gas but lower than oil. These types obviously cost more due to the greater complexity of pipework and typically range in price from £2,500 - £5,000, again it all depends on what is included so always check that you're comparing like for like with the quotes.", 'Depending on where you live they cost between $50 and $300. Some local governments also require a fire inspection before heating system replacements. The fees for these are dependent on the size of the boiler and tend to run between $40 and $75. If you have a boiler installed or replaced, you might pay anywhere between $3,111 and $7,204. On average, homeowners tend to pay $5,100. These prices are highly dependent on the type of boiler you choose.', 'The time has come to replace our 20 year-old centr...	[1, 0, 0, 0, 0, ...]
  how to cite from website mla	['How to cite a website in a bibliography using MLA. The most basic entry for a website consists of the author name(s), page title, website title, sponsoring institution/publisher, date published, medium, and date accessed. Last Name, First Name. “Page Title.” Website Title .Sponsoring Institution/Publisher. ', 'How to cite websites using the MLA citation format. For more information visit the Citing Sources, Research Tips and Tutorials section of the library website at http://library.noctrl.edu. 1 Education. 2 Standard YouTube License. ', 'Components usually found on a website: 1 Name of the author, comiler, director, editor, narrator, performer, or translator of work. 2 Title of the work (italicized if teh work is independent: in quotation marks if it is part of a larger work). 3 Title of the overall website-italicized (if distinct from item 2). 1 Title of the work (italicized if teh work is independent: in quotation marks if it is part of a larger work). 2 Title of the overall...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 11 characters mean: 34.0 characters max: 84 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  are phone area codes	['In the US and its territories, Canada, and the Caribbean, the organization and allocation of telephone numbers is governed by the North American Numbering Plan Administration (NANPA). The NANPA organizes the allocation of area codes and telephone prefixes to various phone companies. The basic format of a phone number in any of these countries is NPA-NXX-XXXX or (NPA) NXX-XXX. NPA codes are more commonly referred to as area codes. Some area codes are reserved for special purpos', 'AreaCode.org is an area codes locator and comprehensive directory of area codes across the United States, Canada, and within the North American Numbering Plan. Find information on each area code, including coverage area, local dialing instructions, area code map and list of cities within each telephone area code. Other relevant telephone area codes information is found in the Area Code Detail section. Use AreaCode.org for area code lookup, area code search, and as an area code finder. You could say that Area...	[1, 0, 0, 0, 0, ...]
  what makes the muscle cell unique	['The structure of skeletal muscle cells also makes them unique among muscle tissues. Skeletal muscle cells develop from the fusion of many smaller cells during fetal development, resulting in long, straight muscle fibers that contain many nuclei. It forms all of the skeletal muscles, such as the biceps brachii and gluteus maximus, and is found in the eyes, throat, diaphragm, and anus.... Click to read more below. Muscle tissue, one of the four major tissue types, plays the vital role of providing movement and heat generation to the organs of the body.', 'Smooth Muscle cells. The types of muscle cells are very interesting because there are many similarities, but there are also many differences. Smooth muscle is the most unique because it has very little in common with the other types. It is weaker and less organized. Furthermore, cardiac muscle cells also have intercalated disks, or desmosomes, that allow the cells to stick together. Skeletal muscle cells and cardiac cells have striati...	[1, 0, 0, 0, 0, ...]
  what does the bronchi split off into	['A bronchus, also known as a main or primary bronchus, is a passage of airway in the respiratory tract that conducts air into the lungs. There is a right bronchus and a left bronchus and these bronchi branch into smaller secondary and tertiary bronchi which branch into smaller tubes, known as bronchioles. The right bronchus now passes below the artery, and is known as the hyparterial branch which divides into the two lobar bronchi to the middle and lower lobes. The left main bronchus is smaller in caliber but longer than the right, being 5 cm long. It enters the root of the left lung opposite the sixth thoracic vertebra.', 'Glossary of Terms. Bronchus / Bronchi / Bronchioles. The airways of the lungs. When a person takes in a breath of air, the air travels through the nose or mouth, into the larynx, then into the trachea, which is the main passageway into the lungs. The trachea divides into a right and left main bronchus. ', 'The left main bronchus divides into two lobar bronchi, one ...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	-	0.0153 (-0.5251)	0.1909 (-0.1342)	0.0562 (-0.4445)	0.0874 (-0.3679)
0.0002	1	2.1689	-	-	-	-	-
0.0508	250	2.096	-	-	-	-	-
0.1016	500	2.0919	2.0775	0.0644 (-0.4760)	0.3259 (+0.0009)	0.0162 (-0.4845)	0.1355 (-0.3199)
0.1525	750	2.0951	-	-	-	-	-
0.2033	1000	2.0902	2.0762	0.0589 (-0.4815)	0.2673 (-0.0577)	0.0585 (-0.4421)	0.1283 (-0.3271)
0.2541	1250	2.0873	-	-	-	-	-
0.3049	1500	2.0872	2.0761	0.0863 (-0.4541)	0.2916 (-0.0334)	0.0441 (-0.4566)	0.1407 (-0.3147)
0.3558	1750	2.0856	-	-	-	-	-
0.4066	2000	2.0854	2.0758	0.0629 (-0.4775)	0.3020 (-0.0230)	0.0560 (-0.4446)	0.1403 (-0.3150)
0.4574	2250	2.0889	-	-	-	-	-
0.5082	2500	2.0866	2.0756	0.0739 (-0.4666)	0.3024 (-0.0227)	0.0646 (-0.4360)	0.1470 (-0.3084)
0.5591	2750	2.0892	-	-	-	-	-
0.6099	3000	2.089	2.0757	0.0655 (-0.4749)	0.2994 (-0.0257)	0.0482 (-0.4525)	0.1377 (-0.3177)
0.6607	3250	2.0865	-	-	-	-	-
0.7115	3500	2.0891	2.0755	0.0552 (-0.4852)	0.3026 (-0.0225)	0.0428 (-0.4578)	0.1335 (-0.3218)
0.7624	3750	2.086	-	-	-	-	-
0.8132	4000	2.0874	2.0753	0.0588 (-0.4816)	0.3050 (-0.0200)	0.0609 (-0.4398)	0.1415 (-0.3138)
0.8640	4250	2.0883	-	-	-	-	-
0.9148	4500	2.0816	2.0753	0.0577 (-0.4827)	0.2875 (-0.0375)	0.0592 (-0.4414)	0.1348 (-0.3205)
0.9656	4750	2.0802	-	-	-	-	-
-1	-1	-	-	0.0739 (-0.4666)	0.3024 (-0.0227)	0.0646 (-0.4360)	0.1470 (-0.3084)

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.0.0
• Transformers: 4.56.0.dev0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ListNetLoss

@inproceedings{cao2007learning,
    title={Learning to Rank: From Pairwise Approach to Listwise Approach},
    author={Cao, Zhe and Qin, Tao and Liu, Tie-Yan and Tsai, Ming-Feng and Li, Hang},
    booktitle={Proceedings of the 24th international conference on Machine learning},
    pages={129--136},
    year={2007}
}