ai21labs/AI21-Jamba-Large-1.7-FP8

Model Information

Jamba Large 1.7-FP8 offers new improvements to our Jamba open model family. This new version builds on the novel SSM-Transformer hybrid architecture, 256K context window, and efficiency gains of previous versions, while introducing improvements in grounding, instruction-following, and speed.

Key improvements:

• Grounding: Jamba Large 1.7-FP8 provides more complete and accurate answers, grounded fully in the given context.
• Instruction following: Jamba Large 1.7-FP8 improves on steerability.
• Speed: Jamba Large 1.7-FP8 is faster due to FP8 quantizations.

Use cases

Jamba’s long context efficiency, contextual faithfulness, and steerability make it ideal for a variety of business applications and industries, such as:

• Finance: Investment research, digital banking support chatbot, M&A due diligence.
• Healthcare: Procurement (RFP creation & response review), medical publication and reports generation.
• Retail: Brand-aligned product description generation, conversational AI.
• Education & Research: Personalized chatbot tutor, grants applications.

The models are released under the Jamba Open Model License, a permissive license allowing full research use and commercial use under the license terms. If you need to license the model for your needs, talk to us.

Model Details

Developed by: AI21 Model type: Joint Attention and Mamba (Jamba) Model size: 94B active/398B total parameters License: Jamba Open Model License Context length: 256K Knowledge cutoff date: August 22, 2024 Supported languages: English, Spanish, French, Portuguese, Italian, Dutch, German, Arabic and Hebrew

Grounding and instruction-following improvements

Category	Benchmark	Jamba Large 1.6	Jamba Large 1.7
Grounding	FACTS	0.758	0.832
Steerability	IFEcal	0.782	0.84

FP8 Quantization

Jamba Large 1.7-FP8 weights are available in this pre-quantized FP8 format, which is optimal for NVIDIA Hopper architecture machines. As a result:

• The initial GPU memory footprint is lower on inference launch.
• FP8 model weights require almost 50% less disk space.

Usage

Find step-by-step instructions on how to privately deploy Jamba:

Run the model with vLLM

The recommended way to perform efficient inference with Jamba Large 1.7-FP8 is using vLLM. First, make sure to install vLLM (version 0.6.5 or higher is required):

pip install vllm>=0.6.5

Jamba Large 1.7-FP8 is too large to be loaded in full (FP32) or half (FP16/BF16) precision on a single node of 8 80GB GPUs. Using Jamba Large 1.7-FP8, you'll be able to deploy the model on a single node of 8 80GB GPUs.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer


model = "ai21labs/AI21-Jamba-Large-1.7-FP8"


llm = LLM(model=model,
          tensor_parallel_size=8,
          max_model_len=220*1024,
         )


tokenizer = AutoTokenizer.from_pretrained(model)


messages = [
   {"role": "system", "content": "You are an ancient oracle who speaks in cryptic but wise phrases, always hinting at deeper meanings."},
   {"role": "user", "content": "Hello!"},
]


prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)


sampling_params = SamplingParams(temperature=0.4, top_p=0.95, max_tokens=100)
outputs = llm.generate(prompts, sampling_params)


generated_text = outputs[0].outputs[0].text
print(generated_text)

Note: Versions 4.44.0 and 4.44.1 of transformers have a bug that restricts the ability to run the Jamba architecture. Make sure you're not using these versions.

Note: If you're having trouble installing mamba-ssm and causal-conv1d for the optimized Mamba kernels, you can run Jamba Large 1.7-FP8 without them, at the cost of extra latency. In order to do that, add the kwarg use_mamba_kernels=False when loading the model via AutoModelForCausalLM.from_pretained().

You can also find all instructions in our private AI (vLLM) deployment guide.

Run the model with Transformers

To load Jamba Large 1.7 in transformers on a single node of 8 80GB GPUs, we recommend to parallelize it using accelerate:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


# a device map to distribute the model evenly across 8 GPUs
device_map = {'model.embed_tokens': 0, 'model.layers.0': 0, 'model.layers.1': 0, 'model.layers.2': 0, 'model.layers.3': 0, 'model.layers.4': 0, 'model.layers.5': 0, 'model.layers.6': 0, 'model.layers.7': 0, 'model.layers.8': 0, 'model.layers.9': 1, 'model.layers.10': 1, 'model.layers.11': 1, 'model.layers.12': 1, 'model.layers.13': 1, 'model.layers.14': 1, 'model.layers.15': 1, 'model.layers.16': 1, 'model.layers.17': 1, 'model.layers.18': 2, 'model.layers.19': 2, 'model.layers.20': 2, 'model.layers.21': 2, 'model.layers.22': 2, 'model.layers.23': 2, 'model.layers.24': 2, 'model.layers.25': 2, 'model.layers.26': 2, 'model.layers.27': 3, 'model.layers.28': 3, 'model.layers.29': 3, 'model.layers.30': 3, 'model.layers.31': 3, 'model.layers.32': 3, 'model.layers.33': 3, 'model.layers.34': 3, 'model.layers.35': 3, 'model.layers.36': 4, 'model.layers.37': 4, 'model.layers.38': 4, 'model.layers.39': 4, 'model.layers.40': 4, 'model.layers.41': 4, 'model.layers.42': 4, 'model.layers.43': 4, 'model.layers.44': 4, 'model.layers.45': 5, 'model.layers.46': 5, 'model.layers.47': 5, 'model.layers.48': 5, 'model.layers.49': 5, 'model.layers.50': 5, 'model.layers.51': 5, 'model.layers.52': 5, 'model.layers.53': 5, 'model.layers.54': 6, 'model.layers.55': 6, 'model.layers.56': 6, 'model.layers.57': 6, 'model.layers.58': 6, 'model.layers.59': 6, 'model.layers.60': 6, 'model.layers.61': 6, 'model.layers.62': 6, 'model.layers.63': 7, 'model.layers.64': 7, 'model.layers.65': 7, 'model.layers.66': 7, 'model.layers.67': 7, 'model.layers.68': 7, 'model.layers.69': 7, 'model.layers.70': 7, 'model.layers.71': 7, 'model.final_layernorm': 7, 'lm_head': 7}
model = AutoModelForCausalLM.from_pretrained("ai21labs/AI21-Jamba-Large-1.7-FP8",
attn_implementation="flash_attention_2",
device_map=device_map)


tokenizer = AutoTokenizer.from_pretrained("ai21labs/AI21-Jamba-Large-1.7-FP8")


messages = [
   {"role": "system", "content": "You are an ancient oracle who speaks in cryptic but wise phrases, always hinting at deeper meanings."},
   {"role": "user", "content": "Hello!"},
]


input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors='pt').to(model.device)


outputs = model.generate(input_ids, max_new_tokens=216)


# Decode the output
conversation = tokenizer.decode(outputs[0], skip_special_tokens=True)


# Split the conversation to get only the assistant's response
assistant_response = conversation.split(messages[-1]['content'])[1].strip()
print(assistant_response)
# Output: Seek and you shall find. The path is winding, but the journey is enlightening. What wisdom do you seek from the ancient echoes?

Note: Versions 4.44.0 and 4.44.1 of transformers have a bug that restricts the ability to run the Jamba architecture. Make sure you're not using these versions.

Note: If you're having trouble installing mamba-ssm and causal-conv1d for the optimized Mamba kernels, you can run Jamba Large 1.7 without them, at the cost of extra latency. In order to do that, add the kwarg use_mamba_kernels=False when loading the model via AutoModelForCausalLM.from_pretained().

And to get started with our SDK: AI21 Python SDK guide

Further documentation

For more comprehensive guides and advanced usage:

• Tokenization guide - Using ai21-tokenizer
• Quantization guide - ExpertsInt8, bitsandbytes
• Fine-tuning guide - LoRA, qLoRA, and full fine-tuning
• Function-calling guide

For more resources to start building, visit our official documentation.