Revision 3.3.1 model (#3)

- Upload folder using huggingface_hub (00f6bae95602cbd087854eb5f6ffc7783f87ad4f)

README.md

@@ -6,25 +6,29 @@ base_model:
 - ibm-granite/granite-3.3-8b-instruct
 library_name: transformers
 ---
-# Granite-speech-3.3-8b
+# Granite-speech-3.3-8b (revision 3.3.1)
 
 **Model Summary:**
 Granite-speech-3.3-8b is a compact and efficient speech-language model, specifically designed for automatic speech recognition (ASR) and automatic speech translation (AST). Granite-speech-3.3-8b uses a two-pass design, unlike integrated models that combine speech and language into a single pass. Initial calls to granite-speech-3.3-8b will transcribe audio files into text. To process the transcribed text using the underlying Granite language model, users must make a second call as each step must be explicitly initiated.
 
 The model was trained on a collection of public corpora comprising diverse datasets for ASR and AST as well as synthetic datasets tailored to support the speech translation task. Granite-speech-3.3 was trained by modality aligning granite-3.3-8b-instruct (https://huggingface.co/ibm-granite/granite-3.3-8b-instruct) to speech on publicly available open source corpora containing audio inputs and text targets.
 
+Compared to the initial release, revision 3.3.1 has improvements in model performance stemming from two key changes:
+* Training on additional data
+* A deeper acoustic encoder for improved transcription accuracy
+
 We are currently investigating an issue with greedy decoding (```num_beams=1```); the model performs reliably with beam sizes > 1, which we recommend for all use cases. 
 Additionally, the model may occasionally hallucinate on very short audio inputs (<0.1s). These issues are under active investigation, and we will update guidance as fixes become available.
 
 **Evaluations:**
 
-We evaluated granite-speech-3.3-8b alongside other speech-language models (SLMs) in the less than 8b parameter range as well as dedicated ASR and AST systems on standard benchmarks. The evaluation spanned multiple public benchmarks, with particular emphasis on English ASR tasks while also including AST for En-X translation. 
+We evaluated granite-speech-3.3-8b revision 3.3.1 alongside other speech-language models (SLMs) in the less than 8b parameter range as well as dedicated ASR and AST systems on standard benchmarks. The evaluation spanned multiple public benchmarks, with particular emphasis on English ASR tasks while also including AST for En-X translation. 
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/79lNwtsiIFaX7mqKKBwCy.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/wJzTRa6Kox6JBwYzmPk0J.png)
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/Sl3YmG326Rh08pJ0sagey.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/Hxg9YZgHYra3IRjYiEjaD.png)
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/2kIgpCMSsxQg-rEHPnDW1.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/YlUGDJoPD72je_p5qVVRE.png)
 
 **Release Date**: April 15, 2025 
 
@@ -39,7 +43,7 @@ The model is intended to be used in enterprise applications that involve process
 
 ## Generation:
 
-Granite Speech model is supported natively in `transformers` from the `main` branch. Below is a simple example of how to use the `granite-speech-3.3-8b` model.
+Granite Speech model is supported natively in `transformers` from the `main` branch. Below is a simple example of how to use the `granite-speech-3.3-8b` revision 3.3.1 model.
 
 ### Usage with `transformers`
 
@@ -303,14 +307,14 @@ print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}"
 
 The architecture of granite-speech-3.3-8b consists of the following components:
 
-(1) Speech encoder: 10 conformer blocks trained with Connectionist Temporal Classification (CTC) on character-level targets on the subset containing
+(1) Speech encoder: 16 conformer blocks trained with Connectionist Temporal Classification (CTC) on character-level targets on the subset containing
 only ASR corpora (see configuration below). In addition, our CTC encoder uses block-attention with 4-seconds audio blocks and self-conditioned CTC
 from the middle layer.
 
 | Configuration parameter  | Value                | 
 |-----------------|----------------------|
 | Input dimension | 160 (80 logmels x 2) | 
-| Nb. of layers   | 10                   | 
+| Nb. of layers   | 16                   | 
 | Hidden dimension | 1024                | 
 | Nb. of attention heads | 8             | 
 | Attention head size    | 128           | 
@@ -340,7 +344,8 @@ below:
 | Librispeech             | ASR | 1000 |  https://huggingface.co/datasets/openslr/librispeech_asr | 
 | VoxPopuli English       | ASR | 500 |  https://huggingface.co/datasets/facebook/voxpopuli | 
 | AMI                     | ASR | 100 | https://huggingface.co/datasets/edinburghcstr/ami | 
-| YODAS English           | ASR | 10000 |  https://huggingface.co/datasets/espnet/yodas | 
+| YODAS English           | ASR | 10000 |  https://huggingface.co/datasets/espnet/yodas |
+| Earnings-22             | ASR | 120 | https://huggingface.co/datasets/distil-whisper/earnings22 | 
 | Switchboard English     | ASR | 260 | https://catalog.ldc.upenn.edu/LDC97S62 |
 | CallHome English        | ASR | 18  | https://catalog.ldc.upenn.edu/LDC97T14 | 
 | Fisher                  | ASR | 2000 | https://catalog.ldc.upenn.edu/LDC2004S13 | 
@@ -350,7 +355,7 @@ below:
 
 **Infrastructure:**
 We train Granite Speech using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable
-and efficient infrastructure for training our models over thousands of GPUs. The training of this particular model was completed in 9 days on 32
+and efficient infrastructure for training our models over thousands of GPUs. The training of this particular model was completed in 12 days on 32
 H100 GPUs.
 
 **Ethical Considerations and Limitations:**

adapter_config.json

@@ -20,8 +20,8 @@
     "rank_pattern": {},
     "revision": null,
     "target_modules": [
-        "v_proj",
-        "q_proj"
+        "q_proj",
+        "v_proj"
     ],
     "task_type": "CAUSAL_LM",
     "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:02d15987eb81e2faafece88b358d5d3a2d01297c37373f8567eaf7ef5b94eb6f
+oid sha256:dc40f62ec34c6b6d5148187fe9fa2a19ba36243536f78877cf68c61208193613
 size 136336192

config.json

@@ -16,14 +16,13 @@
         "max_pos_emb": 512,
         "model_type": "granite_speech_encoder",
         "num_heads": 8,
-        "num_layers": 10,
+        "num_layers": 16,
         "output_dim": 42
     },
     "has_lora_adapter": true,
     "initializer_range": 0.02,
     "model_type": "granite_speech",
     "projector_config": {
-        "_attn_implementation_autoset": true,
         "attention_probs_dropout_prob": 0.1,
         "cross_attention_frequency": 1,
         "encoder_hidden_size": 1024,
@@ -42,7 +41,7 @@
         "vocab_size": 30522
     },
     "text_config": {
-        "_name_or_path": "ibm-granite/granite-3.3-8b-instruct",
+        "_name_or_path": "/proj/speech/data/granite-3.3-8b-instruct/r250409a",
         "architectures": [
             "GraniteForCausalLM"
         ],

model-00009-of-00009.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4527960664105cdd02deae59d9990960ae93dd9532f6772b86bd5fea86d60294
-size 1023821036
+oid sha256:77c78d245a3306c5675771e910ec8198d6956a058014fb94d3443be83b4930d3
+size 1353359620

model.safetensors.index.json

@@ -1,6 +1,6 @@
 {
   "metadata": {
-    "total_size": 16967021220
+    "total_size": 17296537812
   },
   "weight_map": {
     "encoder.input_linear.bias": "model-00009-of-00009.safetensors",
@@ -71,6 +71,204 @@
     "encoder.layers.1.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
     "encoder.layers.1.post_norm.bias": "model-00009-of-00009.safetensors",
     "encoder.layers.1.post_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.to_kv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.to_out.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.to_out.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.attn.to_q.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.batch_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.batch_norm.num_batches_tracked": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.batch_norm.running_mean": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.batch_norm.running_var": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.batch_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.depth_conv.conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.down_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.down_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.up_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.conv.up_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff1.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff1.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff1.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff1.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff1.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff1.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff2.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff2.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff2.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff2.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff2.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.post_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.10.post_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.to_kv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.to_out.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.to_out.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.attn.to_q.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.batch_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.batch_norm.num_batches_tracked": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.batch_norm.running_mean": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.batch_norm.running_var": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.batch_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.depth_conv.conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.down_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.down_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.up_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.conv.up_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff1.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff1.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff1.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff1.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff1.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff1.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff2.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff2.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff2.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff2.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff2.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.post_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.11.post_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.to_kv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.to_out.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.to_out.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.attn.to_q.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.batch_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.batch_norm.num_batches_tracked": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.batch_norm.running_mean": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.batch_norm.running_var": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.batch_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.depth_conv.conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.down_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.down_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.up_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.conv.up_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff1.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff1.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff1.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff1.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff1.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff1.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff2.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff2.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff2.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff2.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff2.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.post_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.12.post_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.to_kv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.to_out.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.to_out.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.attn.to_q.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.batch_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.batch_norm.num_batches_tracked": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.batch_norm.running_mean": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.batch_norm.running_var": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.batch_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.depth_conv.conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.down_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.down_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.up_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.conv.up_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff1.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff1.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff1.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff1.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff1.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff1.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff2.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff2.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff2.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff2.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff2.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.post_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.13.post_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.to_kv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.to_out.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.to_out.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.attn.to_q.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.batch_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.batch_norm.num_batches_tracked": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.batch_norm.running_mean": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.batch_norm.running_var": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.batch_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.depth_conv.conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.down_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.down_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.up_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.conv.up_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff1.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff1.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff1.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff1.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff1.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff1.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff2.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff2.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff2.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff2.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff2.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.post_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.14.post_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.to_kv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.to_out.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.to_out.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.attn.to_q.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.batch_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.batch_norm.num_batches_tracked": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.batch_norm.running_mean": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.batch_norm.running_var": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.batch_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.depth_conv.conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.down_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.down_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.up_conv.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.conv.up_conv.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff1.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff1.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff1.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff1.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff1.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff1.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff2.down_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff2.down_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff2.pre_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff2.pre_norm.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff2.up_proj.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.ff2.up_proj.weight": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.post_norm.bias": "model-00009-of-00009.safetensors",
+    "encoder.layers.15.post_norm.weight": "model-00009-of-00009.safetensors",
     "encoder.layers.2.attn.pre_norm.bias": "model-00009-of-00009.safetensors",
     "encoder.layers.2.attn.pre_norm.weight": "model-00009-of-00009.safetensors",
     "encoder.layers.2.attn.rel_pos_emb.weight": "model-00009-of-00009.safetensors",