macandchiz/Hermes-4-70B-GGUF

Quantized version of: NousResearch/Hermes-4-70B

Available Files

The following GGUF quantization variants are available:

• hermes-4-70b-q2_k.gguf
• hermes-4-70b-q3_k_s.gguf
• hermes-4-70b-q3_k_m.gguf
• hermes-4-70b-q3_k_l.gguf
• hermes-4-70b-q4_0.gguf
• hermes-4-70b-q4_1.gguf
• hermes-4-70b-q4_k_s.gguf
• hermes-4-70b-q4_k_m.gguf
• hermes-4-70b-q5_0.gguf
• hermes-4-70b-q5_1.gguf
• hermes-4-70b-q5_k_s.gguf
• hermes-4-70b-q5_k_m.gguf
• hermes-4-70b-q6_k.gguf
• hermes-4-70b-q8_0.gguf
• hermes-4-70b-f16.gguf

Quantization Information

• q2_k: Smallest size, lowest quality
• q3_k_s, q3_k_m, q3_k_l: Small size, low quality variants
• q4_0, q4_1, q4_k_s, q4_k_m: Medium size, good quality (recommended for most use cases)
• q5_0, q5_1, q5_k_s, q5_k_m: Larger size, better quality
• q6_k: Large size, high quality
• q8_0: Very large size, very high quality
• f16: Original precision (largest size)

Choose the quantization level that best fits your needs based on the trade-off between file size and model quality.