---
base_model: black-forest-labs/FLUX.1-dev
library_name: diffusers
tags:
- text-to-image
- diffusers-training
- diffusers
- lora
- flux
- flux-diffusers
widget:
- text: >-
    a trtcrd of a young woman sitting cross-legged on a floating cloud, eyes
    closed in meditation, with butterflies made of starlight circling her head,
    holding a crystal orb that shows swirling galaxies inside, her hair flowing
    upward like smoke, "the dreamer"
  output:
    url: images/tarot_merged1.png
- text: >-
    a trtcrd of an elderly person in flowing robes standing before a massive
    library, holding an ornate key in one hand and an open book with glowing
    text in the other, owls perched on floating bookshelves that spiral up into
    darkness, "the keeper"
  output:
    url: images/tarot_merged2.png
- text: >-
    a trtcrd of a strong figure in work clothes kneeling beside a half-built
    stone tower, hammer in hand, with blueprints scattered around, a phoenix
    rising from forge flames in the background, mountains silhouetted against
    dawn, "the builder"
  output:
    url: images/tarot_merged3.png
instance_prompt: null
datasets:
- multimodalart/1920-raider-waite-tarot-public-domain
---
# LoRA for FLUX.1-dev - Tarot Card Style

This repository contains a LoRA (Low-Rank Adaptation) fine-tuned on `black-forest-labs/FLUX.1-dev` to generate images in the artistic style of tarot cards. This work is part of the blog post, "Fine-Tuning FLUX.1-dev on consumer hardware and in FP8".

<Gallery />

## Inference

There are two main ways to use this LoRA for inference: loading the adapter on the fly or merging it with the base model.

### Option 1: Loading LoRA Adapters

This approach offers flexibility, allowing you to easily switch between different LoRA styles.

```python
from diffusers import FluxPipeline
import torch

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, torch_dtype=torch.float16
)
pipeline.load_lora_weights("derekl35/tarot-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.enable_model_cpu_offload()

image = pipeline(
    'a trtcrd of a strong figure in work clothes kneeling beside a half-built stone tower, hammer in hand, with blueprints scattered around, a phoenix rising from forge flames in the background, mountains silhouetted against dawn, "the builder"',
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("tarot_loaded.png")
```

### Option 2: Merging LoRA into Base Model

Merging the LoRA into the base model can lead to slightly faster inference and is useful when you want to use a single style consistently.

```python
from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel
import torch

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
)
pipeline.load_lora_weights("derekl35/tarot-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.fuse_lora()
pipeline.unload_lora_weights()

# You can save the fused transformer for later use
# pipeline.transformer.save_pretrained("fused_transformer")

pipeline.enable_model_cpu_offload()
image = pipeline(
    'a trtcrd of a strong figure in work clothes kneeling beside a half-built stone tower, hammer in hand, with blueprints scattered around, a phoenix rising from forge flames in the background, mountains silhouetted against dawn, "the builder"',
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("tarot_merged.png")
```

you can also requantize model:

```python
from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel, BitsAndBytesConfig
import torch 

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
)
pipeline.load_lora_weights("derekl35/tarot-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.fuse_lora()
pipeline.unload_lora_weights()

pipeline.transformer.save_pretrained("fused_transformer")

ckpt_id = "black-forest-labs/FLUX.1-dev"
bnb_4bit_compute_dtype = torch.bfloat16

nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=bnb_4bit_compute_dtype,
)
transformer = FluxTransformer2DModel.from_pretrained(
    "fused_transformer",
    quantization_config=nf4_config,
    torch_dtype=bnb_4bit_compute_dtype,
)

pipeline = AutoPipelineForText2Image.from_pretrained(
    ckpt_id, transformer=transformer, torch_dtype=bnb_4bit_compute_dtype
)
pipeline.enable_model_cpu_offload()

image = pipeline(
    'a trtcrd of a strong figure in work clothes kneeling beside a half-built stone tower, hammer in hand, with blueprints scattered around, a phoenix rising from forge flames in the background, mountains silhouetted against dawn, "the builder"',
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("tarot_merged.png")
```