How to use diffusers with custom ckpt file

Question

Currently I have the current code which runs a prompt on a model which it downloads from huggingface.

from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler

model_id = "stabilityai/stable-diffusion-2"

# Use the Euler scheduler here instead
scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler)
pipe = pipe.to("mps")
pipe.enable_attention_slicing()


prompt = "a photo of an astronaut riding a horse on mars"
pipe(prompt).images[0]

I wanted to know how can I feed a custom ckpt file to this script instead of it downloading it from stabilityAi repo?

Answer 1

You cannot use a ckpt file with diffusers out of the box. The ckpt file has to be converted to a diffusers friendly format.

You can do that with a tool named StableTuner. Or this utility script on HuggingFace https://huggingface.co/spaces/anzorq/sd-to-diffusers