How to use dreamshaperXL10_alpha2Xl10.safetensors with controlnet-canny-sdxl-1.0 ?

[arnold408]

I want to use dreamshaperXL10_alpha2Xl10.safetensors with controlnet-canny-sdxl-1.0
I downloaded dreamshaperXL10_alpha2Xl10.safetensors file and tried to use :

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
'./dreamshaperXL10_alpha2Xl10.safetensors',
controlnet=controlnet,
use_safetensors=True,
torch_dtype=torch.float16,
variant="fp16"
)

got error :
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
File "/opt/conda/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 908, in from_pretrained
cached_folder = cls.download(
File "/opt/conda/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1330, in download
info = model_info(
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
validate_repo_id(arg_value)
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './dream/dreamshaperXL10_alpha2Xl10.safetensors'. Use repo_type argument if needed.

Previously, I tried to use from_single_file insteaed of from_pretrained.
Got error : from_single_file not available with StableDiffusionXLControlNetPipeline.

Please help.
Thanks

[dai-ichiro]

How about converting dreamshaperXL10_alpha2Xl10.safetensors to diffuser pipeline?

[arnold408]

How about converting dreamshaperXL10_alpha2Xl10.safetensors to diffuser pipeline?

how to convert it to diffuser pipeline and use it in StableDiffusionXLControlNetPipeline ?
Thanks

[dai-ichiro]

Install diffusers from main branch and then see #4461.

How to convert

python scripts/convert_original_stable_diffusion_to_diffusers.py \
    --checkpoint_path ./dreamshaperXL10_alpha2Xl10.safetensors \
    --dump_path ./dreamshaperXL \
    --pipeline_class_name StableDiffusionXLPipeline \
    --from_safetensors

How to use

controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0",
    torch_dtype=torch.float16)

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "dreamshaperXL",
    controlnet=controlnet,
    torch_dtype=torch.float16)

[arnold408]

Thanks a lot, I will try this.

[sayakpaul]

If you succeeded, could we close the issue?

[Killfaeh]

Hello !
The conversion worked for me but not the inference.
I got this error :

Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got
torch.HalfTensor instead (while checking arguments for embedding)

[Killfaeh]

Here is the code

controlnet = ControlNetModel.from_pretrained('diffusers/controlnet-canny-sdxl-1.0', torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("./models/Diffuser/dreamshaperXL10_alpha2Xl10", controlnet=controlnet, torch_dtype=torch.float16)

inputImg = Image.open("input/Body-kun-vador-canny.png").convert("RGB")

output = pipe("darth vader holding his light saber in fighting position, trending on art station", negative_prompt='', image=inputImg, num_inference_steps=20, controlnet_conditioning_scale=0.75)

This code works with "stabilityai/stable-diffusion-xl-base-1.0" but not dreamshaperXL10_alpha2Xl10.

[sayakpaul]

How did you run the conversion? We cannot reproduce what you posted above.

[Killfaeh]

I made a script, here is the code

#!/bin/bash

INPUT_FILE="models/Safetensors/dreamshaperXL10_alpha2Xl10.safetensors"
OUTPUT_DIRECTORY="models/Diffuser/dreamshaperXL10_alpha2Xl10"

python3 diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path $INPUT_FILE --dump_path $OUTPUT_DIRECTORY --pipeline_class_name StableDiffusionXLPipeline --from_safetensors

I have the same error when I run an inference with StableDiffusionXLPipeline and from_single_file to load the .safetensors model.
I have tried with the model CounterfaitXL too. Same error.

I have a Mac Intel if this info can help.

[sayakpaul]

That script does not support converting SDXL ControlNets.

[Killfaeh]

I have copied the command line given by dai-ichiro and replaced the checkpoint_path and dump_path by my own values.
I have checked again, the command line is the same. Is this information wrong ?

[Killfaeh]

I found the solution : I have updated some python modules (transformers and accelerate).
The script support SDXL.

[sayakpaul]

Cool! Closing this issue then. It would be very nice, if you could post an end-to-end Colab Notebook here. I think that will benefit the community a lot.