SDXL Style Transfer with IPAdapter and ControlNet

In this workflow, we'll learn how to setup a ComfyUI workflow that uses IPAdapter to do style transfer either as txt-2-img or img-2-img. We will be using the following custom nodes:

• ComfyUI Essentials
• ComfyUI IPAdapter Plus
• ComfyUI ControlNet Auxiliary Preprocessors

‍

For this post, I'm using a famous painting by Jackson Pollock as my style source.

And when we do img-2-img, I'll be using this cat photo to be restyled.

This workflow was originally posted by Code Crafters Corner on YouTube. You can check out their awesome video here.

Step 0: Start a SDXL Base instance on InstaSD

To get started we'll launch a base SDXL instance. Then, install the custom nodes above.

Step 1: Add IPAdapter

To our basic workflow, we'll add a IPAdapter Unified Loader node, followed by a IPAdapter node. We'll also add a Load Image node for the image that we want to use as our "style source".

The output of our IPAdapter node feeds into our sampler's model input.

Step 2: Add an input image

We can use the flow we've built up to this point as a reliable text-to-image generator with the style of our style input image. But we also want to be able to restyle existing photos.

We'll add a new Apply ControlNet node. This will replace the positive input to our sampler node. Our positive prompt will feed into its conditioning input. For its control_net and image inputs, we'll use a Load ControlNet Model node with a canny model and a Empty Latent Image node, respectively.

Now our workflow has 3 inputs:

• Text prompts (positive and negative): to be used as the prompt when there is no ControlNet input image, or as an augment to it.
• Style input image: this is the most important input. The style of this image will be applied to all output images.
• ControlNet input image: The structure from this image will control the output.

With this setup, we can easily switch from a txt-2-img style transfer to an img-2-img style transfer by modifying the strength of the control net node. When it's at 1.0, the text prompt is ignored. When it's at 0, only the text prompt affects the output.

‍

And here is the output – the cat has been re-styled as a Jackson Pollock painting. I'm calling him Catson:

To get this particular result, I did reduce the weight of the IPAdapter a bit so that the cat figure can come through with higher fidelity.

‍

We have setup this workflow on InstaSD, so you can launch it with a click of the button. You can launch it with a click of a button here.