Introduction

In this post, we'll talk about the impact of choosing custom models and LoRAs to stylize your images. Because Stable Diffusion has been out for longer, we're going to focus on models and LoRAs based on that, but all the principles will hold for any other diffusion model.

This is part 4 of our series on getting started with AI image generation. This series has been designed as an introductory tutorial as you begin your journey in image and video generation with ComfyUI. You can find the previous posts here:

- Part 1: Getting Started - Launching Your First ComfyUI Workflow
- Part 2: Choosing Your Canvas - Model Selection in ComfyUI
- Part 3: The Art of Prompting - Crafting Effective Descriptions

What are models in ComfyUI?

We previously covered the basics of models in Part 2. Lets recap and dive in a bit deeper, shall we?

Understanding Base Models

Base models form the foundation of AI image generation. These are pre-trained neural networks designed to generate images based on input prompts. Think of them as the “blank canvas” with built-in capabilities that artists can enhance or customize.

Key Features of Base Models:

• General-Purpose Design: Base models are trained on broad datasets, making them versatile and capable of producing a wide range of styles and content.
• Adaptability: While they perform well out-of-the-box, base models can be fine-tuned or combined with other tools (like LoRAs) for more specific results.

Popular Base Models in AI Image Generation:

• Stable Diffusion 1.5: The foundation of many fine-tuned models, offering flexibility and ease of use for newcomers and seasoned creators alike.
• SDXL: Builds on SD 1.5’s capabilities with higher resolution, better prompt interpretation, and improved detail.
• FLUX: The cutting-edge of AI art generation, focusing on precision and coherence, ideal for professional-grade results.

Diving Deeper into Fine-Tuned Models

While base models provide general capabilities, fine-tuned models are specialized versions trained on specific styles or themes. These models allow you to:

• Generate highly stylized content (e.g., anime art, photorealism).
• Recreate the visual language of particular artists or artistic movements.
• Save time by using pre-configured settings tailored to your needs.

Example Use Cases:

• Anime-Inspired Artwork: Models like “AnythingV4” are tailored for anime and manga styles, capturing the nuances of this visual medium.
• Photorealistic Portraits: Models fine-tuned for realism excel at generating lifelike human faces or natural landscapes.
• Niche Themes: Community-driven platforms like Civitai host models designed for specific aesthetics, such as “vaporwave” or “medieval art.”

To illustrate the influence of the model on the result, here's an image generated with SDXL base model and then the same image generated with the Samaritan 3D Cartoon model. Only the model has changed. Everything else, including the seed, has been kept constant.

How to Choose the Right Model

Selecting the right model depends on your creative goals:

1. Start Broad: Use base models if you’re exploring general styles or want flexibility in your projects.
2. Go Specific: Opt for fine-tuned models when you have a clear aesthetic or theme in mind.
3. Experiment and Combine: Test multiple models to understand their strengths, and don’t hesitate to mix them with LoRAs for even more unique results.

Pro Tip: Check out example images provided on platforms like Civitai to preview what each model is capable of before using it.

Technical Details

While most users focus on artistic output, understanding the technical differences between models can also help:

• Checkpoint Size: Larger models (e.g., SDXL) may require more resources but can deliver higher-quality results.
• Dataset Influence: A model’s training data impacts its strengths and weaknesses; for instance, some models are better suited for abstract art, while others excel in realism.
• Prompt Sensitivity: Different models may interpret prompts with varying levels of fidelity—test and iterate to find the best match for your workflow.

By understanding these details, users can unlock the full potential of their image generation workflows and make informed choices about the tools they use.

Your choice of model is probably the most impactful decision on the quality of your outputs. It's important to choose well-trained, fit-for-purpose models. For example, if you're building workflows for real estate and interior design, you may find that a general purpose model doesn't understand the details of furniture arrangement as well as one that has been trained for interior design specifically.

What Are LoRAs?

LoRAs, or Low-Rank Adaptations, are lightweight extensions that allow you to fine-tune AI models for specific tasks or artistic styles. Instead of creating an entirely new model from scratch, LoRAs let you “overlay” additional capabilities onto an existing base model.

Think of LoRAs as specialized filters or enhancements:

• They don’t replace the base model but enhance it by focusing on specific themes, styles, or details.
• LoRAs are smaller in size and faster to load, making them ideal for quick customization.

Why Use LoRAs?

LoRAs offer a range of benefits that make them indispensable for AI image generation:

1. Flexibility: Add or remove LoRAs on the fly to explore different styles without switching between models.
2. Efficiency: Unlike training a whole new model, LoRAs require minimal computational resources and time to apply.
3. Customization: Tailor your outputs for niche aesthetics, from recreating classic art styles to experimenting with futuristic sci-fi themes.

How Do LoRAs Work?

LoRAs adapt a model’s existing weights to emphasize specific traits or patterns without altering the original model itself. For example:

• A LoRA trained on a dataset of medieval paintings will enhance the model’s ability to generate images with medieval-style elements.
• Another LoRA might focus on a particular artist’s style, like Van Gogh’s brushstrokes or anime character designs.

These adaptations act like a “layer” over the base model, combining the general capabilities of the base with the fine-tuned focus of the LoRA.

Finding and Using LoRAs

LoRAs are widely shared within the AI art community. Popular sources like Civitai provide access to thousands of LoRAs created by users for various purposes.

Here’s how to get started:

1. Search for LoRAs: Use platforms like Civitai to browse LoRAs by style, theme, or keyword (e.g., “cyberpunk,” “watercolor,” “pixel art”).
2. Preview Examples: Each LoRA includes sample images showing its effect on outputs, helping you decide if it suits your project.
3. Download and Apply: Download the LoRA file and load it into your workflow on InstaSD. Adjust its intensity to control how strongly it influences the results.

Key Use Cases

LoRAs are perfect for:

• Replicating Styles: Want your artwork to resemble a specific painter or art movement? There’s likely a LoRA for that.
• Genre-Specific Creations: Create anime, sci-fi, fantasy, or photorealistic works with specialized LoRAs.
• Experimentation: Test multiple LoRAs in combination with base models to achieve unique artistic expressions.

An Example Workflow

Imagine starting with the base model Stable Diffusion 1.5:

• You want to add a cyberpunk vibe to your scene. Load a LoRA trained on cyberpunk art.
• Adjust the LoRA strength to balance between the base model’s capabilities and the cyberpunk theme.
• Generate a few variations to find the perfect result.

Why LoRAs Are Game-Changing

LoRAs empower creators by offering:

• Precision: Fine-tune specific aspects of a model’s output without overhauling its overall behavior.
• Accessibility: Even users without technical expertise can quickly add a new style or feature using pre-trained LoRAs.
• Speed: LoRAs integrate seamlessly into workflows, allowing you to iterate faster than ever before.

With LoRAs, you have a powerful tool to take your AI art from generic to highly personalized. They unlock the true potential of your creativity by enabling you to customize your outputs in ways that were once only possible with extensive training and resources.

How to Use LoRAs in ComfyUI

Let's see how LoRAs work in practice.

We're going to start with a basic SDXL workflow and use it to generate an image of a puppy, using the prompt: "A cute puppy galavanting in a park, beautiful urban environment, high detailed, uhd".

Here's the important part to pay attention to:

• We load the SDXL model, and we feed it to the KSampler node.
• We also use its "CLIP" output to feed into the prompt encoder nodes.

When we want to use a LORA, the lora needs to tweak the model, so we inject it between the model and the places where the model is used. The steps are:

• Load LoRA.
• Give it the base model.
• Use its modified output in the workflow.

ComfyUI comes out of the box with Load LoRA node. Let's how we can use it to perform the above steps.

In this setup, we fed our base model into the "Load LoRA" node and then we used its output in the rest of the workflow just as before.

To illustrate the impact of the LoRA, I've generated 3 images with SDXL only, SDXL + Cyberpunk Anime LoRA, and SDXL + pixel art LoRA.  It's important to note that LoRAs often have trigger words. So we do have to modify the prompt a bit to ensure that the LoRA is "turned on". For the two LoRAs here, we've appended "pixel art" and "Cyberpunk_Anime", respectively, to the prompt.

‍

‍

How to Import LoRAs and Models in InstaSD

In InstaSD, you can build workflows with any model and LoRA. Once you have an instance of ComfyUI up and running, you can follow these steps to import a model or a LoRA.

• From the InstaSD dock, open the file explorer.
• Find the location on the file system where you need to add the file and click the "Add File" button. For models, this is usually under /ComfyUI/models/checkpoints, and for LoRAs, it's usually /ComfyUI/models/loras. However, remember that some specialized nodes may look for their specific models in different directories.
• On the "Add File" dialog, select the source for the download and provide the download link. Remember the download link must be a link to the direct file download _not_ a link to the page about the model. Here are some examples of where to get the link from Civitai or HuggingFace.

Once you've filled out all the fields, click "Upload" and your model or LoRA will be downloaded. Once the download is complete, be sure to refresh your node definitions so ComfyUI is aware of the new file. Now, you can select the file in your LoRA or model loader nodes.

Conclusion

In this post, we’ve explored how LoRAs expand the creative possibilities of AI image generation, allowing you to fine-tune models for specific styles, themes, and artistic goals. Whether you’re an AI art novice or a seasoned creator, LoRAs are a game-changing tool that empowers you to achieve results that truly match your vision.

By understanding the balance between base models and LoRAs, and knowing when and how to use each, you can build workflows that are not only efficient but also uniquely tailored to your projects. From creating photorealistic landscapes to channeling the essence of a favorite art style, LoRAs open the door to endless artistic exploration.

We’ve compiled some of the best-in-class workflows on InstaSD so you can launch them with just one click. Get started [here](I’ll add the link), or join our Discord to share your creations, exchange tips, and connect with a community of fellow creators.

Stay tuned for the next article in our series: “Refining Your Masterpiece: Sampling Methods Explained,” where we’ll dive deeper into the technical aspects of generating high-quality images.

Your journey into AI art is just beginning—let’s create something extraordinary together!

‍