Introduction to EasyAnimate v5.1 with ComfyUI: AI-Powered Video Generation for Creators

What Is EasyAnimate?

EasyAnimate is an AI model specializing in video generation, offering two primary workflows:

1. Image-to-Video: Animate a static image with smooth motion and camera effects.
2. Text-to-Video: Create a video directly from a descriptive text prompt.

In addition, EasyAnimate incorporates control models to help creators achieve specific styles and effects. These include:

• Depth: Add realistic depth effects to animations.
• Canny: Generate animations with detailed edge control.
• MLSD: Create animations with line simplifications for a stylistic touch.
• Camera Control: Fine-tune camera movements to simulate realistic or creative perspectives.

‍

Key Strengths of EasyAnimate

• Versatility: EasyAnimate supports both image-to-video and text-to-video workflows, catering to a wide range of creative needs.
• Control Models: Fine-tune animations with depth, canny, mlsd, and camera controls for greater artistic freedom.
• Integration with ComfyUI: Seamlessly integrate EasyAnimate into ComfyUI workflows for intuitive and modular operation.
• High-Quality Outputs: Generate visually appealing animations with smooth transitions and customizable effects.

‍

Limitations of EasyAnimate

While EasyAnimate is a powerful tool, it has some limitations:

• Hardware Requirements: Running EasyAnimate efficiently requires a high-end GPU with at least 16GB VRAM, though 24GB is recommended for more complex workflows.
• Learning Curve: Understanding how to use control models effectively may take time for beginners.
• Processing Speed: Generating long-duration or high-resolution videos can be time-intensive, depending on your hardware.

Installing EasyAnimate v5.1 for ComfyUI

Follow these steps to install and run EasyAnimate v5.1 with ComfyUI:

1. Set Up ComfyUI
   • Ensure you have ComfyUI installed and configured on your system. You can find installation instructions on the ComfyUI GitHub page.
2. Install EasyAnimate via ComfyUI Manager
   • Open the ComfyUI Manager within your ComfyUI installation.
   • Search for "EasyAnimate" in the list of available nodes.
   • Select EasyAnimate and install it directly through the manager.
3. Download Models
   • Depending on where you are running ComfyUI, follow the appropriate steps below:

‍For Local Installation:‍

Download the relevant models into ComfyUI/models/EasyAnimate:

You can download these models directly from HuggingFace and place them in the appropriate directories. The entire model directory has to be copied as is in your local ComfyUI/models/EasyAnimate directory. You can download them directly and move them to the right spot. But since each model is 40GB and consists of many files, we've found it easier to use Git LFS, described below for Cloud GPUs, even when downloading these locally.

For Cloud GPUs (e.g., Runpod):

Each of the models listed in the table above is 40GB and consists of many files. We've found the easiest way to install them is via cloning the model repository via Git LFS. To do this, follow these steps:

• Access the terminal for your cloud GPU instance.
• Install git lfs if not already installed:sudo apt install git-lfs

• Pull the model files using:git lfs install
cd /ComfyUI/models/EasyAnimate # or wherever your comfy install directory is
git lfs clone <model>


For example, for the inpaint model, I'd use the command git lfs clone https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP.

Exploring EasyAnimate’s Capabilities

To understand EasyAnimate's power, I started with their image-to-video workflow:

I generated an image using SDXL using a cool LoRA for a dissolve effect:

And with very basic prompting, I could produce something pretty cool.

However, what I really wanted was for the knight to completely disappear into smoke. I found that no matter how much I tried tweaking the prompt, I could not accomplish this effect only by prompting.

The next attempt was to control both the start and end frames of the video. I went back to SDXL to generate a new end frame. Can I get the knight piece to transform into something else? I used BrushNet to do some inpainting and swap the knight piece while keeping the rest of the image the same.

The result ended up being pretty incredible on first try:

But this was still not what I wanted. I wanted the chess piece to dissolve into smoke. So I went back to SDXL and tried to get a new end frame:

There are several issues with this end frame. To start, where the inpainting happened, there's a discontinuity with the light in the back. That looks bad in the image, and it will also affect the result of the video generation. But for the sake of experimentation, I did not want to spend so much time on the perfect end frame, so I proceeded.

To get decent results out of this, I found that I had to tweak both the prompt and the CFG. EasyAnimate team recommend using longer negative prompts and using words such as "solid" and "quiet" to the negative prompt for obtaining more dynamic results. I did the same with the positive prompt as well, since a basic prompt such as "a chesspiece dissolving into smoke" was not yielding the result I wanted.

I also found that lower CFG values yielded better results – the transitions seemed smoother. With higher values, the output maintained the start frame for far too long and then within very few frames, it would transition abruptly to the end frame, creating a jarring effect.

This was a lot of fun experimentations. The best two results I got were these:

I feel like a big element that would increase the quality of these results would be having a better end frame image. Experimenting with that and camera trajectory controls will be next on my list.

Conclusion

EasyAnimate v5.1 is a powerful tool that pushes the boundaries of video generation. Whether you're a content creator, animator, or AI enthusiast, its versatile capabilities and seamless integration with ComfyUI make it an excellent choice for your projects. With features like image-to-video, text-to-video, control conditions, and camera control, EasyAnimate offers endless possibilities for creative expression.

Ready to dive in? Install EasyAnimate today and unlock your creative potential!

‍