The LivePortraitProcess node is part of the ComfyUI-LivePortraitKJ repository and is designed to process source and driving images using the LivePortrait pipeline. This node enables the transformation of static source images into dynamic, animated portraits driven by a series of driving images. By utilizing advanced machine learning models, it maps the motion from the driving images onto the source image to create realistic animations.
The primary functionality of the LivePortraitProcess node is to apply motion from a sequence of driving images to a static source image, resulting in an animated output. It leverages the LivePortrait pipeline, which includes various specialized modules for extracting and transferring motion and appearance features between images.
The LivePortraitProcess node accepts the following inputs:
pipeline: This is the instantiated LivePortrait pipeline, which includes preloaded models and configuration settings required for processing the images. The pipeline carries the necessary modules and settings for the LivePortrait algorithm.
crop_info: Information about how the images should be cropped before processing. This can include details such as scaling, offset, and rotation parameters.
source_image: The static image that will be animated. It is input as a tensor and serves as the base to which the motion from the driving images will be applied.
driving_images: A sequence of images that provide the motion cues used to animate the source image. These images are input as a tensor.
lip_zero: A boolean parameter that controls whether the lip retargeting should be minimized.
lip_zero_threshold: A float specifying the threshold for the lip zero feature.
stitching: A boolean indicating whether stitching should be applied to the output images to improve continuity between frames.
delta_multiplier: A float to adjust the motion intensity. This multiplier affects how strongly the driving image motion is applied to the source image.
mismatch_method: A string parameter allowing selection of the method to handle frame mismatch: "constant", "cycle", "mirror", or "cut".
relative_motion_mode: Determines how motion from driving images is interpreted. Options include "relative", "source_video_smoothed", "relative_rotation_only", "single_frame", or "off".
driving_smooth_observation_variance: A float controlling the observation variance for smoothing driving motion.
opt_retargeting_info (optional): A dictionary containing settings for retargeting the eyes and lips separately.
expression_friendly (optional): A boolean toggle for making the output with single source images more expressive.
expression_friendly_multiplier (optional): A float to scale the expressiveness when the expression_friendly option is enabled.
The LivePortraitProcess node produces the following outputs:
cropped_image: A tensor representing the cropped and processed version of the source image that has been animated with the driving images' motion.
output: Additional processed data from the LivePortrait pipeline. This includes motion and feature extraction results which inform the animation process.
In ComfyUI workflows, the LivePortraitProcess node can be used in combination with nodes responsible for preparing inputs, such as loading images and configuring the LivePortrait pipeline. A typical workflow might include the following steps:
DownloadAndLoadLivePortraitModels node.LivePortraitCropper node.LivePortraitProcess to animate the source image.Device Optimization: The node supports different precision settings (e.g., fp16, fp32), allowing users to balance performance and quality based on their hardware capabilities.
Retargeting Options: Advanced retargeting settings allow for detailed control over how features like eyes and lips are animated, providing customization for creating more natural or expressive animations.
Batch Processing: The node is designed to handle sequences of images, facilitating batch processing and animation of multiple frames efficiently.
Compatibility: Consider the device compatibility (e.g., CPU, CUDA) when configuring other components in the workflow, as these can impact the performance and functionality of the node.
Troubleshooting: Ensure that all dependencies are correctly installed, as the node relies on several external libraries for image processing and model inference.
For more information on setting up and using the ComfyUI-LivePortraitKJ repository, visit the GitHub repository.