Inpainting in ComfyUI has not been as easy and intuitive as in AUTOMATIC1111. The resources for inpainting workflow are scarce and riddled with errors. This post hopes to bridge the gap by providing the following bare-bone inpainting examples with detailed instructions in ComfyUI.
- Inpainting with a standard Stable Diffusion model
- Inpainting with an inpainting model
- ControlNet inpainting
- Automatic inpainting to fix faces
Software
Read the ComfyUI installation guide and ComfyUI beginner’s guide if you are new to ComfyUI.
Using the workflow file
You will find many workflow JSON files in this tutorial. Drop them to ComfyUI to use them. Do the following steps if it doesn’t work.
Every time you try to run a new workflow, you may need to do some or all of the following steps.
- Install ComfyUI Manager
- Install missing nodes
- Update everything
Install ComfyUI Manager
Install ComfyUI manager if you haven’t done so already. It provides an easy way to update ComfyUI and install missing nodes.
To install this custom node, go to the custom nodes folder in the PowerShell (Windows) or Terminal (Mac) App:
cd ComfyUI/custom_nodesInstall ComfyUI by cloning the repository under the custom_nodes folder.
git clone https://github.com/ltdrdata/ComfyUI-ManagerRestart ComfyUI completely. You should see a new Manager button appearing on the menu.
If you don’t see the Manager button, check the terminal for error messages. One common issue is GIT not installed. Installing it and repeat the steps should resolve the issue.
Install missing custom nodes
To install the custom nodes that are used by the workflow but you don’t have:
- Click Manager in the Menu.
- Click Install Missing custom Nodes.
- Restart ComfyUI completely.
Update everything
You can use ComfyUI manager to update custom nodes and ComfyUI itself.
- Click Manager in the Menu.
- Click Updates All. It may take a while to be done.
- Restart the ComfyUI and refresh the ComfyUI page.
Inpainting with a standard model
This workflow is for inpainting with a standard Stable Diffusion model (not an inpainting model). This workflow is similar to inpainting the whole picture in AUTOMATIC1111.
Download the ComfyUI workflow for inpainting below.
This inpainting workflow works for both Stable Diffusion v1.5 and Stable Diffusion XL models.
This workflow only works with a standard Stable Diffusion model, not an Inpainting model. (See the next section for a workflow using the inpaint model)
How it works
The trick is NOT to use the VAE Encode (Inpaint) node (which is meant to be used with an inpainting model), but:
- Encode the pixel images with the VAE Encode node.
- Use the Set Latent Noise Mask to attach the inpaint mask to the latent sample.
The KSampler node will apply the mask to the latent image during sampling.
How to use
Step 1: Load a checkpoint model
This workflow uses the Realistic Vision v5.1 model. (A SD v1.5 fine-tuned model).
Download the Realistic Vision model, put it in the folder ComfyUI > models > checkpoints.
Refresh the page and select the model in the Load Checkpoint node’s dropdown menu.
Step 2: Upload an image
Pick an image that you want to inpaint.
You can download the image in PNG format here.
Upload it to the workflow.
Step 3: Create an inpaint mask
Right-click the image. Select Open in MaskEditor in the context menu.
Mask the area you want to regenerate.
Click Save to node.
Step 4: Adjust parameters
Select a Checkpoint model. Note that an inpainting model doesn’t work with the workflow.
Change the prompt:
a man wearing sunglasses, epic style, super hero, highly detailed
The original denoising strength (denoise) is too high. Set it to 0.6.
Step 5: Generate inpainting
Finally, press the Queue Prompt to perform inpainting.
Note
You can adjust the denoising strength in the KSampler node. Lower it to change less. Increase it to change more. Setting it too high will result in incoherent images.
This workflow is for use with a low to moderate denoising strength. If you must use a high denoising strength, you need to use the next two workflows: Inpaint with an inpainting model and ControlNet inpainting.
Inpaint with an inpainting model
An inpainting model is specially designed for inpainting. Its model architecture is different from a standard model. You cannot use the workflow in the previous section with an inpainting model.
Download the ComfyUI inpaint workflow with an inpainting model below.
How it works
This workflow uses the VAE Enocde (for inpainting) node to attach the inpaint mask to the latent image.
Note that this workflow only works when the denoising strength is set to 1. Don’t change it to any other value!
How to use
The usage is the same as the first method, so I won’t repeat it here.
You will need to use an inpainting model, such as the Realistic Vision Inpainting model.
ControlNet inpainting
The two workflows above have their own drawbacks: You cannot use a high denoising strength with a standard model, and a low denoising strength with an inpainting model.
ControlNet inpainting allows you to use a standard Stable Diffusion model with a high denoising strength.
You can download the ComfyUI workflow for ControlNet inpainting below.
How it works
You make the workflow just like any other ControlNets. But you use the Inpaint Preprocessor node. It takes the pixel image and the inpaint mask as the input, and output to the Apply ControlNet node.
The ControlNet conditioning is applied through positive conditioning as usual.
How to use
Download the Realistic Vision model. Put it in Comfyui > models > checkpoints folder. Refresh the page and select the Realistic model in the Load Checkpoint node.
Download the ControlNet inpaint model. Put it in ComfyUI > models > controlnet folder. Refresh the page and select the inpaint model in the Load ControlNet Model node.
You can use it like the first example.
You can set the denoising strength to a high value without sacrificing global coherence.
Automatic inpainting to fix faces
Adetailer is an AUTOMATIC1111 extension for inpainting faces automatically. In Stable Diffusion, faces are often garbled if they are too small. Adetailer crops out a face, inpaints it at a higher resolution, and puts it back.
You can do the same with the FaceDetailer node with the Impact Pack in ComfyUI.
Text-to-image with Face Detailer
Download the ComfyUI Detailer text-to-image workflow below.
How it works
The main node that does the heavy lifting is the FaceDetailer node. It uses a face-detection model (Yolo) to detect the face. It then crops it out, inpaints it at a higher resolution, and puts it back.
Compare the original image on the left and the face-fixed image on the right below.
How to use
You need to set the checkpoint model in the Load Checkpoint node. Adjust the prompt as needed. Press Queue Prompt.
it works the same as the Adetailer’s default setting.
You can adjust the denoising strength in the FaceDetailer node to adjust how much the face should change.
Face Detailer on an uploaded image
The following workflow automatically fixes faces on an uploaded image.
To use the workflow, upload an image to the Load Image node and press Queue Prompt.
You can optionally revise the prompt to customize the face.