Looking for a way to put someone in Stable Diffusion? InstantID is a Stable Diffusion addon for copying a face and add style. No model-training is required! Below is an example.

In this post, I will cover:

• What is InstantID
• How to use InstantID in AUTOMATIC1111
• How to use InstantID in ComfyUI
• Tips on using InstantID

What is InstantID?

InstantID uses InsightFace to detect, crop and extract a face embedding from the reference face. The embedding is then used with the IP-adpater to control image generation. This part is very similar to the IP-Adapter Face ID.

In addition, it detects and fixes several facial landmarks (eyes, nose, and mouth) with ControlNet.

The combination of using IP-Adapter Face ID and ControlNet enables copying and styling the reference image with high fidelity.

InstantID uses Stable Diffusion XL models. There’s no Stable Diffusion 1.5 version at the time of writing.

Since IP-Adapter Face ID doesn’t work as well with the SDXL models, InstantID is a good choice for face swap with SDXL.

Alternative to InstantID

InstantID is not the only face swap technology.

To put a face, or anything, in Stable Diffusion, you can train a checkpoint or a LoRA models. This approach usually gives the best result but it is time-consuming and requires some skill in training models.

You can also use training-free methods like the IP-adapter Face ID or the ReActor extension.

Using InstantID in AUTOMATIC1111

Software setup

https://github.com/Mikubill/sd-webui-controlnet

You will need to select an SDXL checkpoint model to select an SDXL InstantID. The list won’t refresh automatically. You can change the Control Type and change back to see the new options.

Updating the ControlNet extension

Your ControlNet extension may be outdated if you already have it.

Extensions need to be updated regularly to get bug fixes or new functionality. To update an extension:

1. Go to the Extensions page.
2. Click Installed tab.
3. Click Check for updates.

If an update to an extension is available, you will see a new commits checkbox in the Update column.

Leave the checkbox checked for the extensions you wish to update.

Click Apply and Restart UI to update the extensions.

You may need to restart the AUTOMATIC1111 server manually for it to work.

Install InstantID models

Windows or Mac

You need to download the following two model files.

Download the IP Adapter model for InstantID. Rename it to

ip-adapter_instant_id_sdxl.bin

Put it in the folder stable-diffusion-webui > models > ControlNet.

Download the InstantID controlnet model. Rename it to

control_instant_id_sdxl.safetensors

Put it in the folder stable-diffusion-webui > models > ControlNet.

Google Colab

If you use our AUTOMATIC1111 Colab notebook, download and rename the two models above and put them in your Google Drive under AI_PICS > ControlNet folder.

You must select the ControlNet extension to use the model when starting the notebook.

How to use InstantID on AUTOMATIC1111

Here are a few things to pay attention to when using the InstantID model.

• Use an SDXL model.
• Use a lower CFG scale than you normally would.
• Use two ControlNets for InstantID.
• Reduce the Control Weights and Ending Control Steps of the two controlNets.

Step 1: Select a SDXL model

Select an SDXL Turbo model in the Stable Diffusion checkpoint dropdown menu.

We will use the Dreamshaper SDXL Turbo model. Download the model and put it in the folder stable-diffusion-webui > models > Stable-Diffusion.

Step 2: Enter the txt2img setting

Go to the txt2img page.

You can control the style by the prompt

watercolors portrait of a woman (happy laughing:1.15), masterpiece, artistry

And the negative prompt.

low quality, blurry, malformed, distorted

Use the following sampler for the DreamShaper XL Turbo model.

Sampling Method: DPM++ SDE Karras

Sampling steps: 7

The default image size of 1024×1024 doesn’t work well with Instant ID. Use something close but not exactly 1024×1024.

Width: 1016

Height: 1016

The CFG Scale must be set quite low for Instant ID to work.

CFG scale: 2

Step 3: Enter ControlNet settings

You need to use both InstantID models. In other words, you will use two ControlNets.

I will use the following reference image for ControlNet 0 and ControlNet 1.

ControlNet 0 settings

The first ControlNet uses InsightFace to extract the facial features.

• Enable: Yes
• Pixel Perfect: Yes
• Control Type: Instant_ID
• Preprocessor: instant_id_face_embedding
• Model: ip-adapter_instant_id_sdxl
• Control weight: 1
• Starting control step: 0
• Ending control step: 1

Note: If you don’t see Instant_ID in the Control Type and preprocessor list, your ControlNet extension is outdated. See how to update an extension.

ControlNet 1 settings

The second ControlNet is used to extract facial keypoints such as positions of the eyes, the nose and the mouth.

Upload the SAME reference image as ControtNet 0.

• Enable: Yes
• Pixel Perfect: Yes
• Control Type: Instant_ID
• Preprocessor: instant_id_face_keypoints
• Model: control_instant_id_sdxl
• Control weight: 0.4
• Starting control step: 0
• Ending control step: 0.5

Step 4: Generate images

Click Generate. You should have images like this.

Tips for getting good images

The style is controlled by the prompt. You can use the prompts for SDXL.

The settings are highly dependent on the prompt and style. You may need to adjust the following settings.

• Control Weight (ControlNet 0 and 1) – 0.5 to 1
• Ending Control Step (ControlNet 0 and 1) – 0.2 to 0.6
• CFG Scale – 2 to 3.

The first Control image sets the facial feature. The second image sets the keypoints of the face. You can use different images but I found using the same image improves the result.

The InstantID function is flinchy. Watch the terminal for error messages. It is not uncommon to see errors even if you do everything right. Try turning off the ControlNets, generate an image, and turn it back on. Or refresh the page.

How to use InstantID on ComfyUI

Software setup

You need to have the ComfyUI Manger installed to follow this workflow.

Install InsightFace

Like the IP-Adapter FaceID, InstantID requires the InsightFace librrary to run. You should be all set if you have the IP-Adapter FaceID up and running.

If you haven’t installed InsightFace on ComfyUI, follow this instruction to install InsightFace on ComfyUI.

Step 1: Load the workflow

Download the InstantID basic workflow below.

Drag and drop it to ComfyUI to load.

Step 2: Install missing nodes

You should see some nodes in red.

Click Manager > Install Missing Custom Nodes. Install all nodes that are missing.

On the ComfyUI Manager menu, click Update All to update all custom nodes and ComfyUI iteself.

Step 3: Download models

Create the folder ComfyUI > models > instantid.

Download the InstandID IP-Adpater model. Put it in the newly created instantid folder.

Download the InstantID ControlNet model. Put it in the folder ComfyUI > models > controlnet.

Download the antelopev2 face model. Extract the zip files and put the .onnx files in the folder ComfyUI > models > insightface > models > antelopev2. (You need to create the last folder.)

Restart ComfyUI and refresh the ComfyUI page.

Step 4: Run the workflow

Now you should have everything you need to run the workflow.

Select a SDXL Turbo checkpoint model in the Load Checkpoint node. I use the DreamShaper SDXL Turbo model in this example.

Drop an face image to the Load Image node.

Select the InstantID IP-Adapter model in the Load InstatID Model node.

Select the InstantID ControlNet model in the Load ControlNet Model node.

Click Queue Prompt to generate an image.

Tips for using InstantID on ComfyUI

You can find more InstantID workflows here.

Avoid using image size of 1024×1024. For example, use a slight offset 1016×1016 to get the best result.

The default setting should work for most images. Tweak the following settings for for best results:

• Strength and the ending percent of the ControlNet.
• Weight and end_at of the InstantID nodes.
• CFG scale of the KSampler node.