Looking for a way to put someone in Stable Diffusion? InstantID is a Stable Diffusion addon for copying a face and add style. No model-training is required! Below is an example.
In this post, I will cover:
- What is InstantID
- How to use InstantID in AUTOMATIC1111
- How to use InstantID in ComfyUI
- Tips on using InstantID
What is InstantID?
InstantID uses InsightFace to detect, crop and extract a face embedding from the reference face. The embedding is then used with the IP-adpater to control image generation. This part is very similar to the IP-Adapter Face ID.
In addition, it detects and fixes several facial landmarks (eyes, nose, and mouth) with ControlNet.
The combination of using IP-Adapter Face ID and ControlNet enables copying and styling the reference image with high fidelity.
InstantID uses Stable Diffusion XL models. There’s no Stable Diffusion 1.5 version at the time of writing.
Since IP-Adapter Face ID doesn’t work as well with the SDXL models, InstantID is a good choice for face swap with SDXL.
Alternative to InstantID
InstantID is not the only face swap technology.
To put a face, or anything, in Stable Diffusion, you can train a checkpoint or a LoRA models. This approach usually gives the best result but it is time-consuming and requires some skill in training models.
You can also use training-free methods like the IP-adapter Face ID or the ReActor extension.
Using InstantID in AUTOMATIC1111
Software setup
We will use AUTOMATIC1111 , a popular and free Stable Diffusion software. Check out the installation guides on Windows, Mac, or Google Colab.
If you are new to Stable Diffusion, check out the Quick Start Guide.
Take the Stable Diffusion course if you want to build solid skills and understanding.
Check out the AUTOMATIC1111 Guide if you are new to AUTOMATIC1111.
Running InstantID requires close to 20 GB of VRAM in my test. If you run into memory issue, you can try the Low VRAM setting in ControlNet, and the SDXL memory optimization options.
Install ControlNet Extension
You will need to install the ControlNet extension to use InstantID.
- Start AUTOMATIC1111 Web-UI normally.
2. Navigate to the Extension Page.
3. Click the Install from URL tab.
4. Enter the following URL in the URL for extension’s git repository field.
https://github.com/Mikubill/sd-webui-controlnet5. Click the Install button.
6. Wait for the confirmation message that the installation is complete.
7. Restart AUTOMATIC1111.
You will need to select an SDXL checkpoint model to select an SDXL InstantID. The list won’t refresh automatically. You can change the Control Type and change back to see the new options.
Updating the ControlNet extension
Your ControlNet extension may be outdated if you already have it.
Extensions need to be updated regularly to get bug fixes or new functionality. To update an extension:
- Go to the Extensions page.
- Click Installed tab.
- Click Check for updates.
If an update to an extension is available, you will see a new commits checkbox in the Update column.
Leave the checkbox checked for the extensions you wish to update.
Click Apply and Restart UI to update the extensions.
You may need to restart the AUTOMATIC1111 server manually for it to work.
Install InstantID models
Windows or Mac
You need to download the following two model files.
Download the IP Adapter model for InstantID. Rename it to
ip-adapter_instant_id_sdxl.binPut it in the folder stable-diffusion-webui > models > ControlNet.
Download the InstantID controlnet model. Rename it to
control_instant_id_sdxl.safetensorsPut it in the folder stable-diffusion-webui > models > ControlNet.
Google Colab
If you use our AUTOMATIC1111 Colab notebook, download and rename the two models above and put them in your Google Drive under AI_PICS > ControlNet folder.
You must select the ControlNet extension to use the model when starting the notebook.
How to use InstantID on AUTOMATIC1111
Here are a few things to pay attention to when using the InstantID model.
- Use an SDXL model.
- Use a lower CFG scale than you normally would.
- Use two ControlNets for InstantID.
- Reduce the Control Weights and Ending Control Steps of the two controlNets.
Step 1: Select a SDXL model
Select an SDXL Turbo model in the Stable Diffusion checkpoint dropdown menu.
We will use the Dreamshaper SDXL Turbo model. Download the model and put it in the folder stable-diffusion-webui > models > Stable-Diffusion.
Step 2: Enter the txt2img setting
Go to the txt2img page.
You can control the style by the prompt
watercolors portrait of a woman (happy laughing:1.15), masterpiece, artistry
And the negative prompt.
low quality, blurry, malformed, distorted
Use the following sampler for the DreamShaper XL Turbo model.
Sampling Method: DPM++ SDE Karras
Sampling steps: 7
The default image size of 1024×1024 doesn’t work well with Instant ID. Use something close but not exactly 1024×1024.
Width: 1016
Height: 1016
The CFG Scale must be set quite low for Instant ID to work.
CFG scale: 2
Step 3: Enter ControlNet settings
You need to use both InstantID models. In other words, you will use two ControlNets.
I will use the following reference image for ControlNet 0 and ControlNet 1.
ControlNet 0 settings
The first ControlNet uses InsightFace to extract the facial features.
- Enable: Yes
- Pixel Perfect: Yes
- Control Type: Instant_ID
- Preprocessor: instant_id_face_embedding
- Model: ip-adapter_instant_id_sdxl
- Control weight: 1
- Starting control step: 0
- Ending control step: 1
Note: If you don’t see Instant_ID in the Control Type and preprocessor list, your ControlNet extension is outdated. See how to update an extension.
ControlNet 1 settings
The second ControlNet is used to extract facial keypoints such as positions of the eyes, the nose and the mouth.
Upload the SAME reference image as ControtNet 0.
- Enable: Yes
- Pixel Perfect: Yes
- Control Type: Instant_ID
- Preprocessor: instant_id_face_keypoints
- Model: control_instant_id_sdxl
- Control weight: 0.4
- Starting control step: 0
- Ending control step: 0.5
Step 4: Generate images
Click Generate. You should have images like this.
Tips for getting good images
The style is controlled by the prompt. You can use the prompts for SDXL.
The settings are highly dependent on the prompt and style. You may need to adjust the following settings.
- Control Weight (ControlNet 0 and 1) – 0.5 to 1
- Ending Control Step (ControlNet 0 and 1) – 0.2 to 0.6
- CFG Scale – 2 to 3.
The first Control image sets the facial feature. The second image sets the keypoints of the face. You can use different images but I found using the same image improves the result.
The InstantID function is flinchy. Watch the terminal for error messages. It is not uncommon to see errors even if you do everything right. Try turning off the ControlNets, generate an image, and turn it back on. Or refresh the page.
How to use InstantID on ComfyUI
Software setup
We will use ComfyUI, an alternative to AUTOMATIC1111.
Read the ComfyUI installation guide and ComfyUI beginner’s guide if you are new to ComfyUI.
Take the ComfyUI course to learn ComfyUI step-by-step.
You need to have the ComfyUI Manger installed to follow this workflow.
Install InsightFace
Like the IP-Adapter FaceID, InstantID requires the InsightFace librrary to run. You should be all set if you have the IP-Adapter FaceID up and running.
If you haven’t installed InsightFace on ComfyUI, follow this instruction to install InsightFace on ComfyUI.
Step 1: Load the workflow
Download the InstantID basic workflow below.
Drag and drop it to ComfyUI to load.
Step 2: Install missing nodes
You should see some nodes in red.
Click Manager > Install Missing Custom Nodes. Install all nodes that are missing.
On the ComfyUI Manager menu, click Update All to update all custom nodes and ComfyUI iteself.
Step 3: Download models
Create the folder ComfyUI > models > instantid.
Download the InstandID IP-Adpater model. Put it in the newly created instantid folder.
Download the InstantID ControlNet model. Put it in the folder ComfyUI > models > controlnet.
Download the antelopev2 face model. Extract the zip files and put the .onnx files in the folder ComfyUI > models > insightface > models > antelopev2. (You need to create the last folder.)
Restart ComfyUI and refresh the ComfyUI page.
Step 4: Run the workflow
Now you should have everything you need to run the workflow.
Select a SDXL Turbo checkpoint model in the Load Checkpoint node. I use the DreamShaper SDXL Turbo model in this example.
Drop an face image to the Load Image node.
Select the InstantID IP-Adapter model in the Load InstatID Model node.
Select the InstantID ControlNet model in the Load ControlNet Model node.
Click Queue Prompt to generate an image.
Tips for using InstantID on ComfyUI
You can find more InstantID workflows here.
Avoid using image size of 1024×1024. For example, use a slight offset 1016×1016 to get the best result.
The default setting should work for most images. Tweak the following settings for for best results:
- Strength and the ending percent of the ControlNet.
- Weight and end_at of the InstantID nodes.
- CFG scale of the KSampler node.