ControlNet is an indispensable tool for controlling the precise composition of AI images. You can use ControlNet to specify human poses and compositions in Stable Diffusion. You can now use ControlNet with the Flux.1 dev model.
In this article, I will guide you through setting up ControlNet for Flux AI models locally in ComfyUI.
Software
We will use ComfyUI, an alternative to AUTOMATIC1111.
Read the ComfyUI installation guide and ComfyUI beginner’s guide if you are new to ComfyUI. See the Quick Start Guide if you are new to AI images and videos.
Take the ComfyUI course to learn ComfyUI step-by-step.
Follow the Flux1.dev model installation guide if you have not installed the Flux model on ComfyUI.
Install ControlNet for Flux.1 dev model
Xlabs AI has developed custom nodes and ControlNet models for Flux on ComfyUI. You will need the x-flux-comfyui custom nodes and the corresponding ControlNet models to run ControlNet.
Step 1: Update ComfyUI
Upload ComfyUI to the latest version. This can be done using the ComfyUI Manager.
In ComfyUI, click Manager > Update ComfyUI.
Step 2: Load ControlNet workflow
Download the workflow JSON file below.
Drop it in ComfyUI.
Step 3: Install missing nodes
You should see some red nodes if you need to install additional nodes.
Click Manager > Install missing custom nodes.
Install the missing nodes and restart ComfyUI.
Refresh the ComfyUI page.
Step 4: Download the Flux.1 dev model
If you have used Flux on ComfyUI, you may have these files already.
Download the flux1-dev-fp8.safetensors model and put it in ComfyUI > models > unet.
Download the following two CLIP models and put them in ComfyUI > models > clip.
Download the Flux VAE model file and put it in ComfyUI > models > vae.
Step 5: Download the Canny ControlNet model
Download the Canny ControlNet model flux-canny-controlnet-v3.safetensors. Put it in ComfyUI > models > xlabs > controlnets.
Step 6: Run the workflow
Your workflow should be ready to run.
Upload a reference image to the Load Image node. You can also use the image below for testing.
Click Queue Prompt to generate an image.
Tips for using ControlNet for Flux
The strength value in the Apply Flux ControlNet cannot be too high. If you see artifacts on the generated image, you can lower its value.
Adjust the low_threshold and high_threshold of the Canny Edge node to control how much detail to copy from the reference image.
Change the image size in the Empty Latent Image node. Use a reference image with the same aspect ratio for the best results.
Additional ControlNets for Flux
If you have followed the installation steps above, you can easily add additional ControlNets to ComfyUI.
Depth ControlNet for Flux
The depth ControlNet copies the depth information from a reference image. It is useful for copying the perceived 3D composition from the reference image.
Follow the steps below to install the Depth ControlNet for Flux.
Download the workflow JSON file below.
Drop it in ComfyUI.
Download the Depth ControlNet model flux-depth-controlnet-v3.safetensors. Put it in ComfyUI > models > xlabs > controlnets.
Upload a reference image to the Load Image node.
Click Queue Prompt to run.
HED ControlNet for Flux
The HED ControlNet copies the rough outline from a reference image. It is useful for copying the general shape but not fine details from the reference image.
Follow the steps below to install the HED ControlNet for Flux.
Download the workflow JSON file below.
Drop it in ComfyUI.
Download the HED ControlNet model .flux-hed-controlnet-v3.safetensors. Put it in ComfyUI > models > xlabs > controlnets.
Upload a reference image to the Load Image node.
Click Queue Prompt to run.
